#EmacsConf backstage: looking at EmacsConf's growth over 5 years, and how to do pivot tables and graphs with Org Mode and the Python pandas library

| emacsconf, emacs, org, python

Having helped organize EmacsConf for a number of years now, I know that I usually panic about whether we have submissions partway through the call for participation. This causes us to extend the CFP deadline and ask people to ask more people to submit things, and then we end up with a wonderful deluge of talks that I then have to somehow squeeze into a reasonable-looking schedule.

This year, I managed to not panic and I also resisted the urge to extend the CFP deadline, trusting that there will actually be tons of cool stuff. It helped that my schedule SVG code let me visualize what the conference could feel like with the submissions so far, so we started with a reasonably nice one-track conference and built up from there. It also helped that I'd gone back to the submissions for 2022 and plotted them by the number of weeks before the CFP deadline, and I knew that there'd be a big spike from all those people whose Org DEADLINE: properties would nudge them into finalizing their proposals.

Out of curiosity, I wanted to see how the stats for this year compared with previous years. I wrote a small function to collect the data that I wanted to summarize:

emacsconf-count-submissions-by-week: Count submissions in INFO by distance to CFP-DEADLINE.
(defun emacsconf-count-submissions-by-week (&optional info cfp-deadline)
  "Count submissions in INFO by distance to CFP-DEADLINE."
  (setq cfp-deadline (or cfp-deadline emacsconf-cfp-deadline))
  (setq info (or info (emacsconf-get-talk-info)))
  (cons '("Weeks to CFP end date" "Count" "Hours")
        (mapcar (lambda (entry)
                  (list (car entry)
                        (length (cdr entry))
                        (apply '+ (mapcar 'cdr (cdr entry)))))
                (seq-group-by
                 'car
                 (sort
                  (seq-keep
                   (lambda (o)
                     (and (emacsconf-publish-talk-p o)
                          (plist-get o :date-submitted)
                          (cons (floor (/ (days-between (plist-get o :date-submitted) cfp-deadline)
                                          7.0))
                                (string-to-number
                                 (or (plist-get o :video-duration)
                                     (plist-get o :time)
                                     "0")))))
                   info)
                  (lambda (a b) (< (car a) (car b))))))))

and then I ran it against the different files for each year, filling in the previous years' data as needed. The resulting table is pretty long, so I've put that in a collapsible section. 

(let ((years `((2023 "~/proj/emacsconf/2023/private/conf.org" "2023-09-15")
               (2022 "~/proj/emacsconf/2022/private/conf.org" "2022-09-18")
               (2021 "~/proj/emacsconf/2021/private/conf.org" "2021-09-30")
               (2020 "~/proj/emacsconf/wiki/2020/submissions.org" "2020-09-30")
               (2019 "~/proj/emacsconf/2019/private/conf.org" "2019-08-31"))))
  (append
   '(("Weeks to CFP" "Year" "Count" "Minutes"))
   (seq-mapcat
    (lambda (year-info)
      (let ((emacsconf-org-file (elt year-info 1))
            (emacsconf-cfp-deadline (elt year-info 2))
            (year (car year-info)))
        (mapcar (lambda (o) (list (car o) year (cadr o) (elt o 2)))
                (cdr (emacsconf-count-submissions-by-week (emacsconf-get-talk-info) emacsconf-cfp-deadline)))))
    years)))
Table
Weeks to CFP Year Count Minutes
-12 2023 4 70
-9 2023 2 30
-7 2023 2 30
-5 2023 2 30
-4 2023 2 60
-3 2023 3 40
-2 2023 5 130
-1 2023 10 180
0 2023 8 140
1 2023 1 20
-8 2022 2 25
-5 2022 2 31
-3 2022 2 31
-2 2022 2 17
-1 2022 8 191
0 2022 8 110
1 2022 5 107
-8 2021 4 50
-7 2021 2 17
-6 2021 1 7
-5 2021 2 22
-4 2021 2 19
-3 2021 5 73
-2 2021 1 10
-1 2021 12 163
0 2021 13 197
1 2021 1 10
2 2021 1 10
-5 2020 1 10
-4 2020 1 15
-2 2020 1 30
-1 2020 4 68
0 2020 21 424
1 2020 7 152
-5 2019 2 45
-4 2019 1 21
-2 2019 6 126
-1 2019 9 82
0 2019 9 148
2 2019 1 7

Some talks were proposed off-list and are not captured here, and cancelled or withdrawn talks weren't included either. The times for previous years use the actual video time, and the times for this year use proposed times.

Off the top of my head, I didn't know of an easy way to make a pivot table or cross-tab using just Org Mode or Emacs Lisp. I tried using datamash, but I was having a hard time getting my output just the way I wanted it. Fortunately, it was super-easy to get my data from an Org table into Python so I could use pandas.pivot_table. Because I had used #+NAME: submissions-by-week to label the table, I could use :var data=submissions-by-week to refer to the data in my Python program. Then I could summarize them by week.

Here's the number of submissions by the number of weeks to the original CFP deadline, so we can see people generally like to target the CFP date. 

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True)
fig, ax = plt.subplots()
figure = df.plot(title='Number of submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('number-of-submissions.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 4
-9 0 0 0 0 2
-8 0 0 4 2 0
-7 0 0 2 0 2
-6 0 0 1 0 0
-5 2 1 2 2 2
-4 1 1 2 0 2
-3 0 0 5 2 3
-2 6 1 1 2 5
-1 9 4 12 8 10
0 9 21 13 8 8
1 0 7 1 5 1
2 1 0 1 0 0
number-of-submissions.png

Calculating the cumulative number of submissions might be more useful. Here, each row shows the number received so far. 

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax = plt.subplots()
figure = df.plot(title='Cumulative submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('cumulative-submissions.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 4
-9 0 0 0 0 6
-8 0 0 4 2 6
-7 0 0 6 2 8
-6 0 0 7 2 8
-5 2 1 9 4 10
-4 3 2 11 4 12
-3 3 2 16 6 15
-2 9 3 17 8 20
-1 18 7 29 16 30
0 27 28 42 24 38
1 27 35 43 29 39
2 28 35 44 29 39
cumulative-submissions.png
Figure 1: Cumulative submissions by number of weeks to CFP end date

And here's the cumulative number of minutes based on the proposals. 

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Minutes', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax = plt.subplots()
figure = df.plot(title='Cumulative minutes by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('cumulative-minutes.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 70
-9 0 0 0 0 100
-8 0 0 50 25 100
-7 0 0 67 25 130
-6 0 0 74 25 130
-5 45 10 96 56 160
-4 66 25 115 56 220
-3 66 25 188 87 260
-2 192 55 198 104 390
-1 274 123 361 295 570
0 422 547 558 405 710
1 422 699 568 512 730
2 429 699 578 512 730
cumulative-minutes.png
Figure 2: Cumulative minutes by number of weeks to the CFP end date

So… yeah… 730 minutes of talks for this year… I might've gotten a little carried away. But I like all the talks! And I want them to be captured in videos and maybe even transcribed by people who will take the time to change misrecognized words like Emax into Emacs! And I want people to be able to connect with other people who are interested in the sorts of stuff they're doing! So we're going to make it happen. The draft schedule's looking pretty full, but I think it'll work out, especially if the speakers send in their videos on time. Let's see how it all works out!

(…and look, I even got to learn how to do pivot tables and graphs with Python!)

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

2023-10-02 Emacs news

| emacs, emacs-news

Links from reddit.com/r/emacs, r/orgmode, r/spacemacs, r/planetemacs, Hacker News, communick.news, lobste.rs, kbin, programming.dev, lemmy, planet.emacslife.com, YouTube, the Emacs NEWS file, Emacs Calendar, and emacs-devel. Thanks to Andrés Ramírez for emacs-devel links, and to David and Jeremy for other links. Do you have an Emacs-related link or announcement? Please e-mail me at sacha@sachachua.com. Thank you!

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

#EmacsConf backstage: using e-mail templates for confirmations and acceptances

| emacsconf, emacs

We send a number of e-mails to speakers while coordinating EmacsConf. Here's a rough list of the standard e-mails:

  • confirmation that we've received their proposal and that we'll review it over the following week
  • acceptance
  • coordination with related talks
  • checking tentative schedule to see if people have any requests
  • instructions for uploading files
  • instructions for accessing the backstage area
  • confirmation that we've received their uploaded video
  • captions for review
  • miscellaneous todos
  • any large schedule changes
  • schedule confirmation and check-in instructions
  • thanks and resources
  • thanks and follow-up questions

Sending e-mails from within Emacs makes it super-easy to automate, of course.

I started off with putting e-mail templates in our public organizers' notebook because that made it easy to link to the drafts when asking other volunteers for feedback. As the templates settle down, I've been gradually moving some of them into our emacsconf-mail library so that I don't need to copy the template into each year's notebook.

The e-mail templates use the same template functions we use to make the wiki pages. Here's the function for confirming that we've received a submission and letting the speaker know when to expect comments:

emacsconf-mail-review: Let the speaker know we’ve received their proposal.
(defun emacsconf-mail-review (talk)
  "Let the speaker know we've received their proposal."
  (interactive (list (emacsconf-complete-talk-info)))
  (emacsconf-mail-prepare '(:subject "${conf-name} ${year} review: ${title}"
                       :cc "emacsconf-submit@gnu.org"
                       :reply-to "emacsconf-submit@gnu.org, ${email}, ${user-email}"
                       :mail-followup-to "emacsconf-submit@gnu.org, ${email}, ${user-email}"
                       :body "
Hi, ${speakers-short}!

Thanks for submitting your proposal! (ZZZ TODO: feedback)

We'll wait a week (~ ${notification-date}) in case the other volunteers want to chime in regarding your talk. =)

${signature}
")
            (plist-get talk :email)
            (list
             :user-email user-mail-address
             :signature user-full-name
             :title (plist-get talk :title)
             :email (plist-get talk :email)
             :conf-name emacsconf-name
             :speakers-short (plist-get talk :speakers-short)
             :year emacsconf-year
             :notification-date (plist-get talk :date-to-notify))))

I recently extended emacsconf-mail-prepare so that it can insert the template into a reply instead of always starting a new message. This allows the messages to be properly threaded in the emacsconf-submit list archives, which makes it easier to verify that all the submissions have been acknowledged.

emacsconf-mail-prepare: Prepare the e-mail following TEMPLATE. Send it to EMAIL.
(defun emacsconf-mail-prepare (template email attrs)
  "Prepare the e-mail following TEMPLATE. Send it to EMAIL.
Use ATTRS to fill in the template.
If `emacsconf-mail-batch-test' is non-nil, put the message in that buffer instead."
  (let ((fields '((:reply-to "Reply-To")
                  (:mail-followup-to "Mail-Followup-To")
                  (:cc "Cc"))))
    (if emacsconf-mail-batch-test
        (emacsconf-mail-prepare-for-batch-test template email attrs fields)
      ;; prepare to send the mail
      (if (and (derived-mode-p 'message-mode) (string-match "unsent mail" (buffer-name)))
          ;; add to headers
          (emacsconf-mail-update-reply-headers template email attrs fields) 
        ;; compose a new message
        (compose-mail
         email
         (emacsconf-replace-plist-in-string attrs (or (plist-get template :subject) ""))
         (seq-keep (lambda (field)
                     (when (plist-get template (car field))
                       (cons
                        (cadr field)
                        (emacsconf-replace-plist-in-string
                         attrs
                         (plist-get template (car field))))))
                   fields)))
      (message-sort-headers)
      (message-goto-body)
      (save-excursion
        (insert (string-trim (emacsconf-replace-plist-in-string attrs (plist-get template :body)))
                "\n\n")
        (goto-char (point-min))
        (emacsconf-mail-merge-wrap))
      (when (plist-get template :log-note)
        (mapc (lambda (talk)
                (emacsconf-mail-log-message-when-sent talk (plist-get template :log-note)))
              (cdr group))))))

There's a little function that I can add to message-send-hook to prevent me from sending a message if it still has ZZZ in it.

emacsconf-mail-check-for-zzz-before-sending: Throw an error if the ZZZ todo marker is still in the message.
(defun emacsconf-mail-check-for-zzz-before-sending ()
  "Throw an error if the ZZZ todo marker is still in the message.
Good for adding to `message-send-hook'."
  (save-excursion
    (goto-char (point-min))
    (when (re-search-forward "ZZZ" nil t)
      (unless (yes-or-no-p "ZZZ marker found. Send anyway? ")
        (error "ZZZ marker found.")))))

Here's the function for sending an acceptance letter:

emacsconf-mail-accept-talk: Send acceptance letter.
(defun emacsconf-mail-accept-talk (talk)
  "Send acceptance letter."
  (interactive (list (emacsconf-complete-talk-info)))
  (emacsconf-mail-prepare '(:subject "${conf-name} ${year} acceptance: ${title}"
                       :cc "emacsconf-submit@gnu.org"
                       :slugs nil
                       :reply-to "emacsconf-submit@gnu.org, ${email}, ${user-email}"
                       :mail-followup-to "emacsconf-submit@gnu.org, ${email}, ${user-email}"
                       :body
                       "
Hi, ${speakers-short}!

Looks like all systems are a go for your talk. Thanks for
proposing it! Your talk page is now at ${url} . Please feel free
to update it or e-mail us if you'd like help with any changes.${fill}

If you want to get started on your talk early, we have some
instructions at ${base}${year}/prepare/ that might help.
We strongly encourage speakers to prepare a talk video by
${video-target-date} in order to reduce technical risks and make
things flow more smoothly. Plus, we might be able to get it captioned
by volunteers, just like the talks last year. We'll save ${time} minutes
for your talk, not including time for Q&A. Don't sweat it if
you're a few minutes over or under. If it looks like a much shorter or
longer talk once you start getting into it, let us know and we might
be able to adjust.${wrap}

I'll follow up with the specific schedule for your talk once things
settle down. In the meantime, please let us know if you have any
questions or if there's anything we can do to help out!

${signature}"
                       :function emacsconf-mail-accept-talk
                       :log-note "accepted talk")
   (plist-get talk :email)
   (list
    :base emacsconf-base-url
    :user-email user-mail-address
    :year emacsconf-year
    :signature user-full-name
    :conf-name emacsconf-name
    :title (plist-get talk :title)
    :email (plist-get talk :email)
    :time (plist-get talk :time)
    :speakers-short (plist-get talk :speakers-short)
    :url (concat emacsconf-base-url (plist-get talk :url))
    :video-target-date emacsconf-video-target-date)))

We send confirmations and acceptances one at a time. Other e-mails are sent to all the speakers and it's easier to draft them in a batch. I'll cover that kind of mail merge in a separate post.

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

#EmacsConf backstage: adding a talk to the wiki

| emacsconf, emacs

The EmacsConf 2023 call for participation has finished, hooray! We've sent out acceptances and added talks to the wiki. We experimented with early acceptances this year, which was really nice because it gives people quick feedback and allows people to get started on their videos early. That meant that I needed to be able to easily add talks to the wiki throughout the call for participation. We use templates and an Ikiwiki directive to make it easier to consistently format talk pages. This post covers adding a talk to the wiki, and these functions are in the emacsconf-el repository.

Overview

Amin Bandali picked Ikiwiki for the wiki for emacsconf.org. I think it's because Ikiwiki works with plain text in a Git repository, which fits nicely with our workflow. I can use Emacs Lisp to generate files that are included in other files, and I can also use Emacs Lisp to generate the starting point for files that may be manually edited later on.

We organize conference pages by year. Under the directory for this year (2023/), we have:

talks
talk descriptions that can be manually edited
talks/SLUG.md
the description for the talk. Includes ../info/SLUG-nav.md, ../info/SLUG-before.md, and ../info/SLUG-after.md
info
automatically-generated files that are included before and after the talk description, and navigation links between talks
info/SLUG-nav.md
navigation links
info/SLUG-before.md
navigation links
info/SLUG-after.md
navigation links
schedule-details.md
list of talks
organizers-notebook/index.org
public planning notebook

The filenames and URLs for each talk are based on the ID for a talk. I store that in the SLUG property to make it easy to differentiate from CUSTOM_ID, since CUSTOM_ID is useful for lots of other things in Org Mode. I usually assign the slugs when I add the talks to our private conf.org file, although sometimes people suggest a specific ID.

Templating

Publishing to wiki pages and replying to e-mails are easier if I can substitute text into readable templates. There are a number of templating functions for Emacs Lisp, like the built-in tempo.el or s-lex-format from s.el. I ended up writing something that works with plists instead, since we use property lists (plists) all over the emacsconf-el library.

emacsconf-replace-plist-in-string: Replace ${keyword} from ATTRS in STRING.
(defun emacsconf-replace-plist-in-string (attrs string)
  "Replace ${keyword} from ATTRS in STRING."
  (let ((a attrs) name val)
    (while a
      (setq name (pop a) val (pop a))
      (when (stringp val)
        (setq string
              (replace-regexp-in-string (regexp-quote (concat "${" (substring (symbol-name name) 1) "}"))
                                        (or val "")
                                        string t t))))
    string))

It is also handy to be able to add text around another string only if the string is non-nil, and to provide a different string to use if it isn't specified..

emacsconf-surround: Concat BEFORE, TEXT, and AFTER if TEXT is specified, or return ALTERNATIVE.
(defun emacsconf-surround (before text after &optional alternative)
  "Concat BEFORE, TEXT, and AFTER if TEXT is specified, or return ALTERNATIVE."
  (if (and text (not (string= text "")))
      (concat (or before "") text (or after ""))
    alternative))

Getting the talk information

To get the data to fill in the template, we can run a bunch of different functions. This lets us add or remove functions when we need to. We pass the previous result to the next function in order to accumulate properties or modify them. The result is a property list for the current talk.

emacsconf-get-talk-info-for-subtree: Run ‘emacsconf-talk-info-functions’ to extract the info for this entry.
(defun emacsconf-get-talk-info-for-subtree ()
  "Run `emacsconf-talk-info-functions' to extract the info for this entry."
  (seq-reduce (lambda (prev val) (save-excursion (save-restriction (funcall val prev))))
              emacsconf-talk-info-functions
              nil))

emacsconf-talk-info-functions: Functions to collect information.
(defvar emacsconf-talk-info-functions
  '(emacsconf-get-talk-info-from-properties
    emacsconf-get-talk-categories
    emacsconf-get-talk-abstract-from-subtree
    emacsconf-get-talk-logbook
    emacsconf-add-talk-status
    emacsconf-add-checkin-time
    emacsconf-add-timezone-conversions
    emacsconf-add-speakers-with-pronouns
    emacsconf-add-live-info)
  "Functions to collect information.")

Getting the talk abstract

I add a *** Talk abstract subheading to the talk and put the rest of the submission under a *** Talk details subheading. This allows me to extract the text of the Talk abstract heading (or whatever matches emacsconf-abstract-heading-regexp, which is set to "abstract".).

emacsconf-get-subtree-entry: Return the text for the subtree matching HEADING-REGEXP.
(defun emacsconf-get-subtree-entry (heading-regexp)
  "Return the text for the subtree matching HEADING-REGEXP."
  (car
   (delq nil
         (org-map-entries
          (lambda ()
            (when (string-match heading-regexp (org-entry-get (point) "ITEM"))
              (org-get-entry)))
          nil 'tree))))

emacsconf-get-talk-abstract-from-subtree: Add the abstract from a subheading.
(defun emacsconf-get-talk-abstract-from-subtree (o)
  "Add the abstract from a subheading.
The subheading should match `emacsconf-abstract-heading-regexp'."
  (plist-put o :abstract (substring-no-properties (or (emacsconf-get-subtree-entry "abstract") ""))))

I include emacsconf-get-talk-abstract-from-subtree in emacsconf-talk-info-functions so that it retrieves that information when I call emacsconf-get-talk-info-for-subtree.

Publishing the talk page

We add accepted talks to the wiki so that people can see what kinds of talks will be at EmacsConf 2023. To add the talk to the wiki, I use emacsconf-publish-add-talk. It'll create the talk page without overwriting anything that's already there and redo the automatically-generated info pages that provide navigation, status, and so on.

emacsconf-publish-add-talk: Add the current talk to the wiki.
(defun emacsconf-publish-add-talk ()
  "Add the current talk to the wiki."
  (interactive)
  (emacsconf-publish-talk-page (emacsconf-get-talk-info-for-subtree))
  (emacsconf-publish-info-pages)
  (magit-status-setup-buffer emacsconf-directory))

The talk page includes the description and other resources.

emacsconf-publish-talk-page: Draft the talk page for O unless the page already exists or FORCE is non-nil.
(defun emacsconf-publish-talk-page (o &optional force)
  "Draft the talk page for O unless the page already exists or FORCE is non-nil."
  (interactive (list (emacsconf-get-talk-info-for-subtree)
                     (> (prefix-numeric-value current-prefix-arg) 1)))
  (let ((filename (expand-file-name (format "%s.md" (plist-get o :slug))
                                    (expand-file-name "talks" (expand-file-name emacsconf-year emacsconf-directory)))))
    (unless (file-directory-p (expand-file-name "talks" (expand-file-name emacsconf-year emacsconf-directory)))
      (mkdir (expand-file-name "talks" (expand-file-name emacsconf-year emacsconf-directory))))
    (when (or force (null (file-exists-p filename)))
      (with-temp-file filename
        (insert
         (emacsconf-replace-plist-in-string
          (emacsconf-convert-talk-abstract-to-markdown
           (append o (list
                      :speaker-info (emacsconf-publish-format-speaker-info o)
                      :meta "!meta"
                      :categories (if (plist-get o :categories)
                                      (mapconcat (lambda (o) (format "[[!taglink %s]]" o))
                                                 (plist-get o :categories)
                                                 " ")
                                    ""))))
          "[[${meta} title=\"${title}\"]]
[[${meta} copyright=\"Copyright &copy; ${year} ${speakers}\"]]
[[!inline pages=\"internal(${year}/info/${slug}-nav)\" raw=\"yes\"]]

<!-- Initially generated with emacsconf-publish-talk-page and then left alone for manual editing -->
<!-- You can manually edit this file to update the abstract, add links, etc. --->\n

# ${title}
${speaker-info}

[[!inline pages=\"internal(${year}/info/${slug}-before)\" raw=\"yes\"]]

${abstract-md}

[[!inline pages=\"internal(${year}/info/${slug}-after)\" raw=\"yes\"]]

[[!inline pages=\"internal(${year}/info/${slug}-nav)\" raw=\"yes\"]]

${categories}
"))))))

Ikiwiki uses Markdown, so we can take advantage of Org's Markdown export.

emacsconf-convert-talk-abstract-to-markdown: Set the :abstract-md property to a Markdown version of the abstract.
(defun emacsconf-convert-talk-abstract-to-markdown (o)
  "Set the :abstract-md property to a Markdown version of the abstract."
  (plist-put o :abstract-md (org-export-string-as (or (plist-get o :abstract) "") 'md t)))

Publishing the list of talks

The list of talks at https://emacsconf.org/2023/talks/ is grouped by track.

2023-09-26_14-00-19.png
Figure 1: List of talks

emacsconf-publish-schedule: Generate the schedule or program.
(defun emacsconf-publish-schedule (&optional info)
  "Generate the schedule or program."
  (interactive)
  (emacsconf-publish-schedule-svg-snippets)
  (setq info (or info (emacsconf-publish-prepare-for-display info)))
  (with-temp-file (expand-file-name "schedule-details.md"
                                    (expand-file-name emacsconf-year emacsconf-directory))
    (insert
     (if (member emacsconf-publishing-phase '(cfp program))
         (let ((sorted (emacsconf-publish-prepare-for-display (or info (emacsconf-get-talk-info)))))
           (mapconcat
            (lambda (track)
              (concat
               "Jump to: "
               ;; links to other tracks
               (string-join (seq-keep (lambda (track-link)
                                        (unless (string= (plist-get track-link :id)
                                                         (plist-get track :id))
                                          (format "<a href=\"#%s\">%s</a>"
                                                  (plist-get track-link :id)
                                                  (plist-get track-link :name))))
                                      emacsconf-tracks)
                            " | ")
               "\n\n"
               (let ((track-talks (seq-filter (lambda (o) (string= (plist-get o :track)
                                                                   (plist-get track :name)))
                                              sorted)))
                 (format
                  "<h1 id=\"%s\" class=\"sched-track %s\">%s (%d talks)</h1>\n%s"
                  (plist-get track :id)
                  (plist-get track :name)
                  (plist-get track :name)
                  (length track-talks)
                  (emacsconf-publish-format-main-schedule track-talks)))))
            emacsconf-tracks "\n\n"))
       (emacsconf-publish-format-interleaved-schedule info))))
  (when (member emacsconf-publishing-phase '(cfp program))
    (with-temp-file (expand-file-name
                     "draft-schedule.md"
                     (expand-file-name emacsconf-year emacsconf-directory))
      (insert
       "[[!sidebar content=\"\"]]\n\n" 
       "This is a *DRAFT* schedule.\n"
       (let ((emacsconf-publishing-phase 'schedule))
         (emacsconf-publish-format-interleaved-schedule info))))))

The emacsconf-format-main-schedule function displays the information for the talks in each track. It's pretty straightforward, but I put it in a function because I call it from a number of places.

emacsconf-publish-format-main-schedule: Include the schedule information for INFO.
(defun emacsconf-publish-format-main-schedule (info)
  "Include the schedule information for INFO."
  (mapconcat #'emacsconf-publish-sched-directive info "\n"))

We define an Ikiwiki sched directive that conditionally displays things depending on what we specify, so it's easy to add more information during the schedule or conference phase. This is templates/sched.md in the EmacsConf wiki git repository: 

<div data-start="<TMPL_VAR startutc>" data-end="<TMPL_VAR endutc>" class="sched-entry <TMPL_IF track>track-<TMPL_VAR track></TMPL_IF track>">
<div class="sched-meta">
<TMPL_IF start>
<span class="sched-time"><span class="sched-start"><TMPL_VAR start></span>
<TMPL_IF end> - <span class="sched-end"><TMPL_VAR end></span></TMPL_IF end>
</span></TMPL_IF start>
<TMPL_IF track> <span class="sched-track <TMPL_VAR track>"><TMPL_IF watch><a href="<TMPL_VAR watch>"></TMPL_IF><TMPL_VAR track><TMPL_IF watch></a></TMPL_IF></span></TMPL_IF track>
<TMPL_IF pad> <span class="sched-pad"><a href="<TMPL_VAR pad>">Etherpad</a></TMPL_IF pad>
<TMPL_IF q-and-a> <span class="sched-q-and-a">Q&amp;A: <TMPL_VAR q-and-a></span> </TMPL_IF q-and-a>
</div>
<div class="sched-title"><a href="<TMPL_VAR url>"><TMPL_VAR title></a></div>
<div class="sched-speakers"><TMPL_VAR speakers> <TMPL_IF note>- <TMPL_VAR note></TMPL_IF note></div>
<TMPL_IF resources>
<ul class="resources">
<TMPL_VAR resources>
</ul>
</TMPL_IF resources>
<TMPL_IF time><span class="sched-duration><TMPL_VAR time></span> minutes</TMPL_IF time>
<TMPL_IF slug> <span class="sched-slug">id:<TMPL_VAR slug></span></TMPL_IF slug>
</div>

This Emacs Lisp function converts a talk into that directive.

emacsconf-publish-sched-directive: Format the schedule directive with info for O.
(defun emacsconf-publish-sched-directive (o)
  "Format the schedule directive with info for O."
  (format "[[!template id=sched%s]]"
          (let ((result "")
                (attrs (append
                        (pcase emacsconf-publishing-phase
                          ('program
                           (list
                            :time (plist-get o :time)))
                          ((or 'schedule 'conference)
                           (list
                            :status (pcase (plist-get o :status)
                                      ("CAPTIONED" "captioned")
                                      ("PREREC_RECEIVED" "received")
                                      ("DONE" "done")
                                      ("STARTED" "now playing")
                                      (_ nil))
                            :time (plist-get o :time)
                            :q-and-a (plist-get o :qa-link) 
                            :pad (and emacsconf-publish-include-pads (plist-get o :pad-url))
                            :startutc (format-time-string "%FT%T%z" (plist-get o :start-time) t)
                            :endutc (format-time-string "%FT%T%z" (plist-get o :end-time) t)
                            :start (format-time-string "%-l:%M" (plist-get o :start-time) emacsconf-timezone)
                            :end (format-time-string "%-l:%M" (plist-get o :end-time) emacsconf-timezone)))
                          ('resources
                           (list
                            :pad nil
                            :channel nil
                            :resources (mapconcat (lambda (s) (concat "<li>" s "</li>"))
                                                  (emacsconf-link-file-formats-as-list
                                                   (append o
                                                           (list :base-url (format "%s%s/" emacsconf-media-base-url emacsconf-year)))
                                                   (append emacsconf-main-extensions (list "--answers.webm" "--answers.opus" "--answers.vtt")))
                                                  ""))))
                        (list
                         :title (plist-get o :title)
                         :url (concat "/" (plist-get o :url))
                         :speakers (plist-get o :speakers)
                         :track (if (member emacsconf-publishing-phase '(schedule conference)) (plist-get o :track))
                         :watch (plist-get o :watch-url)
                         :slug (plist-get o :slug)
                         :note
                         (string-join
                          (delq nil
                                (list
                                 (when (plist-get o :captions-edited)
                                   "captioned")
                                 (when (and (plist-get o :public)
                                            (or (plist-get o :toobnix-url)
                                                (plist-get o :video-file)))
                                   "video posted")))
                          ", ")
                         )
                        )))
            (while attrs
              (let ((field (pop attrs))
                    (val (pop attrs)))
                (when val
                  (setq result (concat result " " (substring (symbol-name field) 1) "=\"\"\"" val "\"\"\"")))))
            result)))

Publishing auto-generated navigation

It's nice to be able to navigate between talks without going back to the schedule page each time. This is handled by just keeping two extra copies of the list: one with the first talk popped off, and one with an extra element added to the beginning. Then we can use the heads of those lists for next/previous links.

2023-09-26_14-05-51.png
Figure 2: Navigation

emacsconf-publish-nav-pages: Generate links to the next and previous talks.
(defun emacsconf-publish-nav-pages (&optional talks)
  "Generate links to the next and previous talks.
During the schedule and conference phase, the talks are sorted by time.
Otherwise, they're sorted by track and then schedule."
  (interactive (list (emacsconf-publish-prepare-for-display (or emacsconf-schedule-draft (emacsconf-get-talk-info)))))
  (let* ((next-talks (cdr talks))
         (prev-talks (cons nil talks))
         (label (if (member emacsconf-publishing-phase '(schedule conference))
                    "time"
                  "track")))
    (unless (file-directory-p (expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory)))
      (mkdir (expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory))))
    (while talks
      (let* ((o (pop talks))
             (next-talk (emacsconf-format-talk-link (pop next-talks)))
             (prev-talk (emacsconf-format-talk-link (pop prev-talks))))
        (with-temp-file (expand-file-name (format "%s-nav.md" (plist-get o :slug))
                                          (expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory)))
          (insert (concat "\n<div class=\"talk-nav\">
Back to the [[talks]]  \n"
                          (if prev-talk (format "Previous by %s: %s  \n" label prev-talk) "")
                          (if next-talk (format "Next by %s: %s  \n" label next-talk) "")
                          (if (plist-get o :track) ; tagging doesn't work here because ikiwiki will list the nav page
                              (format "Track: <span class=\"sched-track %s\">%s</span>  \n" (plist-get o :track) (plist-get o :track))
                            "")
                          "</div>
")))))))

Before the talk description

We include some details about the schedule in the talk page, before the description.

2023-09-26_14-09-07.png
Figure 3: Description

emacsconf-publish-before-page: Generate the page that has the info included before the abstract.
(defun emacsconf-publish-before-page (talk &optional info)
  "Generate the page that has the info included before the abstract.
This includes the intro note, the schedule, and talk resources."
  (interactive (list (emacsconf-complete-talk-info)))
  (setq info (or info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
  (with-temp-file (expand-file-name (format "%s-before.md" (plist-get talk :slug))
                                    (expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory)))
    
    (insert "<!-- Automatically generated by emacsconf-publish-before-page -->\n")
    (insert (emacsconf-surround "" (plist-get talk :intro-note) "\n\n" ""))
    (let ((is-live (emacsconf-talk-live-p talk)))
      (when is-live (emacsconf-publish-captions-in-wiki talk))
      (when (member emacsconf-publishing-phase '(schedule conference))
        (insert (emacsconf-publish-format-talk-schedule-image talk info)))
      (insert (emacsconf-publish-format-talk-schedule-info talk) "\n\n")
      (insert
       (if (plist-get talk :public) (emacsconf-wiki-talk-resources talk) "")
       "\n# Description\n"))
    (insert "<!-- End of emacsconf-publish-before-page -->")))

emacsconf-publish-format-talk-schedule-info: Format schedule information for O.
(defun emacsconf-publish-format-talk-schedule-info (o)
  "Format schedule information for O."
  (let ((friendly (concat "/" emacsconf-year "/talks/" (plist-get o :slug) ))
        (timestamp (org-timestamp-from-string (plist-get o :scheduled))))
    (emacsconf-replace-plist-in-string
     (append o
             (list
              :format
              (concat (or (plist-get o :video-time)
                          (plist-get o :time))
                      "-min talk"
                      (if (plist-get o :q-and-a)
                          (format " followed by %s Q&A%s"
                                  (plist-get o :q-and-a)
                                  (if (eq emacsconf-publishing-phase 'conference)
                                      (format " (%s)"
                                              (if (string-match "live" (plist-get o :q-and-a))
                                                  (if (eq 'after (emacsconf-bbb-status o))
                                                      "done"
                                                    (format "<https://emacsconf.org/current/%s/room>" (plist-get o :slug)))
                                                (emacsconf-publish-webchat-link o)))
                                    ""))
                        ""))
              :pad-info
              (if emacsconf-publish-include-pads
                  (format "Etherpad: <https://pad.emacsconf.org/%s-%s>  \n" emacsconf-year (plist-get o :slug))
                "")
              :irc-info
              (format "Discuss on IRC: [#%s](%s)  \n" (plist-get o :channel)
                      (plist-get o :webchat-url))
              :status-info
              (if (member emacsconf-publishing-phase '(cfp program schedule conference)) (format "Status: %s  \n" (plist-get o :status-label)) "")
              :schedule-info
              (if (and (member emacsconf-publishing-phase '(schedule conference))
                       (not (emacsconf-talk-all-done-p o))
                       (not (string= (plist-get o :status) "CANCELLED")))
                  (let ((start (org-timestamp-to-time (org-timestamp-split-range timestamp)))
                        (end (org-timestamp-to-time (org-timestamp-split-range timestamp t))))
                    (format
                     "<div>Times in different timezones:</div><div class=\"times\" start=\"%s\" end=\"%s\"><div class=\"conf-time\">%s</div><div class=\"others\"><div>which is the same as:</div>%s</div></div><div><a href=\"/%s/watch/%s/\">Find out how to watch and participate</a></div>"
                     (format-time-string "%Y-%m-%dT%H:%M:%SZ" start t)
                     (format-time-string "%Y-%m-%dT%H:%M:%SZ" end t)
                     (emacsconf-timezone-string o emacsconf-timezone)
                     (string-join (emacsconf-timezone-strings
                                   o
                                   (seq-remove (lambda (zone) (string= emacsconf-timezone zone))
                                               emacsconf-timezones)) "<br />")
                     emacsconf-year
                     (plist-get (emacsconf-get-track (plist-get o :track)) :id)))
                "")))
     (concat
      "[[!toc  ]]
Format: ${format}  
${pad-info}${irc-info}${status-info}${schedule-info}\n" 
      (if (plist-get o :alternate-apac)
          (format "[[!inline pages=\"internal(%s/inline-alternate)\" raw=\"yes\"]]  \n" emacsconf-year)
        "")
      "\n"))))

After the talk description

After the talk description, we include a footer that makes it easier for people to e-mail questions using either the PUBLIC_EMAIL property of the talk or the emacsconf-org-private e-mail address.

2023-09-26_14-10-35.png

emacsconf-publish-format-email-questions-and-comments: Invite people to e-mail either the public contact for TALK or the private list.
(defun emacsconf-publish-format-email-questions-and-comments (talk)
  "Invite people to e-mail either the public contact for TALK or the private list."
  (format "Questions or comments? Please e-mail %s"
          (emacsconf-publish-format-public-email talk
                                         (or
                                          (and (string= (plist-get talk :public-email) "t")
                                               (plist-get talk :email))
                                          (plist-get talk :public-email)
                                          "emacsconf-org-private@gnu.org"))))

emacsconf-publish-after-page: Generate the page with info included after the abstract.
(defun emacsconf-publish-after-page (talk &optional info)
  "Generate the page with info included after the abstract.
This includes captions, contact, and an invitation to participate."
  (interactive (list (emacsconf-complete-talk-info)))
  ;; Contact information
  (with-temp-file (expand-file-name (format "%s-after.md" (plist-get talk :slug))
                                    (expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory)))
    (insert
     "<!-- Automatically generated by emacsconf-publish-after-page -->\n"
     "\n\n"
     ;; main transcript
     (if (plist-get talk :public) (emacsconf-publish-format-captions talk) "")
     (emacsconf-publish-format-email-questions-and-comments talk) "\n"
     (if (eq emacsconf-publishing-phase 'cfp)
         (format "\n----\nGot an idea for an EmacsConf talk or session? We'd love to hear from you! Check out the [[Call for Participation|/%s/cfp]] for details.\n" emacsconf-year)
       "")
     "\n\n<!-- End of emacsconf-publish-after-page -->\n")))

Whenever the schedule changes

This function makes it easier to regenerate all those dynamic pages that need to be updated whenever the schedule changes.

emacsconf-publish-info-pages: Populate year/info/*-nav, -before, and -after files.
(defun emacsconf-publish-info-pages (&optional info)
  "Populate year/info/*-nav, -before, and -after files."
  (interactive (list nil))
  (setq info (or info (emacsconf-publish-prepare-for-display info)))
  (emacsconf-publish-with-wiki-change
    (emacsconf-publish-nav-pages info)
    (emacsconf-publish-schedule info)
    (mapc (lambda (o)
            (emacsconf-publish-before-page o info)
            (emacsconf-publish-after-page o info))
          info)))

Summary

So once the review period has passed and we're ready to accept the talk, I change the status to WAITING_FOR_PREREC and find room for it in the schedule. Then I use emacsconf-publish-add-talk to add the talk description to the wiki. I review the files it generated, tweak hyperlinks as needed, add the pages to the Git repository, and push the commit to the server. If I rearrange talks or change times, I just need to run emacsconf-publish-info-pages and all the dynamically-generated pages will be updated.

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

Org protocol: following Org links from outside Emacs

| org, emacs

_xor had an interesting idea: can we use org-protocol to link to things inside Emacs, so that we can have a webpage with bookmarks into our Org files? Here's a quick hack that reuses org-store-link and org-link-open. 

(defun org-protocol-open-link (info)
  "Process an org-protocol://open style url with INFO."
  (org-link-open (car (org-element-parse-secondary-string (plist-get info :link) '(link)))))

(defun org-protocol-copy-open-link (arg)
  (interactive "P")
  (kill-new (concat "org-protocol://open?link=" (url-hexify-string (org-store-link arg)))))

(with-eval-after-load 'org
  (add-to-list 'org-protocol-protocol-alist
               '("org-open" :protocol "open" :function org-protocol-open-link)))

To make exporting and following easier, we also need a little code to handle org-protocol links inside Org. 

(defun org-protocol-follow (path &rest _)
  "Follow the org-protocol link for PATH."
  (org-protocol-check-filename-for-protocol (concat "org-protocol:" path) nil nil))

(defun org-protocol-export (path desc format info)
  "Export an org-protocol link."
  (setq path (concat "org-protocol:" path))
  (setq desc (or desc path))
  (pcase format
    (`html (format "<a href=\"%s\">%s</a>" path desc))
    (`11ty (format "<a href=\"%s\">%s</a>" path desc))
    (`latex (org-latex-link path desc info))
    (`ascii (org-ascii-link path desc info))
    (`md (org-md-link path desc info))
    (_ path)))

(with-eval-after-load 'org
  (org-link-set-parameters "org-protocol"
                           :follow #'org-protocol-follow
                           :export #'org-protocol-export))

Now I can use org-protocol-copy-open-link to copy a link to the current location, and I can put it into my Org files.

Example bare link to the Org manual, which will work only if you have open in the org-protocol-protocol-alist:

org-protocol://open?link=%5B%5Binfo%3Aorg%23Protocols%5D%5Borg%23Protocols%5D%5D

With a description:

Org manual - Protocols

This is part of my Emacs configuration.
View or add comments (Disqus), or e-mail me at sacha@sachachua.com

2023-09-25 Emacs news

| emacs, emacs-news

Links from reddit.com/r/emacs, r/orgmode, r/spacemacs, r/planetemacs, Hacker News, lobste.rs, kbin, programming.dev, communick.news, lemmy, planet.emacslife.com, YouTube, the Emacs NEWS file, Emacs Calendar, and emacs-devel. Thanks to Andrés Ramírez for emacs-devel links. Do you have an Emacs-related link or announcement? Please e-mail me at sacha@sachachua.com. Thank you!

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

2023-09-18 Emacs news

| emacs, emacs-news

Links from reddit.com/r/emacs, r/orgmode, r/spacemacs, r/planetemacs, Hacker News, lobste.rs, kbin, programming.dev, communick.news, lemmy, planet.emacslife.com, YouTube, the Emacs NEWS file, Emacs Calendar, and emacs-devel. Thanks to Andrés Ramírez for emacs-devel links. Do you have an Emacs-related link or announcement? Please e-mail me at sacha@sachachua.com. Thank you!

View or add comments (Disqus), or e-mail me at sacha@sachachua.com