Categories: emacsconf

View topic page - RSS - Atom - Subscribe via email

#EmacsConf backstage: Using Spookfox to automate creating BigBlueButton rooms in Mozilla Firefox

| emacsconf, emacs, org

Naming conventions make it easier for other people to find things. Just like with file prefixes, I like to use a standard naming pattern for our BigBlueButton web conference rooms. For EmacsConf 2022, we used ec22-day-am-gen Speaker name (slugs). For EmacsConf 2023, I want to set up the BigBlueButton rooms before the schedule settles down, so I won't encode the time or track information into it. Instead, I'll use Speaker name (slugs) - emacsconf2023.

BigBlueButton does have an API for managing rooms, but that requires a shared secret that I don't know yet. I figured I'd just automate it through my browser. Over the last year, I've started using Spookfox to control the Firefox web browser from Emacs. It's been pretty handy for scrolling webpages up and down, so I wondered if I could replace my old xdotool-based automation. Here's what I came up with for this year.

First, I need a function that creates the BBB room for a group of talks and updates the Org entry with the URL. Adding a slight delay makes it a bit more reliable.

emacsconf-spookfox-create-bbb: Create a BBB room for this group of talks.
(defun emacsconf-spookfox-create-bbb (group)
  "Create a BBB room for this group of talks.
GROUP is (email . (talk talk talk)).
Needs a Spookfox connection."
  (let* ((bbb-name
          (format "%s (%s) - %s%s"
                  (mapconcat (lambda (o) (plist-get o :slug)) (cdr group) ", ")
                  (plist-get (cadr group) :speakers)
                  emacsconf-id
                  emacsconf-year))
         path
         (retrieve-command
          (format
           "window.location.origin + [...document.querySelectorAll('h4.room-name-text')].find((o) => o.textContent.trim() == '%s').closest('tr').querySelector('.delete-room').getAttribute('data-path')"
           bbb-name))
         (create-command (format "document.querySelector('#create-room-block').click();
document.querySelector('#create-room-name').value = \"%s\";
document.querySelector('#room_mute_on_join').click();
document.querySelector('.create-room-button').click();"
                                 bbb-name)))
    (setq path (spookfox-js-injection-eval-in-active-tab retrieve-command t))
    (unless path
      (dolist (cmd (split-string create-command ";"))
        (spookfox-js-injection-eval-in-active-tab cmd t)
        (sleep-for 2))
      (sleep-for 2)
      (setq path (spookfox-js-injection-eval-in-active-tab retrieve-command t)))
    (when path
      (dolist (talk (cdr group))
        (save-window-excursion
          (emacsconf-with-talk-heading talk
            (org-entry-put (point) "ROOM" path))))
      (cons bbb-name path))))

Then I need to iterate over the list of talks that have live Q&A sessions but don't have BBB rooms assigned yet so that I can create them.

emacsconf-spookfox-create-bbb-for-live-talks: Create BBB rooms for talks that don’t have them yet.
(defun emacsconf-spookfox-create-bbb-for-live-talks ()
  "Create BBB rooms for talks that don't have them yet."
  (let* ((talks (seq-filter
                 (lambda (o)
                   (and (string-match "live" (or (plist-get o :q-and-a) ""))
                        (not (string= (plist-get o :status) "CANCELLED"))
                        (not (plist-get o :bbb-room))))
                 (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
         (groups (and talks (emacsconf-mail-groups talks))))
    (dolist (group groups)
      (emacsconf-spookfox-create-bbb group))))

The result: a whole bunch of rooms ready for people to check in.

2023-10-14_09-24-34.png
Figure 1: BigBlueButton rooms

Using Spookfox to communicate with Firefox from Emacs Lisp made it easy to get data in and out of my browser. Handy!

This code is in emacsconf-spookfox.el.

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

#EmacsConf backstage: file prefixes

| emacs, org, emacsconf

Sometimes it makes sense to dynamically generate information related to a talk and then save it as an Org property so that I can manually edit it. For example, we like to name all the talk files using this pattern: "emacsconf-year-slug--title--speakers". That's a lot to type consistently! We can generate most of these prefixes automatically, but some might need tweaking, like when the talk title or speaker names have special characters.

Calculating the file prefix for a talk

First we need something that turns a string into an ID.

emacsconf-slugify: Turn S into an ID.
(defun emacsconf-slugify (s)
  "Turn S into an ID.
Replace spaces with dashes, remove non-alphanumeric characters,
and downcase the string."
  (replace-regexp-in-string
   " +" "-"
   (replace-regexp-in-string
    "[^a-z0-9 ]" ""
    (downcase s))))

Then we can use that to calculate the file prefix for a given talk.

emacsconf-file-prefix: Create the file prefix for TALK
(defun emacsconf-file-prefix (talk)
  "Create the file prefix for TALK."
  (concat emacsconf-id "-"
          emacsconf-year "-"
          (plist-get talk :slug) "--"
          (emacsconf-slugify (plist-get talk :title))
          (if (plist-get talk :speakers)
              (concat "--"
                     (emacsconf-slugify (plist-get talk :speakers)))
            "")))

Then we can map over all the talk entries that don't have FILE_PREFIX defined:

emacsconf-set-file-prefixes: Set the FILE_PREFIX property for each talk entry that needs it.
(defun emacsconf-set-file-prefixes ()
  "Set the FILE_PREFIX property for each talk entry that needs it."
  (interactive)
  (org-map-entries
   (lambda ()
     (org-entry-put
      (point) "FILE_PREFIX"
      (emacsconf-file-prefix (emacsconf-get-talk-info-for-subtree))))
   "SLUG={.}-FILE_PREFIX={.}"))

That stores the file prefix in an Org property, so we can edit it if it needs tweaking.

output-2023-10-10-15:17:17.gif
Figure 1: Setting the FILE_PREFIX for all talks that don't have that yet

Renaming files to match the file prefix

Now that we have that, how can we use it? One way is to rename files from within Emacs. I can mark multiple files with Dired's m command or work on them one at a time. If there are several files with the same extension, I can specify something to add to the filename to tell them apart.

emacsconf-rename-files: Rename the marked files or the current file to match TALK.
(defun emacsconf-rename-files (talk &optional filename)
  "Rename the marked files or the current file to match TALK.
If FILENAME is specified, use that as the extra part of the filename after the prefix.
This is useful for distinguishing files with the same extension.
Return the list of new filenames."
  (interactive (list (emacsconf-complete-talk-info)))
  (prog1
      (mapcar
       (lambda (file)
         (let* ((extra
                 (or filename
                     (read-string (format "Filename (%s): " (file-name-base file)))))
                (new-filename
                 (expand-file-name
                  (concat (plist-get talk :file-prefix)
                          (if (string= extra "")
                              ""
                            (concat "--" extra))
                          "."
                          (file-name-extension file))
                  (file-name-directory file))))
           (rename-file file new-filename t)
           new-filename))
       (or (dired-get-marked-files) (list (buffer-file-name))))
    (when (derived-mode-p 'dired-mode)
      (revert-buffer))))

output-2023-10-10-14:18:34.gif
Figure 2: Renaming multiple files

Working with files on other computers

Because Dired works over TRAMP, I can use that to rename files on a remote server without changing anything about the code. I can open the remote directory with Dired and everything just works.

TRAMP also makes it easy to copy a file to the backstage directory after it's renamed, which saves me having to do that as a separate step.

emacsconf-rename-and-upload-to-backstage: Rename marked files or the current file, then upload to backstage.
(defun emacsconf-rename-and-upload-to-backstage (talk &optional filename)
  "Rename marked files or the current file, then upload to backstage."
  (interactive (list (emacsconf-complete-talk-info)))
  (mapc
   (lambda (file)
     (copy-file
      file
      (expand-file-name
       (file-name-nondirectory file)
       emacsconf-backstage-dir)
      t))
   (emacsconf-rename-files talk)))

So if my emacsconf-backstage-dir is set to /ssh:orga@res:/var/www/res.emacsconf.org/2023/backstage, then it looks up the details for res in my ~/.ssh/config and copies the file there.

Renaming files using information from a JSON

What if I don't want to rename the files from Emacs? If I use Emacs's JSON support to export some information from the talks as a JSON file, then I can easily use that data from the command line.

Here's how I export the talk information:

emacsconf-talks-json: Return JSON format with a subset of talk information.
(defun emacsconf-publish-talks-json ()
  "Return JSON format with a subset of talk information."
  (json-encode
   (list
    :talks
    (mapcar
     (lambda (o)
       (apply
        'list
        (cons :start-time (format-time-string "%FT%T%z" (plist-get o :start-time) t))
        (cons :end-time (format-time-string "%FT%T%z" (plist-get o :end-time) t))
        (mapcar
         (lambda (field)
           (cons field (plist-get o field)))
         '(:slug :title :speakers :pronouns :pronunciation :url :track :file-prefix))))
     (emacsconf-filter-talks (emacsconf-get-talk-info))))))

emacsconf-publish-talks-json-to-files
(defun emacsconf-publish-talks-json-to-files ()
  "Export talk information as JSON so that we can use it in shell scripts."
  (interactive)
  (mapc (lambda (dir)
          (when (and dir (file-directory-p dir))
            (with-temp-file (expand-file-name "talks.json" dir)
              (insert (emacsconf-talks-json)))))
        (list emacsconf-res-dir emacsconf-ansible-directory)))

Then I can use jq to extract the information with 

jq -r '.talks[] | select(.slug=="'$SLUG'")["file-prefix"]' < $TALKS_JSON

Here it is in the context of a shell script that renames the given file to match a talk's prefix. 

#!/bin/bash
# 
# Usage: rename-original.sh $slug $file [$extra] [$talks-json]
SLUG=$1
FILE=$2
TALKS_JSON=${4:-~/current/talks.json}
EXTRA=""
if [ -z ${3-unset} ]; then
    EXTRA=""
elif [ -n "$3" ]; then
    EXTRA="--$3"
elif echo "$FILE" | grep -e '\(webm\|mp4\|mov\)'; then
    EXTRA="--original"
fi
filename=$(basename -- "$FILE")
extension="${filename##*.}"
filename="${filename%.*}"
FILE_PREFIX=$(jq -r '.talks[] | select(.slug=="'$SLUG'")["file-prefix"]' < $TALKS_JSON)
mv "$FILE" $FILE_PREFIX$EXTRA.$extension
echo $FILE_PREFIX$EXTRA.$extension
# Copy to original if needed
if [ -f $FILE_PREFIX--original.webm ] && [ ! -f $FILE_PREFIX--main.$extension ]; then
    cp $FILE_PREFIX--original.$extension $FILE_PREFIX--main.webm
fi

Then I can use something like rename-original.sh emacsconf video.webm to emacsconf-2023-emacsconf--emacsconforg-how-we-use-org-mode-and-tramp-to-organize-and-run-a-multitrack-conference--sacha-chua--original.webm.

Working with PsiTransfer-uploaded files

JSON support is useful for getting files into our system, too. For EmacsConf 2022, we used PsiTransfer as a password-protected web-based file upload service. That was much easier for speakers to deal with than FTP, especially for large files. PsiTransfer makes a JSON file for each batch of uploads, which is handy because the uploaded files are named based on the key instead of keeping their filenames and extensions. I wrote a function to copy an uploaded file from the PsiTransfer directory to the backstage directory, renaming it along the way. That meant that I could open the JSON for the uploaded files via TRAMP and then copy a file between two remote directories without manually downloading it to my computer.

emacsconf-upload-copy-from-json: Parse PsiTransfer JSON files and copy the uploaded file to the backstage directory.
(defun emacsconf-upload-copy-from-json (talk key filename)
  "Parse PsiTransfer JSON files and copy the uploaded file to the backstage directory.
The file is associated with TALK. KEY identifies the file in a multi-file upload.
FILENAME specifies an extra string to add to the file prefix if needed."
  (interactive (let-alist (json-parse-string (buffer-string) :object-type 'alist)
                 (list (emacsconf-complete-talk-info)
                       .metadata.key
                       (read-string (format "Filename: ")))))
  (let ((new-filename (concat (plist-get talk :file-prefix)
                              (if (string= filename "")
                                  filename
                                (concat "--" filename))
                              "."
                              (let-alist (json-parse-string (buffer-string) :object-type 'alist)
                                (file-name-extension .metadata.name)))))
    (copy-file
     key
     (expand-file-name new-filename emacsconf-backstage-dir)
     t)))

So that's how I can handle things that can be mostly automated but that might need a little human intervention: use Emacs Lisp to make a starting point, tweak it a little if needed, and then make it easy to use that value elsewhere. Renaming files can be tricky, so it's good to reduce the chance for typos!

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

#EmacsConf backstage: reviewing the last message from a speaker

| emacs, emacsconf, notmuch

One of the things I keep an eye out for when organizing EmacsConf is the most recent time we heard from a speaker. Sometimes life happens and speakers get too busy to prepare a video, so we might offer to let them do it live. Sometimes e-mail delivery issues get in the way and we don't hear from speakers because some server in between has spam filters set too strong. So I made a function that lists the most recent e-mail we got from the speaker that includes "emacsconf" in it. That was a good excuse to learn more about tabulated-list-mode.

2023-10-07-13-18-04.svg
Figure 1: Redacted view of most recent e-mails from speakers

I started by figuring out how to get all the e-mail addresses associated with a talk.

emacsconf-mail-get-all-email-addresses: Return all the possible e-mail addresses for TALK.
(defun emacsconf-mail-get-all-email-addresses (talk)
  "Return all the possible e-mail addresses for TALK."
  (split-string
   (downcase
    (string-join
     (seq-uniq
      (seq-keep
       (lambda (field) (plist-get talk field))
       '(:email :public-email :email-alias)))
     ","))
   " *, *"))

Then I figured out the notmuch search to use to get all messages. Some people write a lot, so I limited it to just the ones that have emacsconf as well. Notmuch can return JSON, so that's easy to parse.

emacsconf-mail-notmuch-tag: Tag to use when searching the Notmuch database for mail.
(defvar emacsconf-mail-notmuch-tag "emacsconf" "Tag to use when searching the Notmuch database for mail.")

emacsconf-mail-notmuch-last-message-for-talk: Return the most recent message from the speakers for TALK.
(defun emacsconf-mail-notmuch-last-message-for-talk (talk &optional subject)
  "Return the most recent message from the speakers for TALK.
Limit to SUBJECT if specified."
  (let ((message (json-parse-string
                  (shell-command-to-string
                   (format "notmuch search --limit=1 --format=json \"%s%s\""
                           (mapconcat
                            (lambda (email) (concat "from:" (shell-quote-argument email)))
                            (emacsconf-mail-get-all-email-addresses talk)
                            " or ")
                           (emacsconf-surround
                            " and "
                            (and emacsconf-mail-notmuch-tag (shell-quote-argument emacsconf-mail-notmuch-tag))
                            "" "")
                           (emacsconf-surround
                            " and subject:"
                            (and subject (shell-quote-argument subject)) "" "")))
                  :object-type 'alist)))
    (cons `(email . ,(plist-get talk :email))
          (when (> (length message) 0)
            (elt message 0)))))

Then I could display all the groups of speakers so that it's easy to check if any of the speakers haven't e-mailed us in a while.

emacsconf-mail-notmuch-show-latest-messages-from-speakers: Verify that the email addresses in GROUPS have e-mailed recently.
(defun emacsconf-mail-notmuch-show-latest-messages-from-speakers (groups &optional subject)
  "Verify that the email addresses in GROUPS have e-mailed recently.
When called interactively, pop up a report buffer showing the e-mails
and messages by date, with oldest messages on top.
This minimizes the risk of mail delivery issues and radio silence."
  (interactive (list (emacsconf-mail-groups (seq-filter
                               (lambda (o) (not (string= (plist-get o :status) "CANCELLED")))
                               (emacsconf-get-talk-info)))))
  (let ((results
         (sort (mapcar
                (lambda (group)
                  (emacsconf-mail-notmuch-last-message-for-talk (cadr group) subject))
                groups)
               (lambda (a b)
                 (< (or (alist-get 'timestamp a) -1)
                    (or (alist-get 'timestamp b) -1))))))
    (when (called-interactively-p 'any)
      (with-current-buffer (get-buffer-create "*Mail report*")
        (let ((inhibit-read-only t))
          (erase-buffer))
        (tabulated-list-mode)
        (setq
         tabulated-list-entries
         (mapcar
          (lambda (row)
            (list
             (alist-get 'thread row)
             (vector
              (alist-get 'email row)
              (or (alist-get 'date_relative row) "")
              (or (alist-get 'subject row) ""))))
          results))
        (setq tabulated-list-format [("Email" 30 t)
                                     ("Date" 10 nil)
                                     ("Subject" 30 t)])
        (local-set-key (kbd "RET") #'emacsconf-mail-notmuch-visit-thread-from-summary)
        (tabulated-list-print)
        (tabulated-list-init-header)
        (pop-to-buffer (current-buffer))))
    results))

If I press RET on a line, I can open the most recent thread. This is handled by the emacsconf-mail-notmuch-visit-thread-from-summary, which is simplified by using the thread ID as the tabulated list ID.

2023-10-07-18-21-55.svg
Figure 2: Viewing a thread in a different window

emacsconf-mail-notmuch-visit-thread-from-summary: Display the thread from the summary.
(defun emacsconf-mail-notmuch-visit-thread-from-summary ()
  "Display the thread from the summary."
  (interactive)
  (let (message-buffer)
    (save-window-excursion
      (setq message-buffer (notmuch-show (tabulated-list-get-id))))
    (display-buffer message-buffer t)))

We haven't heard from a few speakers in a while, so I'll probably e-mail them this weekend to double-check that I'm not getting delivery issues with my e-mails to them. If that doesn't get a reply, I might try other communication methods. If they're just busy, that's cool.

It's a lot easier to spot missing or old entries in a table than it is to try to remember who we haven't heard from recently, so hooray for tabulated-list-mode!

This code is in emacsconf-mail.el.

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

Summarizing #EmacsConf's growth over 5 years by year, and making an animated GIF

| emacs, emacsconf, python

Of course, after I charted EmacsConf's growth in terms of number of submissions and minutes, I realized I also wanted to just sum everything up by year. So here it is: 

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = df.drop('Weeks to CFP', axis=1).groupby(['Year']).sum()
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12,6))
fig1 = df['Count'].plot(kind="bar", ax=ax[0], title='Number of submissions')
fig2 = df['Minutes'].plot(kind="bar", ax=ax[1], title='Number of minutes')
fig.get_figure().savefig('emacsconf-by-year.png')
return df
Year Count Minutes
2019 28 429
2020 35 699
2021 44 578
2022 29 512
2023 39 730
emacsconf-by-year.png

I also wanted to make an animated GIF so that the cumulative graphs could be a little easier to understand. 

import pandas as pd
import matplotlib.pyplot as plt
import imageio as io
df = pd.DataFrame(data[1:], columns=data[0])
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12,6))
count = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
minutes = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Minutes', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
ax[0].set_ylim([0, count.max().max()])
ax[1].set_ylim([0, minutes.max().max()])
with io.get_writer('emacsconf-combined.gif', mode='I', duration=[500, 500, 500, 500, 1000], loop=0) as writer:
    for year in range(2019, 2024):
        count[year].plot(ax=ax[0], title='Cumulative submissions')
        minutes[year].plot(ax=ax[1], title='Cumulative minutes')
        ax[0].legend(loc='upper left')
        ax[1].legend(loc='upper left')
        for axis in ax:
            for line in axis.get_lines():
                if line.get_label() == '2023':
                    line.set_linewidth(5)
            for line in axis.legend().get_lines():
                if line.get_label() == '2023':
                    line.set_linewidth(5)        
        filename = f'emacsconf-combined-${year}.png'
        fig.get_figure().savefig(filename)
        image = io.v3.imread(filename)
        writer.append_data(image)
emacsconf-combined.gif
Figure 1: Animated GIF showing the cumulative total submissions and minutes

I am not quite sure what kind of story this data tells (aside from the fact that there sure are a lot of great talks), but it was fun to learn how to make more kinds of graphs and animate them too. Could be useful someday. =)

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

#EmacsConf backstage: looking at EmacsConf's growth over 5 years, and how to do pivot tables and graphs with Org Mode and the Python pandas library

| emacsconf, emacs, org, python

Having helped organize EmacsConf for a number of years now, I know that I usually panic about whether we have submissions partway through the call for participation. This causes us to extend the CFP deadline and ask people to ask more people to submit things, and then we end up with a wonderful deluge of talks that I then have to somehow squeeze into a reasonable-looking schedule.

This year, I managed to not panic and I also resisted the urge to extend the CFP deadline, trusting that there will actually be tons of cool stuff. It helped that my schedule SVG code let me visualize what the conference could feel like with the submissions so far, so we started with a reasonably nice one-track conference and built up from there. It also helped that I'd gone back to the submissions for 2022 and plotted them by the number of weeks before the CFP deadline, and I knew that there'd be a big spike from all those people whose Org DEADLINE: properties would nudge them into finalizing their proposals.

Out of curiosity, I wanted to see how the stats for this year compared with previous years. I wrote a small function to collect the data that I wanted to summarize:

emacsconf-count-submissions-by-week: Count submissions in INFO by distance to CFP-DEADLINE.
(defun emacsconf-count-submissions-by-week (&optional info cfp-deadline)
  "Count submissions in INFO by distance to CFP-DEADLINE."
  (setq cfp-deadline (or cfp-deadline emacsconf-cfp-deadline))
  (setq info (or info (emacsconf-get-talk-info)))
  (cons '("Weeks to CFP end date" "Count" "Hours")
        (mapcar (lambda (entry)
                  (list (car entry)
                        (length (cdr entry))
                        (apply '+ (mapcar 'cdr (cdr entry)))))
                (seq-group-by
                 'car
                 (sort
                  (seq-keep
                   (lambda (o)
                     (and (emacsconf-publish-talk-p o)
                          (plist-get o :date-submitted)
                          (cons (floor (/ (days-between (plist-get o :date-submitted) cfp-deadline)
                                          7.0))
                                (string-to-number
                                 (or (plist-get o :video-duration)
                                     (plist-get o :time)
                                     "0")))))
                   info)
                  (lambda (a b) (< (car a) (car b))))))))

and then I ran it against the different files for each year, filling in the previous years' data as needed. The resulting table is pretty long, so I've put that in a collapsible section. 

(let ((years `((2023 "~/proj/emacsconf/2023/private/conf.org" "2023-09-15")
               (2022 "~/proj/emacsconf/2022/private/conf.org" "2022-09-18")
               (2021 "~/proj/emacsconf/2021/private/conf.org" "2021-09-30")
               (2020 "~/proj/emacsconf/wiki/2020/submissions.org" "2020-09-30")
               (2019 "~/proj/emacsconf/2019/private/conf.org" "2019-08-31"))))
  (append
   '(("Weeks to CFP" "Year" "Count" "Minutes"))
   (seq-mapcat
    (lambda (year-info)
      (let ((emacsconf-org-file (elt year-info 1))
            (emacsconf-cfp-deadline (elt year-info 2))
            (year (car year-info)))
        (mapcar (lambda (o) (list (car o) year (cadr o) (elt o 2)))
                (cdr (emacsconf-count-submissions-by-week (emacsconf-get-talk-info) emacsconf-cfp-deadline)))))
    years)))
Table
Weeks to CFP Year Count Minutes
-12 2023 4 70
-9 2023 2 30
-7 2023 2 30
-5 2023 2 30
-4 2023 2 60
-3 2023 3 40
-2 2023 5 130
-1 2023 10 180
0 2023 8 140
1 2023 1 20
-8 2022 2 25
-5 2022 2 31
-3 2022 2 31
-2 2022 2 17
-1 2022 8 191
0 2022 8 110
1 2022 5 107
-8 2021 4 50
-7 2021 2 17
-6 2021 1 7
-5 2021 2 22
-4 2021 2 19
-3 2021 5 73
-2 2021 1 10
-1 2021 12 163
0 2021 13 197
1 2021 1 10
2 2021 1 10
-5 2020 1 10
-4 2020 1 15
-2 2020 1 30
-1 2020 4 68
0 2020 21 424
1 2020 7 152
-5 2019 2 45
-4 2019 1 21
-2 2019 6 126
-1 2019 9 82
0 2019 9 148
2 2019 1 7

Some talks were proposed off-list and are not captured here, and cancelled or withdrawn talks weren't included either. The times for previous years use the actual video time, and the times for this year use proposed times.

Off the top of my head, I didn't know of an easy way to make a pivot table or cross-tab using just Org Mode or Emacs Lisp. I tried using datamash, but I was having a hard time getting my output just the way I wanted it. Fortunately, it was super-easy to get my data from an Org table into Python so I could use pandas.pivot_table. Because I had used #+NAME: submissions-by-week to label the table, I could use :var data=submissions-by-week to refer to the data in my Python program. Then I could summarize them by week.

Here's the number of submissions by the number of weeks to the original CFP deadline, so we can see people generally like to target the CFP date. 

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True)
fig, ax = plt.subplots()
figure = df.plot(title='Number of submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('number-of-submissions.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 4
-9 0 0 0 0 2
-8 0 0 4 2 0
-7 0 0 2 0 2
-6 0 0 1 0 0
-5 2 1 2 2 2
-4 1 1 2 0 2
-3 0 0 5 2 3
-2 6 1 1 2 5
-1 9 4 12 8 10
0 9 21 13 8 8
1 0 7 1 5 1
2 1 0 1 0 0
number-of-submissions.png

Calculating the cumulative number of submissions might be more useful. Here, each row shows the number received so far. 

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax = plt.subplots()
figure = df.plot(title='Cumulative submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('cumulative-submissions.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 4
-9 0 0 0 0 6
-8 0 0 4 2 6
-7 0 0 6 2 8
-6 0 0 7 2 8
-5 2 1 9 4 10
-4 3 2 11 4 12
-3 3 2 16 6 15
-2 9 3 17 8 20
-1 18 7 29 16 30
0 27 28 42 24 38
1 27 35 43 29 39
2 28 35 44 29 39
cumulative-submissions.png
Figure 1: Cumulative submissions by number of weeks to CFP end date

And here's the cumulative number of minutes based on the proposals. 

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Minutes', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax = plt.subplots()
figure = df.plot(title='Cumulative minutes by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('cumulative-minutes.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 70
-9 0 0 0 0 100
-8 0 0 50 25 100
-7 0 0 67 25 130
-6 0 0 74 25 130
-5 45 10 96 56 160
-4 66 25 115 56 220
-3 66 25 188 87 260
-2 192 55 198 104 390
-1 274 123 361 295 570
0 422 547 558 405 710
1 422 699 568 512 730
2 429 699 578 512 730
cumulative-minutes.png
Figure 2: Cumulative minutes by number of weeks to the CFP end date

So… yeah… 730 minutes of talks for this year… I might've gotten a little carried away. But I like all the talks! And I want them to be captured in videos and maybe even transcribed by people who will take the time to change misrecognized words like Emax into Emacs! And I want people to be able to connect with other people who are interested in the sorts of stuff they're doing! So we're going to make it happen. The draft schedule's looking pretty full, but I think it'll work out, especially if the speakers send in their videos on time. Let's see how it all works out!

(…and look, I even got to learn how to do pivot tables and graphs with Python!)

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

#EmacsConf backstage: using e-mail templates for confirmations and acceptances

| emacsconf, emacs

We send a number of e-mails to speakers while coordinating EmacsConf. Here's a rough list of the standard e-mails:

  • confirmation that we've received their proposal and that we'll review it over the following week
  • acceptance
  • coordination with related talks
  • checking tentative schedule to see if people have any requests
  • instructions for uploading files
  • instructions for accessing the backstage area
  • confirmation that we've received their uploaded video
  • captions for review
  • miscellaneous todos
  • any large schedule changes
  • schedule confirmation and check-in instructions
  • thanks and resources
  • thanks and follow-up questions

Sending e-mails from within Emacs makes it super-easy to automate, of course.

I started off with putting e-mail templates in our public organizers' notebook because that made it easy to link to the drafts when asking other volunteers for feedback. As the templates settle down, I've been gradually moving some of them into our emacsconf-mail library so that I don't need to copy the template into each year's notebook.

The e-mail templates use the same template functions we use to make the wiki pages. Here's the function for confirming that we've received a submission and letting the speaker know when to expect comments:

emacsconf-mail-review: Let the speaker know we’ve received their proposal.
(defun emacsconf-mail-review (talk)
  "Let the speaker know we've received their proposal."
  (interactive (list (emacsconf-complete-talk-info)))
  (emacsconf-mail-prepare '(:subject "${conf-name} ${year} review: ${title}"
                       :cc "emacsconf-submit@gnu.org"
                       :reply-to "emacsconf-submit@gnu.org, ${email}, ${user-email}"
                       :mail-followup-to "emacsconf-submit@gnu.org, ${email}, ${user-email}"
                       :body "
Hi, ${speakers-short}!

Thanks for submitting your proposal! (ZZZ TODO: feedback)

We'll wait a week (~ ${notification-date}) in case the other volunteers want to chime in regarding your talk. =)

${signature}
")
            (plist-get talk :email)
            (list
             :user-email user-mail-address
             :signature user-full-name
             :title (plist-get talk :title)
             :email (plist-get talk :email)
             :conf-name emacsconf-name
             :speakers-short (plist-get talk :speakers-short)
             :year emacsconf-year
             :notification-date (plist-get talk :date-to-notify))))

I recently extended emacsconf-mail-prepare so that it can insert the template into a reply instead of always starting a new message. This allows the messages to be properly threaded in the emacsconf-submit list archives, which makes it easier to verify that all the submissions have been acknowledged.

emacsconf-mail-prepare: Prepare the e-mail following TEMPLATE. Send it to EMAIL.
(defun emacsconf-mail-prepare (template email attrs)
  "Prepare the e-mail following TEMPLATE. Send it to EMAIL.
Use ATTRS to fill in the template.
If `emacsconf-mail-batch-test' is non-nil, put the message in that buffer instead."
  (let ((fields '((:reply-to "Reply-To")
                  (:mail-followup-to "Mail-Followup-To")
                  (:cc "Cc"))))
    (if emacsconf-mail-batch-test
        (emacsconf-mail-prepare-for-batch-test template email attrs fields)
      ;; prepare to send the mail
      (if (and (derived-mode-p 'message-mode) (string-match "unsent mail" (buffer-name)))
          ;; add to headers
          (emacsconf-mail-update-reply-headers template email attrs fields) 
        ;; compose a new message
        (compose-mail
         email
         (emacsconf-replace-plist-in-string attrs (or (plist-get template :subject) ""))
         (seq-keep (lambda (field)
                     (when (plist-get template (car field))
                       (cons
                        (cadr field)
                        (emacsconf-replace-plist-in-string
                         attrs
                         (plist-get template (car field))))))
                   fields)))
      (message-sort-headers)
      (message-goto-body)
      (save-excursion
        (insert (string-trim (emacsconf-replace-plist-in-string attrs (plist-get template :body)))
                "\n\n")
        (goto-char (point-min))
        (emacsconf-mail-merge-wrap))
      (when (plist-get template :log-note)
        (mapc (lambda (talk)
                (emacsconf-mail-log-message-when-sent talk (plist-get template :log-note)))
              (cdr group))))))

There's a little function that I can add to message-send-hook to prevent me from sending a message if it still has ZZZ in it.

emacsconf-mail-check-for-zzz-before-sending: Throw an error if the ZZZ todo marker is still in the message.
(defun emacsconf-mail-check-for-zzz-before-sending ()
  "Throw an error if the ZZZ todo marker is still in the message.
Good for adding to `message-send-hook'."
  (save-excursion
    (goto-char (point-min))
    (when (re-search-forward "ZZZ" nil t)
      (unless (yes-or-no-p "ZZZ marker found. Send anyway? ")
        (error "ZZZ marker found.")))))

Here's the function for sending an acceptance letter:

emacsconf-mail-accept-talk: Send acceptance letter.
(defun emacsconf-mail-accept-talk (talk)
  "Send acceptance letter."
  (interactive (list (emacsconf-complete-talk-info)))
  (emacsconf-mail-prepare '(:subject "${conf-name} ${year} acceptance: ${title}"
                       :cc "emacsconf-submit@gnu.org"
                       :slugs nil
                       :reply-to "emacsconf-submit@gnu.org, ${email}, ${user-email}"
                       :mail-followup-to "emacsconf-submit@gnu.org, ${email}, ${user-email}"
                       :body
                       "
Hi, ${speakers-short}!

Looks like all systems are a go for your talk. Thanks for
proposing it! Your talk page is now at ${url} . Please feel free
to update it or e-mail us if you'd like help with any changes.${fill}

If you want to get started on your talk early, we have some
instructions at ${base}${year}/prepare/ that might help.
We strongly encourage speakers to prepare a talk video by
${video-target-date} in order to reduce technical risks and make
things flow more smoothly. Plus, we might be able to get it captioned
by volunteers, just like the talks last year. We'll save ${time} minutes
for your talk, not including time for Q&A. Don't sweat it if
you're a few minutes over or under. If it looks like a much shorter or
longer talk once you start getting into it, let us know and we might
be able to adjust.${wrap}

I'll follow up with the specific schedule for your talk once things
settle down. In the meantime, please let us know if you have any
questions or if there's anything we can do to help out!

${signature}"
                       :function emacsconf-mail-accept-talk
                       :log-note "accepted talk")
   (plist-get talk :email)
   (list
    :base emacsconf-base-url
    :user-email user-mail-address
    :year emacsconf-year
    :signature user-full-name
    :conf-name emacsconf-name
    :title (plist-get talk :title)
    :email (plist-get talk :email)
    :time (plist-get talk :time)
    :speakers-short (plist-get talk :speakers-short)
    :url (concat emacsconf-base-url (plist-get talk :url))
    :video-target-date emacsconf-video-target-date)))

We send confirmations and acceptances one at a time. Other e-mails are sent to all the speakers and it's easier to draft them in a batch. I'll cover that kind of mail merge in a separate post.

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

#EmacsConf backstage: adding a talk to the wiki

| emacsconf, emacs

The EmacsConf 2023 call for participation has finished, hooray! We've sent out acceptances and added talks to the wiki. We experimented with early acceptances this year, which was really nice because it gives people quick feedback and allows people to get started on their videos early. That meant that I needed to be able to easily add talks to the wiki throughout the call for participation. We use templates and an Ikiwiki directive to make it easier to consistently format talk pages. This post covers adding a talk to the wiki, and these functions are in the emacsconf-el repository.

Overview

Amin Bandali picked Ikiwiki for the wiki for emacsconf.org. I think it's because Ikiwiki works with plain text in a Git repository, which fits nicely with our workflow. I can use Emacs Lisp to generate files that are included in other files, and I can also use Emacs Lisp to generate the starting point for files that may be manually edited later on.

We organize conference pages by year. Under the directory for this year (2023/), we have:

talks
talk descriptions that can be manually edited
talks/SLUG.md
the description for the talk. Includes ../info/SLUG-nav.md, ../info/SLUG-before.md, and ../info/SLUG-after.md
info
automatically-generated files that are included before and after the talk description, and navigation links between talks
info/SLUG-nav.md
navigation links
info/SLUG-before.md
navigation links
info/SLUG-after.md
navigation links
schedule-details.md
list of talks
organizers-notebook/index.org
public planning notebook

The filenames and URLs for each talk are based on the ID for a talk. I store that in the SLUG property to make it easy to differentiate from CUSTOM_ID, since CUSTOM_ID is useful for lots of other things in Org Mode. I usually assign the slugs when I add the talks to our private conf.org file, although sometimes people suggest a specific ID.

Templating

Publishing to wiki pages and replying to e-mails are easier if I can substitute text into readable templates. There are a number of templating functions for Emacs Lisp, like the built-in tempo.el or s-lex-format from s.el. I ended up writing something that works with plists instead, since we use property lists (plists) all over the emacsconf-el library.

emacsconf-replace-plist-in-string: Replace ${keyword} from ATTRS in STRING.
(defun emacsconf-replace-plist-in-string (attrs string)
  "Replace ${keyword} from ATTRS in STRING."
  (let ((a attrs) name val)
    (while a
      (setq name (pop a) val (pop a))
      (when (stringp val)
        (setq string
              (replace-regexp-in-string (regexp-quote (concat "${" (substring (symbol-name name) 1) "}"))
                                        (or val "")
                                        string t t))))
    string))

It is also handy to be able to add text around another string only if the string is non-nil, and to provide a different string to use if it isn't specified..

emacsconf-surround: Concat BEFORE, TEXT, and AFTER if TEXT is specified, or return ALTERNATIVE.
(defun emacsconf-surround (before text after &optional alternative)
  "Concat BEFORE, TEXT, and AFTER if TEXT is specified, or return ALTERNATIVE."
  (if (and text (not (string= text "")))
      (concat (or before "") text (or after ""))
    alternative))

Getting the talk information

To get the data to fill in the template, we can run a bunch of different functions. This lets us add or remove functions when we need to. We pass the previous result to the next function in order to accumulate properties or modify them. The result is a property list for the current talk.

emacsconf-get-talk-info-for-subtree: Run ‘emacsconf-talk-info-functions’ to extract the info for this entry.
(defun emacsconf-get-talk-info-for-subtree ()
  "Run `emacsconf-talk-info-functions' to extract the info for this entry."
  (seq-reduce (lambda (prev val) (save-excursion (save-restriction (funcall val prev))))
              emacsconf-talk-info-functions
              nil))

emacsconf-talk-info-functions: Functions to collect information.
(defvar emacsconf-talk-info-functions
  '(emacsconf-get-talk-info-from-properties
    emacsconf-get-talk-categories
    emacsconf-get-talk-abstract-from-subtree
    emacsconf-get-talk-logbook
    emacsconf-add-talk-status
    emacsconf-add-checkin-time
    emacsconf-add-timezone-conversions
    emacsconf-add-speakers-with-pronouns
    emacsconf-add-live-info)
  "Functions to collect information.")

Getting the talk abstract

I add a *** Talk abstract subheading to the talk and put the rest of the submission under a *** Talk details subheading. This allows me to extract the text of the Talk abstract heading (or whatever matches emacsconf-abstract-heading-regexp, which is set to "abstract".).

emacsconf-get-subtree-entry: Return the text for the subtree matching HEADING-REGEXP.
(defun emacsconf-get-subtree-entry (heading-regexp)
  "Return the text for the subtree matching HEADING-REGEXP."
  (car
   (delq nil
         (org-map-entries
          (lambda ()
            (when (string-match heading-regexp (org-entry-get (point) "ITEM"))
              (org-get-entry)))
          nil 'tree))))

emacsconf-get-talk-abstract-from-subtree: Add the abstract from a subheading.
(defun emacsconf-get-talk-abstract-from-subtree (o)
  "Add the abstract from a subheading.
The subheading should match `emacsconf-abstract-heading-regexp'."
  (plist-put o :abstract (substring-no-properties (or (emacsconf-get-subtree-entry "abstract") ""))))

I include emacsconf-get-talk-abstract-from-subtree in emacsconf-talk-info-functions so that it retrieves that information when I call emacsconf-get-talk-info-for-subtree.

Publishing the talk page

We add accepted talks to the wiki so that people can see what kinds of talks will be at EmacsConf 2023. To add the talk to the wiki, I use emacsconf-publish-add-talk. It'll create the talk page without overwriting anything that's already there and redo the automatically-generated info pages that provide navigation, status, and so on.

emacsconf-publish-add-talk: Add the current talk to the wiki.
(defun emacsconf-publish-add-talk ()
  "Add the current talk to the wiki."
  (interactive)
  (emacsconf-publish-talk-page (emacsconf-get-talk-info-for-subtree))
  (emacsconf-publish-info-pages)
  (magit-status-setup-buffer emacsconf-directory))

The talk page includes the description and other resources.

emacsconf-publish-talk-page: Draft the talk page for O unless the page already exists or FORCE is non-nil.
(defun emacsconf-publish-talk-page (o &optional force)
  "Draft the talk page for O unless the page already exists or FORCE is non-nil."
  (interactive (list (emacsconf-get-talk-info-for-subtree)
                     (> (prefix-numeric-value current-prefix-arg) 1)))
  (let ((filename (expand-file-name (format "%s.md" (plist-get o :slug))
                                    (expand-file-name "talks" (expand-file-name emacsconf-year emacsconf-directory)))))
    (unless (file-directory-p (expand-file-name "talks" (expand-file-name emacsconf-year emacsconf-directory)))
      (mkdir (expand-file-name "talks" (expand-file-name emacsconf-year emacsconf-directory))))
    (when (or force (null (file-exists-p filename)))
      (with-temp-file filename
        (insert
         (emacsconf-replace-plist-in-string
          (emacsconf-convert-talk-abstract-to-markdown
           (append o (list
                      :speaker-info (emacsconf-publish-format-speaker-info o)
                      :meta "!meta"
                      :categories (if (plist-get o :categories)
                                      (mapconcat (lambda (o) (format "[[!taglink %s]]" o))
                                                 (plist-get o :categories)
                                                 " ")
                                    ""))))
          "[[${meta} title=\"${title}\"]]
[[${meta} copyright=\"Copyright &copy; ${year} ${speakers}\"]]
[[!inline pages=\"internal(${year}/info/${slug}-nav)\" raw=\"yes\"]]

<!-- Initially generated with emacsconf-publish-talk-page and then left alone for manual editing -->
<!-- You can manually edit this file to update the abstract, add links, etc. --->\n

# ${title}
${speaker-info}

[[!inline pages=\"internal(${year}/info/${slug}-before)\" raw=\"yes\"]]

${abstract-md}

[[!inline pages=\"internal(${year}/info/${slug}-after)\" raw=\"yes\"]]

[[!inline pages=\"internal(${year}/info/${slug}-nav)\" raw=\"yes\"]]

${categories}
"))))))

Ikiwiki uses Markdown, so we can take advantage of Org's Markdown export.

emacsconf-convert-talk-abstract-to-markdown: Set the :abstract-md property to a Markdown version of the abstract.
(defun emacsconf-convert-talk-abstract-to-markdown (o)
  "Set the :abstract-md property to a Markdown version of the abstract."
  (plist-put o :abstract-md (org-export-string-as (or (plist-get o :abstract) "") 'md t)))

Publishing the list of talks

The list of talks at https://emacsconf.org/2023/talks/ is grouped by track.

2023-09-26_14-00-19.png
Figure 1: List of talks

emacsconf-publish-schedule: Generate the schedule or program.
(defun emacsconf-publish-schedule (&optional info)
  "Generate the schedule or program."
  (interactive)
  (emacsconf-publish-schedule-svg-snippets)
  (setq info (or info (emacsconf-publish-prepare-for-display info)))
  (with-temp-file (expand-file-name "schedule-details.md"
                                    (expand-file-name emacsconf-year emacsconf-directory))
    (insert
     (if (member emacsconf-publishing-phase '(cfp program))
         (let ((sorted (emacsconf-publish-prepare-for-display (or info (emacsconf-get-talk-info)))))
           (mapconcat
            (lambda (track)
              (concat
               "Jump to: "
               ;; links to other tracks
               (string-join (seq-keep (lambda (track-link)
                                        (unless (string= (plist-get track-link :id)
                                                         (plist-get track :id))
                                          (format "<a href=\"#%s\">%s</a>"
                                                  (plist-get track-link :id)
                                                  (plist-get track-link :name))))
                                      emacsconf-tracks)
                            " | ")
               "\n\n"
               (let ((track-talks (seq-filter (lambda (o) (string= (plist-get o :track)
                                                                   (plist-get track :name)))
                                              sorted)))
                 (format
                  "<h1 id=\"%s\" class=\"sched-track %s\">%s (%d talks)</h1>\n%s"
                  (plist-get track :id)
                  (plist-get track :name)
                  (plist-get track :name)
                  (length track-talks)
                  (emacsconf-publish-format-main-schedule track-talks)))))
            emacsconf-tracks "\n\n"))
       (emacsconf-publish-format-interleaved-schedule info))))
  (when (member emacsconf-publishing-phase '(cfp program))
    (with-temp-file (expand-file-name
                     "draft-schedule.md"
                     (expand-file-name emacsconf-year emacsconf-directory))
      (insert
       "[[!sidebar content=\"\"]]\n\n" 
       "This is a *DRAFT* schedule.\n"
       (let ((emacsconf-publishing-phase 'schedule))
         (emacsconf-publish-format-interleaved-schedule info))))))

The emacsconf-format-main-schedule function displays the information for the talks in each track. It's pretty straightforward, but I put it in a function because I call it from a number of places.

emacsconf-publish-format-main-schedule: Include the schedule information for INFO.
(defun emacsconf-publish-format-main-schedule (info)
  "Include the schedule information for INFO."
  (mapconcat #'emacsconf-publish-sched-directive info "\n"))

We define an Ikiwiki sched directive that conditionally displays things depending on what we specify, so it's easy to add more information during the schedule or conference phase. This is templates/sched.md in the EmacsConf wiki git repository: 

<div data-start="<TMPL_VAR startutc>" data-end="<TMPL_VAR endutc>" class="sched-entry <TMPL_IF track>track-<TMPL_VAR track></TMPL_IF track>">
<div class="sched-meta">
<TMPL_IF start>
<span class="sched-time"><span class="sched-start"><TMPL_VAR start></span>
<TMPL_IF end> - <span class="sched-end"><TMPL_VAR end></span></TMPL_IF end>
</span></TMPL_IF start>
<TMPL_IF track> <span class="sched-track <TMPL_VAR track>"><TMPL_IF watch><a href="<TMPL_VAR watch>"></TMPL_IF><TMPL_VAR track><TMPL_IF watch></a></TMPL_IF></span></TMPL_IF track>
<TMPL_IF pad> <span class="sched-pad"><a href="<TMPL_VAR pad>">Etherpad</a></TMPL_IF pad>
<TMPL_IF q-and-a> <span class="sched-q-and-a">Q&amp;A: <TMPL_VAR q-and-a></span> </TMPL_IF q-and-a>
</div>
<div class="sched-title"><a href="<TMPL_VAR url>"><TMPL_VAR title></a></div>
<div class="sched-speakers"><TMPL_VAR speakers> <TMPL_IF note>- <TMPL_VAR note></TMPL_IF note></div>
<TMPL_IF resources>
<ul class="resources">
<TMPL_VAR resources>
</ul>
</TMPL_IF resources>
<TMPL_IF time><span class="sched-duration><TMPL_VAR time></span> minutes</TMPL_IF time>
<TMPL_IF slug> <span class="sched-slug">id:<TMPL_VAR slug></span></TMPL_IF slug>
</div>

This Emacs Lisp function converts a talk into that directive.

emacsconf-publish-sched-directive: Format the schedule directive with info for O.
(defun emacsconf-publish-sched-directive (o)
  "Format the schedule directive with info for O."
  (format "[[!template id=sched%s]]"
          (let ((result "")
                (attrs (append
                        (pcase emacsconf-publishing-phase
                          ('program
                           (list
                            :time (plist-get o :time)))
                          ((or 'schedule 'conference)
                           (list
                            :status (pcase (plist-get o :status)
                                      ("CAPTIONED" "captioned")
                                      ("PREREC_RECEIVED" "received")
                                      ("DONE" "done")
                                      ("STARTED" "now playing")
                                      (_ nil))
                            :time (plist-get o :time)
                            :q-and-a (plist-get o :qa-link) 
                            :pad (and emacsconf-publish-include-pads (plist-get o :pad-url))
                            :startutc (format-time-string "%FT%T%z" (plist-get o :start-time) t)
                            :endutc (format-time-string "%FT%T%z" (plist-get o :end-time) t)
                            :start (format-time-string "%-l:%M" (plist-get o :start-time) emacsconf-timezone)
                            :end (format-time-string "%-l:%M" (plist-get o :end-time) emacsconf-timezone)))
                          ('resources
                           (list
                            :pad nil
                            :channel nil
                            :resources (mapconcat (lambda (s) (concat "<li>" s "</li>"))
                                                  (emacsconf-link-file-formats-as-list
                                                   (append o
                                                           (list :base-url (format "%s%s/" emacsconf-media-base-url emacsconf-year)))
                                                   (append emacsconf-main-extensions (list "--answers.webm" "--answers.opus" "--answers.vtt")))
                                                  ""))))
                        (list
                         :title (plist-get o :title)
                         :url (concat "/" (plist-get o :url))
                         :speakers (plist-get o :speakers)
                         :track (if (member emacsconf-publishing-phase '(schedule conference)) (plist-get o :track))
                         :watch (plist-get o :watch-url)
                         :slug (plist-get o :slug)
                         :note
                         (string-join
                          (delq nil
                                (list
                                 (when (plist-get o :captions-edited)
                                   "captioned")
                                 (when (and (plist-get o :public)
                                            (or (plist-get o :toobnix-url)
                                                (plist-get o :video-file)))
                                   "video posted")))
                          ", ")
                         )
                        )))
            (while attrs
              (let ((field (pop attrs))
                    (val (pop attrs)))
                (when val
                  (setq result (concat result " " (substring (symbol-name field) 1) "=\"\"\"" val "\"\"\"")))))
            result)))

Publishing auto-generated navigation

It's nice to be able to navigate between talks without going back to the schedule page each time. This is handled by just keeping two extra copies of the list: one with the first talk popped off, and one with an extra element added to the beginning. Then we can use the heads of those lists for next/previous links.

2023-09-26_14-05-51.png
Figure 2: Navigation

emacsconf-publish-nav-pages: Generate links to the next and previous talks.
(defun emacsconf-publish-nav-pages (&optional talks)
  "Generate links to the next and previous talks.
During the schedule and conference phase, the talks are sorted by time.
Otherwise, they're sorted by track and then schedule."
  (interactive (list (emacsconf-publish-prepare-for-display (or emacsconf-schedule-draft (emacsconf-get-talk-info)))))
  (let* ((next-talks (cdr talks))
         (prev-talks (cons nil talks))
         (label (if (member emacsconf-publishing-phase '(schedule conference))
                    "time"
                  "track")))
    (unless (file-directory-p (expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory)))
      (mkdir (expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory))))
    (while talks
      (let* ((o (pop talks))
             (next-talk (emacsconf-format-talk-link (pop next-talks)))
             (prev-talk (emacsconf-format-talk-link (pop prev-talks))))
        (with-temp-file (expand-file-name (format "%s-nav.md" (plist-get o :slug))
                                          (expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory)))
          (insert (concat "\n<div class=\"talk-nav\">
Back to the [[talks]]  \n"
                          (if prev-talk (format "Previous by %s: %s  \n" label prev-talk) "")
                          (if next-talk (format "Next by %s: %s  \n" label next-talk) "")
                          (if (plist-get o :track) ; tagging doesn't work here because ikiwiki will list the nav page
                              (format "Track: <span class=\"sched-track %s\">%s</span>  \n" (plist-get o :track) (plist-get o :track))
                            "")
                          "</div>
")))))))

Before the talk description

We include some details about the schedule in the talk page, before the description.

2023-09-26_14-09-07.png
Figure 3: Description

emacsconf-publish-before-page: Generate the page that has the info included before the abstract.
(defun emacsconf-publish-before-page (talk &optional info)
  "Generate the page that has the info included before the abstract.
This includes the intro note, the schedule, and talk resources."
  (interactive (list (emacsconf-complete-talk-info)))
  (setq info (or info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
  (with-temp-file (expand-file-name (format "%s-before.md" (plist-get talk :slug))
                                    (expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory)))
    
    (insert "<!-- Automatically generated by emacsconf-publish-before-page -->\n")
    (insert (emacsconf-surround "" (plist-get talk :intro-note) "\n\n" ""))
    (let ((is-live (emacsconf-talk-live-p talk)))
      (when is-live (emacsconf-publish-captions-in-wiki talk))
      (when (member emacsconf-publishing-phase '(schedule conference))
        (insert (emacsconf-publish-format-talk-schedule-image talk info)))
      (insert (emacsconf-publish-format-talk-schedule-info talk) "\n\n")
      (insert
       (if (plist-get talk :public) (emacsconf-wiki-talk-resources talk) "")
       "\n# Description\n"))
    (insert "<!-- End of emacsconf-publish-before-page -->")))

emacsconf-publish-format-talk-schedule-info: Format schedule information for O.
(defun emacsconf-publish-format-talk-schedule-info (o)
  "Format schedule information for O."
  (let ((friendly (concat "/" emacsconf-year "/talks/" (plist-get o :slug) ))
        (timestamp (org-timestamp-from-string (plist-get o :scheduled))))
    (emacsconf-replace-plist-in-string
     (append o
             (list
              :format
              (concat (or (plist-get o :video-time)
                          (plist-get o :time))
                      "-min talk"
                      (if (plist-get o :q-and-a)
                          (format " followed by %s Q&A%s"
                                  (plist-get o :q-and-a)
                                  (if (eq emacsconf-publishing-phase 'conference)
                                      (format " (%s)"
                                              (if (string-match "live" (plist-get o :q-and-a))
                                                  (if (eq 'after (emacsconf-bbb-status o))
                                                      "done"
                                                    (format "<https://emacsconf.org/current/%s/room>" (plist-get o :slug)))
                                                (emacsconf-publish-webchat-link o)))
                                    ""))
                        ""))
              :pad-info
              (if emacsconf-publish-include-pads
                  (format "Etherpad: <https://pad.emacsconf.org/%s-%s>  \n" emacsconf-year (plist-get o :slug))
                "")
              :irc-info
              (format "Discuss on IRC: [#%s](%s)  \n" (plist-get o :channel)
                      (plist-get o :webchat-url))
              :status-info
              (if (member emacsconf-publishing-phase '(cfp program schedule conference)) (format "Status: %s  \n" (plist-get o :status-label)) "")
              :schedule-info
              (if (and (member emacsconf-publishing-phase '(schedule conference))
                       (not (emacsconf-talk-all-done-p o))
                       (not (string= (plist-get o :status) "CANCELLED")))
                  (let ((start (org-timestamp-to-time (org-timestamp-split-range timestamp)))
                        (end (org-timestamp-to-time (org-timestamp-split-range timestamp t))))
                    (format
                     "<div>Times in different timezones:</div><div class=\"times\" start=\"%s\" end=\"%s\"><div class=\"conf-time\">%s</div><div class=\"others\"><div>which is the same as:</div>%s</div></div><div><a href=\"/%s/watch/%s/\">Find out how to watch and participate</a></div>"
                     (format-time-string "%Y-%m-%dT%H:%M:%SZ" start t)
                     (format-time-string "%Y-%m-%dT%H:%M:%SZ" end t)
                     (emacsconf-timezone-string o emacsconf-timezone)
                     (string-join (emacsconf-timezone-strings
                                   o
                                   (seq-remove (lambda (zone) (string= emacsconf-timezone zone))
                                               emacsconf-timezones)) "<br />")
                     emacsconf-year
                     (plist-get (emacsconf-get-track (plist-get o :track)) :id)))
                "")))
     (concat
      "[[!toc  ]]
Format: ${format}  
${pad-info}${irc-info}${status-info}${schedule-info}\n" 
      (if (plist-get o :alternate-apac)
          (format "[[!inline pages=\"internal(%s/inline-alternate)\" raw=\"yes\"]]  \n" emacsconf-year)
        "")
      "\n"))))

After the talk description

After the talk description, we include a footer that makes it easier for people to e-mail questions using either the PUBLIC_EMAIL property of the talk or the emacsconf-org-private e-mail address.

2023-09-26_14-10-35.png

emacsconf-publish-format-email-questions-and-comments: Invite people to e-mail either the public contact for TALK or the private list.
(defun emacsconf-publish-format-email-questions-and-comments (talk)
  "Invite people to e-mail either the public contact for TALK or the private list."
  (format "Questions or comments? Please e-mail %s"
          (emacsconf-publish-format-public-email talk
                                         (or
                                          (and (string= (plist-get talk :public-email) "t")
                                               (plist-get talk :email))
                                          (plist-get talk :public-email)
                                          "emacsconf-org-private@gnu.org"))))

emacsconf-publish-after-page: Generate the page with info included after the abstract.
(defun emacsconf-publish-after-page (talk &optional info)
  "Generate the page with info included after the abstract.
This includes captions, contact, and an invitation to participate."
  (interactive (list (emacsconf-complete-talk-info)))
  ;; Contact information
  (with-temp-file (expand-file-name (format "%s-after.md" (plist-get talk :slug))
                                    (expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory)))
    (insert
     "<!-- Automatically generated by emacsconf-publish-after-page -->\n"
     "\n\n"
     ;; main transcript
     (if (plist-get talk :public) (emacsconf-publish-format-captions talk) "")
     (emacsconf-publish-format-email-questions-and-comments talk) "\n"
     (if (eq emacsconf-publishing-phase 'cfp)
         (format "\n----\nGot an idea for an EmacsConf talk or session? We'd love to hear from you! Check out the [[Call for Participation|/%s/cfp]] for details.\n" emacsconf-year)
       "")
     "\n\n<!-- End of emacsconf-publish-after-page -->\n")))

Whenever the schedule changes

This function makes it easier to regenerate all those dynamic pages that need to be updated whenever the schedule changes.

emacsconf-publish-info-pages: Populate year/info/*-nav, -before, and -after files.
(defun emacsconf-publish-info-pages (&optional info)
  "Populate year/info/*-nav, -before, and -after files."
  (interactive (list nil))
  (setq info (or info (emacsconf-publish-prepare-for-display info)))
  (emacsconf-publish-with-wiki-change
    (emacsconf-publish-nav-pages info)
    (emacsconf-publish-schedule info)
    (mapc (lambda (o)
            (emacsconf-publish-before-page o info)
            (emacsconf-publish-after-page o info))
          info)))

Summary

So once the review period has passed and we're ready to accept the talk, I change the status to WAITING_FOR_PREREC and find room for it in the schedule. Then I use emacsconf-publish-add-talk to add the talk description to the wiki. I review the files it generated, tweak hyperlinks as needed, add the pages to the Git repository, and push the commit to the server. If I rearrange talks or change times, I just need to run emacsconf-publish-info-pages and all the dynamically-generated pages will be updated.

View or add comments (Disqus), or e-mail me at sacha@sachachua.com