Categories: emacsconf

View topic page - RSS - Atom - Subscribe via email

#EmacsConf backstage: adding notes to Org logbook drawers from e-mails

| emacs, emacsconf, notmuch

Sometimes I want to work with all the talks associated with an email in my inbox. For example, maybe a speaker said that the draft schedules are fine, and I want to make a note of that in the conference Org file.

First we start with a function that gets the e-mail addresses for a talk. Some speakers have different e-mail addresses for public contact or private contact, and some e-mail us from other addresses.

emacsconf-mail-get-all-email-addresses: Return all the possible e-mail addresses for TALK.
(defun emacsconf-mail-get-all-email-addresses (talk)
  "Return all the possible e-mail addresses for TALK."
  (split-string
   (downcase
    (string-join
     (seq-uniq
      (seq-keep
       (lambda (field) (plist-get talk field))
       '(:email :public-email :email-alias)))
     ","))
   " *, *"))

Then we can use that to find the talks for a given e-mail address.

emacsconf-mail-talks: Return a list of talks matching EMAIL.
(defun emacsconf-mail-talks (email)
  "Return a list of talks matching EMAIL."
  (setq email (downcase (mail-strip-quoted-names email)))
  (seq-filter
   (lambda (o) (member email (emacsconf-mail-get-all-email-addresses o)))
   (emacsconf-get-talk-info)))

We can loop over that to add a note for the e-mail.

emacsconf-mail-add-to-logbook: Add to logbook for all matching talks from this speaker.
(defun emacsconf-mail-add-to-logbook (email note)
  "Add to logbook for all matching talks from this speaker."
  (interactive
   (let* ((email (mail-strip-quoted-names
                  (plist-get (plist-get (notmuch-show-get-message-properties) :headers)
                             :From)))
          (talks (emacsconf-mail-talks email)))
     (list
      email
      (read-string (format "Note for %s: "
                           (mapconcat (lambda (o) (plist-get o :slug))
                                      talks", "))))))
  (save-window-excursion
    (mapc
     (lambda (talk)
       (emacsconf-add-to-talk-logbook talk note))
     (emacsconf-mail-talks email))))

The actual addition of notes is handled by these functions.

emacsconf-add-to-logbook: Add NOTE as a logbook entry for the current subtree.
(defun emacsconf-add-to-logbook (note)
  "Add NOTE as a logbook entry for the current subtree."
  (move-marker org-log-note-return-to (point))
  (move-marker org-log-note-marker (point))
  (with-temp-buffer
    (insert note)
    (let ((org-log-note-purpose 'note))
      (org-store-log-note))))

Then we have a function that looks for the heading for a note and then adds a logbook entry to it.

emacsconf-add-to-talk-logbook: Add NOTE as a logbook entry for TALK.
(defun emacsconf-add-to-talk-logbook (talk note)
  "Add NOTE as a logbook entry for TALK."
  (interactive (list (emacsconf-complete-talk) (read-string "Note: ")))
  (save-excursion
    (emacsconf-with-talk-heading talk
      (emacsconf-add-to-logbook note))))

All together, that makes it easy to use Emacs as a very simple contact relationship management system where I can take notes based on the e-mails that come in.

output-2023-10-14-10:23:29.gif
Figure 1: Logging notes from e-mail

These functions are in emacsconf-mail.el.

#EmacsConf backstage: Using Spookfox to automate creating BigBlueButton rooms in Mozilla Firefox

| emacsconf, emacs, org

Naming conventions make it easier for other people to find things. Just like with file prefixes, I like to use a standard naming pattern for our BigBlueButton web conference rooms. For EmacsConf 2022, we used ec22-day-am-gen Speaker name (slugs). For EmacsConf 2023, I want to set up the BigBlueButton rooms before the schedule settles down, so I won't encode the time or track information into it. Instead, I'll use Speaker name (slugs) - emacsconf2023.

BigBlueButton does have an API for managing rooms, but that requires a shared secret that I don't know yet. I figured I'd just automate it through my browser. Over the last year, I've started using Spookfox to control the Firefox web browser from Emacs. It's been pretty handy for scrolling webpages up and down, so I wondered if I could replace my old xdotool-based automation. Here's what I came up with for this year.

First, I need a function that creates the BBB room for a group of talks and updates the Org entry with the URL. Adding a slight delay makes it a bit more reliable.

emacsconf-spookfox-create-bbb: Create a BBB room for this group of talks.
(defun emacsconf-spookfox-create-bbb (group)
  "Create a BBB room for this group of talks.
GROUP is (email . (talk talk talk)).
Needs a Spookfox connection."
  (let* ((bbb-name
          (format "%s (%s) - %s%s"
                  (mapconcat (lambda (o) (plist-get o :slug)) (cdr group) ", ")
                  (plist-get (cadr group) :speakers)
                  emacsconf-id
                  emacsconf-year))
         path
         (retrieve-command
          (format
           "window.location.origin + [...document.querySelectorAll('h4.room-name-text')].find((o) => o.textContent.trim() == '%s').closest('tr').querySelector('.delete-room').getAttribute('data-path')"
           bbb-name))
         (create-command (format "document.querySelector('#create-room-block').click();
document.querySelector('#create-room-name').value = \"%s\";
document.querySelector('#room_mute_on_join').click();
document.querySelector('.create-room-button').click();"
                                 bbb-name)))
    (setq path (spookfox-js-injection-eval-in-active-tab retrieve-command t))
    (unless path
      (dolist (cmd (split-string create-command ";"))
        (spookfox-js-injection-eval-in-active-tab cmd t)
        (sleep-for 2))
      (sleep-for 2)
      (setq path (spookfox-js-injection-eval-in-active-tab retrieve-command t)))
    (when path
      (dolist (talk (cdr group))
        (save-window-excursion
          (emacsconf-with-talk-heading talk
            (org-entry-put (point) "ROOM" path))))
      (cons bbb-name path))))

Then I need to iterate over the list of talks that have live Q&A sessions but don't have BBB rooms assigned yet so that I can create them.

emacsconf-spookfox-create-bbb-for-live-talks: Create BBB rooms for talks that don’t have them yet.
(defun emacsconf-spookfox-create-bbb-for-live-talks ()
  "Create BBB rooms for talks that don't have them yet."
  (let* ((talks (seq-filter
                 (lambda (o)
                   (and (string-match "live" (or (plist-get o :q-and-a) ""))
                        (not (string= (plist-get o :status) "CANCELLED"))
                        (not (plist-get o :bbb-room))))
                 (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
         (groups (and talks (emacsconf-mail-groups talks))))
    (dolist (group groups)
      (emacsconf-spookfox-create-bbb group))))

The result: a whole bunch of rooms ready for people to check in.

2023-10-14_09-24-34.png
Figure 1: BigBlueButton rooms

Using Spookfox to communicate with Firefox from Emacs Lisp made it easy to get data in and out of my browser. Handy!

This code is in emacsconf-spookfox.el.

#EmacsConf backstage: file prefixes

| emacs, org, emacsconf

Sometimes it makes sense to dynamically generate information related to a talk and then save it as an Org property so that I can manually edit it. For example, we like to name all the talk files using this pattern: "emacsconf-year-slug--title--speakers". That's a lot to type consistently! We can generate most of these prefixes automatically, but some might need tweaking, like when the talk title or speaker names have special characters.

Calculating the file prefix for a talk

First we need something that turns a string into an ID.

emacsconf-slugify: Turn S into an ID.
(defun emacsconf-slugify (s)
  "Turn S into an ID.
Replace spaces with dashes, remove non-alphanumeric characters,
and downcase the string."
  (replace-regexp-in-string
   " +" "-"
   (replace-regexp-in-string
    "[^a-z0-9 ]" ""
    (downcase s))))

Then we can use that to calculate the file prefix for a given talk.

emacsconf-file-prefix: Create the file prefix for TALK
(defun emacsconf-file-prefix (talk)
  "Create the file prefix for TALK."
  (concat emacsconf-id "-"
          emacsconf-year "-"
          (plist-get talk :slug) "--"
          (emacsconf-slugify (plist-get talk :title))
          (if (plist-get talk :speakers)
              (concat "--"
                     (emacsconf-slugify (plist-get talk :speakers)))
            "")))

Then we can map over all the talk entries that don't have FILE_PREFIX defined:

emacsconf-set-file-prefixes: Set the FILE_PREFIX property for each talk entry that needs it.
(defun emacsconf-set-file-prefixes ()
  "Set the FILE_PREFIX property for each talk entry that needs it."
  (interactive)
  (org-map-entries
   (lambda ()
     (org-entry-put
      (point) "FILE_PREFIX"
      (emacsconf-file-prefix (emacsconf-get-talk-info-for-subtree))))
   "SLUG={.}-FILE_PREFIX={.}"))

That stores the file prefix in an Org property, so we can edit it if it needs tweaking.

output-2023-10-10-15:17:17.gif
Figure 1: Setting the FILE_PREFIX for all talks that don't have that yet

Renaming files to match the file prefix

Now that we have that, how can we use it? One way is to rename files from within Emacs. I can mark multiple files with Dired's m command or work on them one at a time. If there are several files with the same extension, I can specify something to add to the filename to tell them apart.

emacsconf-rename-files: Rename the marked files or the current file to match TALK.
(defun emacsconf-rename-files (talk &optional filename)
  "Rename the marked files or the current file to match TALK.
If FILENAME is specified, use that as the extra part of the filename after the prefix.
This is useful for distinguishing files with the same extension.
Return the list of new filenames."
  (interactive (list (emacsconf-complete-talk-info)))
  (prog1
      (mapcar
       (lambda (file)
         (let* ((extra
                 (or filename
                     (read-string (format "Filename (%s): " (file-name-base file)))))
                (new-filename
                 (expand-file-name
                  (concat (plist-get talk :file-prefix)
                          (if (string= extra "")
                              ""
                            (concat "--" extra))
                          "."
                          (file-name-extension file))
                  (file-name-directory file))))
           (rename-file file new-filename t)
           new-filename))
       (or (dired-get-marked-files) (list (buffer-file-name))))
    (when (derived-mode-p 'dired-mode)
      (revert-buffer))))

output-2023-10-10-14:18:34.gif
Figure 2: Renaming multiple files

Working with files on other computers

Because Dired works over TRAMP, I can use that to rename files on a remote server without changing anything about the code. I can open the remote directory with Dired and everything just works.

TRAMP also makes it easy to copy a file to the backstage directory after it's renamed, which saves me having to do that as a separate step.

emacsconf-rename-and-upload-to-backstage: Rename marked files or the current file, then upload to backstage.
(defun emacsconf-rename-and-upload-to-backstage (talk &optional filename)
  "Rename marked files or the current file, then upload to backstage."
  (interactive (list (emacsconf-complete-talk-info)))
  (mapc
   (lambda (file)
     (copy-file
      file
      (expand-file-name
       (file-name-nondirectory file)
       emacsconf-backstage-dir)
      t))
   (emacsconf-rename-files talk)))

So if my emacsconf-backstage-dir is set to /ssh:orga@res:/var/www/res.emacsconf.org/2023/backstage, then it looks up the details for res in my ~/.ssh/config and copies the file there.

Renaming files using information from a JSON

What if I don't want to rename the files from Emacs? If I use Emacs's JSON support to export some information from the talks as a JSON file, then I can easily use that data from the command line.

Here's how I export the talk information:

emacsconf-talks-json: Return JSON format with a subset of talk information.
(defun emacsconf-publish-talks-json ()
  "Return JSON format with a subset of talk information."
  (json-encode
   (list
    :talks
    (mapcar
     (lambda (o)
       (apply
        'list
        (cons :start-time (format-time-string "%FT%T%z" (plist-get o :start-time) t))
        (cons :end-time (format-time-string "%FT%T%z" (plist-get o :end-time) t))
        (mapcar
         (lambda (field)
           (cons field (plist-get o field)))
         '(:slug :title :speakers :pronouns :pronunciation :url :track :file-prefix))))
     (emacsconf-filter-talks (emacsconf-get-talk-info))))))

emacsconf-publish-talks-json-to-files
(defun emacsconf-publish-talks-json-to-files ()
  "Export talk information as JSON so that we can use it in shell scripts."
  (interactive)
  (mapc (lambda (dir)
          (when (and dir (file-directory-p dir))
            (with-temp-file (expand-file-name "talks.json" dir)
              (insert (emacsconf-talks-json)))))
        (list emacsconf-res-dir emacsconf-ansible-directory)))

Then I can use jq to extract the information with

jq -r '.talks[] | select(.slug=="'$SLUG'")["file-prefix"]' < $TALKS_JSON

Here it is in the context of a shell script that renames the given file to match a talk's prefix.

#!/bin/bash
# 
# Usage: rename-original.sh $slug $file [$extra] [$talks-json]
SLUG=$1
FILE=$2
TALKS_JSON=${4:-~/current/talks.json}
EXTRA=""
if [ -z ${3-unset} ]; then
    EXTRA=""
elif [ -n "$3" ]; then
    EXTRA="--$3"
elif echo "$FILE" | grep -e '\(webm\|mp4\|mov\)'; then
    EXTRA="--original"
fi
filename=$(basename -- "$FILE")
extension="${filename##*.}"
filename="${filename%.*}"
FILE_PREFIX=$(jq -r '.talks[] | select(.slug=="'$SLUG'")["file-prefix"]' < $TALKS_JSON)
mv "$FILE" $FILE_PREFIX$EXTRA.$extension
echo $FILE_PREFIX$EXTRA.$extension
# Copy to original if needed
if [ -f $FILE_PREFIX--original.webm ] && [ ! -f $FILE_PREFIX--main.$extension ]; then
    cp $FILE_PREFIX--original.$extension $FILE_PREFIX--main.webm
fi

Then I can use something like rename-original.sh emacsconf video.webm to emacsconf-2023-emacsconf--emacsconforg-how-we-use-org-mode-and-tramp-to-organize-and-run-a-multitrack-conference--sacha-chua--original.webm.

Working with PsiTransfer-uploaded files

JSON support is useful for getting files into our system, too. For EmacsConf 2022, we used PsiTransfer as a password-protected web-based file upload service. That was much easier for speakers to deal with than FTP, especially for large files. PsiTransfer makes a JSON file for each batch of uploads, which is handy because the uploaded files are named based on the key instead of keeping their filenames and extensions. I wrote a function to copy an uploaded file from the PsiTransfer directory to the backstage directory, renaming it along the way. That meant that I could open the JSON for the uploaded files via TRAMP and then copy a file between two remote directories without manually downloading it to my computer.

emacsconf-upload-copy-from-json: Parse PsiTransfer JSON files and copy the uploaded file to the backstage directory.
(defun emacsconf-upload-copy-from-json (talk key filename)
  "Parse PsiTransfer JSON files and copy the uploaded file to the backstage directory.
The file is associated with TALK. KEY identifies the file in a multi-file upload.
FILENAME specifies an extra string to add to the file prefix if needed."
  (interactive (let-alist (json-parse-string (buffer-string) :object-type 'alist)
                 (list (emacsconf-complete-talk-info)
                       .metadata.key
                       (read-string (format "Filename: ")))))
  (let ((new-filename (concat (plist-get talk :file-prefix)
                              (if (string= filename "")
                                  filename
                                (concat "--" filename))
                              "."
                              (let-alist (json-parse-string (buffer-string) :object-type 'alist)
                                (file-name-extension .metadata.name)))))
    (copy-file
     key
     (expand-file-name new-filename emacsconf-backstage-dir)
     t)))

So that's how I can handle things that can be mostly automated but that might need a little human intervention: use Emacs Lisp to make a starting point, tweak it a little if needed, and then make it easy to use that value elsewhere. Renaming files can be tricky, so it's good to reduce the chance for typos!

#EmacsConf backstage: reviewing the last message from a speaker

| emacs, emacsconf, notmuch

One of the things I keep an eye out for when organizing EmacsConf is the most recent time we heard from a speaker. Sometimes life happens and speakers get too busy to prepare a video, so we might offer to let them do it live. Sometimes e-mail delivery issues get in the way and we don't hear from speakers because some server in between has spam filters set too strong. So I made a function that lists the most recent e-mail we got from the speaker that includes "emacsconf" in it. That was a good excuse to learn more about tabulated-list-mode.

2023-10-07-13-18-04.svg
Figure 1: Redacted view of most recent e-mails from speakers

I started by figuring out how to get all the e-mail addresses associated with a talk.

emacsconf-mail-get-all-email-addresses: Return all the possible e-mail addresses for TALK.
(defun emacsconf-mail-get-all-email-addresses (talk)
  "Return all the possible e-mail addresses for TALK."
  (split-string
   (downcase
    (string-join
     (seq-uniq
      (seq-keep
       (lambda (field) (plist-get talk field))
       '(:email :public-email :email-alias)))
     ","))
   " *, *"))

Then I figured out the notmuch search to use to get all messages. Some people write a lot, so I limited it to just the ones that have emacsconf as well. Notmuch can return JSON, so that's easy to parse.

emacsconf-mail-notmuch-tag: Tag to use when searching the Notmuch database for mail.
(defvar emacsconf-mail-notmuch-tag "emacsconf" "Tag to use when searching the Notmuch database for mail.")

emacsconf-mail-notmuch-last-message-for-talk: Return the most recent message from the speakers for TALK.
(defun emacsconf-mail-notmuch-last-message-for-talk (talk &optional subject)
  "Return the most recent message from the speakers for TALK.
Limit to SUBJECT if specified."
  (let ((message (json-parse-string
                  (shell-command-to-string
                   (format "notmuch search --limit=1 --format=json \"%s%s\""
                           (mapconcat
                            (lambda (email) (concat "from:" (shell-quote-argument email)))
                            (emacsconf-mail-get-all-email-addresses talk)
                            " or ")
                           (emacsconf-surround
                            " and "
                            (and emacsconf-mail-notmuch-tag (shell-quote-argument emacsconf-mail-notmuch-tag))
                            "" "")
                           (emacsconf-surround
                            " and subject:"
                            (and subject (shell-quote-argument subject)) "" "")))
                  :object-type 'alist)))
    (cons `(email . ,(plist-get talk :email))
          (when (> (length message) 0)
            (elt message 0)))))

Then I could display all the groups of speakers so that it's easy to check if any of the speakers haven't e-mailed us in a while.

emacsconf-mail-notmuch-show-latest-messages-from-speakers: Verify that the email addresses in GROUPS have e-mailed recently.
(defun emacsconf-mail-notmuch-show-latest-messages-from-speakers (groups &optional subject)
  "Verify that the email addresses in GROUPS have e-mailed recently.
When called interactively, pop up a report buffer showing the e-mails
and messages by date, with oldest messages on top.
This minimizes the risk of mail delivery issues and radio silence."
  (interactive (list (emacsconf-mail-groups (seq-filter
                               (lambda (o) (not (string= (plist-get o :status) "CANCELLED")))
                               (emacsconf-get-talk-info)))))
  (let ((results
         (sort (mapcar
                (lambda (group)
                  (emacsconf-mail-notmuch-last-message-for-talk (cadr group) subject))
                groups)
               (lambda (a b)
                 (< (or (alist-get 'timestamp a) -1)
                    (or (alist-get 'timestamp b) -1))))))
    (when (called-interactively-p 'any)
      (with-current-buffer (get-buffer-create "*Mail report*")
        (let ((inhibit-read-only t))
          (erase-buffer))
        (tabulated-list-mode)
        (setq
         tabulated-list-entries
         (mapcar
          (lambda (row)
            (list
             (alist-get 'thread row)
             (vector
              (alist-get 'email row)
              (or (alist-get 'date_relative row) "")
              (or (alist-get 'subject row) ""))))
          results))
        (setq tabulated-list-format [("Email" 30 t)
                                     ("Date" 10 nil)
                                     ("Subject" 30 t)])
        (local-set-key (kbd "RET") #'emacsconf-mail-notmuch-visit-thread-from-summary)
        (tabulated-list-print)
        (tabulated-list-init-header)
        (pop-to-buffer (current-buffer))))
    results))

If I press RET on a line, I can open the most recent thread. This is handled by the emacsconf-mail-notmuch-visit-thread-from-summary, which is simplified by using the thread ID as the tabulated list ID.

2023-10-07-18-21-55.svg
Figure 2: Viewing a thread in a different window

emacsconf-mail-notmuch-visit-thread-from-summary: Display the thread from the summary.
(defun emacsconf-mail-notmuch-visit-thread-from-summary ()
  "Display the thread from the summary."
  (interactive)
  (let (message-buffer)
    (save-window-excursion
      (setq message-buffer (notmuch-show (tabulated-list-get-id))))
    (display-buffer message-buffer t)))

We haven't heard from a few speakers in a while, so I'll probably e-mail them this weekend to double-check that I'm not getting delivery issues with my e-mails to them. If that doesn't get a reply, I might try other communication methods. If they're just busy, that's cool.

It's a lot easier to spot missing or old entries in a table than it is to try to remember who we haven't heard from recently, so hooray for tabulated-list-mode!

This code is in emacsconf-mail.el.

Summarizing #EmacsConf's growth over 5 years by year, and making an animated GIF

| emacs, emacsconf, python

Of course, after I charted EmacsConf's growth in terms of number of submissions and minutes, I realized I also wanted to just sum everything up by year. So here it is:

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = df.drop('Weeks to CFP', axis=1).groupby(['Year']).sum()
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12,6))
fig1 = df['Count'].plot(kind="bar", ax=ax[0], title='Number of submissions')
fig2 = df['Minutes'].plot(kind="bar", ax=ax[1], title='Number of minutes')
fig.get_figure().savefig('emacsconf-by-year.png')
return df
Year Count Minutes
2019 28 429
2020 35 699
2021 44 578
2022 29 512
2023 39 730
emacsconf-by-year.png

I also wanted to make an animated GIF so that the cumulative graphs could be a little easier to understand.

import pandas as pd
import matplotlib.pyplot as plt
import imageio as io
df = pd.DataFrame(data[1:], columns=data[0])
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12,6))
count = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
minutes = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Minutes', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
ax[0].set_ylim([0, count.max().max()])
ax[1].set_ylim([0, minutes.max().max()])
with io.get_writer('emacsconf-combined.gif', mode='I', duration=[500, 500, 500, 500, 1000], loop=0) as writer:
    for year in range(2019, 2024):
        count[year].plot(ax=ax[0], title='Cumulative submissions')
        minutes[year].plot(ax=ax[1], title='Cumulative minutes')
        ax[0].legend(loc='upper left')
        ax[1].legend(loc='upper left')
        for axis in ax:
            for line in axis.get_lines():
                if line.get_label() == '2023':
                    line.set_linewidth(5)
            for line in axis.legend().get_lines():
                if line.get_label() == '2023':
                    line.set_linewidth(5)        
        filename = f'emacsconf-combined-${year}.png'
        fig.get_figure().savefig(filename)
        image = io.v3.imread(filename)
        writer.append_data(image)
emacsconf-combined.gif
Figure 1: Animated GIF showing the cumulative total submissions and minutes

I am not quite sure what kind of story this data tells (aside from the fact that there sure are a lot of great talks), but it was fun to learn how to make more kinds of graphs and animate them too. Could be useful someday. =)

#EmacsConf backstage: looking at EmacsConf's growth over 5 years, and how to do pivot tables and graphs with Org Mode and the Python pandas library

| emacsconf, emacs, org, python

Having helped organize EmacsConf for a number of years now, I know that I usually panic about whether we have submissions partway through the call for participation. This causes us to extend the CFP deadline and ask people to ask more people to submit things, and then we end up with a wonderful deluge of talks that I then have to somehow squeeze into a reasonable-looking schedule.

This year, I managed to not panic and I also resisted the urge to extend the CFP deadline, trusting that there will actually be tons of cool stuff. It helped that my schedule SVG code let me visualize what the conference could feel like with the submissions so far, so we started with a reasonably nice one-track conference and built up from there. It also helped that I'd gone back to the submissions for 2022 and plotted them by the number of weeks before the CFP deadline, and I knew that there'd be a big spike from all those people whose Org DEADLINE: properties would nudge them into finalizing their proposals.

Out of curiosity, I wanted to see how the stats for this year compared with previous years. I wrote a small function to collect the data that I wanted to summarize:

emacsconf-count-submissions-by-week: Count submissions in INFO by distance to CFP-DEADLINE.
(defun emacsconf-count-submissions-by-week (&optional info cfp-deadline)
  "Count submissions in INFO by distance to CFP-DEADLINE."
  (setq cfp-deadline (or cfp-deadline emacsconf-cfp-deadline))
  (setq info (or info (emacsconf-get-talk-info)))
  (cons '("Weeks to CFP end date" "Count" "Hours")
        (mapcar (lambda (entry)
                  (list (car entry)
                        (length (cdr entry))
                        (apply '+ (mapcar 'cdr (cdr entry)))))
                (seq-group-by
                 'car
                 (sort
                  (seq-keep
                   (lambda (o)
                     (and (emacsconf-publish-talk-p o)
                          (plist-get o :date-submitted)
                          (cons (floor (/ (days-between (plist-get o :date-submitted) cfp-deadline)
                                          7.0))
                                (string-to-number
                                 (or (plist-get o :video-duration)
                                     (plist-get o :time)
                                     "0")))))
                   info)
                  (lambda (a b) (< (car a) (car b))))))))

and then I ran it against the different files for each year, filling in the previous years' data as needed. The resulting table is pretty long, so I've put that in a collapsible section.

(let ((years `((2023 "~/proj/emacsconf/2023/private/conf.org" "2023-09-15")
               (2022 "~/proj/emacsconf/2022/private/conf.org" "2022-09-18")
               (2021 "~/proj/emacsconf/2021/private/conf.org" "2021-09-30")
               (2020 "~/proj/emacsconf/wiki/2020/submissions.org" "2020-09-30")
               (2019 "~/proj/emacsconf/2019/private/conf.org" "2019-08-31"))))
  (append
   '(("Weeks to CFP" "Year" "Count" "Minutes"))
   (seq-mapcat
    (lambda (year-info)
      (let ((emacsconf-org-file (elt year-info 1))
            (emacsconf-cfp-deadline (elt year-info 2))
            (year (car year-info)))
        (mapcar (lambda (o) (list (car o) year (cadr o) (elt o 2)))
                (cdr (emacsconf-count-submissions-by-week (emacsconf-get-talk-info) emacsconf-cfp-deadline)))))
    years)))
Table
Weeks to CFP Year Count Minutes
-12 2023 4 70
-9 2023 2 30
-7 2023 2 30
-5 2023 2 30
-4 2023 2 60
-3 2023 3 40
-2 2023 5 130
-1 2023 10 180
0 2023 8 140
1 2023 1 20
-8 2022 2 25
-5 2022 2 31
-3 2022 2 31
-2 2022 2 17
-1 2022 8 191
0 2022 8 110
1 2022 5 107
-8 2021 4 50
-7 2021 2 17
-6 2021 1 7
-5 2021 2 22
-4 2021 2 19
-3 2021 5 73
-2 2021 1 10
-1 2021 12 163
0 2021 13 197
1 2021 1 10
2 2021 1 10
-5 2020 1 10
-4 2020 1 15
-2 2020 1 30
-1 2020 4 68
0 2020 21 424
1 2020 7 152
-5 2019 2 45
-4 2019 1 21
-2 2019 6 126
-1 2019 9 82
0 2019 9 148
2 2019 1 7

Some talks were proposed off-list and are not captured here, and cancelled or withdrawn talks weren't included either. The times for previous years use the actual video time, and the times for this year use proposed times.

Off the top of my head, I didn't know of an easy way to make a pivot table or cross-tab using just Org Mode or Emacs Lisp. I tried using datamash, but I was having a hard time getting my output just the way I wanted it. Fortunately, it was super-easy to get my data from an Org table into Python so I could use pandas.pivot_table. Because I had used #+NAME: submissions-by-week to label the table, I could use :var data=submissions-by-week to refer to the data in my Python program. Then I could summarize them by week.

Here's the number of submissions by the number of weeks to the original CFP deadline, so we can see people generally like to target the CFP date.

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True)
fig, ax = plt.subplots()
figure = df.plot(title='Number of submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('number-of-submissions.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 4
-9 0 0 0 0 2
-8 0 0 4 2 0
-7 0 0 2 0 2
-6 0 0 1 0 0
-5 2 1 2 2 2
-4 1 1 2 0 2
-3 0 0 5 2 3
-2 6 1 1 2 5
-1 9 4 12 8 10
0 9 21 13 8 8
1 0 7 1 5 1
2 1 0 1 0 0
number-of-submissions.png

Calculating the cumulative number of submissions might be more useful. Here, each row shows the number received so far.

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax = plt.subplots()
figure = df.plot(title='Cumulative submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('cumulative-submissions.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 4
-9 0 0 0 0 6
-8 0 0 4 2 6
-7 0 0 6 2 8
-6 0 0 7 2 8
-5 2 1 9 4 10
-4 3 2 11 4 12
-3 3 2 16 6 15
-2 9 3 17 8 20
-1 18 7 29 16 30
0 27 28 42 24 38
1 27 35 43 29 39
2 28 35 44 29 39
cumulative-submissions.png
Figure 1: Cumulative submissions by number of weeks to CFP end date

And here's the cumulative number of minutes based on the proposals.

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Minutes', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax = plt.subplots()
figure = df.plot(title='Cumulative minutes by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('cumulative-minutes.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 70
-9 0 0 0 0 100
-8 0 0 50 25 100
-7 0 0 67 25 130
-6 0 0 74 25 130
-5 45 10 96 56 160
-4 66 25 115 56 220
-3 66 25 188 87 260
-2 192 55 198 104 390
-1 274 123 361 295 570
0 422 547 558 405 710
1 422 699 568 512 730
2 429 699 578 512 730
cumulative-minutes.png
Figure 2: Cumulative minutes by number of weeks to the CFP end date

So… yeah… 730 minutes of talks for this year… I might've gotten a little carried away. But I like all the talks! And I want them to be captured in videos and maybe even transcribed by people who will take the time to change misrecognized words like Emax into Emacs! And I want people to be able to connect with other people who are interested in the sorts of stuff they're doing! So we're going to make it happen. The draft schedule's looking pretty full, but I think it'll work out, especially if the speakers send in their videos on time. Let's see how it all works out!

(…and look, I even got to learn how to do pivot tables and graphs with Python!)

#EmacsConf backstage: using e-mail templates for confirmations and acceptances

| emacsconf, emacs

We send a number of e-mails to speakers while coordinating EmacsConf. Here's a rough list of the standard e-mails:

  • confirmation that we've received their proposal and that we'll review it over the following week
  • acceptance
  • coordination with related talks
  • checking tentative schedule to see if people have any requests
  • instructions for uploading files
  • instructions for accessing the backstage area
  • confirmation that we've received their uploaded video
  • captions for review
  • miscellaneous todos
  • any large schedule changes
  • schedule confirmation and check-in instructions
  • thanks and resources
  • thanks and follow-up questions

Sending e-mails from within Emacs makes it super-easy to automate, of course.

I started off with putting e-mail templates in our public organizers' notebook because that made it easy to link to the drafts when asking other volunteers for feedback. As the templates settle down, I've been gradually moving some of them into our emacsconf-mail library so that I don't need to copy the template into each year's notebook.

The e-mail templates use the same template functions we use to make the wiki pages. Here's the function for confirming that we've received a submission and letting the speaker know when to expect comments:

emacsconf-mail-review: Let the speaker know we’ve received their proposal.
(defun emacsconf-mail-review (talk)
  "Let the speaker know we've received their proposal."
  (interactive (list (emacsconf-complete-talk-info)))
  (emacsconf-mail-prepare '(:subject "${conf-name} ${year} review: ${title}"
                       :cc "emacsconf-submit@gnu.org"
                       :reply-to "emacsconf-submit@gnu.org, ${email}, ${user-email}"
                       :mail-followup-to "emacsconf-submit@gnu.org, ${email}, ${user-email}"
                       :body "
Hi, ${speakers-short}!

Thanks for submitting your proposal! (ZZZ TODO: feedback)

We'll wait a week (~ ${notification-date}) in case the other volunteers want to chime in regarding your talk. =)

${signature}
")
            (plist-get talk :email)
            (list
             :user-email user-mail-address
             :signature user-full-name
             :title (plist-get talk :title)
             :email (plist-get talk :email)
             :conf-name emacsconf-name
             :speakers-short (plist-get talk :speakers-short)
             :year emacsconf-year
             :notification-date (plist-get talk :date-to-notify))))

I recently extended emacsconf-mail-prepare so that it can insert the template into a reply instead of always starting a new message. This allows the messages to be properly threaded in the emacsconf-submit list archives, which makes it easier to verify that all the submissions have been acknowledged.

emacsconf-mail-prepare: Prepare the e-mail following TEMPLATE. Send it to EMAIL.
(defun emacsconf-mail-prepare (template email attrs)
  "Prepare the e-mail following TEMPLATE. Send it to EMAIL.
Use ATTRS to fill in the template.
If `emacsconf-mail-batch-test' is non-nil, put the message in that buffer instead."
  (let ((fields '((:reply-to "Reply-To")
                  (:mail-followup-to "Mail-Followup-To")
                  (:cc "Cc"))))
    (if emacsconf-mail-batch-test
        (emacsconf-mail-prepare-for-batch-test template email attrs fields)
      ;; prepare to send the mail
      (if (and (derived-mode-p 'message-mode) (string-match "unsent mail" (buffer-name)))
          ;; add to headers
          (emacsconf-mail-update-reply-headers template email attrs fields) 
        ;; compose a new message
        (compose-mail
         email
         (emacsconf-replace-plist-in-string attrs (or (plist-get template :subject) ""))
         (seq-keep (lambda (field)
                     (when (plist-get template (car field))
                       (cons
                        (cadr field)
                        (emacsconf-replace-plist-in-string
                         attrs
                         (plist-get template (car field))))))
                   fields)))
      (message-sort-headers)
      (message-goto-body)
      (save-excursion
        (insert (string-trim (emacsconf-replace-plist-in-string attrs (plist-get template :body)))
                "\n\n")
        (goto-char (point-min))
        (emacsconf-mail-merge-wrap))
      (when (plist-get template :log-note)
        (mapc (lambda (talk)
                (emacsconf-mail-log-message-when-sent talk (plist-get template :log-note)))
              (cdr group))))))

There's a little function that I can add to message-send-hook to prevent me from sending a message if it still has ZZZ in it.

emacsconf-mail-check-for-zzz-before-sending: Throw an error if the ZZZ todo marker is still in the message.
(defun emacsconf-mail-check-for-zzz-before-sending ()
  "Throw an error if the ZZZ todo marker is still in the message.
Good for adding to `message-send-hook'."
  (save-excursion
    (goto-char (point-min))
    (when (re-search-forward "ZZZ" nil t)
      (unless (yes-or-no-p "ZZZ marker found. Send anyway? ")
        (error "ZZZ marker found.")))))

Here's the function for sending an acceptance letter:

emacsconf-mail-accept-talk: Send acceptance letter.
(defun emacsconf-mail-accept-talk (talk)
  "Send acceptance letter."
  (interactive (list (emacsconf-complete-talk-info)))
  (emacsconf-mail-prepare '(:subject "${conf-name} ${year} acceptance: ${title}"
                       :cc "emacsconf-submit@gnu.org"
                       :slugs nil
                       :reply-to "emacsconf-submit@gnu.org, ${email}, ${user-email}"
                       :mail-followup-to "emacsconf-submit@gnu.org, ${email}, ${user-email}"
                       :body
                       "
Hi, ${speakers-short}!

Looks like all systems are a go for your talk. Thanks for
proposing it! Your talk page is now at ${url} . Please feel free
to update it or e-mail us if you'd like help with any changes.${fill}

If you want to get started on your talk early, we have some
instructions at ${base}${year}/prepare/ that might help.
We strongly encourage speakers to prepare a talk video by
${video-target-date} in order to reduce technical risks and make
things flow more smoothly. Plus, we might be able to get it captioned
by volunteers, just like the talks last year. We'll save ${time} minutes
for your talk, not including time for Q&A. Don't sweat it if
you're a few minutes over or under. If it looks like a much shorter or
longer talk once you start getting into it, let us know and we might
be able to adjust.${wrap}

I'll follow up with the specific schedule for your talk once things
settle down. In the meantime, please let us know if you have any
questions or if there's anything we can do to help out!

${signature}"
                       :function emacsconf-mail-accept-talk
                       :log-note "accepted talk")
   (plist-get talk :email)
   (list
    :base emacsconf-base-url
    :user-email user-mail-address
    :year emacsconf-year
    :signature user-full-name
    :conf-name emacsconf-name
    :title (plist-get talk :title)
    :email (plist-get talk :email)
    :time (plist-get talk :time)
    :speakers-short (plist-get talk :speakers-short)
    :url (concat emacsconf-base-url (plist-get talk :url))
    :video-target-date emacsconf-video-target-date)))

We send confirmations and acceptances one at a time. Other e-mails are sent to all the speakers and it's easier to draft them in a batch. I'll cover that kind of mail merge in a separate post.