I figured out multi-track streaming so close to EmacsConf 2022 that there wasn't enough time to get other volunteers used to working with the setup, especially since I was still scrambling to figure out more infrastructure as the conference approached. We decided I'd run both streams myself, which meant I needed to make things as automatic as possible so that I wouldn't go crazy. I wanted a lot of things to happen automatically: playing recorded intros and videos, browsing to the right URLs depending on the type of Q&A, publishing updates to the wiki, and so on.

I used timers and TODO state changes to execute commands via TRAMP, which was pretty cool for the most part. But it turned out TRAMP doesn't like being called when it's already running, like when it's being called from two timers going off at the same time. It gives a "Forbidden reentrant call of TRAMP". We found a couple of quick workarounds: I could reschedule the talks to be a minute apart, or I could cancel the conflicting timer and just start them with the shell scripts.

Last year, we had a shell script that played the intro and the main talk, and other scripts to handle the Q&A by opening BigBlueButton, Etherpad, or the IRC channel. Much of the logic was in Emacs Lisp because it was easy to write it that way. For this year, I wanted to write a script that handled the intro, video, and Q&A portions. This is now in roles/obs/templates/handle-session.

#!/bin/bash
# 
#
# Handle the intro/talk/Q&A for a session
# Usage: handle-session $SLUG

YEAR=""
BASE_DIR=""
FIREFOX_NAME=firefox-esr
SLUG=$1

# Kill background music if playing
if screen -list | grep -q background; then
    screen -S background -X quit
fi

# Update the status
sudo -u  talk $SLUG PLAYING &

# Update the overlay
overlay $SLUG

# Play the intro if it exists. If it doesn't exist, switch to the intro slide and stop processing.

if [[ -f $BASE_DIR/assets/intros/$SLUG.webm ]]; then
  killall -s TERM $FIREFOX_NAME
  mpv $BASE_DIR/assets/intros/$SLUG.webm
else
  firefox --kiosk $BASE_DIR/assets/in-between/$SLUG.png
  exit 0
fi

# Play the video if it exists. If it doesn't exist, switch to the BBB room and stop processing.
if [ "x$TEST_MODE" = "x" ]; then
  LIST=($BASE_DIR/assets/stream/--$SLUG*--main.webm)
else
  LIST=($BASE_DIR/assets/test/--$SLUG*--main.webm)
fi
FILE="${LIST[0]}"
if [ ! -f "$FILE" ]; then
    # Is there an original file?
    LIST=($BASE_DIR/assets/stream/--$SLUG*--original.{webm,mp4,mov})
    FILE="${LIST[0]}"
fi

if [[ -f $FILE ]]; then
  killall -s TERM $FIREFOX_NAME
  mpv $FILE
else
  /usr/local/bin/bbb $SLUG
  exit 0
fi

sudo -u  talk $SLUG CLOSED_Q &

# Open the appropriate Q&A URL
QA=$(jq -r '.talks[] | select(.slug=="'$SLUG'")["qa-backstage-url"]' < $BASE_DIR/talks.json)
QA_TYPE=$(jq -r '.talks[] | select(.slug=="'$SLUG'")["qa-type"]' < $BASE_DIR/talks.json)
echo "QA_TYPE $QA_TYPE QA $QA"
if [ "$QA_TYPE" = "live" ]; then
  /usr/local/bin/bbb $SLUG
elif [ "$QA" != "null" ]; then
  /usr/local/bin/music &
  /usr/bin/firefox $QA
  # i3-msg 'layout splith'
fi
wait

It builds on roles/obs/templates/bbb, roles/obs/templates/overlay, and roles/obs/templates/music. I also have a roles/prerec/templates/talk script that uses emacsclient to update the status of the talk.

I wrote some Tampermonkey scripts to automate joining the web conference and the IRC channel.

Now that we have a script that handles all the different things related to a session, it's easier to schedule the execution of that script. Instead of using Emacs timers and running into that problem with tramp, I want to try using cron. Cron is a standard UNIX and Linux tool for scheduling things to run at certain times. You make a plain text file in a particular format: minute, hour, day of month, month, day of week, and then the command, and then you tell cron to use that file with something like crontab your-file. Since it's plain text, we can generate it with Emacs Lisp and format-time-string, save with TRAMP, and install with ssh. Each track has its own user account for streaming, so each track can have its own file.

(defun emacsconf-stream-format-crontab (track talks &optional test-mode)
  "Return crontab entries for TALKS.
Use the display specified in TRACK.
If TEST-MODE is non-nil, load the videos from the test directory."
  (concat
   (format
    "PATH=/usr/local/bin:/usr/bin
MAILTO=\"\"
XDG_RUNTIME_DIR=\"/run/user/%d\"
" (plist-get track :uid))
   (mapconcat
    (lambda (talk)
      (format "%s /usr/bin/screen -dmS play-%s bash -c \"DISPLAY=%s TEST_MODE=%s /usr/local/bin/handle-session %s | tee -a ~/track.log\"\n"
              ;; cron times are UTC
              (format-time-string "%-M %-H %-d %m *" (plist-get talk :start-time))
              (plist-get talk :slug)
              (plist-get track :vnc-display)
              (if test-mode "1" "")
              (plist-get talk :slug)))
    (emacsconf-filter-talks talks))))

(defun emacsconf-stream-crontabs (&optional test-mode info)
  "Write the streaming users' crontab files.
If TEST-MODE is non-nil, use the videos in the test directory.
If INFO is non-nil, use that as the schedule instead."
  (interactive)
  (let ((emacsconf-publishing-phase 'conference))
    (setq info (or info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
    (dolist (track emacsconf-tracks)
      (let ((talks (seq-filter (lambda (talk)
                                 (string= (plist-get talk :track)
                                          (plist-get track :name)))
                               info))
            (crontab (expand-file-name (concat (plist-get track :id) ".crontab")
                                       (concat (plist-get track :tramp) "~"))))
        (with-temp-file crontab
          (when (plist-get track :autopilot)
            (insert (emacsconf-stream-format-crontab track talks test-mode))))
        (emacsconf-stream-track-ssh track (concat "crontab ~/" (plist-get track :id) ".crontab"))))))

I want to test the whole setup before the conference, of course. First, I needed test videos. This generates test videos and subtitles following our naming convention.

(defun emacsconf-stream-generate-test-videos (&optional info)
  "Generate 1-minute test videos for INFO."
  (interactive)
  (setq info (or info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
  (let* ((dir (expand-file-name "test" emacsconf-stream-asset-dir))
         (default-directory dir)
         (subed-default-subtitle-length 1000)
         (test-length 60))
    (unless (file-directory-p dir)
      (make-directory dir t))
    (shell-command
     (format "ffmpeg -y -f lavfi -i testsrc=duration=%d:size=1280x720:rate=10 -i background-music.opus -shortest %s "
             test-length (expand-file-name "template.webm" dir)))
    (dolist (talk info)
      (with-temp-file (expand-file-name (concat (plist-get talk :file-prefix) "--main.vtt") dir)
        (subed-vtt-mode)
        (subed-auto-insert)
        (dotimes (i test-length)
          (subed-append-subtitle
           nil
           (* i 1000)
           (1- (* i 1000))
           (format "%s %02d %s"
                   (plist-get talk :slug)
                   i
                   (substring "123456789 123456789 123456789 123456789 123456789 123456789 "
                              (1+ (length (format "%s %02d" (plist-get talk :slug) i))))))))
      (copy-file
       (expand-file-name "template.webm" dir)
       (expand-file-name (concat (plist-get talk :file-prefix) "--main.webm") dir)
       t))))

Then I needed to write a crontab based on a different schedule. This code sets up a series of test videos to start about a minute after I run the code, with the dev stream set up to start a minute after the gen stream.

(let* ((offset-seconds 60)
       (start-time (time-add (current-time) offset-seconds))
       (emacsconf-schedule-validation-functions nil)
       (emacsconf-schedule-default-buffer-minutes 1)
       (emacsconf-schedule-default-buffer-minutes-for-live-q-and-a 1)
       (emacsconf-schedule-strategies '(emacsconf-schedule-allocate-buffer-time
                                        emacsconf-schedule-copy-previous-track))
       (schedule (emacsconf-schedule-prepare
                  (emacsconf-schedule-inflate-sexp
                   `(("GEN"
                      :start ,(format-time-string "%Y-%m-%d %H:%M" start-time)
                      :set-track "General")
                     (sat-open :time 1)
                     (uni :time 1) ; live Q&A
                     (adventure :time 1) ; pad Q&A
                     ("DEV"
                      :start
                      ,(format-time-string "%Y-%m-%d %H:%M" (time-add start-time 60))
                      :set-track "Development")
                     (repl :time 1) ; IRC
                     (matplotllm :time 1) ; pad
                     (voice :time 1) ; live
                     )))))
  (emacsconf-stream-crontabs t schedule))

That generates gen.crontab and dev.crontab. This is what gen.crontab looks like for testing:

PATH=/usr/local/bin:/usr/bin
MAILTO=""
XDG_RUNTIME_DIR="/run/user/2002"
35 11 26 10 * /usr/bin/screen -dmS play-sat-open bash -c "DISPLAY=:5 TEST_MODE=1 /usr/local/bin/handle-session sat-open | tee -a ~/track.log"
36 11 26 10 * /usr/bin/screen -dmS play-uni bash -c "DISPLAY=:5 TEST_MODE=1 /usr/local/bin/handle-session uni | tee -a ~/track.log"
38 11 26 10 * /usr/bin/screen -dmS play-adventure bash -c "DISPLAY=:5 TEST_MODE=1 /usr/local/bin/handle-session adventure | tee -a ~/track.log"

The result: for both tracks, the intro videos play, the test videos play, and web browsers go to the right places for the Q&A.

In case I need to resume manual control:

(defun emacsconf-stream-cancel-crontab (track)
  "Remove crontab for TRACK."
  (interactive (list (emacsconf-complete-track)))
  (plist-put track :autopilot nil)
  (emacsconf-stream-track-ssh track "crontab -r"))

(defun emacsconf-stream-cancel-all-crontabs ()
  "Remove crontabs."
  (interactive)
  (dolist (track emacsconf-tracks)
    (plist-put track :autopilot nil)
    (emacsconf-stream-track-ssh track "crontab -r")))

Here are some things I learned along the way:

• I needed to use timedatectl set-timezone America/Toronto to change the server's timezone to America/Toronto so that the crontab would run at the right time.

  In Ansible terms, that's:

  	- name: Set system timezone
  		tags: tz
  		community.general.timezone:
  			name: ""
  	- name: Restart cron
  		tags: tz
  		ansible.builtin.service:
  			name: cron
  			state: restarted
  

• I also needed to specify the PATH so that I didn't need to add the absolute paths in all the other shell scripts, XDG_RUNTIME_DIR to get audio working, and DISPLAY so that windows showed up in the right place.

I think this will let me run both tracks for EmacsConf with more ease and less frantic juggling. We'll see!

During EmacsConf, we join BigBlueButton webconferences for live presentations and Q&A sessions so that the speakers and the host can be on stream. I wanted to reduce the number of manual steps needed to join the web conference, since any clicks or keystrokes would need to be done via a VNC connection. I used Tampermonkey to write a script to join BigBlueButton and set things up the way we want to.

I don't have the Tampermonkey installation and setup automated with Ansible yet, but here's what I did for each track:

1. Install the Tampermonkey extension by going to https://addons.mozilla.org/en-US/firefox/addon/tampermonkey/ .

2. Install the script by clicking on the Tampermonkey extension, choosing Install New Script, and pasting in the following:

      // ==UserScript==
      // @name         Emacsconf BBB setup
      // @namespace    https://emacsconf.org/
      // @version      0.1
      // @description  Join BBB and set things up
      // @author       You
      // @match        https://bbb.emacsverse.org/*
      // @icon         https://www.google.com/s2/favicons?sz=64&domain=emacsverse.org
      // @grant        none
      // ==/UserScript==
      (
          async function() {
              'use strict';
              const NAME = 'emacsconf';
              async function waitUntil(conditionFunc, interval=500, timeout=null) {
                  let initResult = conditionFunc();
                  if (initResult) return initResult;
                  return new Promise((resolve, reject) => {
                      let timeSoFar = 0;
                      let timer = setInterval(() => {
                          let result = conditionFunc();
                          if (result) {
                              clearInterval(timer);
                              resolve(result);
                          }
                          timeSoFar += interval;
                          if (timeout && timeSoFar > timeout) {
                              clearInterval(timer);
                              reject();
                          }
                      }, interval);
                  });
              }
              if (document.querySelector('input.join-form')) {
                  document.querySelector('input.join-form').value = NAME;
                  document.querySelector('#room-join').click();
                  return;
              }
              await waitUntil(() => document.querySelector('.icon-bbb-listen')).then((e) => e.closest('button').click());
              await waitUntil(() => document.querySelector('.icon-bbb-user')).then((e) => e.closest('button').click());
      })();
   

   Press Ctrl+s to save.

3. Add this script to join the IRC channel:

      // ==UserScript==
      // @name         Connect to EmacsConf chat automatically
      // @namespace    https://emacsconf.org/
      // @version      0.1
      // @description  try to take over the world!
      // @author       You
      // @match        https://chat.emacsconf.org/*
      // @icon         https://www.google.com/s2/favicons?sz=64&domain=emacsconf.org
      // @grant        none
      // ==/UserScript==
   
      (function() {
          'use strict';
          setTimeout(() => {
              if (document.querySelector('.connect-row')) {
                  document.querySelector('.connect-row').closest('form').querySelector('button').click();
              }
          }, 1000);
      })();
   

4. Join an BBB meeting. Check the address bar to see if autoplay is disabled (crossed-out autoplay icon). If it is, click on it and change Block audio to Allow audio and video.

Now I can use this Ansible template for a shell script to connect to the BBB session (roles/obs/templates/bbb):

#!/bin/bash
# Open the Big Blue Button room using the backstage link
# 

# Kill the background music if playing
if screen -list | grep -q background; then
    screen -S background -X quit
fi

# Update the overlay
SLUG=$1
overlay $SLUG
killall -s TERM firefox-esr
firefox https://media.emacsconf.org//backstage/assets/redirects/open/bbb-$SLUG.html &
sleep 5
xdotool search --class firefox windowactivate --sync
xdotool key Return
xdotool key F11
wait

The overlay is handled by this script (roles/obs/templates/overlay):

#!/bin/bash
# 

SLUG=$(echo "$1" | perl -ne 'if (/emacsconf-[0-9]*-(.*?)--/) { print $1; } else { print; }')

if [[ -f /assets/overlays/$SLUG-other.svg.png ]]; then
    echo "Found other overlay for $SLUG, copying"
    cp /assets/overlays/$SLUG-other.svg.png ~/other.png
elif [[ -f /assets/overlays/$SLUG-video.svg.png ]]; then
    echo "Found video overlay for $SLUG, copying"
    cp /assets/overlays/$SLUG-video.svg.png ~/other.png
else
    echo "Could not find /assets/overlays/$SLUG-other.svg.png, please override ~/other.png manually"
    cp /assets/overlays/blank-other.svg.png ~/other.png
fi
if [[ -f /assets/overlays/$SLUG-video.svg.png ]]; then
    echo "Found video overlay for $SLUG, copying"
    cp /assets/overlays/$SLUG-video.svg.png ~/video.png
else
    echo "Could not find /assets/overlays/$SLUG-video.svg.png, override ~/video.png manually"
    cp /assets/overlays/blank-video.svg.png ~/video.png
fi

The result: I can run bbb uni and it'll automatically join the Q&A session for the uni talk in listen-only mode and with the user list hidden.

I wanted to use Mermaid to make diagrams, but I ran into this issue when trying to run it:

Error: Could not find Chromium (rev. 1108766). This can occur if either
 1. you did not perform an installation before running the script (e.g. `npm install`) or
 2. your cache path is incorrectly configured (which is: /home/sacha/.cache/puppeteer).
For (2), check out our guide on configuring puppeteer at https://pptr.dev/guides/configuration.

It turns out that I needed to do the following:

sudo npm install -g puppeteer mermaid @mermaid-js/mermaid-cli --unsafe-perm
# Cache Chromium for my own user
node /usr/lib/node_modules/puppeteer/install.js --unsafe-perm
sudo npm install -g mermaid @mermaid-js/mermaid-cli
ln -s ~/.cache/puppeteer/chrome/linux-117.0.5938.149 ~/.cache/puppeteer/chrome/linux-1108766
ln -s ~/.cache/puppeteer/chrome/linux-117.0.5938.149/chrome-linux64 ~/.cache/puppeteer/chrome/linux-117.0.5938.149/chrome-linux

(The exact versions might be different for your installation.)

Then I could make a Mermaid file and try it out with mmdc -i input.mmd -o output.svg, and then I could confirm that it works directly from Org with ob-mermaid:

sequenceDiagram
    input ->> res: original.mp4;
    res ->> backstage: reencoded.webm;
    backstage ->> laptop: cached files;
    laptop ->> backstage: index;
    input ->> res: main.vtt;
    res ->> backstage: main.webm;
    backstage ->> laptop: cached files;
    laptop ->> backstage: index;
    input ->> res: normalized.opus;
    res ->> backstage: main.webm;
    backstage ->> laptop: cached files;
    laptop ->> backstage: index;
    backstage ->> media: public files;

Sometimes I want to work with all the talks associated with an email in my inbox. For example, maybe a speaker said that the draft schedules are fine, and I want to make a note of that in the conference Org file.

First we start with a function that gets the e-mail addresses for a talk. Some speakers have different e-mail addresses for public contact or private contact, and some e-mail us from other addresses.

(defun emacsconf-mail-get-all-email-addresses (talk)
  "Return all the possible e-mail addresses for TALK."
  (split-string
   (downcase
    (string-join
     (seq-uniq
      (seq-keep
       (lambda (field) (plist-get talk field))
       '(:email :public-email :email-alias)))
     ","))
   " *, *"))

Then we can use that to find the talks for a given e-mail address.

(defun emacsconf-mail-talks (email)
  "Return a list of talks matching EMAIL."
  (setq email (downcase (mail-strip-quoted-names email)))
  (seq-filter
   (lambda (o) (member email (emacsconf-mail-get-all-email-addresses o)))
   (emacsconf-get-talk-info)))

We can loop over that to add a note for the e-mail.

(defun emacsconf-mail-add-to-logbook (email note)
  "Add to logbook for all matching talks from this speaker."
  (interactive
   (let* ((email (mail-strip-quoted-names
                  (plist-get (plist-get (notmuch-show-get-message-properties) :headers)
                             :From)))
          (talks (emacsconf-mail-talks email)))
     (list
      email
      (read-string (format "Note for %s: "
                           (mapconcat (lambda (o) (plist-get o :slug))
                                      talks", "))))))
  (save-window-excursion
    (mapc
     (lambda (talk)
       (emacsconf-add-to-talk-logbook talk note))
     (emacsconf-mail-talks email))))

The actual addition of notes is handled by these functions.

(defun emacsconf-add-to-logbook (note)
  "Add NOTE as a logbook entry for the current subtree."
  (move-marker org-log-note-return-to (point))
  (move-marker org-log-note-marker (point))
  (with-temp-buffer
    (insert note)
    (let ((org-log-note-purpose 'note))
      (org-store-log-note))))

Then we have a function that looks for the heading for a note and then adds a logbook entry to it.

(defun emacsconf-add-to-talk-logbook (talk note)
  "Add NOTE as a logbook entry for TALK."
  (interactive (list (emacsconf-complete-talk) (read-string "Note: ")))
  (save-excursion
    (emacsconf-with-talk-heading talk
      (emacsconf-add-to-logbook note))))

All together, that makes it easy to use Emacs as a very simple contact relationship management system where I can take notes based on the e-mails that come in.

These functions are in emacsconf-mail.el.

Naming conventions make it easier for other people to find things. Just like with file prefixes, I like to use a standard naming pattern for our BigBlueButton web conference rooms. For EmacsConf 2022, we used ec22-day-am-gen Speaker name (slugs). For EmacsConf 2023, I want to set up the BigBlueButton rooms before the schedule settles down, so I won't encode the time or track information into it. Instead, I'll use Speaker name (slugs) - emacsconf2023.

BigBlueButton does have an API for managing rooms, but that requires a shared secret that I don't know yet. I figured I'd just automate it through my browser. Over the last year, I've started using Spookfox to control the Firefox web browser from Emacs. It's been pretty handy for scrolling webpages up and down, so I wondered if I could replace my old xdotool-based automation. Here's what I came up with for this year.

First, I need a function that creates the BBB room for a group of talks and updates the Org entry with the URL. Adding a slight delay makes it a bit more reliable.

(defun emacsconf-spookfox-create-bbb (group)
  "Create a BBB room for this group of talks.
GROUP is (email . (talk talk talk)).
Needs a Spookfox connection."
  (let* ((bbb-name
          (format "%s (%s) - %s%s"
                  (mapconcat (lambda (o) (plist-get o :slug)) (cdr group) ", ")
                  (plist-get (cadr group) :speakers)
                  emacsconf-id
                  emacsconf-year))
         path
         (retrieve-command
          (format
           "window.location.origin + [...document.querySelectorAll('h4.room-name-text')].find((o) => o.textContent.trim() == '%s').closest('tr').querySelector('.delete-room').getAttribute('data-path')"
           bbb-name))
         (create-command (format "document.querySelector('#create-room-block').click();
document.querySelector('#create-room-name').value = \"%s\";
document.querySelector('#room_mute_on_join').click();
document.querySelector('.create-room-button').click();"
                                 bbb-name)))
    (setq path (spookfox-js-injection-eval-in-active-tab retrieve-command t))
    (unless path
      (dolist (cmd (split-string create-command ";"))
        (spookfox-js-injection-eval-in-active-tab cmd t)
        (sleep-for 2))
      (sleep-for 2)
      (setq path (spookfox-js-injection-eval-in-active-tab retrieve-command t)))
    (when path
      (dolist (talk (cdr group))
        (save-window-excursion
          (emacsconf-with-talk-heading talk
            (org-entry-put (point) "ROOM" path))))
      (cons bbb-name path))))

Then I need to iterate over the list of talks that have live Q&A sessions but don't have BBB rooms assigned yet so that I can create them.

(defun emacsconf-spookfox-create-bbb-for-live-talks ()
  "Create BBB rooms for talks that don't have them yet."
  (let* ((talks (seq-filter
                 (lambda (o)
                   (and (string-match "live" (or (plist-get o :q-and-a) ""))
                        (not (string= (plist-get o :status) "CANCELLED"))
                        (not (plist-get o :bbb-room))))
                 (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
         (groups (and talks (emacsconf-mail-groups talks))))
    (dolist (group groups)
      (emacsconf-spookfox-create-bbb group))))

The result: a whole bunch of rooms ready for people to check in.

Using Spookfox to communicate with Firefox from Emacs Lisp made it easy to get data in and out of my browser. Handy!

This code is in emacsconf-spookfox.el.

Sometimes it makes sense to dynamically generate information related to a talk and then save it as an Org property so that I can manually edit it. For example, we like to name all the talk files using this pattern: "emacsconf-year-slug--title--speakers". That's a lot to type consistently! We can generate most of these prefixes automatically, but some might need tweaking, like when the talk title or speaker names have special characters.

Calculating the file prefix for a talk

First we need something that turns a string into an ID.

emacsconf-slugify: Turn S into an ID.

(defun emacsconf-slugify (s)
  "Turn S into an ID.
Replace spaces with dashes, remove non-alphanumeric characters,
and downcase the string."
  (replace-regexp-in-string
   " +" "-"
   (replace-regexp-in-string
    "[^a-z0-9 ]" ""
    (downcase s))))

Then we can use that to calculate the file prefix for a given talk.

emacsconf-file-prefix: Create the file prefix for TALK

(defun emacsconf-file-prefix (talk)
  "Create the file prefix for TALK."
  (concat emacsconf-id "-"
          emacsconf-year "-"
          (plist-get talk :slug) "--"
          (emacsconf-slugify (plist-get talk :title))
          (if (plist-get talk :speakers)
              (concat "--"
                     (emacsconf-slugify (plist-get talk :speakers)))
            "")))

Then we can map over all the talk entries that don't have FILE_PREFIX defined:

emacsconf-set-file-prefixes: Set the FILE_PREFIX property for each talk entry that needs it.

(defun emacsconf-set-file-prefixes ()
  "Set the FILE_PREFIX property for each talk entry that needs it."
  (interactive)
  (org-map-entries
   (lambda ()
     (org-entry-put
      (point) "FILE_PREFIX"
      (emacsconf-file-prefix (emacsconf-get-talk-info-for-subtree))))
   "SLUG={.}-FILE_PREFIX={.}"))

That stores the file prefix in an Org property, so we can edit it if it needs tweaking.

Renaming files to match the file prefix

Now that we have that, how can we use it? One way is to rename files from within Emacs. I can mark multiple files with Dired's m command or work on them one at a time. If there are several files with the same extension, I can specify something to add to the filename to tell them apart.

emacsconf-rename-files: Rename the marked files or the current file to match TALK.

(defun emacsconf-rename-files (talk &optional filename)
  "Rename the marked files or the current file to match TALK.
If FILENAME is specified, use that as the extra part of the filename after the prefix.
This is useful for distinguishing files with the same extension.
Return the list of new filenames."
  (interactive (list (emacsconf-complete-talk-info)))
  (prog1
      (mapcar
       (lambda (file)
         (let* ((extra
                 (or filename
                     (read-string (format "Filename (%s): " (file-name-base file)))))
                (new-filename
                 (expand-file-name
                  (concat (plist-get talk :file-prefix)
                          (if (string= extra "")
                              ""
                            (concat "--" extra))
                          "."
                          (file-name-extension file))
                  (file-name-directory file))))
           (rename-file file new-filename t)
           new-filename))
       (or (dired-get-marked-files) (list (buffer-file-name))))
    (when (derived-mode-p 'dired-mode)
      (revert-buffer))))

(defun emacsconf-rename-and-upload-to-backstage (talk &optional filename)
  "Rename marked files or the current file, then upload to backstage."
  (interactive (list (emacsconf-complete-talk-info)))
  (mapc
   (lambda (file)
     (copy-file
      file
      (expand-file-name
       (file-name-nondirectory file)
       emacsconf-backstage-dir)
      t))
   (emacsconf-rename-files talk)))

(defun emacsconf-publish-talks-json ()
  "Return JSON format with a subset of talk information."
  (json-encode
   (list
    :talks
    (mapcar
     (lambda (o)
       (apply
        'list
        (cons :start-time (format-time-string "%FT%T%z" (plist-get o :start-time) t))
        (cons :end-time (format-time-string "%FT%T%z" (plist-get o :end-time) t))
        (mapcar
         (lambda (field)
           (cons field (plist-get o field)))
         '(:slug :title :speakers :pronouns :pronunciation :url :track :file-prefix))))
     (emacsconf-filter-talks (emacsconf-get-talk-info))))))

(defun emacsconf-publish-talks-json-to-files ()
  "Export talk information as JSON so that we can use it in shell scripts."
  (interactive)
  (mapc (lambda (dir)
          (when (and dir (file-directory-p dir))
            (with-temp-file (expand-file-name "talks.json" dir)
              (insert (emacsconf-talks-json)))))
        (list emacsconf-res-dir emacsconf-ansible-directory)))

jq -r '.talks[] | select(.slug=="'$SLUG'")["file-prefix"]' < $TALKS_JSON

#!/bin/bash
# 
# Usage: rename-original.sh $slug $file [$extra] [$talks-json]
SLUG=$1
FILE=$2
TALKS_JSON=${4:-~/current/talks.json}
EXTRA=""
if [ -z ${3-unset} ]; then
    EXTRA=""
elif [ -n "$3" ]; then
    EXTRA="--$3"
elif echo "$FILE" | grep -e '\(webm\|mp4\|mov\)'; then
    EXTRA="--original"
fi
filename=$(basename -- "$FILE")
extension="${filename##*.}"
filename="${filename%.*}"
FILE_PREFIX=$(jq -r '.talks[] | select(.slug=="'$SLUG'")["file-prefix"]' < $TALKS_JSON)
mv "$FILE" $FILE_PREFIX$EXTRA.$extension
echo $FILE_PREFIX$EXTRA.$extension
# Copy to original if needed
if [ -f $FILE_PREFIX--original.webm ] && [ ! -f $FILE_PREFIX--main.$extension ]; then
    cp $FILE_PREFIX--original.$extension $FILE_PREFIX--main.webm
fi

(defun emacsconf-upload-copy-from-json (talk key filename)
  "Parse PsiTransfer JSON files and copy the uploaded file to the backstage directory.
The file is associated with TALK. KEY identifies the file in a multi-file upload.
FILENAME specifies an extra string to add to the file prefix if needed."
  (interactive (let-alist (json-parse-string (buffer-string) :object-type 'alist)
                 (list (emacsconf-complete-talk-info)
                       .metadata.key
                       (read-string (format "Filename: ")))))
  (let ((new-filename (concat (plist-get talk :file-prefix)
                              (if (string= filename "")
                                  filename
                                (concat "--" filename))
                              "."
                              (let-alist (json-parse-string (buffer-string) :object-type 'alist)
                                (file-name-extension .metadata.name)))))
    (copy-file
     key
     (expand-file-name new-filename emacsconf-backstage-dir)
     t)))

One of the things I keep an eye out for when organizing EmacsConf is the most recent time we heard from a speaker. Sometimes life happens and speakers get too busy to prepare a video, so we might offer to let them do it live. Sometimes e-mail delivery issues get in the way and we don't hear from speakers because some server in between has spam filters set too strong. So I made a function that lists the most recent e-mail we got from the speaker that includes "emacsconf" in it. That was a good excuse to learn more about tabulated-list-mode.

I started by figuring out how to get all the e-mail addresses associated with a talk.

(defun emacsconf-mail-get-all-email-addresses (talk)
  "Return all the possible e-mail addresses for TALK."
  (split-string
   (downcase
    (string-join
     (seq-uniq
      (seq-keep
       (lambda (field) (plist-get talk field))
       '(:email :public-email :email-alias)))
     ","))
   " *, *"))

Then I figured out the notmuch search to use to get all messages. Some people write a lot, so I limited it to just the ones that have emacsconf as well. Notmuch can return JSON, so that's easy to parse.

(defvar emacsconf-mail-notmuch-tag "emacsconf" "Tag to use when searching the Notmuch database for mail.")

(defun emacsconf-mail-notmuch-last-message-for-talk (talk &optional subject)
  "Return the most recent message from the speakers for TALK.
Limit to SUBJECT if specified."
  (let ((message (json-parse-string
                  (shell-command-to-string
                   (format "notmuch search --limit=1 --format=json \"%s%s\""
                           (mapconcat
                            (lambda (email) (concat "from:" (shell-quote-argument email)))
                            (emacsconf-mail-get-all-email-addresses talk)
                            " or ")
                           (emacsconf-surround
                            " and "
                            (and emacsconf-mail-notmuch-tag (shell-quote-argument emacsconf-mail-notmuch-tag))
                            "" "")
                           (emacsconf-surround
                            " and subject:"
                            (and subject (shell-quote-argument subject)) "" "")))
                  :object-type 'alist)))
    (cons `(email . ,(plist-get talk :email))
          (when (> (length message) 0)
            (elt message 0)))))

Then I could display all the groups of speakers so that it's easy to check if any of the speakers haven't e-mailed us in a while.

(defun emacsconf-mail-notmuch-show-latest-messages-from-speakers (groups &optional subject)
  "Verify that the email addresses in GROUPS have e-mailed recently.
When called interactively, pop up a report buffer showing the e-mails
and messages by date, with oldest messages on top.
This minimizes the risk of mail delivery issues and radio silence."
  (interactive (list (emacsconf-mail-groups (seq-filter
                               (lambda (o) (not (string= (plist-get o :status) "CANCELLED")))
                               (emacsconf-get-talk-info)))))
  (let ((results
         (sort (mapcar
                (lambda (group)
                  (emacsconf-mail-notmuch-last-message-for-talk (cadr group) subject))
                groups)
               (lambda (a b)
                 (< (or (alist-get 'timestamp a) -1)
                    (or (alist-get 'timestamp b) -1))))))
    (when (called-interactively-p 'any)
      (with-current-buffer (get-buffer-create "*Mail report*")
        (let ((inhibit-read-only t))
          (erase-buffer))
        (tabulated-list-mode)
        (setq
         tabulated-list-entries
         (mapcar
          (lambda (row)
            (list
             (alist-get 'thread row)
             (vector
              (alist-get 'email row)
              (or (alist-get 'date_relative row) "")
              (or (alist-get 'subject row) ""))))
          results))
        (setq tabulated-list-format [("Email" 30 t)
                                     ("Date" 10 nil)
                                     ("Subject" 30 t)])
        (local-set-key (kbd "RET") #'emacsconf-mail-notmuch-visit-thread-from-summary)
        (tabulated-list-print)
        (tabulated-list-init-header)
        (pop-to-buffer (current-buffer))))
    results))

If I press RET on a line, I can open the most recent thread. This is handled by the emacsconf-mail-notmuch-visit-thread-from-summary, which is simplified by using the thread ID as the tabulated list ID.

(defun emacsconf-mail-notmuch-visit-thread-from-summary ()
  "Display the thread from the summary."
  (interactive)
  (let (message-buffer)
    (save-window-excursion
      (setq message-buffer (notmuch-show (tabulated-list-get-id))))
    (display-buffer message-buffer t)))

We haven't heard from a few speakers in a while, so I'll probably e-mail them this weekend to double-check that I'm not getting delivery issues with my e-mails to them. If that doesn't get a reply, I might try other communication methods. If they're just busy, that's cool.

It's a lot easier to spot missing or old entries in a table than it is to try to remember who we haven't heard from recently, so hooray for tabulated-list-mode!

This code is in emacsconf-mail.el.