Categories: emacsconf

RSS - Atom - Subscribe via email

#EmacsConf backstage: coordinating captioning volunteers using a backstage area

| emacs, emacsconf
Figure 1: The backstage area

One of the benefits of volunteering for or speaking at EmacsConf is that you get early access to the talks. We upload the files to a password-protected web server. It's nothing fancy, just a directory that has basic authentication and a static index.html generated by Emacs Lisp.

I organize the talks by TODO status so that people can easily see which talks are ready to be captioned. Since captioning can take a little while, we reserve the talks we want to caption. That way, people don't accidentally work on a talk that someone else is already captioning. When people e-mail me to reserve a talk, I move the talk from TO_ASSIGN to TO_CAPTION and add their name in the :CAPTIONER: property.

Publishing the backstage index is done by an Emacs Lisp function that smooshes the information together and then writes the files directly to the server using TRAMP.

(defun emacsconf-publish-backstage-index (&optional filename)
  (setq filename (or filename (expand-file-name "index.html" emacsconf-backstage-dir)))
  (let ((info (or emacsconf-schedule-draft (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
        (emacsconf-main-extensions (append emacsconf-main-extensions emacsconf-publish-backstage-extensions)))
    (with-temp-file filename
      (let* ((talks
               (lambda (o) (append
                            (list :captions-edited t) o))
               (seq-filter (lambda (o) (plist-get o :speakers))
                           (emacsconf-active-talks (emacsconf-filter-talks info)))))
             (by-status (seq-group-by (lambda (o) (plist-get o :status)) talks))
             (files (directory-files emacsconf-backstage-dir)))
         "<html><head><meta charset=\"UTF-8\"><link rel=\"stylesheet\" href=\"/style.css\" /></head><body>"
         (if (file-exists-p (expand-file-name "include-in-index.html" emacsconf-cache-dir))
             (with-temp-buffer (insert-file-contents (expand-file-name "include-in-index.html" emacsconf-cache-dir)) (buffer-string))
         "<p>Schedule by status: (gray: waiting, light yellow: processing, yellow: to assign, light green: captioning, green: captioned and ready)<br />Updated by and the wiki repository</br />"
         (let* ((emacsconf-schedule-svg-modify-functions '(emacsconf-schedule-svg-color-by-status))
                (img (emacsconf-schedule-svg 800 200 info)))
             (mapc (lambda (node)
                      node 'href
                      (concat "#" (dom-attr node 'data-slug))))
                   (dom-by-tag img 'a))
             (svg-print img)
         (if (eq emacsconf-backstage-phase 'prerec)
             (format "<p>Waiting for %d talks (~%d minutes) out of %d total</p>"
                     (length (assoc-default "WAITING_FOR_PREREC" by-status))
                     (emacsconf-sum :time (assoc-default "WAITING_FOR_PREREC" by-status))
                     (length talks))
          (lambda (status)
            (concat "<li>"
                    (if (string= status "TO_ASSIGN")
                        "TO_ASSIGN (waiting for volunteers)"
                    ": "
                    (mapconcat (lambda (o) (format "<a href=\"#%s\">%s</a>"
                                                   (plist-get o :slug)
                                                   (plist-get o :slug)))
                               (assoc-default status by-status)
                               ", ")
          (pcase emacsconf-backstage-phase
            ('harvest '("TO_ARCHIVE" "TO_REVIEW_QA" "TO_INDEX_QA" "TO_CAPTION_QA")))
         (pcase emacsconf-backstage-phase
             (emacsconf-publish-backstage-processing by-status files)
             (emacsconf-publish-backstage-to-assign by-status files)
             (emacsconf-publish-backstage-to-caption by-status files)
             (emacsconf-publish-backstage-to-stream by-status files)))
            (let ((stages
                   '(("TO_REVIEW_QA" .
                      "Please review the --bbb-webcams.webm file and/or the --bbb-webcams.vtt and tell us ( if a Q&amp;A session can be published or if it needs to be trimmed (lots of silence at the end of the recording, accidentally included sensitive information, etc.).")
                     ("TO_INDEX_QA" .
                      "Please review the --answers.webm and --answers.vtt files to make chapter markers so that people can jump to specific parts of the Q&amp;A session. The <a href=\"\">harvesting page on the wiki</a> has some notes on the process. That way, it's easier for people to see the questions and
answers without needing to listen to everything again. You can see <a href=\"\">asmblox</a> for an example of the Q&amp;A chapter markers.")
                     ("TO_CAPTION_QA" .
                      "Please edit the --answers.vtt for the Q&amp;A talk you're interested in, correcting misrecognized words and cleaning it up so that it's nice to use as closed captions. All talks should now have large-model VTTs to make it easier to edit."))))
               (lambda (stage)
                 (let ((status (car stage)))
                    "<h1>%s: %d talk(s) (%d minutes)</h1>%s<ol class=\"videos\">%s</ol>"
                    (length (assoc-default status by-status))
                    (emacsconf-sum :video-time (assoc-default status by-status))
                    (cdr stage)
                     (lambda (f)
                       (format  "<li><a name=\"%s\"></a><strong><a href=\"%s%s\">%s</a></strong><br />%s (id:%s)<br />%s</li>"
                                (plist-get f :slug)
                                (plist-get f :url)
                                (plist-get f :title)
                                (plist-get f :speakers-with-pronouns)
                                (plist-get f :slug)
                                 (append (list
                                          (unless (file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--bbb-webcams.webm") emacsconf-cache-dir))
                                            "<div>No Q&A video for this talk</div>")
                                           ((file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--answers.webm") emacsconf-cache-dir))
                                            (concat (plist-get f :file-prefix) "--answers.webm"))
                                           ((file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--bbb-webcams.webm") emacsconf-cache-dir))
                                            (concat (plist-get f :file-prefix) "--bbb-webcams.webm"))
                                           (t t)) ;; omit video
                                          :video-id "-qanda"
                                           (emacsconf-surround "QA note: " (plist-get f :qa-note) "<br />")
                                           (format "Q&A archiving: <a href=\"%s-%s.txt\">IRC: %s-%s</a>"
                                                   (format-time-string "%Y-%m-%d" (plist-get f :start-time))
                                                   (plist-get (emacsconf-get-track f) :channel)
                                                   (format-time-string "%Y-%m-%d" (plist-get f :start-time))
                                                   (plist-get (emacsconf-get-track f) :channel))
                                           (emacsconf-surround ", <a href=\""
                                                               (if (file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--pad.txt")
                                                                   (concat (plist-get f :file-prefix) "--pad.txt"))
                                                               "\">Etherpad (Markdown)</a>" "")
                                           (emacsconf-surround ", <a href=\"" (plist-get f :bbb-playback) "\">BBB playback</a>" "")
                                           (emacsconf-surround ", <a href=\""
                                                               (if (file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--bbb.txt")
                                                                   (concat (plist-get f :file-prefix) "--bbb.txt"))
                                                               "\">BBB text chat</a>" "")
                                           (emacsconf-surround ", <a href=\""
                                                               (if (file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--bbb-webcams.opus")
                                                                   (concat (plist-get f :file-prefix) "--bbb-webcams.opus"))
                                                               "\">BBB audio only</a>" ""))
                                          :files (emacsconf-publish-talk-files f files))
                     (assoc-default status by-status) "\n"))))
         (if (file-exists-p (expand-file-name "include-in-index-footer.html" emacsconf-cache-dir))
             (with-temp-buffer (insert-file-contents (expand-file-name "include-in-index-footer.html" emacsconf-cache-dir)) (buffer-string))

For example, the section for talks that are waiting for volunteers is handled by the function emacsconf-publish-backstage-to-assign.

(defun emacsconf-publish-backstage-to-assign (by-status files)
  (let ((list (assoc-default "TO_ASSIGN" by-status)))
    (format "<h1>%s talk(s) to be captioned, waiting for volunteers (%d minutes)</h1><p>You can e-mail <a href=\"\"></a> to call dibs on editing the captions for one of these talks. We use OpenAI Whisper to provide auto-generated VTT that you can use as a starting point, but you can also write the captions from scratch if you like. If you're writing them from scratch, you can choose to include timing information, or we can probably figure them out afterwards with a forced alignment tool. More info: <a href=\"\">captioning tips</a></p><ul class=\"videos\">%s</ul>"
            (length list)
            (emacsconf-sum :video-time list)
             (lambda (f)
               (setq f (append
                        (list :extra
                              (if (plist-get f :caption-note) (concat "<div class=\"caption-note\">" (plist-get f :caption-note) "</div>") "")
                              (emacsconf-publish-talk-files f files))))
               (format  "<li><a name=\"%s\"></a><strong><a href=\"%s\">%s</a></strong><br />%s (id:%s)<br />%s</li>"
                        (plist-get f :slug)
                        (plist-get f :absolute-url)
                        (plist-get f :title)
                        (plist-get f :speakers)
                        (plist-get f :slug)
                        (emacsconf-publish-index-card f)))

Each talk has a little card that includes its video and links to files.

emacsconf-publish-index-card: Format an HTML card for TALK, linking the files in EXTENSIONS.
(defun emacsconf-publish-index-card (talk &optional extensions)
  "Format an HTML card for TALK, linking the files in EXTENSIONS."
  (let* ((file-prefix (plist-get talk :file-prefix))
         (video-file (plist-get talk :video-file))
         (video (and file-prefix
                      (or (plist-get talk :video-id)
                          (concat (plist-get talk :slug) "-mainVideo"))
                      video-file talk extensions))))
    ;; Add extra information to the talk
    (setq talk
            :time-info (emacsconf-surround "Duration: " (plist-get talk :video-duration) " minutes" "")
            :video-html (or (plist-get video :video) "")
            :audio-html (or (plist-get video :audio) "")
            :chapter-list (or (plist-get video :chapter-list) "")
            :resources (or (plist-get video :resources) "")
            :extra (or (plist-get talk :extra) "")
            :speaker-info (or (plist-get talk :speakers-with-pronouns) ""))))
     "<div class=\"vid\">${video-html}${audio-html}<div>${extra}</div>${time-info}${resources}${chapter-list}</div>")))

So it's all built on templates and a little bit of code to make the appropriate lists. You can find this code in emacsconf-publish.el.

#EmacsConf backstage: automatically updating talk status from the crontab

| emacs, emacsconf, org

Now that I've figured out how to automatically play EmacsConf talks with crontab, I want to update our approach to using TRAMP and timers to run two tracks semi-automatically.

When a talk starts playing, I would like to:

  • announce it on IRC so that the talk details split up the chat log and make it easier to read
  • publish the media files to (currently password-protected while testing) so that people can view them
  • update the talk wiki page with the media files and the captions
  • update the Youtube and Toobnix videos so that they're public: this is manual for the moment, since I haven't put time into automating it yet

I have code for most of this, and we've done it successfully for the past couple of years. I just need to refamiliarize myself with how to do it and how to set it up for testing during the dry run, and modify it to work with the new crontab-based system.

Setting things up

Last year, I set up the server with the ability to act as the controller, so that it wasn't limited to my laptop. The organizers notebook says I put it in the orga@ user's account, and I probably ran it under a screen session. I've submitted a talk for this year's conference, so I can use a test video for that one. First, I need to update the Ansible configuration for publishing and editing:

ansible-playbook -i inventory.yml prod-playbook.yml --tags publish,edit

I needed to add a version attribute to the git repo checkout in the Ansible playbook, since we'd switched from master to main. I also needed to set emacs_version to 29.1 since I started using seq-keep in my Emacs Lisp functions. For testing, I set emacsconf-publishing-phase to conference.

Act on TODO state changes

org-after-todo-state-change-hook makes it easy to automatically run functions when the TODO state changes. I add this hook that runs a list of functions and passes the talk information so that I don't have to parse the talk info in each function.

emacsconf-org-after-todo-state-change: Run all the hooks in ‘emacsconf-todo-hooks’.
(defun emacsconf-org-after-todo-state-change ()
  "Run all the hooks in `emacsconf-todo-hooks'.
If an `emacsconf-todo-hooks' entry is a list, run it only for the
tracks with the ID in the cdr of that list."
  (let* ((talk (emacsconf-get-talk-info-for-subtree))
         (track (emacsconf-get-track (plist-get talk :track))))
     (lambda (hook-entry)
        ((symbolp hook-entry) (funcall hook-entry talk))
        ((member (plist-get track :id) (cdr hook-entry))
         (funcall (car hook-entry) talk))))

This can be enabled and disabled with the following functions.

emacsconf-add-org-after-todo-state-change-hook: Add FUNC to ‘org-after-todo-stage-change-hook’.
(defun emacsconf-add-org-after-todo-state-change-hook ()
  "Add FUNC to `org-after-todo-stage-change-hook'."
  (with-current-buffer (find-buffer-visiting emacsconf-org-file)
    (add-hook 'org-after-todo-state-change-hook #'emacsconf-org-after-todo-state-change nil t)))

emacsconf-remove-org-after-todo-state-change-hook: Remove FUNC from ‘org-after-todo-stage-change-hook’.
(defun emacsconf-remove-org-after-todo-state-change-hook ()
  "Remove FUNC from `org-after-todo-stage-change-hook'."
  (with-current-buffer (find-buffer-visiting emacsconf-org-file)
    (remove-hook 'org-after-todo-state-change-hook
                 #'emacsconf-org-after-todo-state-change  t)))

Announce on IRC

This is still much the same as last year.

emacsconf-erc-announce-on-change: Announce talk.
(defun emacsconf-erc-announce-on-change (talk)
  "Announce talk."
  (let ((func
         (pcase org-state
           ("PLAYING" #'erc-cmd-NOWPLAYING)
           ("CLOSED_Q" #'erc-cmd-NOWCLOSEDQ)
           ("OPEN_Q" #'erc-cmd-NOWOPENQ)
           ("UNSTREAMED_Q" #'erc-cmd-NOWUNSTREAMEDQ)
           ("TO_ARCHIVE" #'erc-cmd-NOWDONE))))
    (when func
      (funcall func talk))))

Here's a sample command that announces that the talk is now playing.

erc-cmd-NOWPLAYING: Set the channel topics to announce TALK.
(defun erc-cmd-NOWPLAYING (talk)
  "Set the channel topics to announce TALK."
  (interactive (list (emacsconf-complete-talk-info)))
  (when (stringp talk)
    (setq talk (or (emacsconf-find-talk-info talk) (error "Could not find talk %s" talk))))
  ;; Announce it in the track's channel
  (if (emacsconf-erc-recently-announced (format "---- %s:" (plist-get talk :slug)))
      (message "Recently announced, skipping")
    (when (plist-get talk :track)
      (emacsconf-erc-with-channels (list (concat "#" (plist-get talk :channel)))
          "%s: %s (%s) pad: %s Q&A: %s | %s"
          (plist-get talk :slug)
          (plist-get talk :title)
          (plist-get talk :speakers)
          (plist-get talk :pad-url)
          (plist-get talk :qa-info)
          (car (assoc-default
                (concat "#" (plist-get talk :channel))
        (erc-send-message (format "---- %s: %s - %s ----"
                                  (plist-get talk :slug)
                                  (plist-get talk :title)
                                  (plist-get talk :speakers-with-pronouns)))
         (concat "Add your notes/questions to the pad: " (plist-get talk :pad-url)))
         ((string-match "live" (or (plist-get talk :q-and-a) ""))
          (erc-send-message (concat "Live Q&A: " (plist-get talk :bbb-redirect))))
         ((plist-get talk :irc)
          (erc-send-message (format "or discuss the talk on IRC (nick: %s)"
                                    (plist-get talk :irc)))))))
    ;; Short announcement in #emacsconf
    (emacsconf-erc-with-channels (list emacsconf-erc-hallway emacsconf-erc-org)
      (erc-send-message (format "-- %s track: %s: %s (watch: %s, pad: %s, channel: #%s)"
                                (plist-get talk :track)
                                (plist-get talk :slug)
                                (plist-get talk :title)
                                (plist-get talk :watch-url)
                                (plist-get talk :pad-url)
                                (plist-get talk :channel))))))

Because the commands change the topic, I need to op the user in all the Emacsconf channels.

(defun erc-cmd-OPALL (&optional nick)
  (emacsconf-erc-with-channels (mapcar 'car emacsconf-topic-templates)
    (if nick
        (erc-cmd-OP nick)

This code is in emacsconf-erc.el.

Publish media files

We used to scramble to upload all the videos in the days or weeks after the conference, since the presentations were live. Since we switched to encouraging speakers to upload videos before the conference, we've been able to release the videos pretty much as soon as the talk starts playing. This code in emacsconf-publish.el takes care of copying the files from the backstage to the public media directory, republishing the index, and republishing the playlist. That way, people who come in late or who want to refer to the video can easily get the full video right away.

emacsconf-publish-media-files-on-change: Publish the files and update the index.
(defun emacsconf-publish-media-files-on-change (talk)
  "Publish the files and update the index."
  (interactive (list (emacsconf-complete-talk-info)))
  (let ((org-state (if (boundp 'org-state) org-state (plist-get talk :status))))
    (if (plist-get talk :public)
        ;; Copy main extension files from backstage to public
        (let ((files (directory-files emacsconf-backstage-dir nil
                                      (concat "^"
                                              (regexp-quote (plist-get talk :file-prefix))
                                              (regexp-opt emacsconf-main-extensions)))))
          (mapc (lambda (file)
                  (when (and
                         (not (file-exists-p (expand-file-name file emacsconf-public-media-directory)))
                         (or (not (string-match "--main.vtt$" file))
                             (plist-get talk :captions-edited)))
                    (copy-file (expand-file-name file emacsconf-backstage-dir)
                               (expand-file-name file emacsconf-public-media-directory) t)))
      ;; Remove files from public
      (let ((files (directory-files emacsconf-public-media-directory nil
                                    (concat "^"
                                            (regexp-quote (plist-get talk :file-prefix)
        (mapc (lambda (file)
                (delete-file (expand-file-name file emacsconf-public-media-directory)))
     (expand-file-name "index.m3u" emacsconf-public-media-directory)
     (concat emacsconf-name " " emacsconf-year)
     (emacsconf-public-talks (emacsconf-get-talk-info)))))

The :public property is automatically added by this function based on the TODO status or the time:

emacsconf-add-talk-status: Add status label and public info.
(defun emacsconf-add-talk-status (o)
  "Add status label and public info."
  (plist-put o :status-label
             (or (assoc-default (plist-get o :status)
                                emacsconf-status-types 'string= "")
                 (plist-get o :status)))
  (when (or
         (member (plist-get o :status)
         (time-less-p (plist-get o :start-time)
    (plist-put o :public t))

Update the wiki page

This function updates the schedule page and the page for the talk. It's also in emacsconf-publish.el.

emacsconf-publish-update-talk: Publish the schedule page and the page for this talk.
(defun emacsconf-publish-update-talk (talk)
  "Publish the schedule page and the page for this talk."
  (interactive (list (emacsconf-complete-talk-info)))
  (when (stringp talk) (setq talk (emacsconf-resolve-talk talk)))
  (when (functionp 'emacsconf-upcoming-insert-or-update)
    (let ((info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
      (emacsconf-publish-before-page talk info)
      (emacsconf-publish-after-page talk info)
      (emacsconf-publish-schedule info))))

It uses the publishing functions I described in this post on adding talks to the wiki.

This macro commits changes when emacsconf-publish-autocommit-wiki is t, so I need to set that also.

(defmacro emacsconf-publish-with-wiki-change (&rest body)
  (declare (indent 0) (debug t))
      (and (stringp ,(car body)) ,(car body)))))

Make the videos public on YouTube and Toobnix

This is low-priority, but it might be nice to figure out. The easiest way is probably to use open the Youtube/Toobnix URLs on my computer and then use either Tampermonkey or Spookfox to set the talk to public. Someday!

Update the talk status on the server

Last year, I experimented with having the shell scripts automatically update the status of the talk from TO_STREAM to PLAYING and from PLAYING to CLOSED_Q. Since I've moved the talk-running into track-specific crontabs, now I need to sudo back to the orga user and set XDG_RUNTIME_DIR in order to use emacsclient. I can call this with sudo -u orga talk $slug $status in the roles/obs/templates/handle-session script.

Here's the Ansible template for roles/prerec/templates/talk. It uses getent to look up the user ID.

# How to use: talk slug from-status-regexp to-status
# or talk slug to-status


if [ "x$TO_STATUS" == "x" ]; then
#echo "Pulling"
#git pull
echo "Updating status..."
XDG_RUNTIME_DIR=$XDG_RUNTIME_DIR emacsclient --eval "(emacsconf-with-todo-hooks (emacsconf-update-talk-status \"$SLUG\" \"$FROM_STATUS\" \"$TO_STATUS\"))" -a emacs
#echo "Committing and pushing in the background"
#git commit -m "Update task status for $SLUG from $FROM_STATUS to $TO_STATUS"
#git push &

Testing notes

Looks like everything works fine when I run it from the crontab: the talk status is updated, the media files are published, the wiki is updated, and the talks are announced on IRC. Backup plan A is to manually control the talk status using Emacs on the server. Backup plan B is to control the talk status using Emacs on my laptop. Backup plan C is to call the individual functions instead of relying on the todo state change functions. I think it'll all work out, although I'll probably want to do another dry run at some point to make sure. Slowly getting there…

#EmacsConf backstage: autopilot with crontab

| emacs, emacsconf, subed

[2023-10-26 Thu]: updated handle-session and added talk

I figured out multi-track streaming so close to EmacsConf 2022 that there wasn't enough time to get other volunteers used to working with the setup, especially since I was still scrambling to figure out more infrastructure as the conference approached. We decided I'd run both streams myself, which meant I needed to make things as automatic as possible so that I wouldn't go crazy. I wanted a lot of things to happen automatically: playing recorded intros and videos, browsing to the right URLs depending on the type of Q&A, publishing updates to the wiki, and so on.

I used timers and TODO state changes to execute commands via TRAMP, which was pretty cool for the most part. But it turned out TRAMP doesn't like being called when it's already running, like when it's being called from two timers going off at the same time. It gives a "Forbidden reentrant call of TRAMP". We found a couple of quick workarounds: I could reschedule the talks to be a minute apart, or I could cancel the conflicting timer and just start them with the shell scripts.

Last year, we had a shell script that played the intro and the main talk, and other scripts to handle the Q&A by opening BigBlueButton, Etherpad, or the IRC channel. Much of the logic was in Emacs Lisp because it was easy to write it that way. For this year, I wanted to write a script that handled the intro, video, and Q&A portions. This is now in roles/obs/templates/handle-session.

# Handle the intro/talk/Q&A for a session
# Usage: handle-session $SLUG


# Kill background music if playing
if screen -list | grep -q background; then
    screen -S background -X quit

# Update the status
sudo -u  talk $SLUG PLAYING &

# Update the overlay
overlay $SLUG

# Play the intro if it exists. If it doesn't exist, switch to the intro slide and stop processing.

if [[ -f $BASE_DIR/assets/intros/$SLUG.webm ]]; then
  killall -s TERM $FIREFOX_NAME
  mpv $BASE_DIR/assets/intros/$SLUG.webm
  firefox --kiosk $BASE_DIR/assets/in-between/$SLUG.png
  exit 0

# Play the video if it exists. If it doesn't exist, switch to the BBB room and stop processing.
if [ "x$TEST_MODE" = "x" ]; then
if [ ! -f "$FILE" ]; then
    # Is there an original file?

if [[ -f $FILE ]]; then
  killall -s TERM $FIREFOX_NAME
  mpv $FILE
  /usr/local/bin/bbb $SLUG
  exit 0

sudo -u  talk $SLUG CLOSED_Q &

# Open the appropriate Q&A URL
QA=$(jq -r '.talks[] | select(.slug=="'$SLUG'")["qa-backstage-url"]' < $BASE_DIR/talks.json)
QA_TYPE=$(jq -r '.talks[] | select(.slug=="'$SLUG'")["qa-type"]' < $BASE_DIR/talks.json)
if [ "$QA_TYPE" = "live" ]; then
  /usr/local/bin/bbb $SLUG
elif [ "$QA" != "null" ]; then
  /usr/local/bin/music &
  /usr/bin/firefox $QA
  # i3-msg 'layout splith'

It builds on roles/obs/templates/bbb, roles/obs/templates/overlay, and roles/obs/templates/music. I also have a roles/prerec/templates/talk script that uses emacsclient to update the status of the talk.

I wrote some Tampermonkey scripts to automate joining the web conference and the IRC channel.

Now that we have a script that handles all the different things related to a session, it's easier to schedule the execution of that script. Instead of using Emacs timers and running into that problem with tramp, I want to try using cron. Cron is a standard UNIX and Linux tool for scheduling things to run at certain times. You make a plain text file in a particular format: minute, hour, day of month, month, day of week, and then the command, and then you tell cron to use that file with something like crontab your-file. Since it's plain text, we can generate it with Emacs Lisp and format-time-string, save with TRAMP, and install with ssh. Each track has its own user account for streaming, so each track can have its own file.

emacsconf-stream-format-crontab: Return crontab entries for TALKS.
(defun emacsconf-stream-format-crontab (track talks &optional test-mode)
  "Return crontab entries for TALKS.
Use the display specified in TRACK.
If TEST-MODE is non-nil, load the videos from the test directory."
" (plist-get track :uid))
    (lambda (talk)
      (format "%s /usr/bin/screen -dmS play-%s bash -c \"DISPLAY=%s TEST_MODE=%s /usr/local/bin/handle-session %s | tee -a ~/track.log\"\n"
              ;; cron times are UTC
              (format-time-string "%-M %-H %-d %m *" (plist-get talk :start-time))
              (plist-get talk :slug)
              (plist-get track :vnc-display)
              (if test-mode "1" "")
              (plist-get talk :slug)))
    (emacsconf-filter-talks talks))))

emacsconf-stream-crontabs: Write the streaming users’ crontab files.
(defun emacsconf-stream-crontabs (&optional test-mode info)
  "Write the streaming users' crontab files.
If TEST-MODE is non-nil, use the videos in the test directory.
If INFO is non-nil, use that as the schedule instead."
  (let ((emacsconf-publishing-phase 'conference))
    (setq info (or info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
    (dolist (track emacsconf-tracks)
      (let ((talks (seq-filter (lambda (talk)
                                 (string= (plist-get talk :track)
                                          (plist-get track :name)))
            (crontab (expand-file-name (concat (plist-get track :id) ".crontab")
                                       (concat (plist-get track :tramp) "~"))))
        (with-temp-file crontab
          (when (plist-get track :autopilot)
            (insert (emacsconf-stream-format-crontab track talks test-mode))))
        (emacsconf-stream-track-ssh track (concat "crontab ~/" (plist-get track :id) ".crontab"))))))

I want to test the whole setup before the conference, of course. First, I needed test videos. This generates test videos and subtitles following our naming convention.

(defun emacsconf-stream-generate-test-videos (&optional info)
  "Generate 1-minute test videos for INFO."
  (setq info (or info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
  (let* ((dir (expand-file-name "test" emacsconf-stream-asset-dir))
         (default-directory dir)
         (subed-default-subtitle-length 1000)
         (test-length 60))
    (unless (file-directory-p dir)
      (make-directory dir t))
     (format "ffmpeg -y -f lavfi -i testsrc=duration=%d:size=1280x720:rate=10 -i background-music.opus -shortest %s "
             test-length (expand-file-name "template.webm" dir)))
    (dolist (talk info)
      (with-temp-file (expand-file-name (concat (plist-get talk :file-prefix) "--main.vtt") dir)
        (dotimes (i test-length)
           (* i 1000)
           (1- (* i 1000))
           (format "%s %02d %s"
                   (plist-get talk :slug)
                   (substring "123456789 123456789 123456789 123456789 123456789 123456789 "
                              (1+ (length (format "%s %02d" (plist-get talk :slug) i))))))))
       (expand-file-name "template.webm" dir)
       (expand-file-name (concat (plist-get talk :file-prefix) "--main.webm") dir)

Then I needed to write a crontab based on a different schedule. This code sets up a series of test videos to start about a minute after I run the code, with the dev stream set up to start a minute after the gen stream.

(let* ((offset-seconds 60)
       (start-time (time-add (current-time) offset-seconds))
       (emacsconf-schedule-validation-functions nil)
       (emacsconf-schedule-default-buffer-minutes 1)
       (emacsconf-schedule-default-buffer-minutes-for-live-q-and-a 1)
       (emacsconf-schedule-strategies '(emacsconf-schedule-allocate-buffer-time
       (schedule (emacsconf-schedule-prepare
                      :start ,(format-time-string "%Y-%m-%d %H:%M" start-time)
                      :set-track "General")
                     (sat-open :time 1)
                     (uni :time 1) ; live Q&A
                     (adventure :time 1) ; pad Q&A
                      ,(format-time-string "%Y-%m-%d %H:%M" (time-add start-time 60))
                      :set-track "Development")
                     (repl :time 1) ; IRC
                     (matplotllm :time 1) ; pad
                     (voice :time 1) ; live
  (emacsconf-stream-crontabs t schedule))

That generates gen.crontab and dev.crontab. This is what gen.crontab looks like for testing:

35 11 26 10 * /usr/bin/screen -dmS play-sat-open bash -c "DISPLAY=:5 TEST_MODE=1 /usr/local/bin/handle-session sat-open | tee -a ~/track.log"
36 11 26 10 * /usr/bin/screen -dmS play-uni bash -c "DISPLAY=:5 TEST_MODE=1 /usr/local/bin/handle-session uni | tee -a ~/track.log"
38 11 26 10 * /usr/bin/screen -dmS play-adventure bash -c "DISPLAY=:5 TEST_MODE=1 /usr/local/bin/handle-session adventure | tee -a ~/track.log"

The result: for both tracks, the intro videos play, the test videos play, and web browsers go to the right places for the Q&A.

In case I need to resume manual control:

emacsconf-stream-cancel-crontab: Remove crontab for TRACK.
(defun emacsconf-stream-cancel-crontab (track)
  "Remove crontab for TRACK."
  (interactive (list (emacsconf-complete-track)))
  (plist-put track :autopilot nil)
  (emacsconf-stream-track-ssh track "crontab -r"))

emacsconf-stream-cancel-all-crontabs: Remove crontabs.
(defun emacsconf-stream-cancel-all-crontabs ()
  "Remove crontabs."
  (dolist (track emacsconf-tracks)
    (plist-put track :autopilot nil)
    (emacsconf-stream-track-ssh track "crontab -r")))

Here are some things I learned along the way:

  • I needed to use timedatectl set-timezone America/Toronto to change the server's timezone to America/Toronto so that the crontab would run at the right time.

    In Ansible terms, that's:

    	- name: Set system timezone
    		tags: tz
    			name: ""
    	- name: Restart cron
    		tags: tz
    			name: cron
    			state: restarted
  • I also needed to specify the PATH so that I didn't need to add the absolute paths in all the other shell scripts, XDG_RUNTIME_DIR to get audio working, and DISPLAY so that windows showed up in the right place.

I think this will let me run both tracks for EmacsConf with more ease and less frantic juggling. We'll see!

#EmacsConf backstage: Automatically join BigBlueButton web conferences using Tampermonkey

| emacs, emacsconf

[2023-10-22 Sun] Added IRC script

During EmacsConf, we join BigBlueButton webconferences for live presentations and Q&A sessions so that the speakers and the host can be on stream. I wanted to reduce the number of manual steps needed to join the web conference, since any clicks or keystrokes would need to be done via a VNC connection. I used Tampermonkey to write a script to join BigBlueButton and set things up the way we want to.

I don't have the Tampermonkey installation and setup automated with Ansible yet, but here's what I did for each track:

  1. Install the Tampermonkey extension by going to .
  2. Install the script by clicking on the Tampermonkey extension, choosing Install New Script, and pasting in the following:

       // ==UserScript==
       // @name         Emacsconf BBB setup
       // @namespace
       // @version      0.1
       // @description  Join BBB and set things up
       // @author       You
       // @match*
       // @icon
       // @grant        none
       // ==/UserScript==
           async function() {
               'use strict';
               const NAME = 'emacsconf';
               async function waitUntil(conditionFunc, interval=500, timeout=null) {
                   let initResult = conditionFunc();
                   if (initResult) return initResult;
                   return new Promise((resolve, reject) => {
                       let timeSoFar = 0;
                       let timer = setInterval(() => {
                           let result = conditionFunc();
                           if (result) {
                           timeSoFar += interval;
                           if (timeout && timeSoFar > timeout) {
                       }, interval);
               if (document.querySelector('input.join-form')) {
                   document.querySelector('input.join-form').value = NAME;
               await waitUntil(() => document.querySelector('.icon-bbb-listen')).then((e) => e.closest('button').click());
               await waitUntil(() => document.querySelector('.icon-bbb-user')).then((e) => e.closest('button').click());

    Press Ctrl+s to save.

  3. Add this script to join the IRC channel:

       // ==UserScript==
       // @name         Connect to EmacsConf chat automatically
       // @namespace
       // @version      0.1
       // @description  try to take over the world!
       // @author       You
       // @match*
       // @icon
       // @grant        none
       // ==/UserScript==
       (function() {
           'use strict';
           setTimeout(() => {
               if (document.querySelector('.connect-row')) {
           }, 1000);
  4. Join an BBB meeting. Check the address bar to see if autoplay is disabled (crossed-out autoplay icon). If it is, click on it and change Block audio to Allow audio and video.

Now I can use this Ansible template for a shell script to connect to the BBB session (roles/obs/templates/bbb):

# Open the Big Blue Button room using the backstage link

# Kill the background music if playing
if screen -list | grep -q background; then
    screen -S background -X quit

# Update the overlay
overlay $SLUG
killall -s TERM firefox-esr
firefox$SLUG.html &
sleep 5
xdotool search --class firefox windowactivate --sync
xdotool key Return
xdotool key F11

The overlay is handled by this script (roles/obs/templates/overlay):


SLUG=$(echo "$1" | perl -ne 'if (/emacsconf-[0-9]*-(.*?)--/) { print $1; } else { print; }')

if [[ -f /assets/overlays/$SLUG-other.svg.png ]]; then
    echo "Found other overlay for $SLUG, copying"
    cp /assets/overlays/$SLUG-other.svg.png ~/other.png
elif [[ -f /assets/overlays/$SLUG-video.svg.png ]]; then
    echo "Found video overlay for $SLUG, copying"
    cp /assets/overlays/$SLUG-video.svg.png ~/other.png
    echo "Could not find /assets/overlays/$SLUG-other.svg.png, please override ~/other.png manually"
    cp /assets/overlays/blank-other.svg.png ~/other.png
if [[ -f /assets/overlays/$SLUG-video.svg.png ]]; then
    echo "Found video overlay for $SLUG, copying"
    cp /assets/overlays/$SLUG-video.svg.png ~/video.png
    echo "Could not find /assets/overlays/$SLUG-video.svg.png, override ~/video.png manually"
    cp /assets/overlays/blank-video.svg.png ~/video.png

The result: I can run bbb uni and it'll automatically join the Q&A session for the uni talk in listen-only mode and with the user list hidden.

Getting Mermaid JS and ob-mermaid running on my system - needed to symlink Chromium for Puppeteer

| emacsconf, nodejs, org, emacs

I wanted to use Mermaid to make diagrams, but I ran into this issue when trying to run it:

Error: Could not find Chromium (rev. 1108766). This can occur if either
 1. you did not perform an installation before running the script (e.g. `npm install`) or
 2. your cache path is incorrectly configured (which is: /home/sacha/.cache/puppeteer).
For (2), check out our guide on configuring puppeteer at

It turns out that I needed to do the following:

sudo npm install -g puppeteer mermaid @mermaid-js/mermaid-cli --unsafe-perm
# Cache Chromium for my own user
node /usr/lib/node_modules/puppeteer/install.js --unsafe-perm
sudo npm install -g mermaid @mermaid-js/mermaid-cli
ln -s ~/.cache/puppeteer/chrome/linux-117.0.5938.149 ~/.cache/puppeteer/chrome/linux-1108766
ln -s ~/.cache/puppeteer/chrome/linux-117.0.5938.149/chrome-linux64 ~/.cache/puppeteer/chrome/linux-117.0.5938.149/chrome-linux

(The exact versions might be different for your installation.)

Then I could make a Mermaid file and try it out with mmdc -i input.mmd -o output.svg, and then I could confirm that it works directly from Org with ob-mermaid:

    input ->> res: original.mp4;
    res ->> backstage: reencoded.webm;
    backstage ->> laptop: cached files;
    laptop ->> backstage: index;
    input ->> res: main.vtt;
    res ->> backstage: main.webm;
    backstage ->> laptop: cached files;
    laptop ->> backstage: index;
    input ->> res: normalized.opus;
    res ->> backstage: main.webm;
    backstage ->> laptop: cached files;
    laptop ->> backstage: index;
    backstage ->> media: public files;

#EmacsConf backstage: adding notes to Org logbook drawers from e-mails

| emacs, emacsconf, notmuch

Sometimes I want to work with all the talks associated with an email in my inbox. For example, maybe a speaker said that the draft schedules are fine, and I want to make a note of that in the conference Org file.

First we start with a function that gets the e-mail addresses for a talk. Some speakers have different e-mail addresses for public contact or private contact, and some e-mail us from other addresses.

emacsconf-mail-get-all-email-addresses: Return all the possible e-mail addresses for TALK.
(defun emacsconf-mail-get-all-email-addresses (talk)
  "Return all the possible e-mail addresses for TALK."
       (lambda (field) (plist-get talk field))
       '(:email :public-email :email-alias)))
   " *, *"))

Then we can use that to find the talks for a given e-mail address.

emacsconf-mail-talks: Return a list of talks matching EMAIL.
(defun emacsconf-mail-talks (email)
  "Return a list of talks matching EMAIL."
  (setq email (downcase (mail-strip-quoted-names email)))
   (lambda (o) (member email (emacsconf-mail-get-all-email-addresses o)))

We can loop over that to add a note for the e-mail.

emacsconf-mail-add-to-logbook: Add to logbook for all matching talks from this speaker.
(defun emacsconf-mail-add-to-logbook (email note)
  "Add to logbook for all matching talks from this speaker."
   (let* ((email (mail-strip-quoted-names
                  (plist-get (plist-get (notmuch-show-get-message-properties) :headers)
          (talks (emacsconf-mail-talks email)))
      (read-string (format "Note for %s: "
                           (mapconcat (lambda (o) (plist-get o :slug))
                                      talks", "))))))
     (lambda (talk)
       (emacsconf-add-to-talk-logbook talk note))
     (emacsconf-mail-talks email))))

The actual addition of notes is handled by these functions.

emacsconf-add-to-logbook: Add NOTE as a logbook entry for the current subtree.
(defun emacsconf-add-to-logbook (note)
  "Add NOTE as a logbook entry for the current subtree."
  (move-marker org-log-note-return-to (point))
  (move-marker org-log-note-marker (point))
    (insert note)
    (let ((org-log-note-purpose 'note))

Then we have a function that looks for the heading for a note and then adds a logbook entry to it.

emacsconf-add-to-talk-logbook: Add NOTE as a logbook entry for TALK.
(defun emacsconf-add-to-talk-logbook (talk note)
  "Add NOTE as a logbook entry for TALK."
  (interactive (list (emacsconf-complete-talk) (read-string "Note: ")))
    (emacsconf-with-talk-heading talk
      (emacsconf-add-to-logbook note))))

All together, that makes it easy to use Emacs as a very simple contact relationship management system where I can take notes based on the e-mails that come in.

Figure 1: Logging notes from e-mail

These functions are in emacsconf-mail.el.

#EmacsConf backstage: Using Spookfox to automate creating BigBlueButton rooms in Mozilla Firefox

| emacsconf, emacs, org

Naming conventions make it easier for other people to find things. Just like with file prefixes, I like to use a standard naming pattern for our BigBlueButton web conference rooms. For EmacsConf 2022, we used ec22-day-am-gen Speaker name (slugs). For EmacsConf 2023, I want to set up the BigBlueButton rooms before the schedule settles down, so I won't encode the time or track information into it. Instead, I'll use Speaker name (slugs) - emacsconf2023.

BigBlueButton does have an API for managing rooms, but that requires a shared secret that I don't know yet. I figured I'd just automate it through my browser. Over the last year, I've started using Spookfox to control the Firefox web browser from Emacs. It's been pretty handy for scrolling webpages up and down, so I wondered if I could replace my old xdotool-based automation. Here's what I came up with for this year.

First, I need a function that creates the BBB room for a group of talks and updates the Org entry with the URL. Adding a slight delay makes it a bit more reliable.

emacsconf-spookfox-create-bbb: Create a BBB room for this group of talks.
(defun emacsconf-spookfox-create-bbb (group)
  "Create a BBB room for this group of talks.
GROUP is (email . (talk talk talk)).
Needs a Spookfox connection."
  (let* ((bbb-name
          (format "%s (%s) - %s%s"
                  (mapconcat (lambda (o) (plist-get o :slug)) (cdr group) ", ")
                  (plist-get (cadr group) :speakers)
           "window.location.origin + [...document.querySelectorAll('')].find((o) => o.textContent.trim() == '%s').closest('tr').querySelector('.delete-room').getAttribute('data-path')"
         (create-command (format "document.querySelector('#create-room-block').click();
document.querySelector('#create-room-name').value = \"%s\";
    (setq path (spookfox-js-injection-eval-in-active-tab retrieve-command t))
    (unless path
      (dolist (cmd (split-string create-command ";"))
        (spookfox-js-injection-eval-in-active-tab cmd t)
        (sleep-for 2))
      (sleep-for 2)
      (setq path (spookfox-js-injection-eval-in-active-tab retrieve-command t)))
    (when path
      (dolist (talk (cdr group))
          (emacsconf-with-talk-heading talk
            (org-entry-put (point) "ROOM" path))))
      (cons bbb-name path))))

Then I need to iterate over the list of talks that have live Q&A sessions but don't have BBB rooms assigned yet so that I can create them.

emacsconf-spookfox-create-bbb-for-live-talks: Create BBB rooms for talks that don’t have them yet.
(defun emacsconf-spookfox-create-bbb-for-live-talks ()
  "Create BBB rooms for talks that don't have them yet."
  (let* ((talks (seq-filter
                 (lambda (o)
                   (and (string-match "live" (or (plist-get o :q-and-a) ""))
                        (not (string= (plist-get o :status) "CANCELLED"))
                        (not (plist-get o :bbb-room))))
                 (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
         (groups (and talks (emacsconf-mail-groups talks))))
    (dolist (group groups)
      (emacsconf-spookfox-create-bbb group))))

The result: a whole bunch of rooms ready for people to check in.

Figure 1: BigBlueButton rooms

Using Spookfox to communicate with Firefox from Emacs Lisp made it easy to get data in and out of my browser. Handy!

This code is in emacsconf-spookfox.el.