#EmacsConf backstage: autopilot with crontab

| emacs, emacsconf, subed

[2023-10-26 Thu]: updated handle-session and added talk

I figured out multi-track streaming so close to EmacsConf 2022 that there wasn't enough time to get other volunteers used to working with the setup, especially since I was still scrambling to figure out more infrastructure as the conference approached. We decided I'd run both streams myself, which meant I needed to make things as automatic as possible so that I wouldn't go crazy. I wanted a lot of things to happen automatically: playing recorded intros and videos, browsing to the right URLs depending on the type of Q&A, publishing updates to the wiki, and so on.

I used timers and TODO state changes to execute commands via TRAMP, which was pretty cool for the most part. But it turned out TRAMP doesn't like being called when it's already running, like when it's being called from two timers going off at the same time. It gives a "Forbidden reentrant call of TRAMP". We found a couple of quick workarounds: I could reschedule the talks to be a minute apart, or I could cancel the conflicting timer and just start them with the shell scripts.

Last year, we had a shell script that played the intro and the main talk, and other scripts to handle the Q&A by opening BigBlueButton, Etherpad, or the IRC channel. Much of the logic was in Emacs Lisp because it was easy to write it that way. For this year, I wanted to write a script that handled the intro, video, and Q&A portions. This is now in roles/obs/templates/handle-session.

handle-session 
#!/bin/bash
# 
#
# Handle the intro/talk/Q&A for a session
# Usage: handle-session $SLUG

YEAR=""
BASE_DIR=""
FIREFOX_NAME=firefox-esr
SLUG=$1

# Kill background music if playing
if screen -list | grep -q background; then
    screen -S background -X quit
fi

# Update the status
sudo -u  talk $SLUG PLAYING &

# Update the overlay
overlay $SLUG

# Play the intro if it exists. If it doesn't exist, switch to the intro slide and stop processing.

if [[ -f $BASE_DIR/assets/intros/$SLUG.webm ]]; then
  killall -s TERM $FIREFOX_NAME
  mpv $BASE_DIR/assets/intros/$SLUG.webm
else
  firefox --kiosk $BASE_DIR/assets/in-between/$SLUG.png
  exit 0
fi

# Play the video if it exists. If it doesn't exist, switch to the BBB room and stop processing.
if [ "x$TEST_MODE" = "x" ]; then
  LIST=($BASE_DIR/assets/stream/--$SLUG*--main.webm)
else
  LIST=($BASE_DIR/assets/test/--$SLUG*--main.webm)
fi
FILE="${LIST[0]}"
if [ ! -f "$FILE" ]; then
    # Is there an original file?
    LIST=($BASE_DIR/assets/stream/--$SLUG*--original.{webm,mp4,mov})
    FILE="${LIST[0]}"
fi

if [[ -f $FILE ]]; then
  killall -s TERM $FIREFOX_NAME
  mpv $FILE
else
  /usr/local/bin/bbb $SLUG
  exit 0
fi

sudo -u  talk $SLUG CLOSED_Q &

# Open the appropriate Q&A URL
QA=$(jq -r '.talks[] | select(.slug=="'$SLUG'")["qa-backstage-url"]' < $BASE_DIR/talks.json)
QA_TYPE=$(jq -r '.talks[] | select(.slug=="'$SLUG'")["qa-type"]' < $BASE_DIR/talks.json)
echo "QA_TYPE $QA_TYPE QA $QA"
if [ "$QA_TYPE" = "live" ]; then
  /usr/local/bin/bbb $SLUG
elif [ "$QA" != "null" ]; then
  /usr/local/bin/music &
  /usr/bin/firefox $QA
  # i3-msg 'layout splith'
fi
wait

It builds on roles/obs/templates/bbb, roles/obs/templates/overlay, and roles/obs/templates/music. I also have a roles/prerec/templates/talk script that uses emacsclient to update the status of the talk.

I wrote some Tampermonkey scripts to automate joining the web conference and the IRC channel.

Now that we have a script that handles all the different things related to a session, it's easier to schedule the execution of that script. Instead of using Emacs timers and running into that problem with tramp, I want to try using cron. Cron is a standard UNIX and Linux tool for scheduling things to run at certain times. You make a plain text file in a particular format: minute, hour, day of month, month, day of week, and then the command, and then you tell cron to use that file with something like crontab your-file. Since it's plain text, we can generate it with Emacs Lisp and format-time-string, save with TRAMP, and install with ssh. Each track has its own user account for streaming, so each track can have its own file.

emacsconf-stream-format-crontab: Return crontab entries for TALKS.
(defun emacsconf-stream-format-crontab (track talks &optional test-mode)
  "Return crontab entries for TALKS.
Use the display specified in TRACK.
If TEST-MODE is non-nil, load the videos from the test directory."
  (concat
   (format
    "PATH=/usr/local/bin:/usr/bin
MAILTO=\"\"
XDG_RUNTIME_DIR=\"/run/user/%d\"
" (plist-get track :uid))
   (mapconcat
    (lambda (talk)
      (format "%s /usr/bin/screen -dmS play-%s bash -c \"DISPLAY=%s TEST_MODE=%s /usr/local/bin/handle-session %s | tee -a ~/track.log\"\n"
              ;; cron times are UTC
              (format-time-string "%-M %-H %-d %m *" (plist-get talk :start-time))
              (plist-get talk :slug)
              (plist-get track :vnc-display)
              (if test-mode "1" "")
              (plist-get talk :slug)))
    (emacsconf-filter-talks talks))))

emacsconf-stream-crontabs: Write the streaming users’ crontab files.
(defun emacsconf-stream-crontabs (&optional test-mode info)
  "Write the streaming users' crontab files.
If TEST-MODE is non-nil, use the videos in the test directory.
If INFO is non-nil, use that as the schedule instead."
  (interactive)
  (let ((emacsconf-publishing-phase 'conference))
    (setq info (or info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
    (dolist (track emacsconf-tracks)
      (let ((talks (seq-filter (lambda (talk)
                                 (string= (plist-get talk :track)
                                          (plist-get track :name)))
                               info))
            (crontab (expand-file-name (concat (plist-get track :id) ".crontab")
                                       (concat (plist-get track :tramp) "~"))))
        (with-temp-file crontab
          (when (plist-get track :autopilot)
            (insert (emacsconf-stream-format-crontab track talks test-mode))))
        (emacsconf-stream-track-ssh track (concat "crontab ~/" (plist-get track :id) ".crontab"))))))

I want to test the whole setup before the conference, of course. First, I needed test videos. This generates test videos and subtitles following our naming convention.

emacsconf-stream-generate-test-videos
(defun emacsconf-stream-generate-test-videos (&optional info)
  "Generate 1-minute test videos for INFO."
  (interactive)
  (setq info (or info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
  (let* ((dir (expand-file-name "test" emacsconf-stream-asset-dir))
         (default-directory dir)
         (subed-default-subtitle-length 1000)
         (test-length 60))
    (unless (file-directory-p dir)
      (make-directory dir t))
    (shell-command
     (format "ffmpeg -y -f lavfi -i testsrc=duration=%d:size=1280x720:rate=10 -i background-music.opus -shortest %s "
             test-length (expand-file-name "template.webm" dir)))
    (dolist (talk info)
      (with-temp-file (expand-file-name (concat (plist-get talk :file-prefix) "--main.vtt") dir)
        (subed-vtt-mode)
        (subed-auto-insert)
        (dotimes (i test-length)
          (subed-append-subtitle
           nil
           (* i 1000)
           (1- (* i 1000))
           (format "%s %02d %s"
                   (plist-get talk :slug)
                   i
                   (substring "123456789 123456789 123456789 123456789 123456789 123456789 "
                              (1+ (length (format "%s %02d" (plist-get talk :slug) i))))))))
      (copy-file
       (expand-file-name "template.webm" dir)
       (expand-file-name (concat (plist-get talk :file-prefix) "--main.webm") dir)
       t))))

Then I needed to write a crontab based on a different schedule. This code sets up a series of test videos to start about a minute after I run the code, with the dev stream set up to start a minute after the gen stream. 

(let* ((offset-seconds 60)
       (start-time (time-add (current-time) offset-seconds))
       (emacsconf-schedule-validation-functions nil)
       (emacsconf-schedule-default-buffer-minutes 1)
       (emacsconf-schedule-default-buffer-minutes-for-live-q-and-a 1)
       (emacsconf-schedule-strategies '(emacsconf-schedule-allocate-buffer-time
                                        emacsconf-schedule-copy-previous-track))
       (schedule (emacsconf-schedule-prepare
                  (emacsconf-schedule-inflate-sexp
                   `(("GEN"
                      :start ,(format-time-string "%Y-%m-%d %H:%M" start-time)
                      :set-track "General")
                     (sat-open :time 1)
                     (uni :time 1) ; live Q&A
                     (adventure :time 1) ; pad Q&A
                     ("DEV"
                      :start
                      ,(format-time-string "%Y-%m-%d %H:%M" (time-add start-time 60))
                      :set-track "Development")
                     (repl :time 1) ; IRC
                     (matplotllm :time 1) ; pad
                     (voice :time 1) ; live
                     )))))
  (emacsconf-stream-crontabs t schedule))

That generates gen.crontab and dev.crontab. This is what gen.crontab looks like for testing: 

PATH=/usr/local/bin:/usr/bin
MAILTO=""
XDG_RUNTIME_DIR="/run/user/2002"
35 11 26 10 * /usr/bin/screen -dmS play-sat-open bash -c "DISPLAY=:5 TEST_MODE=1 /usr/local/bin/handle-session sat-open | tee -a ~/track.log"
36 11 26 10 * /usr/bin/screen -dmS play-uni bash -c "DISPLAY=:5 TEST_MODE=1 /usr/local/bin/handle-session uni | tee -a ~/track.log"
38 11 26 10 * /usr/bin/screen -dmS play-adventure bash -c "DISPLAY=:5 TEST_MODE=1 /usr/local/bin/handle-session adventure | tee -a ~/track.log"

The result: for both tracks, the intro videos play, the test videos play, and web browsers go to the right places for the Q&A.

In case I need to resume manual control:

emacsconf-stream-cancel-crontab: Remove crontab for TRACK.
(defun emacsconf-stream-cancel-crontab (track)
  "Remove crontab for TRACK."
  (interactive (list (emacsconf-complete-track)))
  (plist-put track :autopilot nil)
  (emacsconf-stream-track-ssh track "crontab -r"))

emacsconf-stream-cancel-all-crontabs: Remove crontabs.
(defun emacsconf-stream-cancel-all-crontabs ()
  "Remove crontabs."
  (interactive)
  (dolist (track emacsconf-tracks)
    (plist-put track :autopilot nil)
    (emacsconf-stream-track-ssh track "crontab -r")))

Here are some things I learned along the way:

  • I needed to use timedatectl set-timezone America/Toronto to change the server's timezone to America/Toronto so that the crontab would run at the right time.

    In Ansible terms, that's: 

    	- name: Set system timezone
		tags: tz
		community.general.timezone:
			name: ""
	- name: Restart cron
		tags: tz
		ansible.builtin.service:
			name: cron
			state: restarted
  • I also needed to specify the PATH so that I didn't need to add the absolute paths in all the other shell scripts, XDG_RUNTIME_DIR to get audio working, and DISPLAY so that windows showed up in the right place.

I think this will let me run both tracks for EmacsConf with more ease and less frantic juggling. We'll see!

You can comment with Disqus or you can e-mail me at sacha@sachachua.com.