Categories: emacsconf

RSS - Atom - Subscribe via email

EmacsConf backstage: jumping to and working with talks using Embark

| emacs, emacsconf

In the course of organizing and running EmacsConf, I often need to jump to or act on specific talks. I have a function that jumps to the talk heading so that I can look up additional information or add notes.

output-2023-09-10-14:04:30.gif
Figure 1: Jumping to a talk

emacsconf-go-to-talk: Jump to the talk heading matching SEARCH.
(defun emacsconf-go-to-talk (search)
  "Jump to the talk heading matching SEARCH."
  (interactive (list (emacsconf-complete-talk)))
  (find-file emacsconf-org-file)
  (widen)
  (cond
   ((plist-get search :slug)
    (goto-char (org-find-property "SLUG" (plist-get search :slug))))
   ((emacsconf-get-slug-from-string search)
    (goto-char (org-find-property "SLUG" (emacsconf-get-slug-from-string search))))
   (t
    (goto-char
     (catch 'found
       (org-map-entries
        (lambda ()
          (when (string-match search
                              (cons
                               (concat (org-entry-get (point) "SLUG") " - "
                                       (org-entry-get (point) "ITEM") " - "
                                       (org-entry-get (point) "NAME") " - "
                                       (org-entry-get (point) "EMAIL"))
                               (point)))
            (throw 'found (point))))
        "SLUG={.}")))))
  (org-reveal))

Most of the work is done in a completion function that makes it easy to specify a talk using the slug (talk ID), title, or speaker names.

emacsconf-complete-talk: Offer talks for completion.
(defun emacsconf-complete-talk (&optional info)
  "Offer talks for completion.
If INFO is specified, limit it to that list."
  (let ((choices
         (if (and (null info) emacsconf-complete-talk-cache)
             emacsconf-complete-talk-cache
           (mapcar (lambda (o)
                     (string-join
                      (delq nil
                            (mapcar (lambda (f) (plist-get o f))
                                    '(:slug :title :speakers :irc)))
                      " - "))
                   (or info (emacsconf-get-talk-info))))))
    (completing-read
     "Talk: " 
     (lambda (string predicate action)
       (if (eq action 'metadata)
           '(metadata (category . emacsconf))
         (complete-with-action action choices string predicate))))))

In addition to jumping to the Org heading for a talk, there are a bunch of other things I might want to do. Embark lets me add a bunch of shortcuts for working with a talk. I could open the caption file, edit the talk's wiki page, change a talk's property, e-mail the speaker, or more. Here's the Embark-related code from emacsconf.el:

Embark-related code
;;; Embark
(defun emacsconf-embark-finder ()
  "Identify when we're on a talk subtree."
  (when (and (derived-mode-p 'org-mode)
             (org-entry-get-with-inheritance "SLUG"))
    (cons 'emacsconf (org-entry-get-with-inheritance "SLUG"))))

(defun emacsconf-insert-talk-title (search)
  "Insert the talk title matching SEARCH."
  (interactive (list (emacsconf-complete-talk)))
  (insert (plist-get (emacsconf-search-talk-info search) :title)))

(with-eval-after-load 'embark
  (add-to-list 'embark-target-finders 'emacsconf-embark-finder)
  (defvar-keymap embark-emacsconf-actions
    :doc "Keymap for emacsconf-related things"
    "a" #'emacsconf-announce
    "c" #'emacsconf-find-captions-from-slug
    "d" #'emacsconf-find-caption-directives-from-slug
    "p" #'emacsconf-set-property-from-slug
    "w" #'emacsconf-edit-wiki-page
    "s" #'emacsconf-set-start-time-for-slug
    "W" #'emacsconf-browse-wiki-page
    "u" #'emacsconf-update-talk
    "t" #'emacsconf-insert-talk-title
    "m" #'emacsconf-mail-speaker-from-slug
    "n" #'emacsconf-notmuch-search-mail-from-entry
    "f" #'org-forward-heading-same-level
    "b" #'org-backward-heading-same-level
    "RET" #'emacsconf-go-to-talk)
  (add-to-list 'embark-keymap-alist '(emacsconf . embark-emacsconf-actions)))

;;; Status updates

For example, I sometimes need to open the wiki page for a talk in order to update the talk description.

emacsconf-edit-wiki-page: Open the wiki page for the talk matching SEARCH.
;;; Embark
(defun emacsconf-embark-finder ()
  "Identify when we're on a talk subtree."
  (when (and (derived-mode-p 'org-mode)
             (org-entry-get-with-inheritance "SLUG"))
    (cons 'emacsconf (org-entry-get-with-inheritance "SLUG"))))

Embark can also act on completion candidates, so I can call any of those actions from my C-c e t shortcut for emacsconf-go-to-talk. This is specified by the (metadata (category . emacsconf)) in emacsconf-complete-talk and the (add-to-list 'embark-keymap-alist '(emacsconf . embark-emacsconf-actions)) in my Embark configuration.

C-. is the embark-act shortcut in my configuration. When I need to remember what the shortcuts are, I can use C-h (embark-keymap-help) to list the keyboard shortcuts or select the command with completion.

output-2023-09-10-20:45:31.gif
Figure 2: Embark help for Emacsconf talks

The code above and related functions are in emacsconf.el or other files in the emacsconf-el repository.

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

EmacsConf backstage: scheduling with SVGs

| emacs, emacsconf

The EmacsConf 2023 call for participation deadline is coming up next Friday (Sept 15), so I'm getting ready to draft the schedule.

Here's a quick overview of how I experiment with schedules for EmacsConf. I have all the talk details in Org subtrees. There's a SLUG property that has the talk ID, a TIME property that says how long a talk is, and a Q_AND_A property that says what kind of Q&A the speaker wants: live, IRC, Etherpad, or after the event. Some talks have fixed starting times, like the opening remarks. Others start when the Q&A for the previous session ends. I generate an SVG so that I can quickly see how the schedule looks. Are there big gaps? Did I follow everyone's availability constraints? Does the schedule flow reasonably logically?

Code to define a schedule
(require 'emacsconf-schedule)
(setq emacsconf-schedule-tracks
      '((:label "Saturday"
                :start "2023-12-02 9:00"
                :end "2023-12-02 18:00"
                :tracks ("General"))
        (:label "Sunday"
                :start "2023-12-03 9:00"
                :end "2023-12-03 18:00"
                :tracks ("General"))))
(setq emacsconf-schedule-default-buffer-minutes 10
      emacsconf-schedule-default-buffer-minutes-for-live-q-and-a 30
      emacsconf-schedule-break-time 10
      emacsconf-schedule-lunch-time 60
      emacsconf-schedule-max-time 30
      emacsconf-schedule-strategies
      '(emacsconf-schedule-allocate-buffer-time
        emacsconf-schedule-set-all-tracks-to-general))
(setq emacsconf-schedule-plan
      '(("GEN Saturday, Dec 2" :start "2023-12-02 09:00")
        sat-open
        adventure writing one uni lunch hyperdrive ref mentor flat        (hn :start "15:30")
        (web :start "16:00") 
        ("GEN Sunday, Dec 3" :start "2023-12-03 09:00")
        sun-open
        windows extending lunch lspocaml sharing emacsen
        voice llm))
(setq emacsconf-schedule-draft (emacsconf-schedule-prepare (emacsconf-schedule-inflate-sexp emacsconf-schedule-plan)))
(emacsconf-schedule-validate emacsconf-schedule-draft)
(let ((emacsconf-schedule-svg-modify-functions
       '(emacsconf-schedule-svg-color-by-status)))
  (with-temp-file "schedule.svg"
    (svg-print (emacsconf-schedule-svg 800 200 emacsconf-schedule-draft))))
(clear-image-cache)
schedule.svg
Figure 1: Sample schedule draft

Common functions are in emacsconf-schedule.el in the emacsconf-el repository, while conference-specific code is in our private conf.org file.

Motivation

I was working on the schedule for EmacsConf 2022 when I ran into a problem. I wanted more talks than could fit into the time that we had, even if I scheduled it as tightly as possible with just a few minutes of transition between talks. This had been the scheduling strategy we used in previous years, squishing all the talks together with just enough time to let people know where they could go to ask questions. We had experimented with an alternate track for Q&A during EmacsConf 2021, so that had given us a little more space for discussion, but it wasn't going to be enough to fit in all the talks we wanted for EmacsConf 2022.

Could I convince the other organizers to take on the extra work needed for a multi-track conference? I knew it would be a lot of extra technical risk, and we'd need another host and streamer too. Could I make the schedule easy to understand on the wiki if we had two tracks? The Org Mode table I'd been using for previous years was not going to be enough. I was getting lost in the text.

2023-09-10-11-41-42.svg
Figure 2: Scheduling using Org Mode tables

Visualizing the schedule as an SVG

I'd been playing around with SVGs in Emacs. It was pretty straightforward to write a function that used time for the X axis and displayed different tracks. That showed us how full the conference was going to be if I tried to pack everything into two days.

schedule-option-one-full-day.svg
Figure 3: One full day

emacsconf-schedule-svg-track: Draw the actual rectangles and text for the talks.
(defun emacsconf-schedule-svg-track (svg base-x base-y width height start-time end-time info)
  "Draw the actual rectangles and text for the talks."
  (let ((scale (/ width (float-time (time-subtract end-time start-time)))))
    (mapc
     (lambda (o)
       (let* ((offset (floor (* scale (float-time (time-subtract (plist-get o :start-time) start-time)))))
              (size (floor (* scale (float-time (time-subtract (plist-get o :end-time) (plist-get o :start-time))))))
              (x (+ base-x offset))
              (y base-y)
              (node (dom-node
                     'rect
                     (list
                      (cons 'x x)
                      (cons 'y y)
                      (cons 'opacity "0.8")
                      (cons 'width size)
                      (cons 'height (1- height))
                      (cons 'stroke "black")
                      (cons 'stroke-dasharray
                            (if (string-match "live" (or (plist-get o :q-and-a) "live"))
                                ""
                              "5,5,5"))
                      (cons 'fill
                            (cond
                             ((string-match "BREAK\\|LUNCH" (plist-get o :title)) "white")
                             ((plist-get o :invalid) "red")
                             ((string-match "EST"
                                            (or (plist-get o :availability) ""))
                              "lightgray")
                             (t "lightgreen"))))))
              (parent (dom-node
                       'a
                       (list
                        (cons 'href
                              (concat
                               (if emacsconf-use-absolute-url
                                   emacsconf-base-url
                                 "/")
                               (plist-get o :url)))
                        (cons 'title (plist-get o :title))
                        (cons 'data-slug (plist-get o :slug)))
                       (dom-node 'title nil
                                 (concat (format-time-string "%l:%M-" (plist-get o :start-time) emacsconf-timezone)
                                         (format-time-string "%l:%M " (plist-get o :end-time) emacsconf-timezone)
                                         (plist-get o :title)))
                       node
                       (dom-node
                        'g
                        `((transform . ,(format "translate(%d,%d)"
                                                (+ x size -2) (+ y height -2))))
                        (dom-node
                         'text
                         (list
                          (cons 'fill "black")
                          (cons 'x 0)
                          (cons 'y 0)
                          (cons 'font-size 10)
                          (cons 'transform "rotate(-90)"))
                         (svg--encode-text (or (plist-get o :slug) (plist-get o :title))))))))
         (run-hook-with-args
          'emacsconf-schedule-svg-modify-functions
          o node parent)
         (dom-append-child
          svg
          parent)))
     info)))

emacsconf-schedule-svg-day: Add the time scale and the talks on a given day.
(defun emacsconf-schedule-svg-day (elem label width height start end tracks)
  "Add the time scale and the talks on a given day."
  (let* ((label-margin 15)
         (track-height (/ (- height (* 2 label-margin)) (length tracks)))
         (x 0) (y label-margin)
         (scale (/ width (float-time (time-subtract end start))))
         (time start))
    (dom-append-child elem (dom-node 'title nil (concat "Schedule for " label)))
    (svg-rectangle elem 0 0 width height :fill "white")
    (svg-text elem label :x 3 :y (- label-margin 3) :fill "black" :font-size "10")
    (mapc (lambda (track)
            (emacsconf-schedule-svg-track
             elem x y width track-height
             start end track)
            (setq y (+ y track-height)))
          tracks)
    ;; draw grid
    (while (time-less-p time end)
      (let ((x (* (float-time (time-subtract time start)) scale)))
        (dom-append-child
         elem
         (dom-node
          'g
          `((transform . ,(format "translate(%d,%d)" x label-margin)))
          (dom-node
           'line
           `((stroke . "darkgray")
             (x1 . 0)
             (y1 . 0)
             (x2 . 0)
             (y2 . ,(- height label-margin label-margin))))
          (dom-node
           'text
           `((fill . "black")
             (x . 0)
             (y . ,(- height 2 label-margin))
             (font-size . 10)
             (text-anchor . "left"))
           (svg--encode-text (format-time-string "%-l %p" time emacsconf-timezone)))))
        (setq time (time-add time (seconds-to-time 3600)))))
    elem))

emacsconf-schedule-svg-days: Display multiple days.
(defun emacsconf-schedule-svg-days (width height days)
  "Display multiple days."
  (let ((svg (svg-create width height))
        (day-height (/ height (length days)))
        (y 0))
    (dom-append-child svg (dom-node 'title nil "Graphical view of the schedule"))
    (mapc
     (lambda (day)
       (let ((group (dom-node 'g `((transform . ,(format "translate(0,%d)" y))))))
         (dom-append-child svg group)
         (emacsconf-schedule-svg-day group
                   (plist-get day :label)
                   width day-height
                   (date-to-time (plist-get day :start))
                   (date-to-time (plist-get day :end))
                   (plist-get day :tracks)))
       (setq y (+ y day-height)))
     days)
    svg))

emacsconf-schedule-svg: Make the schedule SVG for INFO.
(defun emacsconf-schedule-svg (width height &optional info)
  "Make the schedule SVG for INFO."
  (setq info (emacsconf-prepare-for-display (or info (emacsconf-get-talk-info))))
  (let ((days (seq-group-by (lambda (o)
                              (format-time-string "%Y-%m-%d" (plist-get o :start-time) emacsconf-timezone))
                            (sort (seq-filter (lambda (o)
                                                (or (plist-get o :slug)
                                                    (plist-get o :include-in-info)))
                                              info)
                                  #'emacsconf-sort-by-scheduled))))
    (emacsconf-schedule-svg-days
     width height
     (mapcar (lambda (o)
               (let ((start (concat (car o) "T" emacsconf-schedule-start-time emacsconf-timezone-offset))
                     (end (concat (car o) "T" emacsconf-schedule-end-time emacsconf-timezone-offset)))
                 (list :label (format-time-string "%A" (date-to-time (car o)))
                       :start start
                       :end end
                       :tracks (emacsconf-by-track (cdr o)))))
             days))))

With that, I was able to show the other organizers what a two-track conference could look like.

schedule-option-gen-and-dev.svg
Figure 4: Two-track conference
Defining a schedule with two tracks
(emacsconf-schedule-test
 filename
 (emacsconf-time-constraints '(("LUNCH" "11:30" "13:30")))
 (emacsconf-schedule-default-buffer-minutes 15)
 (emacsconf-schedule-default-buffer-minutes-for-live-q-and-a 25)
 (arranged
  (emacsconf-schedule-inflate-sexp
   '(("GEN Saturday, December 3" . "2022-12-03 09:00")
     "Saturday opening remarks"
     survey orgyear rolodex
     break
     links buttons 
     lunch
     hyperorg        realestate    health 
     break
     jupyter workflows
     
     ("Saturday closing remarks" . "2022-12-03 17:00")
     ("GEN Sunday, December 4" . "2022-12-04 09:00")
     "Sunday opening remarks"
     journalism
     handwritten
     break
               school  science   lunch
               meetups buddy
               community 
      orgvm indieweb  fanfare
     ("Sunday closing remarks" . "2022-12-04 17:00")
     ("DEV Saturday, December 3" . "2022-12-03 10:00")
     localizing  treesitter lspbridge
     lunch sqlite
     
mail     eev python break wayland (haskell . "2022-12-03 16:00")
     ("DEV Sunday, December 4" . "2022-12-04 10:00")
     justl eshell 
     detached rde 
     lunch
                tramp async
     break
     asmblox dbus maint           )) )
 (emacsconf-schedule-break-time 10)
 (emacsconf-schedule-lunch-time 60)
 (emacsconf-schedule-max-time 30)
 (emacsconf-schedule-tweaked-allocations '(("indieweb" . 20)
                                           ("maint" . 20)
                                           ("workflows" . 20)))
 (emacsconf-scheduling-strategies '(emacsconf-schedule-override-breaks
                                    emacsconf-schedule-allocate-buffer-time
                                    ))
 (tracks '((:label "Saturday"
                   :start "2022-12-03 9:00"
                   :end "2022-12-03 18:00"
                   :tracks (("^GEN Sat" "^GEN Sun")
                            ("^DEV Sat" "^DEV Sun")))
          (:label "Sunday"
                  :start "2022-12-04 9:00"
                  :end "2022-12-04 18:00"
                  :tracks (("^GEN Sun" "^DEV Sat")
                           ("^DEV Sun"))))))

When the other organizers saw the two schedules, they were strongly in favour of the two-track option. Yay!

Changing scheduling strategies

As I played around with the schedule, I wanted a quick way to test different scheduling strategies, such as changing the length of Q&A sessions for live web conferences versus IRC/Etherpad/email Q&A. Putting those into variables allowed me to easily override them with a let form, and I used a list of functions to calculate or modify the schedule from the Org trees.

output-2023-09-10-12:20:14.gif
Figure 5: Changing the default Q&A times

emacsconf-schedule-allocate-buffer-time: Allocate buffer time based on whether INFO has live Q&A.
(defun emacsconf-schedule-allocate-buffer-time (info)
  "Allocate buffer time based on whether INFO has live Q&A.
Uses `emacsconf-schedule-default-buffer-minutes' and
`emacsconf-schedule-default-buffer-minutes-for-live-q-and-a'."
  (mapcar (lambda (o)
            (when (plist-get o :slug)
              (unless (plist-get o :buffer)
                (plist-put o :buffer
                           (number-to-string 
                            (if (string-match "live" (or (plist-get o :q-and-a) "live"))
                                emacsconf-schedule-default-buffer-minutes-for-live-q-and-a
                              emacsconf-schedule-default-buffer-minutes)))))
            o)
          info))

This is actually applied by emacsconf-schedule-prepare, which runs through the list of strategies defined in emacsconf-schedule-strategies.

emacsconf-schedule-prepare: Apply ‘emacsconf-schedule-strategies’ to INFO to determine the schedule.
(defun emacsconf-schedule-prepare (&optional info)
  "Apply `emacsconf-schedule-strategies' to INFO to determine the schedule."
  (emacsconf-schedule-based-on-info
   (seq-reduce (lambda (prev val) (funcall val prev))
               emacsconf-schedule-strategies
               (or info (emacsconf-get-talk-info)))))

Type of Q&A

I wanted to see which sessions had live Q&A via web conference and which ones had IRC/Etherpad/asynchronous Q&A. I set the talk outlines so that dashed lines mean asynchronous Q&A and solid lines mean live. Live Q&As take a little more work on our end because the host starts it up and reads questions, but they're more interactive. This is handled by 

(cons 'stroke-dasharray
      (if (string-match "live" (or (plist-get o :q-and-a) "live"))
          ""
        "5,5,5"))

in the emacsconf-schedule-svg-track function.

Moving talks around

I wanted to be able to quickly reorganize talks by moving their IDs around in a list instead of just using the order of the subtrees in my Org Mode file. I wrote a function that took a list of symbols, looked up each of the talks, and returned a list of talk info property lists. I also added the ability to override some things about talks, such as whether something started at a fixed time.

emacsconf-schedule-inflate-sexp: Takes a list of talk IDs and returns a list that includes the scheduling info.
(defun emacsconf-schedule-inflate-sexp (sequence &optional info include-time)
  "Takes a list of talk IDs and returns a list that includes the scheduling info.
Pairs with `emacsconf-schedule-dump-sexp'."
  (setq info (or info (emacsconf-get-talk-info)))
  (let ((by-assoc (mapcar (lambda (o) (cons (intern (plist-get o :slug)) o))
                          (emacsconf-filter-talks info)))
        date)
    (mapcar
     (lambda (seq)
       (unless (listp seq) (setq seq (list seq)))
       
       (if include-time
           (error "Not yet implemented")
         (let ((start-prop (or (plist-get (cdr seq) :start)
                               (and (stringp (cdr seq)) (cdr seq))))
               (time-prop (or (plist-get (cdr seq) :time) ; this is duration in minutes
                              (and (numberp (cdr seq)) (cdr seq))))
               (track-prop (plist-get (cdr seq) :track)))
           (append
            ;; overriding 
            (when start-prop
              (if (string-match "-" start-prop)
                  (setq date (format-time-string "%Y-%m-%d" (date-to-time start-prop)))
                (setq start-prop  (concat date " " start-prop)))
              (list
               :scheduled (format-time-string (cdr org-time-stamp-formats) (date-to-time start-prop)
                                              emacsconf-timezone)
               :start-time (date-to-time start-prop)
               :fixed-time t))
            (when track-prop
              (list :track track-prop))
            (when time-prop
              (list :time (if (numberp time-prop) (number-to-string time-prop) time-prop)))
            ;; base entity
            (cond
             ((eq (car seq) 'lunch)
              (list :title "LUNCH" :time (number-to-string emacsconf-schedule-lunch-time)))
             ((eq (car seq) 'break)
              (list :title "BREAK" :time (number-to-string emacsconf-schedule-break-time)))
             ((symbolp (car seq))
              (assoc-default (car seq) by-assoc))
             ((stringp (car seq))
              (or (seq-find (lambda (o) (string= (plist-get o :title) (car seq))) info)
                  (list :title (car seq))))
             (t (error "Unknown %s" (prin1-to-string seq))))))))
     sequence)))

That allowed me to specify a test schedule like this: 

(setq emacsconf-schedule-plan
      '(("GEN Saturday, Dec 2" :start "2023-12-02 09:00")
        sat-open
        adventure writing one uni lunch hyperdrive ref mentor flat
        (hn :start "15:30")
        (web :start "16:00") 
        ("GEN Sunday, Dec 3" :start "2023-12-03 09:00")
        sun-open
        windows extending lunch lspocaml sharing emacsen
        voice llm))
(setq emacsconf-schedule-draft
      (emacsconf-schedule-prepare
       (emacsconf-schedule-inflate-sexp emacsconf-schedule-plan)))

Validating the schedule

Now that I could see the schedule visually, it made sense to add some validation. As mentioned in my post about timezones, I wanted to validate live Q&A against the speaker's availability and colour sessions red if they were outside the times I'd written down.

I also wanted to arrange the schedule so that live Q&A sessions didn't start at the same time, giving me a little time to switch between sessions in case I needed to help out. I wrote a function that checked if the Q&A for a session started within five minutes of the previous one. This turned out to be pretty useful, since I ended up mostly taking care of playing the videos and opening the browsers for both streams. With that in place, I could just move the talks around until everything fit well together.

emacsconf-schedule-validate-live-q-and-a-sessions-are-staggered: Try to avoid overlapping the start of live Q&A sessions.
(defun emacsconf-schedule-validate-live-q-and-a-sessions-are-staggered (schedule)
  "Try to avoid overlapping the start of live Q&A sessions.
Return nil if there are no errors."
  (when emacsconf-schedule-validate-live-q-and-a-sessions-buffer
    (let (last-end)
      (delq nil
            (mapcar (lambda (o)
                      (prog1
                          (when (and last-end
                                     (time-less-p
                                      (plist-get o :end-time)
                                      (time-add last-end (seconds-to-time (* emacsconf-schedule-validate-live-q-and-a-sessions-buffer 60)))))
                            (plist-put o :invalid (format "%s live Q&A starts at %s within %d minutes of previous live Q&A at %s"
                                                          (plist-get o :slug)
                                                          (format-time-string "%m-%d %-l:%M"
                                                                              (plist-get o :end-time))
                                                          emacsconf-schedule-validate-live-q-and-a-sessions-buffer
                                                          (format-time-string "%m-%d %-l:%M"
                                                                              last-end)))
                            (plist-get o :invalid))
                        (setq last-end (plist-get o :end-time))))
                    (sort 
                     (seq-filter (lambda (o) (string-match "live" (or (plist-get o :q-and-a) "")))
                                 schedule)
                     (lambda (a b)
                       (time-less-p (plist-get a :end-time) (plist-get b :end-time)))
                     ))))))

Publishing SVGs in the wiki

When the schedule settled down, it made perfect sense to include the image on the schedule page as well as on each talk page. I wanted people to be able to click on a rectangle and load the talk page. That meant including the SVG in the markup and allowing the attributes in ikiwiki's HTML sanitizer. In our htmlscrubber.pm, I needed to add svg rect text g title line to allow, and add version xmlns x y x1 y1 x2 y2 fill font-size font-weight stroke stroke-width stroke-dasharray transform opacity to the default attributes.

opening-talk.gif
Figure 6: Opening a talk from the SVG

For the public-facing pages, I wanted to colour the talks based on track. I specified the colours in emacsconf-tracks (peachpuff and skyblue) and used them in emacsconf-schedule-svg-color-by-track.

emacsconf-schedule-svg-color-by-track: Color sessions based on track.
(defun emacsconf-schedule-svg-color-by-track (o node &optional parent)
  "Color sessions based on track."
  (let ((track (emacsconf-get-track (plist-get o :track))))
    (when track
      (dom-set-attribute node 'fill (plist-get track :color)))))

I wanted talk pages to highlight the talk on the schedule so that people could easily find other sessions that conflict. Because a number of people in the Emacs community browse with Javascript turned off, I used Emacs Lisp to generate a copy of the SVG with the current talk highlighted.

emacsconf-publish-format-talk-page-schedule: Add the schedule image for TALK based on INFO.
(defun emacsconf-publish-format-talk-page-schedule (talk info)
  "Add the schedule image for TALK based on INFO."
  (concat
   "\nThe following image shows where the talk is in the schedule for "
   (format-time-string "%a %Y-%m-%d" (plist-get talk :start-time) emacsconf-timezone) ". Solid lines show talks with Q&A via BigBlueButton. Dashed lines show talks with Q&A via IRC or Etherpad."
   (format "<div class=\"schedule-in-context schedule-svg-container\" data-slug=\"%s\">\n" (plist-get talk :slug))           
   (let* ((width 800) (height 150)
          (talk-date (format-time-string "%Y-%m-%d" (plist-get talk :start-time) emacsconf-timezone))
          (start (date-to-time (concat talk-date "T" emacsconf-schedule-start-time emacsconf-timezone-offset)))
          (end (date-to-time (concat talk-date "T" emacsconf-schedule-end-time emacsconf-timezone-offset)))
          svg)
     (with-temp-buffer
       (setq svg (emacsconf-schedule-svg-day
                  (svg-create width height)
                  (format-time-string "%A" (plist-get talk :start-time) emacsconf-timezone)
                  width height
                  start end
                  (emacsconf-by-track
                   (seq-filter (lambda (o) (string= (format-time-string "%Y-%m-%d" (plist-get o :start-time) emacsconf-timezone)
                                                    talk-date))
                               info))))
       (mapc (lambda (node)
               (let ((rect (car (dom-by-tag node 'rect))))
                 (if (string= (dom-attr node 'data-slug) (plist-get talk :slug))
                     (progn
                       (dom-set-attribute rect 'opacity "0.8")
                       (dom-set-attribute rect 'stroke-width "3")
                       (dom-set-attribute (car (dom-by-tag node 'text)) 'font-weight "bold"))
                   (dom-set-attribute rect 'opacity "0.5"))))
             (dom-by-tag svg 'a))
       (svg-print svg)
       (buffer-string)))
   "\n</div>\n"
   "\n"
   (emacsconf-format-talk-schedule-info talk) "\n\n"))

Colouring talks based on status

As the conference approached, I wanted to colour talks based on their status, so I could see how much of our schedule already had videos and even how many had been captioned. That was just a matter of adding a modifier function to change the SVG rectangle colour depending on the TODO status.

output-2023-09-10-13:20:12.gif
Figure 7: Changing talk colour based on status

emacsconf-schedule-svg-color-by-status: Set talk color based on status.
(defun emacsconf-schedule-svg-color-by-status (o node &optional _)
  "Set talk color based on status.
Processing: palegoldenrod,
Waiting to be assigned a captioner: yellow,
Captioning in progress: lightgreen,
Ready to stream: green,
Other status: gray"
  (unless (plist-get o :invalid)
    (dom-set-attribute node 'fill
                       (pcase (plist-get o :status)
                         ((rx (or "TO_PROCESS"
                                  "PROCESSING"
                                  "TO_AUTOCAP"))
                          "palegoldenrod")
                         ((rx (or "TO_ASSIGN"))
                          "yellow")
                         ((rx (or "TO_CAPTION"))
                          "lightgreen")
                         ((rx (or "TO_STREAM"))
                          "green")
                         (_ "gray")))))

Summary

Common functions are in emacsconf-schedule.el in the emacsconf-el repository, while conference-specific code is in our private conf.org file.

Some information is much easier to work with graphically than in plain text. Seeing the schedules as images made it easier to spot gaps or errors, tinker with parameters, and communicate with other people. Writing Emacs Lisp functions to modify my data made it easier to try out different things without rearranging the text in my Org file. Because Emacs can display images inside the editor, it was easy for me to make changes and see the images update right away. Using SVG also made it possible to export the image and make it interactive. Handy technique!

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

EmacsConf backstage: converting timezones

| emacsconf, emacs
  • [2023-10-12 Thu]: Update screenshots to use the overlay talk
  • [2023-10-10 Tue]: Updated translation schedule to use emacsconf-mail-format-talk-schedule.
  • 2023-09-07: It looks like I can use Etc/GMT-2 to mean GMT+2 - note the reversed sign.

EmacsConf is a virtual conference with speakers from all over the world. We like to plan the schedule so that the speakers can come for live Q&A sessions without having to wake up too early or stay up too late.

Timezones are tricky for me. Sometimes I mess up timezone names (like the time I misspelled Tbilisi and ended up with UTC conversion) or get the timezone conversion wrong because of daylight savings time, and it's annoying to go to a website to convert the timezones.

Fortunately, the tzc package provides a way to convert times from one timeone to another in Emacs, and it includes a list of timezones in tzc-time-zones loaded from /usr/share/zoneinfo. Here's how I use it to make organizing EmacsConf easier.

Setting the timeone with completion

To reduce data entry errors, I use completion when setting the timezone.

output-2023-10-12-11:31:56.gif
Figure 1: Setting the timezone

emacsconf-timezone-set: Set the timezone for the current Org entry.
(defun emacsconf-timezone-set (timezone)
  "Set the timezone for the current Org entry."
  (interactive
   (list
    (progn
      (require 'tzc)
      (completing-read "Timezone: " tzc-time-zones))))
  (org-entry-put (point) "TIMEZONE" timezone))

Sometimes speakers specify their timezone as an offset from GMT or UTC, such as GMT+2. It turns out that I can use timezones like Etc/GMT-2 to capture that, although it's important to note that the sign for Etc/GMT timezones is reversed (so Etc/GMT-2 = GMT+2).

Converting timezones

In Toronto, we switch from daylight savings time to standard time sometime in November, so I need to make sure that my time conversions for speaker availability uses the date of the conference (emacsconf-date, 2023-12-02 this year). emacsconf-convert-from-timezone makes it easy to convert times on emacsconf-date so that I don't have to keep re-entering the date part.

output-2023-10-12-21:32:34.gif
Figure 2: Converting from a timezone

emacsconf-convert-from-timezone
(defun emacsconf-convert-from-timezone (timezone time)
  (interactive (list (progn
                       (require 'tzc)
                       (if (and (derived-mode-p 'org-mode)
                                (org-entry-get (point) "TIMEZONE"))
                           (completing-read (format "From zone (%s): "
                                                    (org-entry-get (point) "TIMEZONE"))
                                            tzc-time-zones nil nil nil nil
                                            (org-entry-get (point) "TIMEZONE"))
                         (completing-read "From zone: " tzc-time-zones nil t)))
                     (read-string "Time: ")))
  (let* ((from-offset (format-time-string "%z" (date-to-time emacsconf-date) timezone))
         (time
          (date-to-time
           (concat emacsconf-date "T" (string-pad time 5 ?0 t)  ":00.000"
                   from-offset))))
    (message "%s = %s"
             (format-time-string
              "%b %d %H:%M %z"
              time
              timezone)
             (format-time-string
              "%b %d %H:%M %z"
              time
              emacsconf-timezone))))

I can use this to convert times like 8:00 in US/Pacific to 11:00 EST.

Validating schedule constraints

Once I get the availability into a standard format, I can use that to validate that sessions are scheduled during the times that speakers have indicated that they're available. So far, I've been using text like >= 10:00 EST at the beginning of the talk's AVAILABILITY property, since that's easy to parse and validate. I can use that to colour invalid talks red in an SVG, and I can make a list of invalid talks as well.

output-2023-09-06-10:58:41.gif
Figure 3: Validating time constraints in a draft schedule

How does that work? First, we get the time constraint out of the AVAILABILITY property with emacsconf-schedule-get-time-constraint.

emacsconf-schedule-get-time-constraint
(defun emacsconf-schedule-get-time-constraint (o)
  (when (emacsconf-schedule-q-and-a-p o)
    (let ((avail (or (plist-get o :availability) ""))
          hours
          start
          (pos 0)
          (result (list nil nil nil)))
      (while (string-match "\\([<>]\\)=? *\\([0-9]+:[0-9]+\\) *EST" avail pos)
        (setf (elt result (if (string= (match-string 1 avail) ">")
                              0
                            1))
              (match-string 2 avail))
          (setq pos (match-end 0)))
      (when (string-match "[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]" avail)
        (setf (elt result 2) (match-string 0 avail)))
      result)))

Then we can return a warning if a talk is scheduled outside those time constraints.

emacsconf-schedule-check-time: FROM-TIME and TO-TIME should be nil strings like HH:MM in EST.
(defun emacsconf-schedule-check-time (label o &optional from-time to-time day)
  "FROM-TIME and TO-TIME should be nil strings like HH:MM in EST.
DAY should be YYYY-MM-DD if specified.
Both start and end time are tested."
  (let* ((start-time (format-time-string "%H:%M" (plist-get o :start-time)))
         (end-time (format-time-string "%H:%M" (plist-get o :end-time)))
         result)
    (setq result
          (or
           (and (null o) (format "%s: Not found" label))
           (and from-time (string< start-time from-time)
                (format "%s: Starts at %s before %s" label start-time from-time))
           (and to-time (string< to-time end-time)
                (format "%s: Ends at %s after %s" label end-time to-time))
           (and day
                (not (string= (format-time-string "%Y-%m-%d" (plist-get o :start-time))
                              day))
                (format "%s: On %s instead of %s"
                        label
                        (format-time-string "%Y-%m-%d" (plist-get o :start-time))
                        day))))
    (when result (plist-put o :invalid result))
    result))

So then we can check all the talks as scheduled, and set the :invalid property if it's outside the availability constraints.

emacsconf-schedule-validate-time-constraints
(defun emacsconf-schedule-validate-time-constraints (info &rest _)
  (interactive)
  (let* ((info (or info (emacsconf-get-talk-info)))
         (results (delq nil
                        (append
                         (mapcar
                          (lambda (o)
                            (apply #'emacsconf-schedule-check-time
                                   (car o)
                                   (emacsconf-search-talk-info (car o) info)
                                   (cdr o)))
                          emacsconf-time-constraints)
                         (mapcar
                          (lambda (o)
                            (let (result
                                  (constraint (emacsconf-schedule-get-time-constraint o)))
                              (when constraint
                                (setq result (apply #'emacsconf-schedule-check-time
                                                    (plist-get o :slug)
                                                    o
                                                    constraint))
                                (when result (plist-put o :invalid result))
                                result)))
                          info)))))
    (if (called-interactively-p 'any)
        (message "%s" (string-join results "\n"))
      results)))

Here are more details on how I made the schedule SVG. It's handy to have a quick way to check availability in both text and graphical format.

Translating schedules into local times

When we e-mail speakers their schedules, we also include a translation to their local time if we know it.

2023-10-10-16-19-05.svg
Figure 4: Sample e-mail for schedule feedback

That's handled by the emacsconf-mail-format-talk-schedule, which handles three cases:

  • timezone is the same as the conference: show just that time
  • UTC offset is the same as the conferenc, just a different timezone: mention that
  • UTC offset is different: translate to local time and make it clear that this is a translation, not a second event

(If we haven't noted the timezone for the talk, we ask the speaker.)

emacsconf-mail-format-talk-schedule: Format the schedule for O for inclusion in mail messages etc.
(defun emacsconf-mail-format-talk-schedule (o)
  "Format the schedule for O for inclusion in mail messages etc."
  (interactive (list (emacsconf-complete-talk)))
  (when (stringp o)
    (setq o
          (emacsconf-resolve-talk
           (emacsconf-get-slug-from-string o)
           (or emacsconf-schedule-draft (emacsconf-get-talk-info)))))
  (let ((result
         (concat
          (plist-get o :title) "\n"
          (format-time-string "%b %-e %a %-I:%M %#p %Z" (plist-get o :start-time) emacsconf-timezone) "\n"
          (if (and (plist-get o :timezone) (not (string= (plist-get o :timezone) emacsconf-timezone)))
              (if (string= (format-time-string "%z" (plist-get o :start-time) (plist-get o :timezone))
                           (format-time-string "%z" (plist-get o :start-time) emacsconf-timezone))
                  (format "which is the same time in your local timezone %s\n"
                          (emacsconf-schedule-rename-etc-timezone (plist-get o :timezone)))
                (format "translated to your local timezone %s: %s\n"
                        (emacsconf-schedule-rename-etc-timezone (plist-get o :timezone))
                        (format-time-string "%b %-e %a %-I:%M %#p %Z" (plist-get o :start-time) (plist-get o :timezone))))
            ""))))
    (when (called-interactively-p 'any)
      (insert result))
    result))

The Etc/GMT... timezones are a little confusing, because the signs are opposite from what you'd expect (GMT-3 = UTC+0300). So we have a little function that turns those into regular UTC offsets.

emacsconf-schedule-rename-etc-timezone: Change Etc/GMT-3 etc. to UTC+3 etc., since Etc uses negative signs and this is confusing.
(defun emacsconf-schedule-rename-etc-timezone (s)
  "Change Etc/GMT-3 etc. to UTC+3 etc., since Etc uses negative signs and this is confusing."
  (cond ((string-match "Etc/GMT-\\(.*\\)" s) (concat "UTC+" (match-string 1 s)))
        ((string-match "Etc/GMT\\+\\(.*\\)" s) (concat "UTC-" (match-string 1 s)))
        (t s)))

So that's how we work with timezones in EmacsConf!

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

EmacsConf backstage: capturing submissions from e-mails

| emacsconf, emacs, org

2023-09-11: Updated code for recognizing fields.

People submit proposals for EmacsConf sessions via e-mail following this submission template. (You can still submit a proposal until Sept 14!) I mostly handle acceptance and scheduling, so I copy this information into our private conf.org file so that we can use it to plan the draft schedule, mail-merge speakers, and so on. I used to do this manually, but I'm experimenting with using functions to create the heading automatically so that it includes the date, talk title, and e-mail address from the e-mail, and it calculates the notification date for early acceptances as well. I use Notmuch for e-mail, so I can get the properties from (notmuch-show-get-message-properties).

2023-09-05_13-09-57.png
Figure 1: E-mail submission

emacsconf-mail-add-submission: Add the submission from the current e-mail.
(defun emacsconf-mail-add-submission (slug)
  "Add the submission from the current e-mail."
  (interactive "MTalk ID: ")
  (let* ((props (notmuch-show-get-message-properties))
         (from (or (plist-get (plist-get props :headers) :Reply-To)
                   (plist-get (plist-get props :headers) :From)))
         (body (plist-get
                (car
                 (plist-get props :body))
                :content))
         (date (format-time-string "%Y-%m-%d"
                                   (date-to-time (plist-get (plist-get props :headers) :Date))))
         (to-notify (format-time-string
                     "%Y-%m-%d"
                     (time-add
                      (days-to-time emacsconf-review-days)
                      (date-to-time (plist-get (plist-get props :headers) :Date)))))
         (data (emacsconf-mail-parse-submission body)))
    (when (string-match "<\\(.*\\)>" from)
      (setq from (match-string 1 from)))
    (with-current-buffer
        (find-file emacsconf-org-file)
      ;;  go to the submissions entry
      (goto-char (org-find-property "CUSTOM_ID" "submissions"))
      (when (org-find-property "CUSTOM_ID" slug)
        (error "Duplicate talk ID")))
    (find-file emacsconf-org-file)
    (delete-other-windows)
    (outline-next-heading)
    (org-insert-heading)
    (insert " " (or (plist-get data :title) "") "\n")
    (org-todo "TO_REVIEW")
    (org-entry-put (point) "CUSTOM_ID" slug)
    (org-entry-put (point) "SLUG" slug)
    (org-entry-put (point) "TRACK" "General")
    (org-entry-put (point) "EMAIL" from)
    (org-entry-put (point) "DATE_SUBMITTED" date)
    (org-entry-put (point) "DATE_TO_NOTIFY" to-notify)
    (when (plist-get data :time)
      (org-entry-put (point) "TIME" (plist-get data :time)))
    (when (plist-get data :availability)
      (org-entry-put (point) "AVAILABILITY"
                     (replace-regexp-in-string "\n+" " "
                                               (plist-get data :availability))))
    (when (plist-get data :public)
      (org-entry-put (point) "PUBLIC_CONTACT"
                     (replace-regexp-in-string "\n+" " "
                                               (plist-get data :public))))
    (when (plist-get data :private)
      (org-entry-put (point) "EMERGENCY"
                     (replace-regexp-in-string "\n+" " "
                                               (plist-get data :private))))
    (when (plist-get data :q-and-a)
      (org-entry-put (point) "Q_AND_A"
                     (replace-regexp-in-string "\n+" " "
                                               (plist-get data :q-and-a))))
    (save-excursion
      (insert (plist-get data :body)))
    (re-search-backward org-drawer-regexp)
    (org-fold-hide-drawer-toggle 'off)
    (org-end-of-meta-data)
    (split-window-below)))

emacsconf-mail-parse-submission: Extract data from EmacsConf 2023 submissions in BODY.
(defun emacsconf-mail-parse-submission (body)
  "Extract data from EmacsConf 2023 submissions in BODY."
  (when (listp body) (setq body (plist-get (car body) :content)))
  (let ((data (list :body body))
        (fields '((:title "^[* ]*Talk title")
                  (:description "^[* ]*Talk description")
                  (:format "^[* ]*Format")
                  (:intro "^[* ]*Introduction for you and your talk")
                  (:name "^[* ]*Speaker name")
                  (:availability "^[* ]*Speaker availability")
                  (:q-and-a "^[* ]*Preferred Q&A approach")
                  (:public "^[* ]*Public contact information")
                  (:private "^[* ]*Private emergency contact information")
                  (:release "^[* ]*Please include this speaker release"))))
    (with-temp-buffer
      (insert body)
      (goto-char (point-min))
      ;; Try to parse it
      (while fields
        ;; skip the field title
        (when (and (or (looking-at (cadar fields))
                       (re-search-forward (cadar fields) nil t))
                   (re-search-forward "\\(:[ \t\n]+\\|\n\n\\)" nil t))
          ;; get the text between this and the next field
          (setq data (plist-put data (caar fields)
                                (buffer-substring (point)
                                                  (or
                                                   (when (and (cdr fields)
                                                              (re-search-forward (cadr (cadr fields)) nil t))
                                                     (goto-char (match-beginning 0))
                                                     (point))
                                                   (point-max))))))
        (setq fields (cdr fields)))
      (if (string-match "[0-9]+" (or (plist-get data :format) ""))
          (plist-put data :time (match-string 0 (or (plist-get data :format) ""))))
      data)))

The functions above are in the emacsconf-el repository. When I call emacsconf-mail-parse-submission and give it the talk ID I want to use, it makes the Org entry.

2023-09-05_13-12-34.png
Figure 2: Creating the entry

We store structured data in Org Mode properties such as NAME, EMAIL and EMERGENCY. I tend to make mistakes when typing, so I have a short function that sets an Org property based on a region. This is the code from my personal config:

my-org-set-property: In the current entry, set PROPERTY to VALUE.
(defun my-org-set-property (property value)
  "In the current entry, set PROPERTY to VALUE.
Use the region if active."
  (interactive (list (org-read-property-name)
                     (when (region-active-p) (replace-regexp-in-string "[ \n\t]+" " " (buffer-substring (point) (mark))))))
  (org-set-property property value))

I've bound it to C-c C-x p. This is what it looks like when I use it:

output-2023-09-05-13:13:30.gif
Figure 3: Setting Org properties from the region

That helps me reduce errors in entering data. I sometimes forget details, so I ask other people to double-check my work, especially when it comes to speaker availability. That's how I copy the submission e-mails into our Org file.

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

EmacsConf backstage: Using TRAMP and timers to run two tracks semi-automatically

| emacs, emacsconf, org

In previous years, organizers streamed the video feeds for EmacsConf from their own computers to the Icecast server, which was a little challenging because of CPU load. A server shared by a volunteer had a 6-core Intel Xeon E5-2420 with 48 GB of RAM, which turned out to be enough horsepower to run OBS for both the general and development track for EmacsConf 2022. One of the advantages of this setup was that I could write some Emacs Lisp to automatically play recorded intros and talk videos at scheduled times, right from the large Org file that had all the conference details. I used SCHEDULED: properties to indicate when talks should play, and that was picked up by another function that took the Org entry properties and put them into a plist.

This function scheduled the timers:

emacsconf-stream-schedule-timers
(defun emacsconf-stream-schedule-timers (&optional info)
  "Schedule PLAYING for the rest of talks and CLOSED_Q for recorded talks."
  (interactive)
  (emacsconf-stream-cancel-all-timers)
  (setq info (emacsconf-prepare-for-display (emacsconf-filter-talks (or info (emacsconf-get-talk-info)))))
  (let ((now (current-time)))
    (mapc (lambda (talk)
            (when (and (time-less-p now (plist-get talk :start-time)))
              (emacsconf-stream-schedule-talk-status-change talk (plist-get talk :start-time) "PLAYING"
                                                            `(:title (concat "Starting " (plist-get talk :slug)))))
            (when (and
                   (plist-get talk :video-file)
                   (plist-get talk :qa-time)
                   (not (string-match "none" (or (plist-get talk :q-and-a) "none")))
                   (null (plist-get talk :stream-files)) ;; can't tell when this is
                   (time-less-p now (plist-get talk :qa-time)))
              (emacsconf-stream-schedule-talk-status-change talk (plist-get talk :qa-time) "CLOSED_Q"
                                                            `(:title (concat "Q&A for " (plist-get talk :slug) " (" (plist-get talk :q-and-a) ")"))))
            )
          info)))

It turns out that TRAMP doesn't like being called from timers if there's a chance that two TRAMP processes might run at the same time. I got "Forbidden reentrant call of Tramp" errors when that happened. There was an easy fix, though. I adjusted the schedules of the talks so that they started at least a minute apart.

Sometimes I wanted to cancel just one timer:

emacsconf-stream-cancel-timer
(defun emacsconf-stream-cancel-timer (id)
  "Cancel a timer by ID."
  (interactive (list
                (completing-read
                 "ID: "
                 (lambda (string pred action)
                    (if (eq action 'metadata)
                        `(metadata (display-sort-function . ,#'identity))
                      (complete-with-action action
                                            (sort
                                             (seq-filter (lambda (o)
                                                           (and (timerp (cdr o))
                                                                (not (timer--triggered (cdr o)))))
                                                         emacsconf-stream-timers)
                                             (lambda (a b) (string< (car a) (car b))))
                                            string pred))))))
  (when (timerp (assoc-default id emacsconf-stream-timers))
    (cancel-timer (assoc-default id emacsconf-stream-timers))
    (setq emacsconf-stream-timers
          (delq (assoc id emacsconf-stream-timers)
                (seq-filter (lambda (o)
                              (and (timerp (cdr o))
                                   (not (timer--triggered (cdr o)))))
                            emacsconf-stream-timers)))))

and schedule just one timer manually:

emacsconf-stream-schedule-talk-status-change
(defun emacsconf-stream-schedule-talk-status-change (talk time new-status &optional notification)
  "Schedule a one-off timer for TALK at TIME to set it to NEW-STATUS."
  (interactive (list (emacsconf-complete-talk-info)
                     (read-string "Time: ")
                     (completing-read "Status: " (mapcar 'car emacsconf-status-types))))
  (require 'diary-lib)
  (setq talk (emacsconf-resolve-talk talk))
  (let* ((converted
          (cond
           ((listp time) time)
           ((timer-duration time) (timer-relative-time nil (timer-duration time)))
           (t                           ; HH:MM
            (date-to-time (concat (format-time-string "%Y-%m-%d" nil emacsconf-timezone)
                                  "T"
                                  (string-pad time 5 ?0 t) 
                                  emacsconf-timezone-offset)))))
         (timer-id (concat (format-time-string "%m-%dT%H:%M" converted)
                           "-"
                           (plist-get talk :slug)
                           "-"
                           new-status)))
    (emacsconf-stream-cancel-timer timer-id) 
    (add-to-list 'emacsconf-stream-timers
                  (cons
                   timer-id
                   (run-at-time time converted #'emacsconf-stream-update-talk-status-from-timer
                                talk new-status
                                notification)))))

The actual playing of talks happened using functions that were called from org-after-todo-state-change-hook. I wrote a function that extracted the talk information and then called my own list of functions.

emacsconf-org-after-todo-state-change
(defun emacsconf-org-after-todo-state-change ()
  "Run all the hooks in `emacsconf-todo-hooks'.
If an `emacsconf-todo-hooks' entry is a list, run it only for the
tracks with the ID in the cdr of that list."
  (let* ((talk (emacsconf-get-talk-info-for-subtree))
         (track (emacsconf-get-track (plist-get talk :track))))
    (mapc
     (lambda (hook-entry)
       (cond
        ((symbolp hook-entry) (funcall hook-entry talk))
        ((member (plist-get track :id) (cdr hook-entry))
         (funcall (car hook-entry) talk))))
     emacsconf-todo-hooks)))

For example, this function played the recorded intro and the talk:

emacsconf-stream-play-talk-on-change
(defun emacsconf-stream-play-talk-on-change (talk)
  "Play the talk."
  (interactive (list (emacsconf-complete-talk-info)))
  (setq talk (emacsconf-resolve-talk talk))
  (when (or (not (boundp 'org-state)) (string= org-state "PLAYING"))
    (if (plist-get talk :stream-files)
        (progn
          (emacsconf-stream-track-ssh
           talk
           "overlay"
           (plist-get talk :slug))
          (emacsconf-stream-track-ssh
           talk
           (append
            (list
             "nohup"
             "mpv")
            (split-string-and-unquote (plist-get talk :stream-files))
            (list "&"))))
      (emacsconf-stream-track-ssh
       talk
       (cons
        "nohup"
        (cond
         ((and
           (plist-get talk :recorded-intro)
           (plist-get talk :video-file)) ;; recorded intro and recorded talk
          (message "should automatically play intro and recording")
          (list "play-with-intro" (plist-get talk :slug))) ;; todo deal with stream files
         ((and
           (plist-get talk :recorded-intro)
           (null (plist-get talk :video-file))) ;; recorded intro and live talk; play the intro and join BBB
          (message "should automatically play intro; join %s" (plist-get talk :bbb-backstage))
          (list "intro" (plist-get talk :slug)))
         ((and
           (null (plist-get talk :recorded-intro))
           (plist-get talk :video-file)) ;; live intro and recorded talk, show slide and use Mumble; manually play talk
          (message "should show intro slide; play %s afterwards" (plist-get talk :slug))
          (list "intro" (plist-get talk :slug)))
         ((and
           (null (plist-get talk :recorded-intro))
           (null (plist-get talk :video-file))) ;; live intro and live talk, join the BBB
          (message "join %s for live intro and talk" (plist-get talk :bbb-backstage))
          (list "bbb" (plist-get talk :slug)))))))))

and this function handled IRC announcements when the talk state changed:

emacsconf-erc-announce-on-change
(defun emacsconf-erc-announce-on-change (talk)
  "Announce talk."
  (let ((func
         (pcase org-state
           ("PLAYING" #'erc-cmd-NOWPLAYING)
           ("CLOSED_Q" #'erc-cmd-NOWCLOSEDQ)
           ("OPEN_Q" #'erc-cmd-NOWOPENQ)
           ("UNSTREAMED_Q" #'erc-cmd-NOWUNSTREAMEDQ)
           ("TO_ARCHIVE" #'erc-cmd-NOWDONE))))
    (when func
      (funcall func talk))))

The actual announcements were handled by something like this:

erc-cmd-NOWCLOSEDQ
(defun erc-cmd-NOWCLOSEDQ (talk)
  "Announce TALK has started Q&A, but the host has not yet opened it up."
  (interactive (list (emacsconf-complete-talk-info)))
  (when (stringp talk) (setq talk (or (emacsconf-find-talk-info talk) (error "Could not find talk %s" talk))))
  (if (emacsconf-erc-recently-announced (format "-- Q&A beginning for \"%s\"" (plist-get talk :slug)))
      (message "Recently announced, skipping")
    (emacsconf-erc-with-channels (list (concat "#" (plist-get talk :channel)))
      (erc-send-message (format "-- Q&A beginning for \"%s\" (%s) Watch: %s Add notes/questions: %s"
                                (plist-get talk :title)
                                (plist-get talk :qa-info)
                                (plist-get talk :watch-url)
                                (plist-get talk :pad-url))))  
    (emacsconf-erc-with-channels (list emacsconf-erc-hallway emacsconf-erc-org)
      (erc-send-message (format "-- Q&A beginning for \"%s\" in the %s track (%s) Watch: %s Add notes/questions: %s . Chat: #%s"
                                (plist-get talk :title)
                                (plist-get talk :track)
                                (plist-get talk :qa-info)
                                (plist-get talk :watch-url)
                                (plist-get talk :pad-url)
                                (plist-get talk :channel))))))

All that code meant that during the actual conference, my role was mostly just worrying, and occasionally starting up the Q&A (if I wasn't sure if the code would do it right). The shell scripts I wrote made it easy for the other organizers to take over the second part as they saw how it worked.

Yay timers, Emacs, and TRAMP!

You can find the latest versions of these functions in the emacsconf-el repository.

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

Using Emacs and Python to record an animation and synchronize it with audio

| emacs, emacsconf, python, subed, video

[2023-01-14 Sat]: Removed my fork since upstream now has the :eval function.

The Q&A session for Things I'd like to see in Emacs (Richard Stallman) from EmacsConf 2022 was done over Mumble. Amin pasted the questions into the Mumble chat buffer and I copied them into a larger buffer as the speaker answered them, but I didn't do it consistently. I figured it might be worth making another video with easier-to-read visuals. At first, I thought about using LaTeX to create Beamer slides with the question text, which I could then turn into a video using ffmpeg. Then I decided to figure out how to animate the text in Emacs, because why not? I figured a straightforward typing animation would probably be less distracting than animate-string, and emacs-director seems to handle that nicely. I forked it to add a few things I wanted, like variables to make the typing speed slower (so that it could more reliably type things on my old laptop, since sometimes the timers seemed to have hiccups) and an :eval step for running things without needing to log them. (2023-01-14: Upstream has the :eval feature now.)

To make it easy to synchronize the resulting animation with the chapter markers I derived from the transcript of the audio file, I decided to beep between scenes. First step: make a beep file. 

ffmpeg -y -f lavfi -i 'sine=frequency=1000:duration=0.1' beep.wav

Next, I animated the text, with a beep between scenes. I used subed-parse-file to read the question text directly from the chapter markers, and I used simplescreenrecorder to set up the recording settings (including audio). 

(defun my-beep ()
  (interactive)
  (save-window-excursion
    (shell-command "aplay ~/recordings/beep.wav &" nil nil)))

(require 'director)
(defvar emacsconf-recording-process nil)
(shell-command "xdotool getwindowfocus windowsize 1282 720")
(progn
  (switch-to-buffer (get-buffer-create "*Questions*"))
  (erase-buffer)
  (org-mode)
  (face-remap-add-relative 'default :height 300)
  (setq-local mode-line-format "   Q&A for EmacsConf 2022: What I'd like to see in Emacs (Richard M. Stallman) - emacsconf.org/2022/talks/rms")
  (sit-for 3)
  (delete-other-windows)
  (hl-line-mode -1)
  (when (process-live-p emacsconf-recording-process) (kill-process emacsconf-recording-process))
  (setq emacsconf-recording-process (start-process "ssr" (get-buffer-create "*ssr*")
                                                   "simplescreenrecorder"
                                                   "--start-recording"
                                                   "--start-hidden"))
  (sit-for 3)
  (director-run
   :version 1
   :log-target '(file . "/tmp/director.log")
   :before-start
   (lambda ()
     (switch-to-buffer (get-buffer-create "*Questions*"))
     (delete-other-windows))
   :steps
   (let ((subtitles (subed-parse-file "~/proj/emacsconf/rms/emacsconf-2022-rms--what-id-like-to-see-in-emacs--answers--chapters.vtt")))
     (apply #'append
            (list
             (list :eval '(my-beep))
             (list :type "* Q&A for Richard Stallman's EmacsConf 2022 talk: What I'd like to see in Emacs\nhttps://emacsconf.org/2022/talks/rms\n\n"))
            (mapcar
             (lambda (sub)
               (list
                (list :log (elt sub 3))
                (list :eval '(progn (org-end-of-subtree)
                                    (unless (bolp) (insert "\n"))))
                (list :type (concat "** " (elt sub 3) "\n\n"))
                (list :eval '(org-back-to-heading))
                (list :wait 5)
                (list :eval '(my-beep))))
             subtitles)))
   :typing-style 'human
   :delay-between-steps 0
   :after-end (lambda ()
                (process-send-string emacsconf-recording-process "record-save\nwindow-show\nquit\n"))
   :on-failure (lambda ()
                 (process-send-string emacsconf-recording-process "record-save\nwindow-show\nquit\n"))
   :on-error (lambda ()
               (process-send-string emacsconf-recording-process "record-save\nwindow-show\nquit\n"))))

I used the following code to copy the latest recording to animation.webm and extract the audio to animation.wav. my-latest-file and my-recordings-dir are in my Emacs config. 

(let ((name "animation.webm"))
  (copy-file (my-latest-file my-recordings-dir) name t)
  (shell-command
   (format "ffmpeg -y -i %s -ar 8000 -ac 1 %s.wav"
           (shell-quote-argument name)
           (shell-quote-argument (file-name-sans-extension name)))))

Then I needed to get the timestamps of the beeps in the recording. I subtracted a little bit (0.82 seconds) based on comparing the waveform with the results. 

filename = "animation.wav"
from scipy.io import wavfile
from scipy import signal
import numpy as np
import re
rate, source = wavfile.read(filename)
peaks = signal.find_peaks(source, height=1000, distance=1000)
base_times = (peaks[0] / rate) - 0.82
print(base_times)

I noticed that the first question didn't seem to get beeped properly, so I tweaked the times. Then I wrote some code to generate a very long ffmpeg command that used trim and tpad to select the segments and extend them to the right durations. There was some drift when I did it without the audio track, but the timestamps seemed to work right when I included the Q&A audio track as well. 

import webvtt
import subprocess
chapters_filename =  "emacsconf-2022-rms--what-id-like-to-see-in-emacs--answers--chapters.vtt"
answers_filename = "answers.wav"
animation_filename = "animation.webm"
def get_length(filename):
    result = subprocess.run(["ffprobe", "-v", "error", "-show_entries",
                             "format=duration", "-of",
                             "default=noprint_wrappers=1:nokey=1", filename],
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT)
    return float(result.stdout)

def get_frames(filename):
    result = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0", "-count_packets",
                             "-show_entries", "stream=nb_read_packets", "-of",
                             "csv=p=0", filename],
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT)
    return float(result.stdout)

answers_length = get_length(answers_filename)
# override base_times
times = np.asarray([  1.515875,  13.50, 52.32125 ,  81.368625, 116.66625 , 146.023125,
       161.904875, 182.820875, 209.92125 , 226.51525 , 247.93875 ,
       260.971   , 270.87375 , 278.23325 , 303.166875, 327.44925 ,
       351.616375, 372.39525 , 394.246625, 409.36325 , 420.527875,
       431.854   , 440.608625, 473.86825 , 488.539   , 518.751875,
       544.1515  , 555.006   , 576.89225 , 598.157375, 627.795125,
       647.187125, 661.10875 , 695.87175 , 709.750125, 717.359875])
fps = 30.0
times = np.append(times, get_length(animation_filename))
anim_spans = list(zip(times[:-1], times[1:]))
chapters = webvtt.read(chapters_filename)
if chapters[0].start_in_seconds == 0:
    vtt_times = [[c.start_in_seconds, c.text] for c in chapters]
else:
    vtt_times = [[0, "Introduction"]] + [[c.start_in_seconds, c.text] for c in chapters] 
vtt_times = vtt_times + [[answers_length, "End"]]
# Add ending timestamps
vtt_times = [[x[0][0], x[1][0], x[0][1]] for x in zip(vtt_times[:-1], vtt_times[1:])]
test_rate = 1.0

i = 0
concat_list = ""
groups = list(zip(anim_spans, vtt_times))
import ffmpeg
animation = ffmpeg.input('animation.webm').video
audio = ffmpeg.input('rms.opus')

for_overlay = ffmpeg.input('color=color=black:size=1280x720:d=%f' % answers_length, f='lavfi')
params = {"b:v": "1k", "vcodec": "libvpx", "r": "30", "crf": "63"}
test_limit = 1
params = {"vcodec": "libvpx", "r": "30", "copyts": None, "b:v": "1M", "crf": 24}
test_limit = 0
anim_rate = 1
import math
cursor = 0
if test_limit > 0:
    groups = groups[0:test_limit]
clips = []

# cursor is the current time
for anim, vtt in groups:
    padding = vtt[1] - cursor - (anim[1] - anim[0]) / anim_rate
    if (padding < 0):
        print("Squeezing", math.floor((anim[1] - anim[0]) / (anim_rate * 1.0)), 'into', vtt[1] - cursor, padding)
        clips.append(animation.trim(start=anim[0], end=anim[1]).setpts('PTS-STARTPTS')) 
    elif padding == 0:
        clips.append(animation.trim(start=anim[0], end=anim[1]).setpts('PTS-STARTPTS'))
    else:
        print("%f to %f: Padding %f into %f - pad: %f" % (cursor, vtt[1], (anim[1] - anim[0]) / (anim_rate * 1.0), vtt[1] - cursor, padding))
        cursor = cursor + padding + (anim[1] - anim[0]) / anim_rate
        clips.append(animation.trim(start=anim[0], end=anim[1]).setpts('PTS-STARTPTS').filter('tpad', stop_mode="clone", stop_duration=padding))
    for_overlay = for_overlay.overlay(animation.trim(start=anim[0], end=anim[1]).setpts('PTS-STARTPTS+%f' % vtt[0]))
    clips.append(audio.filter('atrim', start=vtt[0], end=vtt[1]).filter('asetpts', 'PTS-STARTPTS'))
args = ffmpeg.concat(*clips, v=1, a=1).output('output.webm', **params).overwrite_output().compile()
print(' '.join(f'"{item}"' for item in args))

Anyway, it's here for future reference. =)

View org source for this post
View or add comments (Disqus), or e-mail me at sacha@sachachua.com

Converting our VTT files to TTML

| emacsconf, geek, ffmpeg

I wanted to convert our VTT files to TTML files so that we might be able to use them for training lachesis for transcript segmentation. I downloaded the VTT files from EmacsConf 2021 to a directory and copied the edited captions from the EmacsConf 2022 backstage area (using head -1 ${FILE} | grep -q "captioned" to distinguish them from the automatic ones). I installed the ttconv python package. Then I used the following command to convert the TTML files: 

for FILE in *.vtt; do
    BASE=$(basename -s .vtt "$FILE");
    ffmpeg -y -i $FILE $BASE.srt; tt convert -i $BASE.srt -o $BASE.ttml
done

I haven't gotten around to installing whanever I need in order to get lachesis to work under Python 2.7, since it hasn't been updated for Python 3. It'll probably be a low-priority project anyway, as EmacsConf is fast approaching. Anyway, I thought I'd stash this in my blog somewhere in case I need to make TTML files again!

View or add comments (Disqus), or e-mail me at sacha@sachachua.com