Categories: geek » emacs » org

View topic page - RSS - Atom - Subscribe via email

Animating SVG topic maps with Inkscape, Emacs, FFmpeg, and Reveal.js

| emacs, drawing, org, ffmpeg, video

tldr (2167 words): I can make animating presentation maps easier by writing my own functions for the Emacs text editor. In this post, I show how I can animate an SVG element by element. I can also add IDs to the path and use CSS to build up an SVG with temporary highlighting in a Reveal.js presentation.

Text from the sketch
  • PNG: Inkscape: trace
  • Supernote (e-ink)
  • iPad: Adobe Fresco

Convert PDF to SVG with Inkscape (Cairo option) or pdftocairo)

  • PNG / Supernote PDF: Combined shapes. Process
    1. Break apart, fracture overlaps
    2. Recombine
    3. Set IDs
    4. Sort paths -> Animation style 1
  • Adobe Fresco: individual elements in order; landscape feels natural

Animation styles

  • Animation style 1: Display elements one after another
  • Animation style 2: Display elements one after another, and also show/hide highlights
    • Table: slide ID, IDs to add, temporary highlights -> Reveal.js: CSS with transitions

Ideas for next steps:

  • Explore graphviz & other diagramming tools
  • Frame-by-frame SVGs
    • on include
    • write to files
  • FFmpeg crossfade
  • Recording Reveal.js presentations
  • Use OCR results?

I often have a hard time organizing my thoughts into a linear sequence. Sketches are nice because they let me jump around and still show the connections between ideas. For presentations, I'd like to walk people through these sketches by highlighting different areas. For example, I might highlight the current topic or show the previous topics that are connected to the current one. Of course, this is something Emacs can help with. Before we dive into it, here are quick previews of the kinds of animation I'm talking about:

animation-loop.gif
Figure 1: Animation style 1: based on drawing order

Animation style 2: building up a map with temporary highlights

Getting the sketches: PDFs are not all the same

Let's start with getting the sketches. I usually export my sketches as PNGs from my Supernote A5X. But if I know that I'm going to animate a sketch, I can export it as a PDF. I've recently been experimenting with Adobe Fresco on the iPad, which can also export to PDF. The PDF I get from Fresco is easier to animate, but I prefer to draw on the Supernote because it's an e-ink device (and because the kiddo usually uses the iPad).

If I start with a PNG, I could use Inkscape to trace the PNG and turn it into an SVG. I think Inkscape uses autotrace behind the scenes. I don't usually put my highlights on a separate layer, so autotrace will make odd shapes.

It's a lot easier if you start off with vector graphics in the first place. I can export a vector PDF from the SuperNote A5X and either import it into Inkscape using the Cairo option or use the command-line pdftocairo tool.

I've been looking into using Adobe Fresco, which is a free app available for the iPad. Fresco's PDF export can be converted to an SVG using Inkscape or PDF to Cairo. What I like about the output of this app is that it gives me individual elements as their own paths and they're listed in order of drawing. This makes it really easy to animate by just going through the paths in order.

Animation style 1: displaying paths in order

Here's a sample SVG file that pdfcairo creates from an Adobe Fresco PDF export:

pdftocairo -svg ~/Downloads/subed-audio.pdf ~/Downloads/subed-audio.svg
Sample SVG
subed-audio.svg

Adobe Fresco also includes built-in time-lapse, but since I often like to move things around or tidy things up, it's easier to just work with the final image, export it as a PDF, and convert it to an SVG.

I can make a very simple animation by setting the opacity of all the paths to 0, then looping through the elements to set the opacity back to 1 and write that version of the SVG to a separate file. From how-can-i-generate-png-frames-that-step-through-the-highlights:

my-animate-svg-paths: Add one path at a time. Save the resulting SVGs to OUTPUT-DIR.
(defun my-animate-svg-paths (filename output-dir)
  "Add one path at a time. Save the resulting SVGs to OUTPUT-DIR."
  (unless (file-directory-p output-dir)
    (make-directory output-dir t))
  (let* ((dom (xml-parse-file filename))
         (paths (seq-filter (lambda (e) (dom-attr e 'style))
                            (dom-by-tag dom 'path)))
         (total (length paths))
         (frame-num (length paths))
         result)
    (dolist (elem paths)
      (dom-set-attribute elem 'style
                         (concat
                          (dom-attr elem 'style)
                          ";mix-blend-mode:darken")))
    (with-temp-file (expand-file-name (format "frame-%03d.svg" (1+ frame-num)) output-dir)
      (xml-print dom))
    (dolist (elem paths)
      (dom-set-attribute elem 'style
                         (concat
                          (dom-attr elem 'style)
                          ";fill-opacity:0")))
    (dolist (elem paths)
      (with-temp-file (expand-file-name
                       (format "frame-%03d.svg"
                               (- total frame-num))
                       output-dir)
        (message "%03d" frame-num)
        (dom-set-attribute elem 'style
                           (concat (dom-attr elem 'style)
                                   ";fill-opacity:1"))
        (push (list (format "frame-%03d.svg"
                            (1+ (- total frame-num)))
                    (dom-attr elem 'id))
              result)
        (setq frame-num (1- frame-num))
        (xml-print dom)))
    (reverse result)))

Here's how I call it:

(my-animate-svg-paths "~/Downloads/subed-audio.svg" "/tmp/subed-audio/frames" t)

Then I can use FFmpeg to combine all of those frames into a video:

ffmpeg -i frame-%03d.svg -vf palettegen -y palette.png
ffmpeg -framerate 30 -i frame-%03d.svg -i palette.png -lavfi "paletteuse" -loop 0 -y animation-loop.gif
animation-loop.gif
Figure 2: Animating SVG paths based on drawing order

Neither Supernote nor Adobe Fresco give me the original stroke information. These are filled shapes, so I can't animate something drawing it. But having different elements appear in sequence is fine for my purposes. If you happen to know how to get stroke information out of Supernote .note files or of an iPad app that exports nice single-line SVGs that have stroke direction, I would love to hear about it.

Identifying paths from Supernote sketches

When I export a PDF from Supernote and convert it to an SVG, each color is a combined shape with all the elements. If I want to animate parts of the image, I have to break it up and recombine selected elements (Inkscape's Ctrl-k shortcut) so that the holes in shapes are properly handled. This is a bit of a tedious process and it usually ends up with elements in a pretty random order. Since I have to reorder elements by hand, I don't really want to animate the sketch letter-by-letter. Instead, I combine them into larger chunks like topics or paragraphs.

The following code takes the PDF, converts it to an SVG, recolours highlights, and then breaks up paths into elements:

my-sketch-convert-pdf-and-break-up-paths: Convert PDF to SVG and break up paths.
(defun my-sketch-convert-pdf-and-break-up-paths (pdf-file &optional rotate)
  "Convert PDF to SVG and break up paths."
  (interactive (list (read-file-name
                      (format "PDF (%s): "
                              (my-latest-file "~/Dropbox/Supernote/EXPORT/" "pdf"))
                      "~/Dropbox/Supernote/EXPORT/"
                      (my-latest-file "~/Dropbox/Supernote/EXPORT/" "pdf")
                      t
                      nil
                      (lambda (s) (string-match "pdf" s)))))
  (unless (file-exists-p (concat (file-name-sans-extension pdf-file) ".svg"))
    (call-process "pdftocairo" nil nil nil "-svg" (expand-file-name pdf-file)
                  (expand-file-name (concat (file-name-sans-extension pdf-file) ".svg"))))
  (let ((dom (xml-parse-file (expand-file-name (concat (file-name-sans-extension pdf-file) ".svg"))))
        highlights)
    (setq highlights (dom-node 'g '((id . "highlights"))))
    (dom-append-child dom highlights)
    (dolist (path (dom-by-tag dom 'path))
      ;;  recolor and move
      (unless (string-match (regexp-quote "rgb(0%,0%,0%)") (or (dom-attr path 'style) ""))
        (dom-remove-node dom path)
        (dom-append-child highlights path)
        (dom-set-attribute
         path 'style
         (replace-regexp-in-string
          (regexp-quote "rgb(78.822327%,78.822327%,78.822327%)")
          "#f6f396"
          (or (dom-attr path 'style) ""))))
      (let ((parent (dom-parent dom path)))
        ;; break apart
        (when (dom-attr path 'd)
          (dolist (part (split-string (dom-attr path 'd) "M " t " +"))
            (dom-append-child
             parent
             (dom-node 'path `((style . ,(dom-attr path 'style))
                               (d . ,(concat "M " part))))))
          (dom-remove-node dom path))))
    ;; remove the use
    (dolist (use (dom-by-tag dom 'use))
      (dom-remove-node dom use))
    (dolist (use (dom-by-tag dom 'image))
      (dom-remove-node dom use))
    ;; move the first g down
    (let ((g (car (dom-by-id dom "surface1"))))
      (setf (cddar dom)
            (seq-remove (lambda (o)
                          (and (listp o) (string= (dom-attr o 'id) "surface1")))
                        (dom-children dom)))
      (dom-append-child dom g)
      (when rotate
        (let* ((old-width (dom-attr dom 'width))
               (old-height (dom-attr dom 'height))
               (view-box (mapcar 'string-to-number (split-string (dom-attr dom 'viewBox))))
               (rotate (format "rotate(90) translate(0 %s)" (- (elt view-box 3)))))
          (dom-set-attribute dom 'width old-height)
          (dom-set-attribute dom 'height old-width)
          (dom-set-attribute dom 'viewBox (format "0 0 %d %d" (elt view-box 3) (elt view-box 2)))
          (dom-set-attribute highlights 'transform rotate)
          (dom-set-attribute g 'transform rotate))))
    (with-temp-file (expand-file-name (concat (file-name-sans-extension pdf-file) "-split.svg"))
      (svg-print (car dom)))))

2023-10-split.svg
Figure 3: Image after splitting up into elements

You can see how the spaces inside letters like "o" end up being black. Selecting and combining those paths fixes that.

Combining paths in Inkscape

If there were shapes that were touching, then I need to draw lines and fracture the shapes in order to break them apart.

Fracturing shapes and checking the highlights

The end result should be an SVG with the different chunks that I might want to animate, but I need to identify the paths first. You can assign object IDs in Inkscape, but this is a bit of an annoying process since I haven't figured out a keyboard-friendly way to set object IDs. I usually find it easier to just set up an Autokey shortcut (or AutoHotkey in Windows) to click on the ID text box so that I can type something in.

Autokey script for clicking
import time
x, y = mouse.get_location()
# Use the coordinates of the ID text field on your screen; xev can help
mouse.click_absolute(3152, 639, 1)
time.sleep(1)
keyboard.send_keys("<ctrl>+a")
mouse.move_cursor(x, y)

Then I can select each element, press the shortcut key, and type an ID into the textbox. I might use "t-…" to indicate the text for a map section, "h-…" to indicate a highlight, and arrows by specifying their start and end.

Setting IDs in Inkscape

To simplify things, I wrote a function in Emacs that will go through the different groups that I've made, show each path in a different color and with a reasonable guess at a bounding box, and prompt me for an ID. This way, I can quickly assign IDs to all of the paths. The completion is mostly there to make sure I don't accidentally reuse an ID, although it can try to combine paths if I specify the ID. It saves the paths after each change so that I can start and stop as needed. Identifying paths in Emacs is usually much nicer than identifying them in Inkscape.

Identifying paths inside Emacs

my-svg-identify-paths: Prompt for IDs for each path in FILENAME.
(defun my-svg-identify-paths (filename)
  "Prompt for IDs for each path in FILENAME."
  (interactive (list (read-file-name "SVG: " nil nil
                                     (lambda (f) (string-match "\\.svg$" f)))))
  (let* ((dom (car (xml-parse-file filename)))
         (paths (dom-by-tag dom 'path))
         (vertico-count 3)
         (ids (seq-keep (lambda (path)
                          (unless (string-match "path[0-9]+" (or (dom-attr path 'id) "path0"))
                            (dom-attr path 'id)))
                        paths))
         (edges (window-inside-pixel-edges (get-buffer-window)))
         id)
    (my-svg-display "*image*" dom nil t)
    (dolist (path paths)
      (when (string-match "path[0-9]+" (or (dom-attr path 'id) "path0"))
        ;; display the image with an outline
        (unwind-protect
            (progn
              (my-svg-display "*image*" dom (dom-attr path 'id) t)
              (setq id (completing-read
                        (format "ID (%s): " (dom-attr path 'id))
                        ids))
              ;; already exists, merge with existing element
              (if-let ((old (dom-by-id dom id)))
                  (progn
                    (dom-set-attribute
                     old
                     'd
                     (concat (dom-attr (dom-by-id dom id) 'd)
                             " "
                             ;; change relative to absolute
                             (replace-regexp-in-string "^m" "M"
                                                       (dom-attr path 'd))))
                    (dom-remove-node dom path)
                    (setq id nil))
                (dom-set-attribute path 'id id)
                (add-to-list 'ids id))))
        ;; save the image just in case we get interrupted halfway through
        (with-temp-file filename
          (svg-print dom))))))

Sorting and animating the paths by IDs

Then I can animate SVGs by specifying the IDs. I can reorder the paths in the SVG itself so that I can animate it group by group, like the way that the Adobe Fresco SVGs were animated element by element.

Reordering paths
(my-svg-reorder-paths "~/proj/2023-12-audio-workflow/map.svg"
                      '("t-start" "h-audio" "h-capture" "t-but" "t-mic" "h-mic"
                        "t-reviewing" "h-reviewing"
                        "t-words" "h-words" "t-workflow" "h-workflow"
                        "t-lapel" "h-lapel"
                        "mic-recorder" "t-recorder" "h-recorder"
                        "t-syncthing" "h-sync"
                        "t-keywords" "h-keywords" "t-keyword-types"
                        "t-lines" "h-lines"
                        "t-align" "h-align"
                        "arrow"
                        "t-org" "h-org" "t-todo" "h-todo" "h-linked"
                        "t-jump" "h-jump"
                        "t-waveform" "h-waveform"
                        "t-someday"
                        "h-sections"
                        "t-speech-recognition" "h-speech-recognition"
                        "t-ai" "h-ai"
                        "t-summary"
                        "extra")
                      "~/proj/2023-12-audio-workflow/map-output.svg")
(my-animate-svg-paths "~/proj/2023-12-audio-workflow/map-output.svg"
                      "~/proj/2023-12-audio-workflow/frames/")
Table of filenames after reordering paths and animating the image
frame-001.svg t-start
frame-002.svg h-audio
frame-003.svg h-capture
frame-004.svg t-but
frame-005.svg t-mic
frame-006.svg h-mic
frame-007.svg t-reviewing
frame-008.svg h-reviewing
frame-009.svg t-words
frame-010.svg h-words
frame-011.svg t-workflow
frame-012.svg h-workflow
frame-013.svg t-lapel
frame-014.svg h-lapel
frame-015.svg mic-recorder
frame-016.svg t-recorder
frame-017.svg h-recorder
frame-018.svg t-syncthing
frame-019.svg h-sync
frame-020.svg t-keywords
frame-021.svg h-keywords
frame-022.svg t-keyword-types
frame-023.svg t-lines
frame-024.svg h-lines
frame-025.svg t-align
frame-026.svg h-align
frame-027.svg arrow
frame-028.svg t-org
frame-029.svg h-org
frame-030.svg t-todo
frame-031.svg h-todo
frame-032.svg h-linked
frame-033.svg t-jump
frame-034.svg h-jump
frame-035.svg t-waveform
frame-036.svg h-waveform
frame-037.svg t-someday
frame-038.svg h-sections
frame-039.svg t-speech-recognition
frame-040.svg h-speech-recognition
frame-041.svg t-ai
frame-042.svg h-ai
frame-043.svg t-summary
frame-044.svg extra

The table of filenames makes it easy to use specific frames as part of a presentation or video.

Here is the result as a video:

(let ((compile-media-output-video-width 1280)
      (compile-media-output-video-height 720))
  (my-ffmpeg-animate-images
   (directory-files "~/proj/2023-12-audio-workflow/frames/" t "\\.svg$")
   (expand-file-name "~/proj/2023-12-audio-workflow/frames/animation.webm")
   4))

Animation of SVG by paths

The way it works is that the my-svg-reorder-paths function removes and readds elements following the list of IDs specified, so everything's ready to go for step-by-step animation. Here's the code:

my-svg-reorder-paths: Sort paths in FILENAME.
(defun my-svg-reorder-paths (filename &optional ids output-filename)
  "Sort paths in FILENAME."
  (interactive (list (read-file-name "SVG: " nil nil (lambda (f) (string-match "\\.svg$" f)))
                     nil (read-file-name "Output: ")))
  (let* ((dom (car (xml-parse-file filename)))
         (paths (dom-by-tag dom 'path))
         (parent (dom-parent dom (car paths)))
         (ids-left
          (nreverse (seq-keep (lambda (path)
                                (unless (string-match "path[0-9]+" (or (dom-attr path 'id) "path0"))
                                  (dom-attr path 'id)))
                              paths)))
         list)
    (when (called-interactively-p)
      (while ids-left
        (my-svg-display "*image*" dom (car ids-left))
        (let ((current (completing-read
                        (format "ID (%s): "
                                (car ids-left))
                        ids-left nil nil nil nil (car ids-left)))
              node)
          (add-to-list 'ids current)
          (setq ids-left (seq-remove (lambda (o) (string= o current)) ids-left)))))
    (if ids ;; reorganize under the first path's parent
        (progn
          (dolist (id ids)
            (if-let ((node (car (dom-by-id dom id))))
                (progn
                  (dom-remove-node dom node)
                  (dom-append-child parent node))
              (message "Could not find %s" id)))
          (with-temp-file (or output-filename filename)
            (svg-print dom))))
    (nreverse (seq-keep (lambda (path)
                          (unless (string-match "path[0-9]+" (or (dom-attr path 'id) "path0"))
                            (dom-attr path 'id)))
                        (dom-by-tag dom 'path)))))

Animation style 2: Building up a map with temporary highlights

I can also use CSS rules to transition between opacity values for more complex animations. For my EmacsConf 2023 presentation, I wanted to make a self-paced, narrated presentation so that people could follow hyperlinks, read the source code, and explore. I wanted to include a map so that I could try to make sense of everything. For this map, I wanted to highlight the previous sections that were connected to the topic for the current section.

I used a custom Org link to include the full contents of the SVG instead of just including it with an img tag.

#+ATTR_HTML: :class r-stretch
my-include:~/proj/emacsconf-2023-emacsconf/map.svg?wrap=export html

my-include-export: Export PATH to FORMAT using the specified wrap parameter.
(defun my-include-export (path _ format _)
  "Export PATH to FORMAT using the specified wrap parameter."
  (let (params body start end)
    (when (string-match "^\\(.*+?\\)\\(?:::\\|\\?\\)\\(.*+\\)" path)
      (setq params (save-match-data (org-protocol-convert-query-to-plist (match-string 2 path)))
            path (match-string 1 path)))
    (with-temp-buffer
      (insert-file-contents-literally path)
      (when (string-match "\\.org$" path)
        (org-mode))
      (if (plist-get params :name)
          (when (org-babel-find-named-block (plist-get params :name))
            (goto-char (org-babel-find-named-block (plist-get params :name)))
            (let ((block (org-element-context)))
              (setq start (org-element-begin block)
                    end (org-element-end block))))
        (goto-char (point-min))
        (when (plist-get params :from-regexp)
          (re-search-forward (url-unhex-string (plist-get params :from-regexp)))
          (goto-char (match-beginning 0)))
        (setq start (point))
        (setq end (point-max))
        (when (plist-get params :to-regexp)
          (re-search-forward (url-unhex-string (plist-get params :to-regexp)))
          (setq end (match-beginning 0))))
      (setq body (buffer-substring start end)))
    (with-temp-buffer
      (when (plist-get params :wrap)
        (let* ((wrap (plist-get params :wrap))
               block args)
          (when (string-match "\\<\\(\\S-+\\)\\( +.*\\)?" wrap)
            (setq block (match-string 1 wrap))
            (setq args (match-string 2 wrap))
            (setq body (format "#+BEGIN_%s%s\n%s\n#+END_%s\n"
                               block (or args "")
                               body
                               block)))))
      (when (plist-get params :summary)
        (setq body (format "#+begin_my_details %s\n%s\n#+end_my_details\n"
                           (plist-get params :summary)
                           body)))
      (insert body)
      (org-export-as format nil nil t))))

I wanted to be able to specify the entire sequence using a table in the Org Mode source for my presentation. Each row had the slide ID, a list of highlights in the form prev1,prev2;current, and a comma-separated list of elements to add to the full-opacity view.

Slide Highlight Additional elements
props-map h-email;h-properties t-email,email-properties,t-properties
file-prefixes h-properties;h-filename t-filename,properties-filename
renaming h-filename;h-renaming t-renaming,filename-renaming
shell-scripts h-renaming;h-shell-scripts renaming-shell-scripts,t-shell-scripts
availability h-properties;h-timezone t-timezone,properties-timezone
schedule h-timezone;h-schedule t-schedule,timezone-schedule
emailing-speakers h-timezone,h-mail-merge;h-emailing-speakers schedule-emailing-speakers,t-emailing-speakers
template h-properties;h-template t-template,properties-template
wiki h-template;h-wiki t-wiki,template-wiki,schedule-wiki
pad h-template;h-pad template-pad,t-pad
mail-merge h-template;h-mail-merge t-mail-merge,template-mail-merge,schedule-mail-merge,emailing-speakers-mail-merge
bbb h-bbb t-bbb
checkin h-mail-merge;h-checkin t-checkin,bbb-checkin
redirect h-bbb;h-redirect t-redirect,bbb-redirect
shortcuts h-email;h-shortcuts t-shortcuts,email-shortcuts
logbook h-shortcuts;h-logbook shortcuts-logbook,t-logbook
captions h-captions t-captions,captions-wiki
tramp h-captions;h-tramp t-tramp,captions-tramp
crontab h-tramp;h-crontab tramp-crontab,bbb-crontab,t-crontab
transitions h-crontab;h-transitions shell-scripts-transitions,t-transitions,shortcuts-transitions,transitions-crontab
irc h-transitions;h-irc t-irc,transitions-irc

Reveal.js adds a "current" class to the slide, so I can use that as a trigger for the transition. I have a bit of Emacs Lisp code that generates some very messy CSS, in which I specify the ID of the slide, followed by all of the elements that need their opacity set to 1, and also specifying the highlights that will be shown in an animated way.

my-reveal-svg-progression-css: Make the CSS.
(defun my-reveal-svg-progression-css (map-progression &optional highlight-duration)
  "Make the CSS.
map-progression should be a list of lists with the following format:
((\"slide-id\" \"prev1,prev2;cur1\" \"id-to-add1,id-to-add2\") ...)."
  (setq highlight-duration (or highlight-duration 2))
  (let (full)
    (format
     "<style>%s</style>"
     (mapconcat
      (lambda (slide)
        (setq full (append (split-string (elt slide 2) ",") full))
        (format "#slide-%s.present path { opacity: 0.2 }
%s { opacity: 1 !important }
%s"
                (car slide)
                (mapconcat (lambda (id) (format "#slide-%s.present #%s" (car slide) id))
                           full
                           ", ")
                (my-reveal-svg-highlight-different-colors slide)))
      map-progression
      "\n"))))
I call it from my Org file like this:

#+NAME: progression-css
#+begin_src emacs-lisp :exports code :var map-progression=progression :var highlight-duration=2 :results silent
(my-reveal-svg-progression-css map-progression highlight-duration)
#+end_src

Here's an excerpt showing the kind of code it makes:

<style>#slide-props-map.present path { opacity: 0.2 }
#slide-props-map.present #t-email, #slide-props-map.present #email-properties, #slide-props-map.present #t-properties { opacity: 1 !important }
#slide-props-map.present #h-email { fill: #c6c6c6; opacity: 1 !important; transition: fill 0.5s; transition-delay: 0.0s }#slide-props-map.present #h-properties { fill: #f6f396; opacity: 1 !important; transition: fill 0.5s; transition-delay: 0.5s }
#slide-file-prefixes.present path { opacity: 0.2 }
#slide-file-prefixes.present #t-filename, #slide-file-prefixes.present #properties-filename, #slide-file-prefixes.present #t-email, #slide-file-prefixes.present #email-properties, #slide-file-prefixes.present #t-properties { opacity: 1 !important }
#slide-file-prefixes.present #h-properties { fill: #c6c6c6; opacity: 1 !important; transition: fill 0.5s; transition-delay: 0.0s }#slide-file-prefixes.present #h-filename { fill: #f6f396; opacity: 1 !important; transition: fill 0.5s; transition-delay: 0.5s }
...</style>

Since it's automatically generated, I don't have to worry about it once I've gotten it to work. It's all hidden in a results drawer. So this CSS highlights specific parts of the SVG with a transition, and the highlight changes over the course of a second or two. It highlights the previous names and then the current one. The topics I'd already discussed would be in black, and the topics that I had yet to discuss would be in very light gray. This could give people a sense of the progress through the presentation.

Code for making the CSS
(defun my-reveal-svg-animation (slide)
  (string-join
   (seq-map-indexed
    (lambda (step-ids i)
      (format "%s { fill: #f6f396; transition: fill %ds; transition-delay: %ds }"
              (mapconcat
               (lambda (id) (format "#slide-%s.present #%s" (car slide) id))
               (split-string step-ids ",")
               ", ")
              highlight-duration
              (* i highlight-duration)))
    (split-string (elt slide 1) ";"))
   "\n"))

(defun my-reveal-svg-highlight-different-colors (slide)
  (let* ((colors '("#f6f396" "#c6c6c6")) ; reverse
         (steps (split-string (elt slide 1) ";"))
         (step-length 0.5))
    (string-join
     (seq-map-indexed
      (lambda (step-ids i)
        (format "%s { fill: %s; opacity: 1 !important; transition: fill %.1fs; transition-delay: %.1fs }"
                (mapconcat
                 (lambda (id) (format "#slide-%s.present #%s" (car slide) id))
                 (split-string step-ids ",")
                 ", ")
                (elt colors (- (length steps) i 1))
                step-length
                (* i 0.5)))
      steps))))

(defun my-reveal-svg-progression-css (map-progression &optional highlight-duration)
  "Make the CSS.
map-progression should be a list of lists with the following format:
((\"slide-id\" \"prev1,prev2;cur1\" \"id-to-add1,id-to-add2\") ...)."
  (setq highlight-duration (or highlight-duration 2))
  (let (full)
    (format
     "<style>%s</style>"
     (mapconcat
      (lambda (slide)
        (setq full (append (split-string (elt slide 2) ",") full))
        (format "#slide-%s.present path { opacity: 0.2 }
%s { opacity: 1 !important }
%s"
                (car slide)
                (mapconcat (lambda (id) (format "#slide-%s.present #%s" (car slide) id))
                           full
                           ", ")
                (my-reveal-svg-highlight-different-colors slide)))
      map-progression
      "\n"))))

As a result, as I go through my presentation, the image appears to build up incrementally, which is the effect that I was going for. I can test this by exporting only my map slides:

(save-excursion
  (goto-char (org-babel-find-named-block "progression-css"))
  (org-babel-execute-src-block))
(let ((org-tags-exclude-from-inheritance "map")
      (org-export-select-tags '("map")))
   (oer-reveal-export-to-html))

Ideas for next steps

  • Graphviz, mermaid-js, and other diagramming tools can make SVGs. I should be able to adapt my code to animate those diagrams by adding other elements in addition to path. Then I'll be able to make diagrams even more easily.
  • Since SVGs can contain CSS, I could make an SVG equivalent of the CSS rules I used for the presentation, maybe calling a function with a Lisp expression that specifies the operations (ex: ("frame-001.svg" "h-foo" opacity 1)). Then I could write frames to SVGs.
  • FFmpeg has a crossfade filter. With a little bit of figuring out, I should be able to make the same kind of animation in a webm form that I can include in my regular videos instead of using Reveal.js and CSS transitions.
  • I've also been thinking about automating the recording of my Reveal.js presentations. For my EmacsConf talk, I opened my presentation, started the recording with the system audio and the screen, and then let it autoplay the presentation. I checked on it periodically to avoid the screensaver/energy saving things from kicking in and so that I could stop the recording when it's finished. If I want to make this take less work, one option is to use ffmpeg's "-t" argument to specify the expected duration of the presentation so that I don't have to manually stop it. I'm also thinking about using Puppeteer to open the presentation, check when it's fully loaded, and start the process to record it - maybe even polling to see whether it's finished. I haven't gotten around to it yet. Anyhow, those are some ideas to explore next time.
  • As for animation, I'm still curious about the possibility of finding a way to access the raw stroke information if it's even available from my Supernote A5X (difficult because it's a proprietary data format) or finding an app for the iPad that exports single line SVGs that use stroke information instead of fill. That would only be if I wanted to do those even fancier animations that look like the whole thing is being drawn for you. I was trying to figure out if I could green screen the Adobe Fresco timelapse videos so that even if I have a pre-sketch to figure out spacing and remind me what to draw, I can just export the finished elements. But there's too much anti-aliasing and I haven't figured out how to do it cleanly yet. Maybe some other day.
  • I use Google Cloud Vision's text detection engine to convert my handwriting to text. It can give me bounding polygons for words or paragraphs. I might be able to figure out which curves are entirely within a word's bounding polygon and combine those automatically.
  • It would be pretty cool if I could combine the words recognized by Google Cloud Vision with the word-level timestamps from speech recognition so that I could get word-synced sketchnote animations with maybe a little manual intervention.

Anyway, those are some workflows for animating sketches with Inkscape and Emacs. Yay Emacs!

View org source for this post

Using Embark and qrencode to show a QR code for the Org Mode link at point

Posted: - Modified: | embark, emacs, org

[2024-01-12 Fri]: Added some code to display the QR code on the right side.

John Kitchin includes little QR codes in his videos. I thought that was a neat touch that makes it easier for people to jump to a link while they're watching. I'd like to make it easier to show QR codes too. The following code lets me show a QR code for the Org link at point. Since many of my links use custom Org link types that aren't that useful for people to scan, the code reuses the link resolution code from https://sachachua.com/dotemacs#web-link so that I can get the regular https: link.

(defun my-org-link-qr (url)
  "Display a QR code for URL in a buffer."
  (let ((buf (save-window-excursion (qrencode--encode-to-buffer (my-org-stored-link-as-url url)))))
    (display-buffer-in-side-window buf '((side . right)))))

(use-package qrencode
  :config
  (with-eval-after-load 'embark
    (define-key embark-org-link-map (kbd "q") #'my-org-link-qr)))
qr-code.svg
Figure 1: Screenshot of QR code for the link at point
View org source for this post
This is part of my Emacs configuration.

Using an Emacs Lisp macro to define quick custom Org Mode links to project files; plus URLs and search

| org, emacs, coding
  • [2024-09-19 Thu]: Added function for replacing current link, bound to C-. r (my-embark-replace-link-with-exported-url)
  • [2024-01-12 Fri] Added embark action to copy the exported link URL.
  • [2024-01-11 Thu] Switched to using Github links since Codeberg's down.
  • [2024-01-11 Thu] Updated my-copy-link to just return the link if called from Emacs Lisp. Fix getting the properties.
  • [2024-01-08 Mon] Add tip from Omar about embark-around-action-hooks
  • [2024-01-08 Mon] Simplify code by using consult--grep-position

Summary (882 words): Emacs macros make it easy to define sets of related functions for custom Org links. This makes it easier to link to projects and export or copy the links to the files in the web-based repos. You can also use that information to consult-ripgrep across lots of projects.

I'd like to get better at writing notes while coding and at turning those notes into blog posts and videos. I want to be able to link to files in projects easily with the ability to complete, follow, and export links. For example, [[subed:subed.el]] should become subed.el, which opens the file if I'm in Emacs and exports a link if I'm publishing a post. I've been making custom link types using org-link-set-parameters. I think it's time to make a macro that defines that set of functions for me. Emacs Lisp macros are a great way to write code to write code.

(defvar my-project-web-base-list nil "Local path . web repo URLs for easy linking.")

(defmacro my-org-project-link (type file-path git-url)
  `(progn
     (defun ,(intern (format "my-org-%s-complete" type)) ()
       ,(format "Complete a file from %s." type)
       (concat ,type ":" (completing-read "File: "
                                          (projectile-project-files ,file-path))))
     (defun ,(intern (format "my-org-%s-follow" type)) (link _)
       ,(format "Open a file from %s." type)
       (find-file
        (expand-file-name
         link
         ,file-path)))
     (defun ,(intern (format "my-org-%s-export" type)) (link desc format _)
       "Export link to file."
       (setq desc (or desc link))
       (when ,git-url
         (setq link (concat ,git-url (replace-regexp-in-string "^/" "" link))))
       (pcase format
         ((or 'html '11ty) (format "<a href=\"%s\">%s</a>"
                                   link
                                   (or desc link)))
         ('md (if desc (format "[%s](%s)" desc link)
                (format "<%s>" link)))
         ('latex (format "\\href{%s}{%s}" link desc))
         ('texinfo (format "@uref{%s,%s}" link desc))
         ('ascii (format "%s (%s)" desc link))
         (_ (format "%s (%s)" desc link))))
     (with-eval-after-load 'org
       (org-link-set-parameters
        ,type
        :complete (quote ,(intern (format "my-org-%s-complete" type)))
        :export (quote ,(intern (format "my-org-%s-export" type)))
        :follow (quote ,(intern (format "my-org-%s-follow" type))))
       (cl-pushnew (cons (expand-file-name ,file-path) ,git-url)
                   my-project-web-base-list
                   :test 'equal))))

Then I can define projects this way:

(my-org-project-link "subed"
                     "~/proj/subed/subed/"
                     "https://github.com/sachac/subed/blob/main/subed/"
                     ;; "https://codeberg.org/sachac/subed/src/branch/main/subed/"
                     )
(my-org-project-link "emacsconf-el"
                     "~/proj/emacsconf/lisp/"
                     "https://git.emacsconf.org/emacsconf-el/tree/")
(my-org-project-link "subed-record"
                     "~/proj/subed-record/"
                     "https://github.com/sachac/subed-record/blob/main/"
                     ;; "https://codeberg.org/sachac/subed-record/src/branch/main/"
                     )
(my-org-project-link "compile-media"
                     "~/proj/compile-media/"
                     "https://github.com/sachac/compile-media/blob/main/"
                     ;; "https://codeberg.org/sachac/compile-media/src/branch/main/"
                     )
(my-org-project-link "ox-11ty"
                     "~/proj/ox-11ty/"
                     "https://github.com/sachac/ox-11ty/blob/master/")

And I can complete them with the usual C-c C-l (org-insert-link) process:

completing-custom-links.gif
Figure 1: Completing a custom link with org-insert-link

Sketches are handled by my Org Mode sketch links, but we can add them anyway.

(cl-pushnew (cons (expand-file-name "~/sync/sketches/") "https://sketches.sachachua.com/filename/")
            my-project-web-base-list
            :test 'equal)

I've been really liking being able to refer to various emacsconf-el files by just selecting the link type and completing the filename, so maybe it'll be easier to write about lots of other stuff if I extend that to my other projects.

Quickly search my code

Since my-project-web-base-list is a list of projects I often think about or write about, I can also make something that searches through them. That way, I don't have to care about where my code is.

(defun my-consult-ripgrep-code ()
  (interactive)
  (consult-ripgrep (mapcar 'car my-project-web-base-list)))

I can add .rgignore files in directories to tell ripgrep to ignore things like node_modules or *.json.

I also want to search my Emacs configuration at the same time, although links to my config are handled by my dotemacs link type so I'll leave the URL as nil. This is also the way I can handle other unpublished directories.

(cl-pushnew (cons (expand-file-name "~/sync/emacs/Sacha.org") nil)
            my-project-web-base-list
            :test 'equal)
(cl-pushnew (cons (expand-file-name "~/proj/static-blog/_includes") nil)
            my-project-web-base-list
            :test 'equal)
(cl-pushnew (cons (expand-file-name "~/bin") nil)
            my-project-web-base-list
            :test 'equal)

Actually, let's throw my blog posts and Org files in there as well, since I often have code snippets. If it gets to be too much, I can always have different commands search different things.

(cl-pushnew (cons (expand-file-name "~/proj/static-blog/blog/") "https://sachachua.com/blog/")
            my-project-web-base-list
            :test 'equal)
(cl-pushnew (cons (expand-file-name "~/sync/orgzly") nil)
            my-project-web-base-list
            :test 'equal)
ripgrep-code.gif
Figure 2: Using my-consult-ripgrep-code

I don't have anything bound to M-s c (code) yet, so let's try that.

(keymap-global-set "M-s c" #'my-consult-ripgrep-code)

At some point, it might be fun to get Embark set up so that I can grab a link to something right from the consult-ripgrep interface. In the meantime, I can always jump to it and get the link.

Tip from Omar: embark-around-action-hooks

[2024-01-07 Sun] I modified oantolin's suggestion from the comments to work with consult-ripgrep, since consult-ripgrep gives me consult-grep targets instead of consult-location:

(cl-defun embark-consult--at-location (&rest args &key target type run &allow-other-keys)
  "RUN action at the target location."
  (save-window-excursion
    (save-excursion
      (save-restriction
        (pcase type
          ('consult-location (consult--jump (consult--get-location target)))
          ('org-heading (org-goto-marker-or-bmk (get-text-property 0 'org-marker target)))
          ('consult-grep (consult--jump (consult--grep-position target)))
          ('file (find-file target)))
        (apply run args)))))

(cl-pushnew #'embark-consult--at-location (alist-get 'org-store-link embark-around-action-hooks))

I think I can use it with M-s c to search for the code, then C-. C-c l on the matching line, where C-c l is my regular keybinding for storing links. Thanks, Omar!

In general, I don't want to have to think about where something is on my laptop or where it's published on the Web, I just want to write about it. One step closer, yay Emacs!

This is part of my Emacs configuration.
View org source for this post

Using consult and org-ql to search my Org Mode agenda files and sort the results to prioritize heading matches

| emacs, org

I want to get better at looking in my Org files for something that I don't exactly remember. I might remember a few words from it but not in order, or I might remember some words from the body, or I might need to fiddle with the keywords until I find it.

I usually use C-u C-c C-w (org-refile with a prefix argument), counting on consult + orderless to let me just put in keywords in any order. This doesn't let me search the body, though.

org-ql seems like a great fit for this. It's fast and flexible, and might be useful for all sorts of queries.

I think by default org-ql matches against all of the text in the entry. You can scope the match to just the heading with a query like heading:your,text. I wanted to see all matches, prioritize heading matches so that they come first. I thought about saving the query by adding advice before org-ql-search and then adding a new comparator function, but that got a bit complicated, so I haven't figured that out yet. It was easier to figure out how to rewrite the query to use heading instead of rifle, do the more constrained query, and then append the other matches that weren't in the heading matches.

Also, I wanted something a little like helm-org-rifle's live previews. I've used helm before, but I was curious about getting it to work with consult.

Here's a quick demo of my-consult-org-ql-agenda-jump, which I've bound to M-s a. The top few tasks have org-ql in the heading, and they're followed by the rest of the matches. I think this might be handy.

my-consult-org-ql-agenda-jump.gif
Figure 1: Screencast of using my-consult-org-ql-agenda-jump
(defun my-consult-org-ql-agenda-jump ()
  "Search agenda files with preview."
  (interactive)
  (let* ((marker (consult--read
                  (consult--dynamic-collection
                   #'my-consult-org-ql-agenda-match)
                  :state (consult--jump-state)
                  :category 'consult-org-heading
                  :prompt "Heading: "
                  :sort nil
                  :lookup #'consult--lookup-candidate))
         (buffer (marker-buffer marker))
         (pos (marker-position marker)))
    ;; based on org-agenda-switch-to
    (unless buffer (user-error "Trying to switch to non-existent buffer"))
    (pop-to-buffer-same-window buffer)
    (goto-char pos)
    (when (derived-mode-p 'org-mode)
      (org-fold-show-context 'agenda)
      (run-hooks 'org-agenda-after-show-hook))))

(defun my-consult-org-ql-agenda-format (o)
  (propertize
   (org-ql-view--format-element o)
   'consult--candidate (org-element-property :org-hd-marker o)))

(defun my-consult-org-ql-agenda-match (string)
  "Return candidates that match STRING.
Sort heading matches first, followed by other matches.
Within those groups, sort by date and priority."
  (let* ((query (org-ql--query-string-to-sexp string))
         (sort '(date reverse priority))
         (heading-query (-tree-map (lambda (x) (if (eq x 'rifle) 'heading x)) query))
         (matched-heading
          (mapcar #'my-consult-org-ql-agenda-format
                  (org-ql-select 'org-agenda-files heading-query
                    :action 'element-with-markers
                    :sort sort)))
         (all-matches
          (mapcar #'my-consult-org-ql-agenda-format
                  (org-ql-select 'org-agenda-files query
                    :action 'element-with-markers
                    :sort sort))))
    (append
     matched-heading
     (seq-difference all-matches matched-heading))))

(use-package org-ql
  :bind ("M-s a" . my-consult-org-ql-agenda-jump))

Along the way, I learned how to use consult to complete using consult--dynamic-collection and add consult--candidate so that I can reuse consult--lookup-candidate and consult--jump-state. Neat!

Someday I'd like to figure out how to add a sorting function and sort by headers without having to reimplement the other sorts. In the meantime, this might be enough to help me get started.

This is part of my Emacs configuration.

Automatically refiling Org Mode headings based on tags

| org, emacs

I have lots of different things in my Org Mode inbox. Following the PARA method, I want to file them under projects, areas, resources, or archive so that I can find related things later. Actually, no, I don't want to refile them. I do want to be able to:

  • find all the pieces related to something when I'm ready to start working on a task
  • find useful links again, especially if I can use my own words

Refiling is annoying on my phone, so I tend to wait until I'm back at my computer. But even with org-refile-use-outline-path set to file and the ability to specify substrings, there's still a bit of friction.

Tagging is a little easier to do on my phone. I can add a few tags when I share a webpage or create a task.

I thought it would be nice to have something that automatically refiles my inbox headings tagged with various tags to other subtrees where I've set a :TAG_TARGET: property or something like that. For example, I can set the TAG_TARGET property to emacsconf to mean that anything tagged with :emacsconf: should get filed under there.

https://emacs.stackexchange.com/questions/36360/recursively-refiling-all-subtrees-with-tag-to-a-destination-org-mode

(defcustom my-org-refile-to-ids nil
  "Searches and IDs."
  :group 'sacha
  :type '(repeat (cons string string)))

(defun my-org-update-tag-targets ()
  (interactive)
  (setq my-org-refile-to-ids
        (let (list)
          (org-map-entries
           (lambda ()
             (cons (concat "+" (org-entry-get (point) "TAG_TARGET"))
                   (org-id-get-create)))
           "TAG_TARGET={.}" 'agenda)))
  (customize-save-variable 'my-org-refile-to-ids my-org-refile-to-ids))

(defun my-org-add-tag-target (tag)
  (interactive "MTag: ")
  (org-entry-put (point) "TAG_TARGET" tag)
  (push (cons (concat "+" tag) (org-id-get-create)) my-org-refile-to-ids)
  (customize-save-variable 'my-org-refile-to-ids my-org-refile-to-ids))

;; Based on https://emacs.stackexchange.com/questions/36360/recursively-refiling-all-subtrees-with-tag-to-a-destination-org-mode
(defun my-org-refile-matches-to-heading (match target-heading-id &optional scope copy)
  "Refile all headings within SCOPE (per `org-map-entries') to TARGET-HEADING-ID."
  (if-let (target-marker (org-id-find target-heading-id t))
      (let* ((target-rfloc (with-current-buffer (marker-buffer target-marker)
                             (goto-char target-marker)
                             (list (org-get-heading)
                                   (buffer-file-name (marker-buffer target-marker))
                                   nil
                                   target-marker)))
             (headings-to-copy (org-map-entries (lambda () (point-marker)) match scope)))
        (mapc
         (lambda (heading-marker)
           (with-current-buffer (marker-buffer heading-marker)
             (goto-char heading-marker)
             (org-refile nil nil target-rfloc (when copy "Copy"))))
         (nreverse headings-to-copy))
        (message "%s %d headings!"
                 (if copy "Copied" "Refiled")
                 (length headings-to-copy)))
    (warn "Could not find target heading %S" target-heading-id)))

(defun my-org-refile-to-tag-targets ()
  (interactive)
  (dolist (rule my-org-refile-to-ids)
    (my-org-refile-matches-to-heading (car rule) (cdr rule))))

So when I'm ready, I can call my-org-refile-to-tag-targets and have lots of things disappear from my inbox.

Next step might be to write a function that will refile just the current subtree (either going straight to the tag target or prompting me for a destination if there isn't a matching one), so I can look at stuff, decide if it needs to be scheduled first or something like that, and then send it somewhere. There must be something I can pass a property match to and it'll tell me if it matches the current subtree - probably something along the lines of org-make-tags-matcher

Anyway, just wanted to share this!

This is part of my Emacs configuration.

#EmacsConf backstage: automatically updating talk status from the crontab

| emacs, emacsconf, org

Now that I've figured out how to automatically play EmacsConf talks with crontab, I want to update our approach to using TRAMP and timers to run two tracks semi-automatically.

When a talk starts playing, I would like to:

  • announce it on IRC so that the talk details split up the chat log and make it easier to read
  • publish the media files to https://media.emacsconf.org/2023 (currently password-protected while testing) so that people can view them
  • update the talk wiki page with the media files and the captions
  • update the Youtube and Toobnix videos so that they're public: this is manual for the moment, since I haven't put time into automating it yet

I have code for most of this, and we've done it successfully for the past couple of years. I just need to refamiliarize myself with how to do it and how to set it up for testing during the dry run, and modify it to work with the new crontab-based system.

Setting things up

Last year, I set up the server with the ability to act as the controller, so that it wasn't limited to my laptop. The organizers notebook says I put it in the orga@ user's account, and I probably ran it under a screen session. I've submitted a talk for this year's conference, so I can use a test video for that one. First, I need to update the Ansible configuration for publishing and editing:

ansible-playbook -i inventory.yml prod-playbook.yml --tags publish,edit

I needed to add a version attribute to the git repo checkout in the Ansible playbook, since we'd switched from master to main. I also needed to set emacs_version to 29.1 since I started using seq-keep in my Emacs Lisp functions. For testing, I set emacsconf-publishing-phase to conference.

Act on TODO state changes

org-after-todo-state-change-hook makes it easy to automatically run functions when the TODO state changes. I add this hook that runs a list of functions and passes the talk information so that I don't have to parse the talk info in each function.

emacsconf-org-after-todo-state-change: Run all the hooks in ‘emacsconf-todo-hooks’.
(defun emacsconf-org-after-todo-state-change ()
  "Run all the hooks in `emacsconf-todo-hooks'.
If an `emacsconf-todo-hooks' entry is a list, run it only for the
tracks with the ID in the cdr of that list."
  (let* ((talk (emacsconf-get-talk-info-for-subtree))
         (track (emacsconf-get-track (plist-get talk :track))))
    (mapc
     (lambda (hook-entry)
       (cond
        ((symbolp hook-entry) (funcall hook-entry talk))
        ((member (plist-get track :id) (cdr hook-entry))
         (funcall (car hook-entry) talk))))
     emacsconf-todo-hooks)))

This can be enabled and disabled with the following functions.

emacsconf-add-org-after-todo-state-change-hook: Add FUNC to ‘org-after-todo-stage-change-hook’.
(defun emacsconf-add-org-after-todo-state-change-hook ()
  "Add FUNC to `org-after-todo-stage-change-hook'."
  (interactive)
  (with-current-buffer (find-buffer-visiting emacsconf-org-file)
    (add-hook 'org-after-todo-state-change-hook #'emacsconf-org-after-todo-state-change nil t)))

emacsconf-remove-org-after-todo-state-change-hook: Remove FUNC from ‘org-after-todo-stage-change-hook’.
(defun emacsconf-remove-org-after-todo-state-change-hook ()
  "Remove FUNC from `org-after-todo-stage-change-hook'."
  (interactive)
  (with-current-buffer (find-buffer-visiting emacsconf-org-file)
    (remove-hook 'org-after-todo-state-change-hook
                 #'emacsconf-org-after-todo-state-change  t)))

Announce on IRC

This is still much the same as last year.

emacsconf-erc-announce-on-change: Announce talk.
(defun emacsconf-erc-announce-on-change (talk)
  "Announce talk."
  (let ((func
         (pcase org-state
           ("PLAYING" #'erc-cmd-NOWPLAYING)
           ("CLOSED_Q" #'erc-cmd-NOWCLOSEDQ)
           ("OPEN_Q" #'erc-cmd-NOWOPENQ)
           ("UNSTREAMED_Q" #'erc-cmd-NOWUNSTREAMEDQ)
           ("TO_ARCHIVE" #'erc-cmd-NOWDONE))))
    (when func
      (funcall func talk))))

Here's a sample command that announces that the talk is now playing.

erc-cmd-NOWPLAYING: Set the channel topics to announce TALK.
(defun erc-cmd-NOWPLAYING (talk)
  "Set the channel topics to announce TALK."
  (interactive (list (emacsconf-complete-talk-info)))
  (when (stringp talk)
    (setq talk (or (emacsconf-find-talk-info talk) (error "Could not find talk %s" talk))))
  ;; Announce it in the track's channel
  (if (emacsconf-erc-recently-announced (format "---- %s:" (plist-get talk :slug)))
      (message "Recently announced, skipping")
    (when (plist-get talk :track)
      (emacsconf-erc-with-channels (list (concat "#" (plist-get talk :channel)))
        (erc-cmd-TOPIC
         (format
          "%s: %s (%s) pad: %s Q&A: %s | %s"
          (plist-get talk :slug)
          (plist-get talk :title)
          (plist-get talk :speakers)
          (plist-get talk :pad-url)
          (plist-get talk :qa-info)
          (car (assoc-default
                (concat "#" (plist-get talk :channel))
                emacsconf-topic-templates))))
        (erc-send-message (format "---- %s: %s - %s ----"
                                  (plist-get talk :slug)
                                  (plist-get talk :title)
                                  (plist-get talk :speakers-with-pronouns)))
        (erc-send-message
         (concat "Add your notes/questions to the pad: " (plist-get talk :pad-url)))
        (cond
         ((string-match "live" (or (plist-get talk :q-and-a) ""))
          (erc-send-message (concat "Live Q&A: " (plist-get talk :bbb-redirect))))
         ((plist-get talk :irc)
          (erc-send-message (format "or discuss the talk on IRC (nick: %s)"
                                    (plist-get talk :irc)))))))
    ;; Short announcement in #emacsconf
    (emacsconf-erc-with-channels (list emacsconf-erc-hallway emacsconf-erc-org)
      (erc-send-message (format "-- %s track: %s: %s (watch: %s, pad: %s, channel: #%s)"
                                (plist-get talk :track)
                                (plist-get talk :slug)
                                (plist-get talk :title)
                                (plist-get talk :watch-url)
                                (plist-get talk :pad-url)
                                (plist-get talk :channel))))))

Because the commands change the topic, I need to op the user in all the Emacsconf channels.

erc-cmd-OPALL
(defun erc-cmd-OPALL (&optional nick)
  (emacsconf-erc-with-channels (mapcar 'car emacsconf-topic-templates)
    (if nick
        (erc-cmd-OP nick)
      (erc-cmd-OPME))))

This code is in emacsconf-erc.el.

Publish media files

We used to scramble to upload all the videos in the days or weeks after the conference, since the presentations were live. Since we switched to encouraging speakers to upload videos before the conference, we've been able to release the videos pretty much as soon as the talk starts playing. This code in emacsconf-publish.el takes care of copying the files from the backstage to the public media directory, republishing the index, and republishing the playlist. That way, people who come in late or who want to refer to the video can easily get the full video right away.

emacsconf-publish-media-files-on-change: Publish the files and update the index.
(defun emacsconf-publish-media-files-on-change (talk)
  "Publish the files and update the index."
  (interactive (list (emacsconf-complete-talk-info)))
  (let ((org-state (if (boundp 'org-state) org-state (plist-get talk :status))))
    (if (plist-get talk :public)
        ;; Copy main extension files from backstage to public
        (let ((files (directory-files emacsconf-backstage-dir nil
                                      (concat "^"
                                              (regexp-quote (plist-get talk :file-prefix))
                                              (regexp-opt emacsconf-main-extensions)))))
          (mapc (lambda (file)
                  (when (and
                         (not (file-exists-p (expand-file-name file emacsconf-public-media-directory)))
                         (or (not (string-match "--main.vtt$" file))
                             (plist-get talk :captions-edited)))
                    (copy-file (expand-file-name file emacsconf-backstage-dir)
                               (expand-file-name file emacsconf-public-media-directory) t)))
                files))
      ;; Remove files from public
      (let ((files (directory-files emacsconf-public-media-directory nil
                                    (concat "^"
                                            (regexp-quote (plist-get talk :file-prefix)
                                                          )))))
        (mapc (lambda (file)
                (delete-file (expand-file-name file emacsconf-public-media-directory)))
              files)))
    (emacsconf-publish-public-index)
    (emacsconf-publish-playlist
     (expand-file-name "index.m3u" emacsconf-public-media-directory)
     (concat emacsconf-name " " emacsconf-year)
     (emacsconf-public-talks (emacsconf-get-talk-info)))))

The :public property is automatically added by this function based on the TODO status or the time:

emacsconf-add-talk-status: Add status label and public info.
(defun emacsconf-add-talk-status (o)
  "Add status label and public info."
  (plist-put o :status-label
             (or (assoc-default (plist-get o :status)
                                emacsconf-status-types 'string= "")
                 (plist-get o :status)))
  (when (or
         (member (plist-get o :status)
                 (split-string "PLAYING CLOSED_Q OPEN_Q UNSTREAMED_Q TO_ARCHIVE TO_EXTRACT TO_FOLLOW_UP DONE"))
         (time-less-p (plist-get o :start-time)
                      (current-time)))
    (plist-put o :public t))
  o)

Update the wiki page

This function updates the schedule page and the page for the talk. It's also in emacsconf-publish.el.

emacsconf-publish-update-talk: Publish the schedule page and the page for this talk.
(defun emacsconf-publish-update-talk (talk)
  "Publish the schedule page and the page for this talk."
  (interactive (list (emacsconf-complete-talk-info)))
  (when (stringp talk) (setq talk (emacsconf-resolve-talk talk)))
  (when (functionp 'emacsconf-upcoming-insert-or-update)
    (emacsconf-upcoming-insert-or-update))
  (emacsconf-publish-with-wiki-change
    (let ((info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
      (emacsconf-publish-before-page talk info)
      (emacsconf-publish-after-page talk info)
      (emacsconf-publish-schedule info))))

It uses the publishing functions I described in this post on adding talks to the wiki.

This macro commits changes when emacsconf-publish-autocommit-wiki is t, so I need to set that also.

emacsconf-publish-with-wiki-change
(defmacro emacsconf-publish-with-wiki-change (&rest body)
  (declare (indent 0) (debug t))
  `(progn
     ,@body
     (emacsconf-publish-commit-and-push-wiki-maybe
      ,emacsconf-publish-autocommit-wiki
      (and (stringp ,(car body)) ,(car body)))))

Make the videos public on YouTube and Toobnix

This is low-priority, but it might be nice to figure out. The easiest way is probably to use open the Youtube/Toobnix URLs on my computer and then use either Tampermonkey or Spookfox to set the talk to public. Someday!

Update the talk status on the server

Last year, I experimented with having the shell scripts automatically update the status of the talk from TO_STREAM to PLAYING and from PLAYING to CLOSED_Q. Since I've moved the talk-running into track-specific crontabs, now I need to sudo back to the orga user and set XDG_RUNTIME_DIR in order to use emacsclient. I can call this with sudo -u orga talk $slug $status in the roles/obs/templates/handle-session script.

Here's the Ansible template for roles/prerec/templates/talk. It uses getent to look up the user ID.

#!/bin/bash
# 
# How to use: talk slug from-status-regexp to-status
# or talk slug to-status

SLUG="$1"
FROM_STATUS="$2"
TO_STATUS="$3"
XDG_RUNTIME_DIR=/run/user/

if [ "x$TO_STATUS" == "x" ]; then
    FROM_STATUS=.
    TO_STATUS="$2"
fi
cd 
#echo "Pulling conf.org..."
#git pull
echo "Updating status..."
XDG_RUNTIME_DIR=$XDG_RUNTIME_DIR emacsclient --eval "(emacsconf-with-todo-hooks (emacsconf-update-talk-status \"$SLUG\" \"$FROM_STATUS\" \"$TO_STATUS\"))" -a emacs
#echo "Committing and pushing in the background"
#git commit -m "Update task status for $SLUG from $FROM_STATUS to $TO_STATUS" conf.org
#git push &

Testing notes

Looks like everything works fine when I run it from the crontab: the talk status is updated, the media files are published, the wiki is updated, and the talks are announced on IRC. Backup plan A is to manually control the talk status using Emacs on the server. Backup plan B is to control the talk status using Emacs on my laptop. Backup plan C is to call the individual functions instead of relying on the todo state change functions. I think it'll all work out, although I'll probably want to do another dry run at some point to make sure. Slowly getting there…

Getting Mermaid JS and ob-mermaid running on my system - needed to symlink Chromium for Puppeteer

| emacsconf, nodejs, org, emacs

I wanted to use Mermaid to make diagrams, but I ran into this issue when trying to run it:

Error: Could not find Chromium (rev. 1108766). This can occur if either
 1. you did not perform an installation before running the script (e.g. `npm install`) or
 2. your cache path is incorrectly configured (which is: /home/sacha/.cache/puppeteer).
For (2), check out our guide on configuring puppeteer at https://pptr.dev/guides/configuration.

It turns out that I needed to do the following:

sudo npm install -g puppeteer mermaid @mermaid-js/mermaid-cli --unsafe-perm
# Cache Chromium for my own user
node /usr/lib/node_modules/puppeteer/install.js --unsafe-perm
sudo npm install -g mermaid @mermaid-js/mermaid-cli
ln -s ~/.cache/puppeteer/chrome/linux-117.0.5938.149 ~/.cache/puppeteer/chrome/linux-1108766
ln -s ~/.cache/puppeteer/chrome/linux-117.0.5938.149/chrome-linux64 ~/.cache/puppeteer/chrome/linux-117.0.5938.149/chrome-linux

(The exact versions might be different for your installation.)

Then I could make a Mermaid file and try it out with mmdc -i input.mmd -o output.svg, and then I could confirm that it works directly from Org with ob-mermaid:

sequenceDiagram
    input ->> res: original.mp4;
    res ->> backstage: reencoded.webm;
    backstage ->> laptop: cached files;
    laptop ->> backstage: index;
    input ->> res: main.vtt;
    res ->> backstage: main.webm;
    backstage ->> laptop: cached files;
    laptop ->> backstage: index;
    input ->> res: normalized.opus;
    res ->> backstage: main.webm;
    backstage ->> laptop: cached files;
    laptop ->> backstage: index;
    backstage ->> media: public files;
media-flow.png