Let's start with getting the sketches. I usually export my sketches as PNGs from my Supernote A5X. But if I know that I'm going to animate a sketch, I can export it as a PDF. I've recently been experimenting with Adobe Fresco on the iPad, which can also export to PDF. The PDF I get from Fresco is easier to animate, but I prefer to draw on the Supernote because it's an e-ink device (and because the kiddo usually uses the iPad).

If I start with a PNG, I could use Inkscape to trace the PNG and turn it into an SVG. I think Inkscape uses autotrace behind the scenes. I don't usually put my highlights on a separate layer, so autotrace will make odd shapes.

It's a lot easier if you start off with vector graphics in the first place. I can export a vector PDF from the SuperNote A5X and either import it into Inkscape using the Cairo option or use the command-line pdftocairo tool.

I've been looking into using Adobe Fresco, which is a free app available for the iPad. Fresco's PDF export can be converted to an SVG using Inkscape or PDF to Cairo. What I like about the output of this app is that it gives me individual elements as their own paths and they're listed in order of drawing. This makes it really easy to animate by just going through the paths in order.

Here's a sample SVG file that pdfcairo creates from an Adobe Fresco PDF export:

pdftocairo -svg ~/Downloads/subed-audio.pdf ~/Downloads/subed-audio.svg

Sample SVG

Adobe Fresco also includes built-in time-lapse, but since I often like to move things around or tidy things up, it's easier to just work with the final image, export it as a PDF, and convert it to an SVG.

I can make a very simple animation by setting the opacity of all the paths to 0, then looping through the elements to set the opacity back to 1 and write that version of the SVG to a separate file. From how-can-i-generate-png-frames-that-step-through-the-highlights:

(defun my-animate-svg-paths (filename output-dir)
  "Add one path at a time. Save the resulting SVGs to OUTPUT-DIR."
  (unless (file-directory-p output-dir)
    (make-directory output-dir t))
  (let* ((dom (xml-parse-file filename))
         (paths (seq-filter (lambda (e) (dom-attr e 'style))
                            (dom-by-tag dom 'path)))
         (total (length paths))
         (frame-num (length paths))
         result)
    (dolist (elem paths)
      (dom-set-attribute elem 'style
                         (concat
                          (dom-attr elem 'style)
                          ";mix-blend-mode:darken")))
    (with-temp-file (expand-file-name (format "frame-%03d.svg" (1+ frame-num)) output-dir)
      (xml-print dom))
    (dolist (elem paths)
      (dom-set-attribute elem 'style
                         (concat
                          (dom-attr elem 'style)
                          ";fill-opacity:0")))
    (dolist (elem paths)
      (with-temp-file (expand-file-name
                       (format "frame-%03d.svg"
                               (- total frame-num))
                       output-dir)
        (message "%03d" frame-num)
        (dom-set-attribute elem 'style
                           (concat (dom-attr elem 'style)
                                   ";fill-opacity:1"))
        (push (list (format "frame-%03d.svg"
                            (1+ (- total frame-num)))
                    (dom-attr elem 'id))
              result)
        (setq frame-num (1- frame-num))
        (xml-print dom)))
    (reverse result)))

Here's how I call it:

(my-animate-svg-paths "~/Downloads/subed-audio.svg" "/tmp/subed-audio/frames" t)

Then I can use FFmpeg to combine all of those frames into a video:

ffmpeg -i frame-%03d.svg -vf palettegen -y palette.png
ffmpeg -framerate 30 -i frame-%03d.svg -i palette.png -lavfi "paletteuse" -loop 0 -y animation-loop.gif

Neither Supernote nor Adobe Fresco give me the original stroke information. These are filled shapes, so I can't animate something drawing it. But having different elements appear in sequence is fine for my purposes. If you happen to know how to get stroke information out of Supernote .note files or of an iPad app that exports nice single-line SVGs that have stroke direction, I would love to hear about it.

When I export a PDF from Supernote and convert it to an SVG, each color is a combined shape with all the elements. If I want to animate parts of the image, I have to break it up and recombine selected elements (Inkscape's Ctrl-k shortcut) so that the holes in shapes are properly handled. This is a bit of a tedious process and it usually ends up with elements in a pretty random order. Since I have to reorder elements by hand, I don't really want to animate the sketch letter-by-letter. Instead, I combine them into larger chunks like topics or paragraphs.

The following code takes the PDF, converts it to an SVG, recolours highlights, and then breaks up paths into elements:

(defun my-sketch-convert-pdf-and-break-up-paths (pdf-file &optional rotate)
  "Convert PDF to SVG and break up paths."
  (interactive (list (read-file-name
                      (format "PDF (%s): "
                              (my-latest-file "~/Dropbox/Supernote/EXPORT/" "pdf"))
                      "~/Dropbox/Supernote/EXPORT/"
                      (my-latest-file "~/Dropbox/Supernote/EXPORT/" "pdf")
                      t
                      nil
                      (lambda (s) (string-match "pdf" s)))))
  (unless (file-exists-p (concat (file-name-sans-extension pdf-file) ".svg"))
    (call-process "pdftocairo" nil nil nil "-svg" (expand-file-name pdf-file)
                  (expand-file-name (concat (file-name-sans-extension pdf-file) ".svg"))))
  (let ((dom (xml-parse-file (expand-file-name (concat (file-name-sans-extension pdf-file) ".svg"))))
        highlights)
    (setq highlights (dom-node 'g '((id . "highlights"))))
    (dom-append-child dom highlights)
    (dolist (path (dom-by-tag dom 'path))
      ;;  recolor and move
      (unless (string-match (regexp-quote "rgb(0%,0%,0%)") (or (dom-attr path 'style) ""))
        (dom-remove-node dom path)
        (dom-append-child highlights path)
        (dom-set-attribute
         path 'style
         (replace-regexp-in-string
          (regexp-quote "rgb(78.822327%,78.822327%,78.822327%)")
          "#f6f396"
          (or (dom-attr path 'style) ""))))
      (let ((parent (dom-parent dom path)))
        ;; break apart
        (when (dom-attr path 'd)
          (dolist (part (split-string (dom-attr path 'd) "M " t " +"))
            (dom-append-child
             parent
             (dom-node 'path `((style . ,(dom-attr path 'style))
                               (d . ,(concat "M " part))))))
          (dom-remove-node dom path))))
    ;; remove the use
    (dolist (use (dom-by-tag dom 'use))
      (dom-remove-node dom use))
    (dolist (use (dom-by-tag dom 'image))
      (dom-remove-node dom use))
    ;; move the first g down
    (let ((g (car (dom-by-id dom "surface1"))))
      (setf (cddar dom)
            (seq-remove (lambda (o)
                          (and (listp o) (string= (dom-attr o 'id) "surface1")))
                        (dom-children dom)))
      (dom-append-child dom g)
      (when rotate
        (let* ((old-width (dom-attr dom 'width))
               (old-height (dom-attr dom 'height))
               (view-box (mapcar 'string-to-number (split-string (dom-attr dom 'viewBox))))
               (rotate (format "rotate(90) translate(0 %s)" (- (elt view-box 3)))))
          (dom-set-attribute dom 'width old-height)
          (dom-set-attribute dom 'height old-width)
          (dom-set-attribute dom 'viewBox (format "0 0 %d %d" (elt view-box 3) (elt view-box 2)))
          (dom-set-attribute highlights 'transform rotate)
          (dom-set-attribute g 'transform rotate))))
    (with-temp-file (expand-file-name (concat (file-name-sans-extension pdf-file) "-split.svg"))
      (svg-print (car dom)))))

You can see how the spaces inside letters like "o" end up being black. Selecting and combining those paths fixes that.

If there were shapes that were touching, then I need to draw lines and fracture the shapes in order to break them apart.

The end result should be an SVG with the different chunks that I might want to animate, but I need to identify the paths first. You can assign object IDs in Inkscape, but this is a bit of an annoying process since I haven't figured out a keyboard-friendly way to set object IDs. I usually find it easier to just set up an Autokey shortcut (or AutoHotkey in Windows) to click on the ID text box so that I can type something in.

import time
x, y = mouse.get_location()
# Use the coordinates of the ID text field on your screen; xev can help
mouse.click_absolute(3152, 639, 1)
time.sleep(1)
keyboard.send_keys("<ctrl>+a")
mouse.move_cursor(x, y)

Then I can select each element, press the shortcut key, and type an ID into the textbox. I might use "t-…" to indicate the text for a map section, "h-…" to indicate a highlight, and arrows by specifying their start and end.

To simplify things, I wrote a function in Emacs that will go through the different groups that I've made, show each path in a different color and with a reasonable guess at a bounding box, and prompt me for an ID. This way, I can quickly assign IDs to all of the paths. The completion is mostly there to make sure I don't accidentally reuse an ID, although it can try to combine paths if I specify the ID. It saves the paths after each change so that I can start and stop as needed. Identifying paths in Emacs is usually much nicer than identifying them in Inkscape.

(defun my-svg-identify-paths (filename)
  "Prompt for IDs for each path in FILENAME."
  (interactive (list (read-file-name "SVG: " nil nil
                                     (lambda (f) (string-match "\\.svg$" f)))))
  (let* ((dom (car (xml-parse-file filename)))
         (paths (dom-by-tag dom 'path))
         (vertico-count 3)
         (ids (seq-keep (lambda (path)
                          (unless (string-match "path[0-9]+" (or (dom-attr path 'id) "path0"))
                            (dom-attr path 'id)))
                        paths))
         (edges (window-inside-pixel-edges (get-buffer-window)))
         id)
    (my-svg-display "*image*" dom nil t)
    (dolist (path paths)
      (when (string-match "path[0-9]+" (or (dom-attr path 'id) "path0"))
        ;; display the image with an outline
        (unwind-protect
            (progn
              (my-svg-display "*image*" dom (dom-attr path 'id) t)
              (setq id (completing-read
                        (format "ID (%s): " (dom-attr path 'id))
                        ids))
              ;; already exists, merge with existing element
              (if-let ((old (dom-by-id dom id)))
                  (progn
                    (dom-set-attribute
                     old
                     'd
                     (concat (dom-attr (dom-by-id dom id) 'd)
                             " "
                             ;; change relative to absolute
                             (replace-regexp-in-string "^m" "M"
                                                       (dom-attr path 'd))))
                    (dom-remove-node dom path)
                    (setq id nil))
                (dom-set-attribute path 'id id)
                (add-to-list 'ids id))))
        ;; save the image just in case we get interrupted halfway through
        (with-temp-file filename
          (svg-print dom))))))

Then I can animate SVGs by specifying the IDs. I can reorder the paths in the SVG itself so that I can animate it group by group, like the way that the Adobe Fresco SVGs were animated element by element.

(my-svg-reorder-paths "~/proj/2023-12-audio-workflow/map.svg"
                      '("t-start" "h-audio" "h-capture" "t-but" "t-mic" "h-mic"
                        "t-reviewing" "h-reviewing"
                        "t-words" "h-words" "t-workflow" "h-workflow"
                        "t-lapel" "h-lapel"
                        "mic-recorder" "t-recorder" "h-recorder"
                        "t-syncthing" "h-sync"
                        "t-keywords" "h-keywords" "t-keyword-types"
                        "t-lines" "h-lines"
                        "t-align" "h-align"
                        "arrow"
                        "t-org" "h-org" "t-todo" "h-todo" "h-linked"
                        "t-jump" "h-jump"
                        "t-waveform" "h-waveform"
                        "t-someday"
                        "h-sections"
                        "t-speech-recognition" "h-speech-recognition"
                        "t-ai" "h-ai"
                        "t-summary"
                        "extra")
                      "~/proj/2023-12-audio-workflow/map-output.svg")
(my-animate-svg-paths "~/proj/2023-12-audio-workflow/map-output.svg"
                      "~/proj/2023-12-audio-workflow/frames/")

The table of filenames makes it easy to use specific frames as part of a presentation or video.

Here is the result as a video:

(let ((compile-media-output-video-width 1280)
      (compile-media-output-video-height 720))
  (my-ffmpeg-animate-images
   (directory-files "~/proj/2023-12-audio-workflow/frames/" t "\\.svg$")
   (expand-file-name "~/proj/2023-12-audio-workflow/frames/animation.webm")
   4))

The way it works is that the my-svg-reorder-paths function removes and readds elements following the list of IDs specified, so everything's ready to go for step-by-step animation. Here's the code:

(defun my-svg-reorder-paths (filename &optional ids output-filename)
  "Sort paths in FILENAME."
  (interactive (list (read-file-name "SVG: " nil nil (lambda (f) (string-match "\\.svg$" f)))
                     nil (read-file-name "Output: ")))
  (let* ((dom (car (xml-parse-file filename)))
         (paths (dom-by-tag dom 'path))
         (parent (dom-parent dom (car paths)))
         (ids-left
          (nreverse (seq-keep (lambda (path)
                                (unless (string-match "path[0-9]+" (or (dom-attr path 'id) "path0"))
                                  (dom-attr path 'id)))
                              paths)))
         list)
    (when (called-interactively-p)
      (while ids-left
        (my-svg-display "*image*" dom (car ids-left))
        (let ((current (completing-read
                        (format "ID (%s): "
                                (car ids-left))
                        ids-left nil nil nil nil (car ids-left)))
              node)
          (add-to-list 'ids current)
          (setq ids-left (seq-remove (lambda (o) (string= o current)) ids-left)))))
    (if ids ;; reorganize under the first path's parent
        (progn
          (dolist (id ids)
            (if-let ((node (car (dom-by-id dom id))))
                (progn
                  (dom-remove-node dom node)
                  (dom-append-child parent node))
              (message "Could not find %s" id)))
          (with-temp-file (or output-filename filename)
            (svg-print dom))))
    (nreverse (seq-keep (lambda (path)
                          (unless (string-match "path[0-9]+" (or (dom-attr path 'id) "path0"))
                            (dom-attr path 'id)))
                        (dom-by-tag dom 'path)))))

I can also use CSS rules to transition between opacity values for more complex animations. For my EmacsConf 2023 presentation, I wanted to make a self-paced, narrated presentation so that people could follow hyperlinks, read the source code, and explore. I wanted to include a map so that I could try to make sense of everything. For this map, I wanted to highlight the previous sections that were connected to the topic for the current section.

I used a custom Org link to include the full contents of the SVG instead of just including it with an img tag.

#+ATTR_HTML: :class r-stretch
my-include:~/proj/emacsconf-2023-emacsconf/map.svg?wrap=export html

(defun my-include-export (path _ format _)
  "Export PATH to FORMAT using the specified wrap parameter."
  (let (params body start end)
    (when (string-match "^\\(.*+?\\)\\(?:::\\|\\?\\)\\(.*+\\)" path)
      (setq params (save-match-data (org-protocol-convert-query-to-plist (match-string 2 path)))
            path (match-string 1 path)))
    (with-temp-buffer
      (insert-file-contents-literally path)
      (when (string-match "\\.org$" path)
        (org-mode))
      (if (plist-get params :name)
          (when (org-babel-find-named-block (plist-get params :name))
            (goto-char (org-babel-find-named-block (plist-get params :name)))
            (let ((block (org-element-context)))
              (setq start (org-element-begin block)
                    end (org-element-end block))))
        (goto-char (point-min))
        (when (plist-get params :from-regexp)
          (re-search-forward (url-unhex-string (plist-get params :from-regexp)))
          (goto-char (match-beginning 0)))
        (setq start (point))
        (setq end (point-max))
        (when (plist-get params :to-regexp)
          (re-search-forward (url-unhex-string (plist-get params :to-regexp)))
          (setq end (match-beginning 0))))
      (setq body (buffer-substring start end)))
    (with-temp-buffer
      (when (plist-get params :wrap)
        (let* ((wrap (plist-get params :wrap))
               block args)
          (when (string-match "\\<\\(\\S-+\\)\\( +.*\\)?" wrap)
            (setq block (match-string 1 wrap))
            (setq args (match-string 2 wrap))
            (setq body (format "#+BEGIN_%s%s\n%s\n#+END_%s\n"
                               block (or args "")
                               body
                               block)))))
      (when (plist-get params :summary)
        (setq body (format "#+begin_my_details %s\n%s\n#+end_my_details\n"
                           (plist-get params :summary)
                           body)))
      (insert body)
      (org-export-as format nil nil t))))

I wanted to be able to specify the entire sequence using a table in the Org Mode source for my presentation. Each row had the slide ID, a list of highlights in the form prev1,prev2;current, and a comma-separated list of elements to add to the full-opacity view.

Reveal.js adds a "current" class to the slide, so I can use that as a trigger for the transition. I have a bit of Emacs Lisp code that generates some very messy CSS, in which I specify the ID of the slide, followed by all of the elements that need their opacity set to 1, and also specifying the highlights that will be shown in an animated way.

(defun my-reveal-svg-progression-css (map-progression &optional highlight-duration)
  "Make the CSS.
map-progression should be a list of lists with the following format:
((\"slide-id\" \"prev1,prev2;cur1\" \"id-to-add1,id-to-add2\") ...)."
  (setq highlight-duration (or highlight-duration 2))
  (let (full)
    (format
     "<style>%s</style>"
     (mapconcat
      (lambda (slide)
        (setq full (append (split-string (elt slide 2) ",") full))
        (format "#slide-%s.present path { opacity: 0.2 }
%s { opacity: 1 !important }
%s"
                (car slide)
                (mapconcat (lambda (id) (format "#slide-%s.present #%s" (car slide) id))
                           full
                           ", ")
                (my-reveal-svg-highlight-different-colors slide)))
      map-progression
      "\n"))))

#+NAME: progression-css
#+begin_src emacs-lisp :exports code :var map-progression=progression :var highlight-duration=2 :results silent
(my-reveal-svg-progression-css map-progression highlight-duration)
#+end_src

Here's an excerpt showing the kind of code it makes:

<style>#slide-props-map.present path { opacity: 0.2 }
#slide-props-map.present #t-email, #slide-props-map.present #email-properties, #slide-props-map.present #t-properties { opacity: 1 !important }
#slide-props-map.present #h-email { fill: #c6c6c6; opacity: 1 !important; transition: fill 0.5s; transition-delay: 0.0s }#slide-props-map.present #h-properties { fill: #f6f396; opacity: 1 !important; transition: fill 0.5s; transition-delay: 0.5s }
#slide-file-prefixes.present path { opacity: 0.2 }
#slide-file-prefixes.present #t-filename, #slide-file-prefixes.present #properties-filename, #slide-file-prefixes.present #t-email, #slide-file-prefixes.present #email-properties, #slide-file-prefixes.present #t-properties { opacity: 1 !important }
#slide-file-prefixes.present #h-properties { fill: #c6c6c6; opacity: 1 !important; transition: fill 0.5s; transition-delay: 0.0s }#slide-file-prefixes.present #h-filename { fill: #f6f396; opacity: 1 !important; transition: fill 0.5s; transition-delay: 0.5s }
...</style>

Since it's automatically generated, I don't have to worry about it once I've gotten it to work. It's all hidden in a results drawer. So this CSS highlights specific parts of the SVG with a transition, and the highlight changes over the course of a second or two. It highlights the previous names and then the current one. The topics I'd already discussed would be in black, and the topics that I had yet to discuss would be in very light gray. This could give people a sense of the progress through the presentation.

(defun my-reveal-svg-animation (slide)
  (string-join
   (seq-map-indexed
    (lambda (step-ids i)
      (format "%s { fill: #f6f396; transition: fill %ds; transition-delay: %ds }"
              (mapconcat
               (lambda (id) (format "#slide-%s.present #%s" (car slide) id))
               (split-string step-ids ",")
               ", ")
              highlight-duration
              (* i highlight-duration)))
    (split-string (elt slide 1) ";"))
   "\n"))

(defun my-reveal-svg-highlight-different-colors (slide)
  (let* ((colors '("#f6f396" "#c6c6c6")) ; reverse
         (steps (split-string (elt slide 1) ";"))
         (step-length 0.5))
    (string-join
     (seq-map-indexed
      (lambda (step-ids i)
        (format "%s { fill: %s; opacity: 1 !important; transition: fill %.1fs; transition-delay: %.1fs }"
                (mapconcat
                 (lambda (id) (format "#slide-%s.present #%s" (car slide) id))
                 (split-string step-ids ",")
                 ", ")
                (elt colors (- (length steps) i 1))
                step-length
                (* i 0.5)))
      steps))))

(defun my-reveal-svg-progression-css (map-progression &optional highlight-duration)
  "Make the CSS.
map-progression should be a list of lists with the following format:
((\"slide-id\" \"prev1,prev2;cur1\" \"id-to-add1,id-to-add2\") ...)."
  (setq highlight-duration (or highlight-duration 2))
  (let (full)
    (format
     "<style>%s</style>"
     (mapconcat
      (lambda (slide)
        (setq full (append (split-string (elt slide 2) ",") full))
        (format "#slide-%s.present path { opacity: 0.2 }
%s { opacity: 1 !important }
%s"
                (car slide)
                (mapconcat (lambda (id) (format "#slide-%s.present #%s" (car slide) id))
                           full
                           ", ")
                (my-reveal-svg-highlight-different-colors slide)))
      map-progression
      "\n"))))

As a result, as I go through my presentation, the image appears to build up incrementally, which is the effect that I was going for. I can test this by exporting only my map slides:

(save-excursion
  (goto-char (org-babel-find-named-block "progression-css"))
  (org-babel-execute-src-block))
(let ((org-tags-exclude-from-inheritance "map")
      (org-export-select-tags '("map")))
   (oer-reveal-export-to-html))

• Graphviz, mermaid-js, and other diagramming tools can make SVGs. I should be able to adapt my code to animate those diagrams by adding other elements in addition to path. Then I'll be able to make diagrams even more easily.
• Since SVGs can contain CSS, I could make an SVG equivalent of the CSS rules I used for the presentation, maybe calling a function with a Lisp expression that specifies the operations (ex: ("frame-001.svg" "h-foo" opacity 1)). Then I could write frames to SVGs.
• FFmpeg has a crossfade filter. With a little bit of figuring out, I should be able to make the same kind of animation in a webm form that I can include in my regular videos instead of using Reveal.js and CSS transitions.
• I've also been thinking about automating the recording of my Reveal.js presentations. For my EmacsConf talk, I opened my presentation, started the recording with the system audio and the screen, and then let it autoplay the presentation. I checked on it periodically to avoid the screensaver/energy saving things from kicking in and so that I could stop the recording when it's finished. If I want to make this take less work, one option is to use ffmpeg's "-t" argument to specify the expected duration of the presentation so that I don't have to manually stop it. I'm also thinking about using Puppeteer to open the presentation, check when it's fully loaded, and start the process to record it - maybe even polling to see whether it's finished. I haven't gotten around to it yet. Anyhow, those are some ideas to explore next time.
• As for animation, I'm still curious about the possibility of finding a way to access the raw stroke information if it's even available from my Supernote A5X (difficult because it's a proprietary data format) or finding an app for the iPad that exports single line SVGs that use stroke information instead of fill. That would only be if I wanted to do those even fancier animations that look like the whole thing is being drawn for you. I was trying to figure out if I could green screen the Adobe Fresco timelapse videos so that even if I have a pre-sketch to figure out spacing and remind me what to draw, I can just export the finished elements. But there's too much anti-aliasing and I haven't figured out how to do it cleanly yet. Maybe some other day.
• I use Google Cloud Vision's text detection engine to convert my handwriting to text. It can give me bounding polygons for words or paragraphs. I might be able to figure out which curves are entirely within a word's bounding polygon and combine those automatically.
• It would be pretty cool if I could combine the words recognized by Google Cloud Vision with the word-level timestamps from speech recognition so that I could get word-synced sketchnote animations with maybe a little manual intervention.

Anyway, those are some workflows for animating sketches with Inkscape and Emacs. Yay Emacs!

I wanted to write a report on EmacsConf 2023 so that we could share it with speakers, volunteers, participants, donors, related organizations like the Free Software Foundation, and other communities. I experimented with livestreaming via YouTube while I worked on the conference highlights.

It's a little over an hour long and probably very boring, but it was nice of people to drop by and say hello.

The main parts are:

• 0:00: reading through other conference reports for inspiration
• 6:54: writing an overview of the talks
• 13:10: adding quotes for specific talks
• 25:00: writing about the overall conference
• 32:00: squeezing in more highlights
• 49:00: fiddling with the formatting and the export

It mostly worked out, aside from a brief moment of "uhhh, I'm looking at our private conf.org file on stream". Fortunately, the e-mail addresses that were showed were the public ones.

Setup:

• I set up environment variables and screen resolution:

    # From pacmd list-sources | egrep '^\s+name'
    LAPEL=alsa_input.usb-Jieli_Technology_USB_Composite_Device_433035383239312E-00.mono-fallback #
    YETI=alsa_input.usb-Blue_Microphones_Yeti_Stereo_Microphone_REV8-00.analog-stereo
    SYSTEM=alsa_output.pci-0000_00_1b.0.analog-stereo.monitor
    # MIC=$LAPEL
    # AUDIO_WEIGHTS="1 1"
    MIC=$YETI
    AUDIO_WEIGHTS="0.5 0.5"
    OFFSET=+1920,430
    SIZE=1280x720
    SCREEN=LVDS-1  # from xrandr
    xrandr --output $SCREEN --mode 1280x720
  

• I switch to a larger size and a light theme. I also turn consult previews off to minimize the risk of leaking data through buffer previews.
  my-emacsconf-prepare-for-screenshots: Set the resolution, change to a light theme, and make the text bigger.

  (defun my-emacsconf-prepare-for-screenshots ()
    (interactive)
    (shell-command "xrandr --output LVDS-1 --mode 1280x720")
    (modus-themes-load-theme 'modus-operandi)
    (my-hl-sexp-update-overlay)
    (set-face-attribute 'default nil :height 170)
    (keycast-mode))
  

Testing:

ffmpeg -f x11grab -video_size $SIZE -i :0.0$OFFSET -y /tmp/test.png; display /tmp/test.png
ffmpeg -f pulse -i $MIC -f pulse -i $SYSTEM -filter_complex amix=inputs=2:weights=$AUDIO_WEIGHTS:duration=longest:normalize=0 -y /tmp/test.mp3; mpv /tmp/test.mp3
DATE=$(date "+%Y-%m-%d-%H-%M-%S")
ffmpeg -f x11grab -framerate 30 -video_size $SIZE -i :0.0$OFFSET -f pulse -i $MIC -f pulse -i $SYSTEM -filter_complex "amix=inputs=2:weights=$AUDIO_WEIGHTS:duration=longest:normalize=0" -c:v libx264 -preset fast -maxrate 690k -bufsize 2000k -g 60 -vf format=yuv420p -c:a aac -b:a 96k -y -flags +global_header "/home/sacha/recordings/$DATE.flv" -f flv

Streaming:

DATE=$(date "+%Y-%m-%d-%H-%M-%S")
ffmpeg -f x11grab -framerate 30 -video_size $SIZE -i :0.0$OFFSET -f pulse -i $MIC -f pulse -i $SYSTEM -filter_complex "amix=inputs=2:weights=$AUDIO_WEIGHTS:duration=longest:normalize=0[audio]" -c:v libx264 -preset fast -maxrate 690k -bufsize 2000k -g 60 -vf format=yuv420p -c:a aac -b:a 96k -y -f tee -map 0:v -map '[audio]' -flags +global_header  "/home/sacha/recordings/$DATE.flv|[f=flv]rtmp://a.rtmp.youtube.com/live2/$YOUTUBE_KEY"

To restore my previous setup:

(defun my-emacsconf-back-to-normal ()
  (interactive)
  (shell-command "xrandr --output LVDS-1 --mode 1366x768")
  (modus-themes-load-theme 'modus-vivendi)
  (my-hl-sexp-update-overlay)
  (set-face-attribute 'default nil :height 115)
  (keycast-mode -1))

I can think of a few workflow tweaks that might be fun:

• a stream notes buffer on the right side of the screen for context information, timestamped notes to make editing/review easier (maybe using org-timer), etc. I experimented with some streaming-related code in my config, so I can dust that off and see what that's like. I also want to have an org-capture template for it so that I can add notes from anywhere.
• a quick way to add a screenshot from my Supernote to my Org files

I think I'll try going through an informal presentation or Emacs News as my next livestream experiment, since that's probably higher information density.

I didn't know how to do any of that yet with ffmpeg, so here's how I started figuring things out. First, I wanted to see how to fade test.jpg into test2.jpg over 4 seconds.

ffmpeg -y -loop 1 -i test.jpg -loop 1 -i test2.jpg -filter_complex "[1:v]fade=t=in:d=4:alpha=1[fadein];[0:v][fadein]overlay[out]" -map "[out]" -r 1 -t 4 -shortest test.webm

Here's another way using the blend filter:

ffmpeg -y -loop 1 -i test.jpg -loop 1 -i test2.jpg -filter_complex "[1:v][0:v]blend=all_expr='A*(if(gte(T,4),1,T/4))+B*(1-(if(gte(T,4),1,T/4)))" -t 4 -r 1 test.webm

Then I looked into chromakeying in the other video. I used balloons instead of sparkles just in case she happened to look at my screen.

ffmpeg -y -i test.webm -i balloons.mp4 -filter_complex "[1:v]chromakey=0x00ff00:0.1:0.2[ckout];[0:v][ckout]overlay[out]" -map "[out]" -shortest -r 1 overlaid.webm

I experimented with the alphamerge filter.

ffmpeg -y -i test.jpg -i test2.jpg -i mask.png -filter_complex "[1:v][2:v]alphamerge[a];[0:v][a]overlay[out]" -map "[out]" masked.jpg

Okay! That overlaid test.jpg with a masked part of test2.jpg. How about alphamerging in a video? First, I need a mask video…

ffmpeg -y -loop 1 -i mask.png  -r 1 -t 4  mask.webm

Then I can combine that:

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i mask.webm -filter_complex "[1:v][2:v]alphamerge[masked];[0:v][masked]overlay[out]" -map "[out]" -r 1 -t 4 alphamerged.webm

Great, let's figure out how to combine chroma-key and alphamerge video. The naive approach doesn't work, probably because they're both messing with the alpha layer.

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i mask.webm -filter_complex "[1:v]chromakey=0x00ff00:0.1:0.2[ckout];[ckout][2:v]alphamerge[masked];[0:v][masked]overlay[out]" -map "[out]" -r 1 -t 4 masked.webm

So I probably need to blend the chromakey and the mask. Let's see if I can extract the chromakey alpha.

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i mask.webm -filter_complex "[1:v]chromakey=0x00ff00:0.1:0.2,format=rgba,alphaextract[out]" -map "[out]" -r 1 -t 4
chroma-alpha.webm

Now let's blend it with the mask.webm.

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i mask.webm -filter_complex "[1:v]chromakey=0x00ff00:0.1:0.2,format=rgba,alphaextract[ckalpha];[ckalpha][2:v]blend=all_mode=and[out]" -map "[out]" -r 1 -t 4 masked-alpha.webm

Then let's use it as the alpha:

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i masked-alpha.webm -filter_complex "[2:v]format=rgba[mask];[1:v][mask]alphamerge[masked];[0:v][masked]overlay[out]" -map "[out]" -r 1 -t 4 alphamerged.webm

Okay, that worked! Now how do I combine everything into one command? Hmm…

ffmpeg -loglevel 32 -y -loop 1 -i test.jpg -t 4 -loop 1 -i test2.jpg -t 4 -i balloons.mp4 -loop 1 -i mask.png -t 4 -filter_complex "[1:v][0:v]blend=all_expr='A*(if(gte(T,4),1,T/4))+B*(1-(if(gte(T,4),1,T/4)))'[fade];[2:v]chromakey=0x00ff00:0.1:0.2,format=rgba,alphaextract[ckalpha];[ckalpha][3:v]blend=all_mode=and,format=rgba[maskedalpha];[2:v][maskedalpha]alphamerge[masked];[fade][masked]overlay[out]" -map "[out]" -r 5 -t 4 alphamerged.webm

Then I wanted to fade the masked video out by the end.

ffmpeg -loglevel 32 -y -loop 1 -i test.jpg -t 4 -loop 1 -i test2.jpg -t 4 -i balloons.mp4 -loop 1 -i mask.png -t 4 -filter_complex "[1:v][0:v]blend=all_expr='A*(if(gte(T,4),1,T/4))+B*(1-(if(gte(T,4),1,T/4)))'[fade];[2:v]chromakey=0x00ff00:0.1:0.2,format=rgba,alphaextract[ckalpha];[ckalpha][3:v]blend=all_mode=and,format=rgba[maskedalpha];[2:v][maskedalpha]alphamerge[masked];[masked]fade=type=out:st=2:d=1:alpha=1[maskedfade];[fade][maskedfade]overlay[out]" -map "[out]" -r 10 -t 4 alphamerged.webm

When A- finally went to bed, we arranged the presents, using the GoPro to take a picture at each step of the way. I cropped and resized the images, using Krita to figure out the cropping rectangle and offset.

for FILE in *.JPG; do convert $FILE -crop 1558x876+473+842 -resize 1280x720 cropped/$FILE; done

I used ImageMagick to calculate the masks automatically.

files=(*.JPG)
i=0
j=1
len="${#files[@]}"
while [ "$j" -lt $len ]; do
  compare -fuzz 15% cropped/${files[$i]} cropped/${files[$j]} -compose Src -highlight-color White -lowlight-color Black masks/${files[$j]}
  convert -morphology Open Disk -morphology Close Disk -blur 20x5 masks/${files[$j]} processed-masks/${files[$j]}
  i=$((i+1))
  j=$((j+1))
done

Then I faded the images together to make a video.

import ffmpeg
import glob
files = glob.glob("images/cropped/*.JPG")
files.sort()
fps = 15
crf = 32
out = ffmpeg.input(files[0], loop=1, r=fps)
duration = 3
for i in range(1, len(files)):
    out = ffmpeg.filter([out, ffmpeg.input(files[i], loop=1, r=fps).filter('fade', t='in', d=duration, st=i*duration, alpha=1)], 'overlay')
args = out.output('images.webm', t=len(files) * duration, r=fps, y=None, crf=crf).compile()
print(' '.join(f'"{item}"' for item in args))

"ffmpeg" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2317.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2318.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2319.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2320.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2321.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2322.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2323.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2324.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2325.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2326.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2327.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2328.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2329.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2330.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2331.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2332.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2333.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2334.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2335.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2336.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2337.JPG" "-filter_complex" "[1]fade=alpha=1:d=3:st=3:t=in[s0];[0][s0]overlay[s1];[2]fade=alpha=1:d=3:st=6:t=in[s2];[s1][s2]overlay[s3];[3]fade=alpha=1:d=3:st=9:t=in[s4];[s3][s4]overlay[s5];[4]fade=alpha=1:d=3:st=12:t=in[s6];[s5][s6]overlay[s7];[5]fade=alpha=1:d=3:st=15:t=in[s8];[s7][s8]overlay[s9];[6]fade=alpha=1:d=3:st=18:t=in[s10];[s9][s10]overlay[s11];[7]fade=alpha=1:d=3:st=21:t=in[s12];[s11][s12]overlay[s13];[8]fade=alpha=1:d=3:st=24:t=in[s14];[s13][s14]overlay[s15];[9]fade=alpha=1:d=3:st=27:t=in[s16];[s15][s16]overlay[s17];[10]fade=alpha=1:d=3:st=30:t=in[s18];[s17][s18]overlay[s19];[11]fade=alpha=1:d=3:st=33:t=in[s20];[s19][s20]overlay[s21];[12]fade=alpha=1:d=3:st=36:t=in[s22];[s21][s22]overlay[s23];[13]fade=alpha=1:d=3:st=39:t=in[s24];[s23][s24]overlay[s25];[14]fade=alpha=1:d=3:st=42:t=in[s26];[s25][s26]overlay[s27];[15]fade=alpha=1:d=3:st=45:t=in[s28];[s27][s28]overlay[s29];[16]fade=alpha=1:d=3:st=48:t=in[s30];[s29][s30]overlay[s31];[17]fade=alpha=1:d=3:st=51:t=in[s32];[s31][s32]overlay[s33];[18]fade=alpha=1:d=3:st=54:t=in[s34];[s33][s34]overlay[s35];[19]fade=alpha=1:d=3:st=57:t=in[s36];[s35][s36]overlay[s37];[20]fade=alpha=1:d=3:st=60:t=in[s38];[s37][s38]overlay[s39]" "-map" "[s39]" "-crf" "32" "-r" "15" "-t" "63" "-y" "images.webm"

Next, I faded the masks together. These ones faded in and out so that only one mask was active at a time.

import ffmpeg
import glob
files = glob.glob("images/processed-masks/*.JPG")
files.sort()
files = files[:-2]  # Omit the last two, where I'm just turning off the lights
fps = 15
crf = 32
out = ffmpeg.input('color=black:s=1280x720', f='lavfi', r=fps)
duration = 3
for i in range(0, len(files)):
    out = ffmpeg.filter([out, ffmpeg.input(files[i], loop=1, r=fps).filter('fade', t='in', d=1, st=(i + 1)*duration, alpha=1).filter('fade', t='out', st=(i + 2)*duration - 1)], 'overlay')
args = out.output('processed-masks.webm', t=len(files) * duration, r=fps, y=None, crf=crf).compile()
print(' '.join(f'"{item}"' for item in args))

"ffmpeg" "-f" "lavfi" "-r" "15" "-i" "color=s=1280x720" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2318.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2319.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2320.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2321.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2322.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2323.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2324.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2325.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2326.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2327.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2328.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2329.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2330.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2331.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2332.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2333.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2334.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2335.JPG" "-filter_complex" "[1]fade=alpha=1:d=1:st=3:t=in[s0];[s0]fade=st=5:t=out[s1];[0][s1]overlay[s2];[2]fade=alpha=1:d=1:st=6:t=in[s3];[s3]fade=st=8:t=out[s4];[s2][s4]overlay[s5];[3]fade=alpha=1:d=1:st=9:t=in[s6];[s6]fade=st=11:t=out[s7];[s5][s7]overlay[s8];[4]fade=alpha=1:d=1:st=12:t=in[s9];[s9]fade=st=14:t=out[s10];[s8][s10]overlay[s11];[5]fade=alpha=1:d=1:st=15:t=in[s12];[s12]fade=st=17:t=out[s13];[s11][s13]overlay[s14];[6]fade=alpha=1:d=1:st=18:t=in[s15];[s15]fade=st=20:t=out[s16];[s14][s16]overlay[s17];[7]fade=alpha=1:d=1:st=21:t=in[s18];[s18]fade=st=23:t=out[s19];[s17][s19]overlay[s20];[8]fade=alpha=1:d=1:st=24:t=in[s21];[s21]fade=st=26:t=out[s22];[s20][s22]overlay[s23];[9]fade=alpha=1:d=1:st=27:t=in[s24];[s24]fade=st=29:t=out[s25];[s23][s25]overlay[s26];[10]fade=alpha=1:d=1:st=30:t=in[s27];[s27]fade=st=32:t=out[s28];[s26][s28]overlay[s29];[11]fade=alpha=1:d=1:st=33:t=in[s30];[s30]fade=st=35:t=out[s31];[s29][s31]overlay[s32];[12]fade=alpha=1:d=1:st=36:t=in[s33];[s33]fade=st=38:t=out[s34];[s32][s34]overlay[s35];[13]fade=alpha=1:d=1:st=39:t=in[s36];[s36]fade=st=41:t=out[s37];[s35][s37]overlay[s38];[14]fade=alpha=1:d=1:st=42:t=in[s39];[s39]fade=st=44:t=out[s40];[s38][s40]overlay[s41];[15]fade=alpha=1:d=1:st=45:t=in[s42];[s42]fade=st=47:t=out[s43];[s41][s43]overlay[s44];[16]fade=alpha=1:d=1:st=48:t=in[s45];[s45]fade=st=50:t=out[s46];[s44][s46]overlay[s47];[17]fade=alpha=1:d=1:st=51:t=in[s48];[s48]fade=st=53:t=out[s49];[s47][s49]overlay[s50];[18]fade=alpha=1:d=1:st=54:t=in[s51];[s51]fade=st=56:t=out[s52];[s50][s52]overlay[s53]" "-map" "[s53]" "-crf" "32" "-r" "15" "-t" "54" "-y" "processed-masks.webm"

I ended up using this particle glitter video because the gifts were small, so I wanted a video that was dense with sparkly things. I also wanted the sparkles to be more concentrated on the area where the gifts were, so I resized it and positioned it.

ffmpeg -loglevel 32 -y -f lavfi -i color=black:s=1280x720 -i sparkles4.webm -ss 13 -filter_complex "[1:v]scale=700:392[sparkles];[0:v][sparkles]overlay=x=582:y=194,setpts=(PTS-STARTPTS)*1.05[out]" -map "[out]" -r 15 -t 53 -shortest sparkles-trimmed.webm
ffmpeg -y -stream_loop 2 -i sparkles-trimmed.webm -t 57 sparkles-looped.webm              

Lastly, I combined the videos with the sparkles.

ffmpeg -loglevel 32 -y -i images.webm -i sparkles-looped.webm -i processed-masks.webm -filter_complex "[1:v]chromakey=0x0a9d06:0.1:0.2,format=rgba,alphaextract[ckalpha];[ckalpha][2:v]blend=all_mode=and,format=rgba[maskedalpha];[1:v][maskedalpha]alphamerge[masked];[masked]fade=t=out:st=57:d=1:alpha=1[maskedfaded];[0:v][maskedfaded]overlay[combined];[combined]tpad=start_mode=clone:start_duration=4:stop_mode=clone:stop_duration=4[out]" -map "[out]" -r 15 -crf 32 output.webm

After many iterations and a very late night, I got (roughly) the video I wanted, which I'm not posting here because of reasons. But it worked, yay! Now I don't have to manually place stars frame-by-frame in Krita, and I can just have all that magic happen semi-automatically.