Building up my tech notes

| geek, supernote
  • [2023-01-04 Wed] Added extra CSS to force images to fit on the page
  • [2023-01-03 Tue] Updated shell script to use EPUB for more formats

A- wants me to sit with her at bedtime. She also wants to read a stack of books until she gets sleepy. This means I sometimes have an hour (or even two) of sitting quietly with her which I can use for writing, drawing, reading, or knitting, as long as I'm quiet. ("Mama, keepp your thoughts to yourself!")

My Supernote A5X supports EPUBs and PDFs, but doesn't support HTML files or my library's e-book platform (Libby), and I'm not too keen on the Kindle app. So I need to load it up with my own collection of books, manuals, API documentation, and notes.

Org Mode can export to EPUBs and PDFs well enough. If I make the output file a symbolic link to the same file in the Dropbox folder that's synchronized with my Supernote, I can re-export the EPUB and it will end up in the right place when I sync. I've started accumulating little snippets from the digest of my reading highlights, since putting them into Org Mode allows me to organize them and summarize them in different ways. It feels good to be collecting and organizing things I'm learning.

I plan to use this reading time to skim documentation for interesting things, since sometimes the challenges are more about knowing something exists and what it's called. Then I can copy the digests into my and export it as an EPUB or PDF, review that periodically, and maybe add some shortcuts to my Emacs configuration so that I can quickly jump to lines in my reference file.


The Supernote doesn't support HTML files, but I can convert HTML to PDFs with pandoc file.html -t latex -o file.pdf. This shell script copies files to my INBOX directory, converting HTML files along the way:

for FILE in "$@"; do
    if [[ "$FILE" == *.html ]]; then
        ebook-convert "$FILE" $INBOX/$(basename "$FILE" .html).epub --extra-css 'img { max-width: 100% !important; max-weight: 100% !important }'
        # or pdf: wkhtmltopdf --no-background "$FILE" $INBOX/$(basename "$FILE" .html).pdf
    elif [[ "$FILE" == *.xml ]]; then
        dbtoepub "$FILE" -o $INBOX/$(basename "$FILE" .xml).epub
    elif [[ "$FILE" == *.texi ]]; then
        texi2pdf "$FILE" -o $INBOX/$(basename "$FILE" .texi).pdf
    elif [[ "$FILE" == *.org ]]; then
        emacs -Q --batch "$FILE" --eval "(progn (package-initialize) (use-package 'ox-epub) (org-epub-export-to-epub))"
        cp "${FILE%.*}".epub $INBOX
        cp "$FILE" $INBOX


I'd like to be able to refer to manpages. I couldn't figure out how to get man -H to work with the Firefox inside a snap (it complained about having elevated permissions). I installed man2html and found the manpage for xdotool. zcat /usr/share/man/man1/xdotool.1.gz | man2html > /tmp/xdotool.html created the HTML file, and then I used ebook-convert /tmp/xdotool.html /tmp/xdotool.epub to create an EPUB file.

I tried getting the filename for the manpage by using the man command in Emacs, but I couldn't figure out how to get the filename from there. I remembered that Emacs has a woman command that displays manpages without using the external man command. That led me to woman-file-name, which gives me the path to the manpage given a command. Emacs handles uncompressing .gz files automatically, so everything's good to go from there.

(defvar my-supernote-inbox "~/Dropbox/Supernote/INBOX")
(defun my-save-manpage-to-supernote (path)
  (interactive (list (woman-file-name nil)))
  (let* ((base (file-name-base path))
         (temp-html (make-temp-file base nil ".html")))
      (insert-file-contents path)
      (call-process-region (point-min) (point-max) "man2html" t t)
      (when (re-search-backward "Invalid Man Page" nil t)
        (delete-file temp-html)
        (error "Could not convert."))
      (write-file temp-html))
    (call-process "ebook-convert" nil (get-buffer-create "*temp*") nil temp-html
                  (expand-file-name (concat base ".epub") my-supernote-inbox))
    (delete-file temp-html)))

Info files

I turned the Elisp reference into a PDF by going to doc/lispref in my Emacs checkout and typing make elisp.pdf. It's 1470 pages long, so that should keep me busy for a while. Org Mode also has a make pdf target that uses texi2pdf to generate doc/org.pdf and doc/orgguide.pdf. Other .texi files could be converted with texi2pdf, or I can use makeinfo to create Docbook files and then use dbtoepub to convert them as in the shell script in the HTML section above.

Python documentation

I wanted to load the API documentation for autokey into one page for easy reference. The documentation at was produced by epydoc, which doesn't support Python 3. I got to work using the sphinx-epytext extension. After I used sphinx-quickstart, I edited to include extensions = ['sphinx.ext.autodoc', 'sphinx_epytext', 'sphinx.ext.autosummary'], and I added the following to index.rst:

Welcome to autokey's documentation!

   .. autoclass:: autokey.scripting.Keyboard

   .. autoclass:: autokey.scripting.Mouse

   .. autoclass:: autokey.scripting.Store

   .. autoclass:: autokey.scripting.QtDialog

   .. autoclass:: autokey.scripting.System

   .. autoclass:: autokey.scripting.QtClipboard

   .. autoclass:: autokey.scripting.Window

   .. autoclass:: autokey.scripting.Engine

Then make pdf created a PDF. There's probably a way to get a proper table of contents, but it was a good start.

Linking to and exporting function definitions in Org Mode

| emacs

I'd like to write more blog posts about little Emacs hacks, and I'd like to do it with less effort. Including source code is handy even when it's missing some context from other functions defined in the same file, since sometimes people pick up ideas and having the source code right there means less flipping between links. When I'm working inside my config file or other literate programming documents, I can just write my blog post around the function definitions. When I'm talking about Emacs Lisp functions defined elsewhere, though, it's a little more annoying to copy the function definition and put it in a source block, especially if there are updates.

The following code creates a defun link type that exports the function definition. It works for functions that can be located with find-function, so only functions loaded from .el files, but that does what I need for now. Probably once I post this, someone will mention a much more elegant way to do things. Anyway, it makes it easier to use org-store-link to capture a link to the function, insert it into a blog post, navigate back to the function, and export HTML.

(defun my-org-defun-complete ()
  "Return function definitions."
   "Function: "
   nil nil
   (and fn (symbol-name fn))))

(defun my-org-defun-export (symbol description format _)
  "Export the function."
    (find-function (intern symbol))
    (let ((function-body (buffer-substring (point)
                                           (progn (forward-sexp) (point)))))
      (pcase format
        ((or '11ty 'html)
         (format "<div class=\"org-src-container\">\n<details><summary>%s</summary><pre class=\"src src-emacs-lisp\">%s</pre></details></div>"
                 (org-html-do-format-code function-body "emacs-lisp" nil nil nil nil)))
        (`ascii function-body)
        (_ function-body)))))

(defun my-org-defun-store ()
  "Store a link to the function."
  (when (derived-mode-p 'emacs-lisp-mode)
    (org-link-store-props :type "defun"
                          :link (concat "defun:" (lisp-current-defun-name)))))

(defun my-org-defun-open (symbol _)
  "Jump to the function definition."
  (find-function (intern symbol)))

(org-link-set-parameters "defun" :follow #'my-org-defun-open
                         :export #'my-org-defun-export
                         :complete #'my-org-defun-complete
                         :store #'my-org-defun-store)

For example, if I have something like the following Org markup:


I can pull in the definition of emacsconf-prep-agenda from emacsconf.el, which you can find in the emacsconf-el repository.

(defun emacsconf-prep-agenda ()
  (let* ((org-agenda-custom-commands
         `(("a" "Agenda"
            ((tags-todo "-PRIORITY=\"C\"-SCHEDULED={.}-nextyear"
                        ((org-agenda-files (list ,emacsconf-notebook))
                         (org-agenda-sorting-strategy '(priority-down effort-up))))
             (agenda ""
                     ((org-agenda-files (list ,emacsconf-notebook))
                      (org-agenda-span 7)))
    (org-agenda nil "a")))

This is part of my Emacs configuration.

EmacsConf backstage: Using TRAMP and timers to run two tracks semi-automatically

| emacs, emacsconf, org

In previous years, organizers streamed the video feeds for EmacsConf from their own computers to the Icecast server, which was a little challenging because of CPU load. A server shared by a volunteer had a 6-core Intel Xeon E5-2420 with 48 GB of RAM, which turned out to be enough horsepower to run OBS for both the general and development track for EmacsConf 2022. One of the advantages of this setup was that I could write some Emacs Lisp to automatically play recorded intros and talk videos at scheduled times, right from the large Org file that had all the conference details. I used SCHEDULED: properties to indicate when talks should play, and that was picked up by another function that took the Org entry properties and put them into a plist.

This function scheduled the timers:

(defun emacsconf-stream-schedule-timers (&optional info)
  "Schedule PLAYING for the rest of talks and CLOSED_Q for recorded talks."
  (setq info (emacsconf-prepare-for-display (emacsconf-filter-talks (or info (emacsconf-get-talk-info)))))
  (let ((now (current-time)))
    (mapc (lambda (talk)
            (when (and (time-less-p now (plist-get talk :start-time)))
              (emacsconf-stream-schedule-talk-status-change talk (plist-get talk :start-time) "PLAYING"
                                                            `(:title (concat "Starting " (plist-get talk :slug)))))
            (when (and
                   (plist-get talk :video-file)
                   (plist-get talk :qa-time)
                   (not (string-match "none" (or (plist-get talk :q-and-a) "none")))
                   (null (plist-get talk :stream-files)) ;; can't tell when this is
                   (time-less-p now (plist-get talk :qa-time)))
              (emacsconf-stream-schedule-talk-status-change talk (plist-get talk :qa-time) "CLOSED_Q"
                                                            `(:title (concat "Q&A for " (plist-get talk :slug) " (" (plist-get talk :q-and-a) ")"))))

It turns out that TRAMP doesn't like being called from timers if there's a chance that two TRAMP processes might run at the same time. I got "Forbidden reentrant call of Tramp" errors when that happened. There was an easy fix, though. I adjusted the schedules of the talks so that they started at least a minute apart.

Sometimes I wanted to cancel just one timer:

(defun emacsconf-stream-cancel-timer (id)
  "Cancel a timer by ID."
  (interactive (list
                 "ID: "
                 (lambda (string pred action)
                    (if (eq action 'metadata)
                        `(metadata (display-sort-function . ,#'identity))
                      (complete-with-action action
                                             (seq-filter (lambda (o)
                                                           (and (timerp (cdr o))
                                                                (not (timer--triggered (cdr o)))))
                                             (lambda (a b) (string< (car a) (car b))))
                                            string pred))))))
  (when (timerp (assoc-default id emacsconf-stream-timers))
    (cancel-timer (assoc-default id emacsconf-stream-timers))
    (setq emacsconf-stream-timers
          (delq (assoc id emacsconf-stream-timers)
                (seq-filter (lambda (o)
                              (and (timerp (cdr o))
                                   (not (timer--triggered (cdr o)))))

and schedule just one timer manually:

(defun emacsconf-stream-schedule-talk-status-change (talk time new-status &optional notification)
  "Schedule a one-off timer for TALK at TIME to set it to NEW-STATUS."
  (interactive (list (emacsconf-complete-talk-info)
                     (read-string "Time: ")
                     (completing-read "Status: " (mapcar 'car emacsconf-status-types))))
  (require 'diary-lib)
  (setq talk (emacsconf-resolve-talk talk))
  (let* ((converted
           ((listp time) time)
           ((timer-duration time) (timer-relative-time nil (timer-duration time)))
           (t                           ; HH:MM
            (date-to-time (concat (format-time-string "%Y-%m-%d" nil emacsconf-timezone)
                                  (string-pad time 5 ?0 t) 
         (timer-id (concat (format-time-string "%m-%dT%H:%M" converted)
                           (plist-get talk :slug)
    (emacsconf-stream-cancel-timer timer-id) 
    (add-to-list 'emacsconf-stream-timers
                   (run-at-time time converted #'emacsconf-stream-update-talk-status-from-timer
                                talk new-status

The actual playing of talks happened using functions that were called from org-after-todo-state-change-hook. I wrote a function that extracted the talk information and then called my own list of functions.

(defun emacsconf-org-after-todo-state-change ()
  "Run all the hooks in `emacsconf-todo-hooks'.
If an `emacsconf-todo-hooks' entry is a list, run it only for the
tracks with the ID in the cdr of that list."
  (let* ((talk (emacsconf-get-talk-info-for-subtree))
         (track (emacsconf-get-track (plist-get talk :track))))
     (lambda (hook-entry)
        ((symbolp hook-entry) (funcall hook-entry talk))
        ((member (plist-get track :id) (cdr hook-entry))
         (funcall (car hook-entry) talk))))

For example, this function played the recorded intro and the talk:

(defun emacsconf-stream-play-talk-on-change (talk)
  "Play the talk."
  (interactive (list (emacsconf-complete-talk-info)))
  (setq talk (emacsconf-resolve-talk talk))
  (when (or (not (boundp 'org-state)) (string= org-state "PLAYING"))
    (if (plist-get talk :stream-files)
           (plist-get talk :slug))
            (split-string-and-unquote (plist-get talk :stream-files))
            (list "&"))))
           (plist-get talk :recorded-intro)
           (plist-get talk :video-file)) ;; recorded intro and recorded talk
          (message "should automatically play intro and recording")
          (list "play-with-intro" (plist-get talk :slug))) ;; todo deal with stream files
           (plist-get talk :recorded-intro)
           (null (plist-get talk :video-file))) ;; recorded intro and live talk; play the intro and join BBB
          (message "should automatically play intro; join %s" (plist-get talk :bbb-backstage))
          (list "intro" (plist-get talk :slug)))
           (null (plist-get talk :recorded-intro))
           (plist-get talk :video-file)) ;; live intro and recorded talk, show slide and use Mumble; manually play talk
          (message "should show intro slide; play %s afterwards" (plist-get talk :slug))
          (list "intro" (plist-get talk :slug)))
           (null (plist-get talk :recorded-intro))
           (null (plist-get talk :video-file))) ;; live intro and live talk, join the BBB
          (message "join %s for live intro and talk" (plist-get talk :bbb-backstage))
          (list "bbb" (plist-get talk :slug)))))))))

and this function handled IRC announcements when the talk state changed:

(defun emacsconf-erc-announce-on-change (talk)
  "Announce talk."
  (let ((func
         (pcase org-state
           ("PLAYING" #'erc-cmd-NOWPLAYING)
           ("CLOSED_Q" #'erc-cmd-NOWCLOSEDQ)
           ("OPEN_Q" #'erc-cmd-NOWOPENQ)
           ("UNSTREAMED_Q" #'erc-cmd-NOWUNSTREAMEDQ)
           ("TO_ARCHIVE" #'erc-cmd-NOWDONE))))
    (when func
      (funcall func talk))))

The actual announcements were handled by something like this:

(defun erc-cmd-NOWCLOSEDQ (talk)
  "Announce TALK has started Q&A, but the host has not yet opened it up."
  (interactive (list (emacsconf-complete-talk-info)))
  (when (stringp talk) (setq talk (or (emacsconf-find-talk-info talk) (error "Could not find talk %s" talk))))
  (if (emacsconf-erc-recently-announced (format "-- Q&A beginning for \"%s\"" (plist-get talk :slug)))
      (message "Recently announced, skipping")
    (emacsconf-erc-with-channels (list (concat "#" (plist-get talk :channel)))
      (erc-send-message (format "-- Q&A beginning for \"%s\" (%s) Watch: %s Add notes/questions: %s"
                                (plist-get talk :title)
                                (plist-get talk :qa-info)
                                (plist-get talk :watch-url)
                                (plist-get talk :pad-url))))  
    (emacsconf-erc-with-channels (list emacsconf-erc-hallway emacsconf-erc-org)
      (erc-send-message (format "-- Q&A beginning for \"%s\" in the %s track (%s) Watch: %s Add notes/questions: %s . Chat: #%s"
                                (plist-get talk :title)
                                (plist-get talk :track)
                                (plist-get talk :qa-info)
                                (plist-get talk :watch-url)
                                (plist-get talk :pad-url)
                                (plist-get talk :channel))))))

All that code meant that during the actual conference, my role was mostly just worrying, and occasionally starting up the Q&A (if I wasn't sure if the code would do it right). The shell scripts I wrote made it easy for the other organizers to take over the second part as they saw how it worked.

Yay timers, Emacs, and TRAMP!

You can find the latest versions of these functions in the emacsconf-el repository.

2023-01-02 Emacs news

| emacs, emacs-news

Links from, r/orgmode, r/spacemacs, r/planetemacs, Hacker News,,, YouTube, the Emacs NEWS file, Emacs Calendar, emacs-devel, and lemmy/c/emacs. Thanks to Andrés Ramírez for emacs-devel links. Do you have an Emacs-related link or announcement? Please e-mail me at Thank you!

2022-12-26 Emacs news

| emacs, emacs-news

Links from, r/orgmode, r/spacemacs, r/planetemacs, Hacker News,,, YouTube, the Emacs NEWS file, Emacs Calendar, emacs-devel, and lemmy/c/emacs. Thanks to Andrés Ramírez for emacs-devel links. Do you have an Emacs-related link or announcement? Please e-mail me at Thank you!

Comparison-shopping with Org Mode

| emacs, org

I don't like shopping. We're lucky to be able to choose, but I get overwhelmed with all the choices. I'm trying to get the hang of it, though, since I'll need to shop for lots of things for A- over the years. One of the things that's stressful is comparing choices between different webpages, especially if I want to get A-'s opinion on something. Between the challenge of remembering things as we flip between pages and the temptations of other products she sees along the way… Ugh.

I think there are web browser extensions for shopping, but I prefer to work within Org Mode so that I can capture links from my phone's web browser, refile entries into different categories, organize them with keyboard shortcuts, and tweak things the way I like. So if I have subheadings with the NAME, PRICE, IMAGE, and URL properties, I can make a table that looks like this:


Figure 1: Comparison-shopping

using code that looks like this:

#+begin_src emacs-lisp :eval yes :exports results :wrap EXPORT html

and I can view the table by exporting the subtree with HTML using org-export-dispatch (C-c C-e C-s h o). When I add new items, I can use C-u C-c C-e to reexport the subtree without navigating up to the root.

Here's the very rough code I use for that:

(defun my-get-shopping-details ()
  (goto-char (point-min))
  (let (data)
     ((re-search-forward "  data-section-data
>" nil t)
      (setq data (json-read))
      (let-alist data
        (list (cons 'name .product.title)
              (cons 'brand .product.vendor)
              (cons 'description .product.description)
              (cons 'image (concat "https:" .product.featured_image))
              (cons 'price (/ .product.price 100.0)))))
     ((and (re-search-forward "<script type=\"application/ld\\+json\">" nil t)
           (null (re-search-forward "Fabric Fabric" nil t))) ; Carter's, Columbia?
      (setq data (json-read))
      (if (vectorp data) (setq data (elt data 0)))
      (if (assoc-default '@graph data)
          (setq data (assoc-default '@graph data)))
      (if (vectorp data) (setq data (elt data 0)))
      (let-alist data
        (list (cons 'name .name)
              (cons 'url (or .url .@id))
              (cons 'brand
              (cons 'description .description)
              (cons 'rating .aggregateRating.ratingValue)
              (cons 'ratingCount .aggregateRating.reviewCount)
              (cons 'image (if (stringp .image) .image (elt .image 0)))
              (cons 'price
                    (assoc-default 'price (if (arrayp .offers)
                                              (elt .offers 0)
     ((re-search-forward "" nil t)
      (goto-char (point-min))
      (re-search-forward "^$")
      (let ((doc (libxml-parse-html-region (point) (point-max))))
        (list (cons 'name (dom-text (dom-by-tag doc 'title)))
              (cons 'description (dom-texts (dom-by-id doc "productDescription")))
              (cons 'image (dom-attr (dom-by-tag (dom-by-id doc "imgTagWrapperId") 'img) 'src))
              (cons 'price
                    (dom-texts (dom-by-id doc "priceblock_ourprice"))))))
      (goto-char (point-min))
      (re-search-forward "^$")
      (let* ((doc (libxml-parse-html-region (point) (point-max)))
              `((name . ,(string-trim (dom-text (dom-by-tag doc "title"))))
                (description . ,(string-trim (dom-text (dom-by-tag doc "title")))))
        (mapc (lambda (property)
                (let ((node
                        (lambda (o)
                          (delq nil
                                (mapcar (lambda (p)
                                          (or (string= (dom-attr o 'property) p)
                                              (string-match p (or (dom-attr o 'class) ""))))
                                        (cdr property)))))))
                  (when node (add-to-list 'result (cons (car property)
                                                        (or (dom-attr node 'content)
                                                            (string-trim (dom-text node))))))))
              '((name "og:title" "pdp-product-title")
                (brand "og:brand")
                (url "og:url")
                (image "og:image")
                (description "og:description")
                (price "og:price:amount" "product:price:amount" "pdp-price-label")))
(defun my-org-insert-shopping-details ()
  (save-excursion (yank))
  (when (org-entry-get (point) "NAME")
    (org-edit-headline (org-entry-get (point) "NAME")))
(defun my-org-update-shopping-details ()
  (when (re-search-forward org-link-any-re (save-excursion (org-end-of-subtree)) t)
    (let* ((link (org-element-property :raw-link (org-element-context)))
      (if (string-match "theshoecompany\\|dsw" link)
            (browse-url link)
            (org-entry-put (point) "URL" link)
            (unless (org-entry-get (point) "IMAGE")
              (org-entry-put (point) "IMAGE" (read-string "Image: ")))
            (unless (org-entry-get (point) "PRICE")
              (org-entry-put (point) "PRICE" (read-string "Price: "))))
        (setq data (with-current-buffer (url-retrieve-synchronously link)
        (when data
          (let-alist data
            (org-entry-put (point) "NAME" .name)
            (org-entry-put (point) "URL" link)
            (org-entry-put (point) "BRAND" .brand)
            (org-entry-put (point) "DESCRIPTION" (replace-regexp-in-string "&#039;" "'" (replace-regexp-in-string "\n" " " (or .description ""))))
            (org-entry-put (point) "IMAGE" .image)
            (org-entry-put (point) "PRICE" (cond ((stringp .price) .price) ((numberp .price) (format "%.2f" .price)) (t ""))) 
            (if .rating (org-entry-put (point) "RATING" (if (stringp .rating) .rating (format "%.1f" .rating))))
            (if .ratingCount (org-entry-put (point) "RATING_COUNT" (if (stringp .ratingCount) .ratingCount (number-to-string .ratingCount))))
(defun my-org-format-shopping-subtree ()
   "<style>body { max-width: 100% !important } #content { max-width: 100% !important } .item img { max-height: 100px; }</style><div style=\"display: flex; flex-wrap: wrap; align-items: flex-start\">"
       (lambda ()
         (if (org-entry-get (point) "URL")
              "<div class=item style=\"width: 200px\"><div><a href=\"%s\"><img src=\"%s\" height=100></a></div>
<div><a href=\"%s\">%s</a></div>
              (org-entry-get (point) "URL")
              (org-entry-get (point) "IMAGE")
              (org-entry-get (point) "PRICE")
              (org-entry-get (point) "URL")
              (url-domain (url-generic-parse-url (org-entry-get (point) "URL")))
              (org-entry-get (point) "NAME")
              (or (org-entry-get (point) "NOTES") ""))
       (if (org-before-first-heading-p) nil 'tree)))

At some point, it would be nice to keep track of how I feel about different return policies, and to add more rules for automatically extracting information from different websites. (org-chef might be a good model.) In the meantime, this makes it a little less stressful to look for stuff.

This is part of my Emacs configuration.

Figuring out how to use ffmpeg to mask a chroma-keyed video based on the differences between images

| linux, geek, ffmpeg

A- is really into Santa and Christmas because of the books she's read. Last year, she wanted to set up the GoPro to capture footage during Christmas Eve. I helped her set it up for a timelapse video. After she went to bed, we gradually positioned the presents. I extracted the frames from the video, removed the ones that caught us moving around, and then used Krita's new animation features to animate sparkles so that the presents magically appeared. She mentioned the sparkles a number of times during her deliberations about whether Santa exists or not.

This year, I want to see if I can use green-screen videos like this reversed-spin sparkle or this other sparkle video. I'm going to take a series of images, with each image adding one more gift. Then I'm going to make a mask in Krita with white covering the gift and a transparent background for the rest of the image. Then I'll use chroma-key to drop out the green screen of the sparkle video and mask it in so that the sparkles only happen within the boundaries of the gift that was added. I also want to fade one image into the other, and I want the sparkles to fade out as the gift appears.

Figuring things out

I didn't know how to do any of that yet with ffmpeg, so here's how I started figuring things out. First, I wanted to see how to fade test.jpg into test2.jpg over 4 seconds.

ffmpeg -y -loop 1 -i test.jpg -loop 1 -i test2.jpg -filter_complex "[1:v]fade=t=in:d=4:alpha=1[fadein];[0:v][fadein]overlay[out]" -map "[out]" -r 1 -t 4 -shortest test.webm

Here's another way using the blend filter:

ffmpeg -y -loop 1 -i test.jpg -loop 1 -i test2.jpg -filter_complex "[1:v][0:v]blend=all_expr='A*(if(gte(T,4),1,T/4))+B*(1-(if(gte(T,4),1,T/4)))" -t 4 -r 1 test.webm

Then I looked into chromakeying in the other video. I used balloons instead of sparkles just in case she happened to look at my screen.

ffmpeg -y -i test.webm -i balloons.mp4 -filter_complex "[1:v]chromakey=0x00ff00:0.1:0.2[ckout];[0:v][ckout]overlay[out]" -map "[out]" -shortest -r 1 overlaid.webm

I experimented with the alphamerge filter.

ffmpeg -y -i test.jpg -i test2.jpg -i mask.png -filter_complex "[1:v][2:v]alphamerge[a];[0:v][a]overlay[out]" -map "[out]" masked.jpg

Okay! That overlaid test.jpg with a masked part of test2.jpg. How about alphamerging in a video? First, I need a mask video…

ffmpeg -y -loop 1 -i mask.png  -r 1 -t 4  mask.webm

Then I can combine that:

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i mask.webm -filter_complex "[1:v][2:v]alphamerge[masked];[0:v][masked]overlay[out]" -map "[out]" -r 1 -t 4 alphamerged.webm

Great, let's figure out how to combine chroma-key and alphamerge video. The naive approach doesn't work, probably because they're both messing with the alpha layer.

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i mask.webm -filter_complex "[1:v]chromakey=0x00ff00:0.1:0.2[ckout];[ckout][2:v]alphamerge[masked];[0:v][masked]overlay[out]" -map "[out]" -r 1 -t 4 masked.webm

So I probably need to blend the chromakey and the mask. Let's see if I can extract the chromakey alpha.

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i mask.webm -filter_complex "[1:v]chromakey=0x00ff00:0.1:0.2,format=rgba,alphaextract[out]" -map "[out]" -r 1 -t 4

Now let's blend it with the mask.webm.

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i mask.webm -filter_complex "[1:v]chromakey=0x00ff00:0.1:0.2,format=rgba,alphaextract[ckalpha];[ckalpha][2:v]blend=all_mode=and[out]" -map "[out]" -r 1 -t 4 masked-alpha.webm

Then let's use it as the alpha:

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i masked-alpha.webm -filter_complex "[2:v]format=rgba[mask];[1:v][mask]alphamerge[masked];[0:v][masked]overlay[out]" -map "[out]" -r 1 -t 4 alphamerged.webm

Okay, that worked! Now how do I combine everything into one command? Hmm…

ffmpeg -loglevel 32 -y -loop 1 -i test.jpg -t 4 -loop 1 -i test2.jpg -t 4 -i balloons.mp4 -loop 1 -i mask.png -t 4 -filter_complex "[1:v][0:v]blend=all_expr='A*(if(gte(T,4),1,T/4))+B*(1-(if(gte(T,4),1,T/4)))'[fade];[2:v]chromakey=0x00ff00:0.1:0.2,format=rgba,alphaextract[ckalpha];[ckalpha][3:v]blend=all_mode=and,format=rgba[maskedalpha];[2:v][maskedalpha]alphamerge[masked];[fade][masked]overlay[out]" -map "[out]" -r 5 -t 4 alphamerged.webm

Then I wanted to fade the masked video out by the end.

ffmpeg -loglevel 32 -y -loop 1 -i test.jpg -t 4 -loop 1 -i test2.jpg -t 4 -i balloons.mp4 -loop 1 -i mask.png -t 4 -filter_complex "[1:v][0:v]blend=all_expr='A*(if(gte(T,4),1,T/4))+B*(1-(if(gte(T,4),1,T/4)))'[fade];[2:v]chromakey=0x00ff00:0.1:0.2,format=rgba,alphaextract[ckalpha];[ckalpha][3:v]blend=all_mode=and,format=rgba[maskedalpha];[2:v][maskedalpha]alphamerge[masked];[masked]fade=type=out:st=2:d=1:alpha=1[maskedfade];[fade][maskedfade]overlay[out]" -map "[out]" -r 10 -t 4 alphamerged.webm

Making the video

When A- finally went to bed, we arranged the presents, using the GoPro to take a picture at each step of the way. I cropped and resized the images, using Krita to figure out the cropping rectangle and offset.

for FILE in *.JPG; do convert $FILE -crop 1558x876+473+842 -resize 1280x720 cropped/$FILE; done

I used ImageMagick to calculate the masks automatically.

while [ "$j" -lt $len ]; do
  compare -fuzz 15% cropped/${files[$i]} cropped/${files[$j]} -compose Src -highlight-color White -lowlight-color Black masks/${files[$j]}
  convert -morphology Open Disk -morphology Close Disk -blur 20x5 masks/${files[$j]} processed-masks/${files[$j]}

Then I faded the images together to make a video.

import ffmpeg
import glob
files = glob.glob("images/cropped/*.JPG")
fps = 15
crf = 32
out = ffmpeg.input(files[0], loop=1, r=fps)
duration = 3
for i in range(1, len(files)):
    out = ffmpeg.filter([out, ffmpeg.input(files[i], loop=1, r=fps).filter('fade', t='in', d=duration, st=i*duration, alpha=1)], 'overlay')
args = out.output('images.webm', t=len(files) * duration, r=fps, y=None, crf=crf).compile()
print(' '.join(f'"{item}"' for item in args))

"ffmpeg" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2317.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2318.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2319.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2320.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2321.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2322.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2323.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2324.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2325.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2326.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2327.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2328.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2329.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2330.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2331.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2332.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2333.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2334.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2335.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2336.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2337.JPG" "-filter_complex" "[1]fade=alpha=1:d=3:st=3:t=in[s0];[0][s0]overlay[s1];[2]fade=alpha=1:d=3:st=6:t=in[s2];[s1][s2]overlay[s3];[3]fade=alpha=1:d=3:st=9:t=in[s4];[s3][s4]overlay[s5];[4]fade=alpha=1:d=3:st=12:t=in[s6];[s5][s6]overlay[s7];[5]fade=alpha=1:d=3:st=15:t=in[s8];[s7][s8]overlay[s9];[6]fade=alpha=1:d=3:st=18:t=in[s10];[s9][s10]overlay[s11];[7]fade=alpha=1:d=3:st=21:t=in[s12];[s11][s12]overlay[s13];[8]fade=alpha=1:d=3:st=24:t=in[s14];[s13][s14]overlay[s15];[9]fade=alpha=1:d=3:st=27:t=in[s16];[s15][s16]overlay[s17];[10]fade=alpha=1:d=3:st=30:t=in[s18];[s17][s18]overlay[s19];[11]fade=alpha=1:d=3:st=33:t=in[s20];[s19][s20]overlay[s21];[12]fade=alpha=1:d=3:st=36:t=in[s22];[s21][s22]overlay[s23];[13]fade=alpha=1:d=3:st=39:t=in[s24];[s23][s24]overlay[s25];[14]fade=alpha=1:d=3:st=42:t=in[s26];[s25][s26]overlay[s27];[15]fade=alpha=1:d=3:st=45:t=in[s28];[s27][s28]overlay[s29];[16]fade=alpha=1:d=3:st=48:t=in[s30];[s29][s30]overlay[s31];[17]fade=alpha=1:d=3:st=51:t=in[s32];[s31][s32]overlay[s33];[18]fade=alpha=1:d=3:st=54:t=in[s34];[s33][s34]overlay[s35];[19]fade=alpha=1:d=3:st=57:t=in[s36];[s35][s36]overlay[s37];[20]fade=alpha=1:d=3:st=60:t=in[s38];[s37][s38]overlay[s39]" "-map" "[s39]" "-crf" "32" "-r" "15" "-t" "63" "-y" "images.webm"

Next, I faded the masks together. These ones faded in and out so that only one mask was active at a time.

import ffmpeg
import glob
files = glob.glob("images/processed-masks/*.JPG")
files = files[:-2]  # Omit the last two, where I'm just turning off the lights
fps = 15
crf = 32
out = ffmpeg.input('color=black:s=1280x720', f='lavfi', r=fps)
duration = 3
for i in range(0, len(files)):
    out = ffmpeg.filter([out, ffmpeg.input(files[i], loop=1, r=fps).filter('fade', t='in', d=1, st=(i + 1)*duration, alpha=1).filter('fade', t='out', st=(i + 2)*duration - 1)], 'overlay')
args = out.output('processed-masks.webm', t=len(files) * duration, r=fps, y=None, crf=crf).compile()
print(' '.join(f'"{item}"' for item in args))

"ffmpeg" "-f" "lavfi" "-r" "15" "-i" "color=s=1280x720" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2318.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2319.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2320.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2321.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2322.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2323.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2324.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2325.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2326.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2327.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2328.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2329.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2330.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2331.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2332.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2333.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2334.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2335.JPG" "-filter_complex" "[1]fade=alpha=1:d=1:st=3:t=in[s0];[s0]fade=st=5:t=out[s1];[0][s1]overlay[s2];[2]fade=alpha=1:d=1:st=6:t=in[s3];[s3]fade=st=8:t=out[s4];[s2][s4]overlay[s5];[3]fade=alpha=1:d=1:st=9:t=in[s6];[s6]fade=st=11:t=out[s7];[s5][s7]overlay[s8];[4]fade=alpha=1:d=1:st=12:t=in[s9];[s9]fade=st=14:t=out[s10];[s8][s10]overlay[s11];[5]fade=alpha=1:d=1:st=15:t=in[s12];[s12]fade=st=17:t=out[s13];[s11][s13]overlay[s14];[6]fade=alpha=1:d=1:st=18:t=in[s15];[s15]fade=st=20:t=out[s16];[s14][s16]overlay[s17];[7]fade=alpha=1:d=1:st=21:t=in[s18];[s18]fade=st=23:t=out[s19];[s17][s19]overlay[s20];[8]fade=alpha=1:d=1:st=24:t=in[s21];[s21]fade=st=26:t=out[s22];[s20][s22]overlay[s23];[9]fade=alpha=1:d=1:st=27:t=in[s24];[s24]fade=st=29:t=out[s25];[s23][s25]overlay[s26];[10]fade=alpha=1:d=1:st=30:t=in[s27];[s27]fade=st=32:t=out[s28];[s26][s28]overlay[s29];[11]fade=alpha=1:d=1:st=33:t=in[s30];[s30]fade=st=35:t=out[s31];[s29][s31]overlay[s32];[12]fade=alpha=1:d=1:st=36:t=in[s33];[s33]fade=st=38:t=out[s34];[s32][s34]overlay[s35];[13]fade=alpha=1:d=1:st=39:t=in[s36];[s36]fade=st=41:t=out[s37];[s35][s37]overlay[s38];[14]fade=alpha=1:d=1:st=42:t=in[s39];[s39]fade=st=44:t=out[s40];[s38][s40]overlay[s41];[15]fade=alpha=1:d=1:st=45:t=in[s42];[s42]fade=st=47:t=out[s43];[s41][s43]overlay[s44];[16]fade=alpha=1:d=1:st=48:t=in[s45];[s45]fade=st=50:t=out[s46];[s44][s46]overlay[s47];[17]fade=alpha=1:d=1:st=51:t=in[s48];[s48]fade=st=53:t=out[s49];[s47][s49]overlay[s50];[18]fade=alpha=1:d=1:st=54:t=in[s51];[s51]fade=st=56:t=out[s52];[s50][s52]overlay[s53]" "-map" "[s53]" "-crf" "32" "-r" "15" "-t" "54" "-y" "processed-masks.webm"

I ended up using this particle glitter video because the gifts were small, so I wanted a video that was dense with sparkly things. I also wanted the sparkles to be more concentrated on the area where the gifts were, so I resized it and positioned it.

ffmpeg -loglevel 32 -y -f lavfi -i color=black:s=1280x720 -i sparkles4.webm -ss 13 -filter_complex "[1:v]scale=700:392[sparkles];[0:v][sparkles]overlay=x=582:y=194,setpts=(PTS-STARTPTS)*1.05[out]" -map "[out]" -r 15 -t 53 -shortest sparkles-trimmed.webm
ffmpeg -y -stream_loop 2 -i sparkles-trimmed.webm -t 57 sparkles-looped.webm              

Lastly, I combined the videos with the sparkles.

ffmpeg -loglevel 32 -y -i images.webm -i sparkles-looped.webm -i processed-masks.webm -filter_complex "[1:v]chromakey=0x0a9d06:0.1:0.2,format=rgba,alphaextract[ckalpha];[ckalpha][2:v]blend=all_mode=and,format=rgba[maskedalpha];[1:v][maskedalpha]alphamerge[masked];[masked]fade=t=out:st=57:d=1:alpha=1[maskedfaded];[0:v][maskedfaded]overlay[combined];[combined]tpad=start_mode=clone:start_duration=4:stop_mode=clone:stop_duration=4[out]" -map "[out]" -r 15 -crf 32 output.webm

After many iterations and a very late night, I got (roughly) the video I wanted, which I'm not posting here because of reasons. But it worked, yay! Now I don't have to manually place stars frame-by-frame in Krita, and I can just have all that magic happen semi-automatically.