Categories: geek » emacs » org

View topic page - RSS - Atom - Subscribe via email

#EmacsConf backstage: Using Spookfox to automate creating BigBlueButton rooms in Mozilla Firefox

| emacsconf, emacs, org

Naming conventions make it easier for other people to find things. Just like with file prefixes, I like to use a standard naming pattern for our BigBlueButton web conference rooms. For EmacsConf 2022, we used ec22-day-am-gen Speaker name (slugs). For EmacsConf 2023, I want to set up the BigBlueButton rooms before the schedule settles down, so I won't encode the time or track information into it. Instead, I'll use Speaker name (slugs) - emacsconf2023.

BigBlueButton does have an API for managing rooms, but that requires a shared secret that I don't know yet. I figured I'd just automate it through my browser. Over the last year, I've started using Spookfox to control the Firefox web browser from Emacs. It's been pretty handy for scrolling webpages up and down, so I wondered if I could replace my old xdotool-based automation. Here's what I came up with for this year.

First, I need a function that creates the BBB room for a group of talks and updates the Org entry with the URL. Adding a slight delay makes it a bit more reliable.

emacsconf-spookfox-create-bbb: Create a BBB room for this group of talks.
(defun emacsconf-spookfox-create-bbb (group)
  "Create a BBB room for this group of talks.
GROUP is (email . (talk talk talk)).
Needs a Spookfox connection."
  (let* ((bbb-name
          (format "%s (%s) - %s%s"
                  (mapconcat (lambda (o) (plist-get o :slug)) (cdr group) ", ")
                  (plist-get (cadr group) :speakers)
                  emacsconf-id
                  emacsconf-year))
         path
         (retrieve-command
          (format
           "window.location.origin + [...document.querySelectorAll('h4.room-name-text')].find((o) => o.textContent.trim() == '%s').closest('tr').querySelector('.delete-room').getAttribute('data-path')"
           bbb-name))
         (create-command (format "document.querySelector('#create-room-block').click();
document.querySelector('#create-room-name').value = \"%s\";
document.querySelector('#room_mute_on_join').click();
document.querySelector('.create-room-button').click();"
                                 bbb-name)))
    (setq path (spookfox-js-injection-eval-in-active-tab retrieve-command t))
    (unless path
      (dolist (cmd (split-string create-command ";"))
        (spookfox-js-injection-eval-in-active-tab cmd t)
        (sleep-for 2))
      (sleep-for 2)
      (setq path (spookfox-js-injection-eval-in-active-tab retrieve-command t)))
    (when path
      (dolist (talk (cdr group))
        (save-window-excursion
          (emacsconf-with-talk-heading talk
            (org-entry-put (point) "ROOM" path))))
      (cons bbb-name path))))

Then I need to iterate over the list of talks that have live Q&A sessions but don't have BBB rooms assigned yet so that I can create them.

emacsconf-spookfox-create-bbb-for-live-talks: Create BBB rooms for talks that don’t have them yet.
(defun emacsconf-spookfox-create-bbb-for-live-talks ()
  "Create BBB rooms for talks that don't have them yet."
  (let* ((talks (seq-filter
                 (lambda (o)
                   (and (string-match "live" (or (plist-get o :q-and-a) ""))
                        (not (string= (plist-get o :status) "CANCELLED"))
                        (not (plist-get o :bbb-room))))
                 (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
         (groups (and talks (emacsconf-mail-groups talks))))
    (dolist (group groups)
      (emacsconf-spookfox-create-bbb group))))

The result: a whole bunch of rooms ready for people to check in.

2023-10-14_09-24-34.png
Figure 1: BigBlueButton rooms

Using Spookfox to communicate with Firefox from Emacs Lisp made it easy to get data in and out of my browser. Handy!

This code is in emacsconf-spookfox.el.

#EmacsConf backstage: file prefixes

| emacs, org, emacsconf

Sometimes it makes sense to dynamically generate information related to a talk and then save it as an Org property so that I can manually edit it. For example, we like to name all the talk files using this pattern: "emacsconf-year-slug--title--speakers". That's a lot to type consistently! We can generate most of these prefixes automatically, but some might need tweaking, like when the talk title or speaker names have special characters.

Calculating the file prefix for a talk

First we need something that turns a string into an ID.

emacsconf-slugify: Turn S into an ID.
(defun emacsconf-slugify (s)
  "Turn S into an ID.
Replace spaces with dashes, remove non-alphanumeric characters,
and downcase the string."
  (replace-regexp-in-string
   " +" "-"
   (replace-regexp-in-string
    "[^a-z0-9 ]" ""
    (downcase s))))

Then we can use that to calculate the file prefix for a given talk.

emacsconf-file-prefix: Create the file prefix for TALK
(defun emacsconf-file-prefix (talk)
  "Create the file prefix for TALK."
  (concat emacsconf-id "-"
          emacsconf-year "-"
          (plist-get talk :slug) "--"
          (emacsconf-slugify (plist-get talk :title))
          (if (plist-get talk :speakers)
              (concat "--"
                     (emacsconf-slugify (plist-get talk :speakers)))
            "")))

Then we can map over all the talk entries that don't have FILE_PREFIX defined:

emacsconf-set-file-prefixes: Set the FILE_PREFIX property for each talk entry that needs it.
(defun emacsconf-set-file-prefixes ()
  "Set the FILE_PREFIX property for each talk entry that needs it."
  (interactive)
  (org-map-entries
   (lambda ()
     (org-entry-put
      (point) "FILE_PREFIX"
      (emacsconf-file-prefix (emacsconf-get-talk-info-for-subtree))))
   "SLUG={.}-FILE_PREFIX={.}"))

That stores the file prefix in an Org property, so we can edit it if it needs tweaking.

output-2023-10-10-15:17:17.gif
Figure 1: Setting the FILE_PREFIX for all talks that don't have that yet

Renaming files to match the file prefix

Now that we have that, how can we use it? One way is to rename files from within Emacs. I can mark multiple files with Dired's m command or work on them one at a time. If there are several files with the same extension, I can specify something to add to the filename to tell them apart.

emacsconf-rename-files: Rename the marked files or the current file to match TALK.
(defun emacsconf-rename-files (talk &optional filename)
  "Rename the marked files or the current file to match TALK.
If FILENAME is specified, use that as the extra part of the filename after the prefix.
This is useful for distinguishing files with the same extension.
Return the list of new filenames."
  (interactive (list (emacsconf-complete-talk-info)))
  (prog1
      (mapcar
       (lambda (file)
         (let* ((extra
                 (or filename
                     (read-string (format "Filename (%s): " (file-name-base file)))))
                (new-filename
                 (expand-file-name
                  (concat (plist-get talk :file-prefix)
                          (if (string= extra "")
                              ""
                            (concat "--" extra))
                          "."
                          (file-name-extension file))
                  (file-name-directory file))))
           (rename-file file new-filename t)
           new-filename))
       (or (dired-get-marked-files) (list (buffer-file-name))))
    (when (derived-mode-p 'dired-mode)
      (revert-buffer))))

output-2023-10-10-14:18:34.gif
Figure 2: Renaming multiple files

Working with files on other computers

Because Dired works over TRAMP, I can use that to rename files on a remote server without changing anything about the code. I can open the remote directory with Dired and everything just works.

TRAMP also makes it easy to copy a file to the backstage directory after it's renamed, which saves me having to do that as a separate step.

emacsconf-rename-and-upload-to-backstage: Rename marked files or the current file, then upload to backstage.
(defun emacsconf-rename-and-upload-to-backstage (talk &optional filename)
  "Rename marked files or the current file, then upload to backstage."
  (interactive (list (emacsconf-complete-talk-info)))
  (mapc
   (lambda (file)
     (copy-file
      file
      (expand-file-name
       (file-name-nondirectory file)
       emacsconf-backstage-dir)
      t))
   (emacsconf-rename-files talk)))

So if my emacsconf-backstage-dir is set to /ssh:orga@res:/var/www/res.emacsconf.org/2023/backstage, then it looks up the details for res in my ~/.ssh/config and copies the file there.

Renaming files using information from a JSON

What if I don't want to rename the files from Emacs? If I use Emacs's JSON support to export some information from the talks as a JSON file, then I can easily use that data from the command line.

Here's how I export the talk information:

emacsconf-talks-json: Return JSON format with a subset of talk information.
(defun emacsconf-publish-talks-json ()
  "Return JSON format with a subset of talk information."
  (json-encode
   (list
    :talks
    (mapcar
     (lambda (o)
       (apply
        'list
        (cons :start-time (format-time-string "%FT%T%z" (plist-get o :start-time) t))
        (cons :end-time (format-time-string "%FT%T%z" (plist-get o :end-time) t))
        (mapcar
         (lambda (field)
           (cons field (plist-get o field)))
         '(:slug :title :speakers :pronouns :pronunciation :url :track :file-prefix))))
     (emacsconf-filter-talks (emacsconf-get-talk-info))))))

emacsconf-publish-talks-json-to-files
(defun emacsconf-publish-talks-json-to-files ()
  "Export talk information as JSON so that we can use it in shell scripts."
  (interactive)
  (mapc (lambda (dir)
          (when (and dir (file-directory-p dir))
            (with-temp-file (expand-file-name "talks.json" dir)
              (insert (emacsconf-talks-json)))))
        (list emacsconf-res-dir emacsconf-ansible-directory)))

Then I can use jq to extract the information with

jq -r '.talks[] | select(.slug=="'$SLUG'")["file-prefix"]' < $TALKS_JSON

Here it is in the context of a shell script that renames the given file to match a talk's prefix.

#!/bin/bash
# 
# Usage: rename-original.sh $slug $file [$extra] [$talks-json]
SLUG=$1
FILE=$2
TALKS_JSON=${4:-~/current/talks.json}
EXTRA=""
if [ -z ${3-unset} ]; then
    EXTRA=""
elif [ -n "$3" ]; then
    EXTRA="--$3"
elif echo "$FILE" | grep -e '\(webm\|mp4\|mov\)'; then
    EXTRA="--original"
fi
filename=$(basename -- "$FILE")
extension="${filename##*.}"
filename="${filename%.*}"
FILE_PREFIX=$(jq -r '.talks[] | select(.slug=="'$SLUG'")["file-prefix"]' < $TALKS_JSON)
mv "$FILE" $FILE_PREFIX$EXTRA.$extension
echo $FILE_PREFIX$EXTRA.$extension
# Copy to original if needed
if [ -f $FILE_PREFIX--original.webm ] && [ ! -f $FILE_PREFIX--main.$extension ]; then
    cp $FILE_PREFIX--original.$extension $FILE_PREFIX--main.webm
fi

Then I can use something like rename-original.sh emacsconf video.webm to emacsconf-2023-emacsconf--emacsconforg-how-we-use-org-mode-and-tramp-to-organize-and-run-a-multitrack-conference--sacha-chua--original.webm.

Working with PsiTransfer-uploaded files

JSON support is useful for getting files into our system, too. For EmacsConf 2022, we used PsiTransfer as a password-protected web-based file upload service. That was much easier for speakers to deal with than FTP, especially for large files. PsiTransfer makes a JSON file for each batch of uploads, which is handy because the uploaded files are named based on the key instead of keeping their filenames and extensions. I wrote a function to copy an uploaded file from the PsiTransfer directory to the backstage directory, renaming it along the way. That meant that I could open the JSON for the uploaded files via TRAMP and then copy a file between two remote directories without manually downloading it to my computer.

emacsconf-upload-copy-from-json: Parse PsiTransfer JSON files and copy the uploaded file to the backstage directory.
(defun emacsconf-upload-copy-from-json (talk key filename)
  "Parse PsiTransfer JSON files and copy the uploaded file to the backstage directory.
The file is associated with TALK. KEY identifies the file in a multi-file upload.
FILENAME specifies an extra string to add to the file prefix if needed."
  (interactive (let-alist (json-parse-string (buffer-string) :object-type 'alist)
                 (list (emacsconf-complete-talk-info)
                       .metadata.key
                       (read-string (format "Filename: ")))))
  (let ((new-filename (concat (plist-get talk :file-prefix)
                              (if (string= filename "")
                                  filename
                                (concat "--" filename))
                              "."
                              (let-alist (json-parse-string (buffer-string) :object-type 'alist)
                                (file-name-extension .metadata.name)))))
    (copy-file
     key
     (expand-file-name new-filename emacsconf-backstage-dir)
     t)))

So that's how I can handle things that can be mostly automated but that might need a little human intervention: use Emacs Lisp to make a starting point, tweak it a little if needed, and then make it easy to use that value elsewhere. Renaming files can be tricky, so it's good to reduce the chance for typos!

#EmacsConf backstage: looking at EmacsConf's growth over 5 years, and how to do pivot tables and graphs with Org Mode and the Python pandas library

| emacsconf, emacs, org, python

Having helped organize EmacsConf for a number of years now, I know that I usually panic about whether we have submissions partway through the call for participation. This causes us to extend the CFP deadline and ask people to ask more people to submit things, and then we end up with a wonderful deluge of talks that I then have to somehow squeeze into a reasonable-looking schedule.

This year, I managed to not panic and I also resisted the urge to extend the CFP deadline, trusting that there will actually be tons of cool stuff. It helped that my schedule SVG code let me visualize what the conference could feel like with the submissions so far, so we started with a reasonably nice one-track conference and built up from there. It also helped that I'd gone back to the submissions for 2022 and plotted them by the number of weeks before the CFP deadline, and I knew that there'd be a big spike from all those people whose Org DEADLINE: properties would nudge them into finalizing their proposals.

Out of curiosity, I wanted to see how the stats for this year compared with previous years. I wrote a small function to collect the data that I wanted to summarize:

emacsconf-count-submissions-by-week: Count submissions in INFO by distance to CFP-DEADLINE.
(defun emacsconf-count-submissions-by-week (&optional info cfp-deadline)
  "Count submissions in INFO by distance to CFP-DEADLINE."
  (setq cfp-deadline (or cfp-deadline emacsconf-cfp-deadline))
  (setq info (or info (emacsconf-get-talk-info)))
  (cons '("Weeks to CFP end date" "Count" "Hours")
        (mapcar (lambda (entry)
                  (list (car entry)
                        (length (cdr entry))
                        (apply '+ (mapcar 'cdr (cdr entry)))))
                (seq-group-by
                 'car
                 (sort
                  (seq-keep
                   (lambda (o)
                     (and (emacsconf-publish-talk-p o)
                          (plist-get o :date-submitted)
                          (cons (floor (/ (days-between (plist-get o :date-submitted) cfp-deadline)
                                          7.0))
                                (string-to-number
                                 (or (plist-get o :video-duration)
                                     (plist-get o :time)
                                     "0")))))
                   info)
                  (lambda (a b) (< (car a) (car b))))))))

and then I ran it against the different files for each year, filling in the previous years' data as needed. The resulting table is pretty long, so I've put that in a collapsible section.

(let ((years `((2023 "~/proj/emacsconf/2023/private/conf.org" "2023-09-15")
               (2022 "~/proj/emacsconf/2022/private/conf.org" "2022-09-18")
               (2021 "~/proj/emacsconf/2021/private/conf.org" "2021-09-30")
               (2020 "~/proj/emacsconf/wiki/2020/submissions.org" "2020-09-30")
               (2019 "~/proj/emacsconf/2019/private/conf.org" "2019-08-31"))))
  (append
   '(("Weeks to CFP" "Year" "Count" "Minutes"))
   (seq-mapcat
    (lambda (year-info)
      (let ((emacsconf-org-file (elt year-info 1))
            (emacsconf-cfp-deadline (elt year-info 2))
            (year (car year-info)))
        (mapcar (lambda (o) (list (car o) year (cadr o) (elt o 2)))
                (cdr (emacsconf-count-submissions-by-week (emacsconf-get-talk-info) emacsconf-cfp-deadline)))))
    years)))
Table
Weeks to CFP Year Count Minutes
-12 2023 4 70
-9 2023 2 30
-7 2023 2 30
-5 2023 2 30
-4 2023 2 60
-3 2023 3 40
-2 2023 5 130
-1 2023 10 180
0 2023 8 140
1 2023 1 20
-8 2022 2 25
-5 2022 2 31
-3 2022 2 31
-2 2022 2 17
-1 2022 8 191
0 2022 8 110
1 2022 5 107
-8 2021 4 50
-7 2021 2 17
-6 2021 1 7
-5 2021 2 22
-4 2021 2 19
-3 2021 5 73
-2 2021 1 10
-1 2021 12 163
0 2021 13 197
1 2021 1 10
2 2021 1 10
-5 2020 1 10
-4 2020 1 15
-2 2020 1 30
-1 2020 4 68
0 2020 21 424
1 2020 7 152
-5 2019 2 45
-4 2019 1 21
-2 2019 6 126
-1 2019 9 82
0 2019 9 148
2 2019 1 7

Some talks were proposed off-list and are not captured here, and cancelled or withdrawn talks weren't included either. The times for previous years use the actual video time, and the times for this year use proposed times.

Off the top of my head, I didn't know of an easy way to make a pivot table or cross-tab using just Org Mode or Emacs Lisp. I tried using datamash, but I was having a hard time getting my output just the way I wanted it. Fortunately, it was super-easy to get my data from an Org table into Python so I could use pandas.pivot_table. Because I had used #+NAME: submissions-by-week to label the table, I could use :var data=submissions-by-week to refer to the data in my Python program. Then I could summarize them by week.

Here's the number of submissions by the number of weeks to the original CFP deadline, so we can see people generally like to target the CFP date.

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True)
fig, ax = plt.subplots()
figure = df.plot(title='Number of submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('number-of-submissions.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 4
-9 0 0 0 0 2
-8 0 0 4 2 0
-7 0 0 2 0 2
-6 0 0 1 0 0
-5 2 1 2 2 2
-4 1 1 2 0 2
-3 0 0 5 2 3
-2 6 1 1 2 5
-1 9 4 12 8 10
0 9 21 13 8 8
1 0 7 1 5 1
2 1 0 1 0 0
number-of-submissions.png

Calculating the cumulative number of submissions might be more useful. Here, each row shows the number received so far.

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax = plt.subplots()
figure = df.plot(title='Cumulative submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('cumulative-submissions.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 4
-9 0 0 0 0 6
-8 0 0 4 2 6
-7 0 0 6 2 8
-6 0 0 7 2 8
-5 2 1 9 4 10
-4 3 2 11 4 12
-3 3 2 16 6 15
-2 9 3 17 8 20
-1 18 7 29 16 30
0 27 28 42 24 38
1 27 35 43 29 39
2 28 35 44 29 39
cumulative-submissions.png
Figure 1: Cumulative submissions by number of weeks to CFP end date

And here's the cumulative number of minutes based on the proposals.

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Minutes', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax = plt.subplots()
figure = df.plot(title='Cumulative minutes by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('cumulative-minutes.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 70
-9 0 0 0 0 100
-8 0 0 50 25 100
-7 0 0 67 25 130
-6 0 0 74 25 130
-5 45 10 96 56 160
-4 66 25 115 56 220
-3 66 25 188 87 260
-2 192 55 198 104 390
-1 274 123 361 295 570
0 422 547 558 405 710
1 422 699 568 512 730
2 429 699 578 512 730
cumulative-minutes.png
Figure 2: Cumulative minutes by number of weeks to the CFP end date

So… yeah… 730 minutes of talks for this year… I might've gotten a little carried away. But I like all the talks! And I want them to be captured in videos and maybe even transcribed by people who will take the time to change misrecognized words like Emax into Emacs! And I want people to be able to connect with other people who are interested in the sorts of stuff they're doing! So we're going to make it happen. The draft schedule's looking pretty full, but I think it'll work out, especially if the speakers send in their videos on time. Let's see how it all works out!

(…and look, I even got to learn how to do pivot tables and graphs with Python!)

Org protocol: following Org links from outside Emacs

| org, emacs

_xor had an interesting idea: can we use org-protocol to link to things inside Emacs, so that we can have a webpage with bookmarks into our Org files? Here's a quick hack that reuses org-store-link and org-link-open.

(defun org-protocol-open-link (info)
  "Process an org-protocol://open style url with INFO."
  (org-link-open (car (org-element-parse-secondary-string (plist-get info :link) '(link)))))

(defun org-protocol-copy-open-link (arg)
  (interactive "P")
  (kill-new (concat "org-protocol://open?link=" (url-hexify-string (org-store-link arg)))))

(with-eval-after-load 'org
  (add-to-list 'org-protocol-protocol-alist
               '("org-open" :protocol "open" :function org-protocol-open-link)))

To make exporting and following easier, we also need a little code to handle org-protocol links inside Org.

(defun org-protocol-follow (path &rest _)
  "Follow the org-protocol link for PATH."
  (org-protocol-check-filename-for-protocol (concat "org-protocol:" path) nil nil))

(defun org-protocol-export (path desc format info)
  "Export an org-protocol link."
  (setq path (concat "org-protocol:" path))
  (setq desc (or desc path))
  (pcase format
    (`html (format "<a href=\"%s\">%s</a>" path desc))
    (`11ty (format "<a href=\"%s\">%s</a>" path desc))
    (`latex (org-latex-link path desc info))
    (`ascii (org-ascii-link path desc info))
    (`md (org-md-link path desc info))
    (_ path)))

(with-eval-after-load 'org
  (org-link-set-parameters "org-protocol"
                           :follow #'org-protocol-follow
                           :export #'org-protocol-export))

Now I can use org-protocol-copy-open-link to copy a link to the current location, and I can put it into my Org files.

Example bare link to the Org manual, which will work only if you have open in the org-protocol-protocol-alist:

org-protocol://open?link=%5B%5Binfo%3Aorg%23Protocols%5D%5Borg%23Protocols%5D%5D

With a description:

Org manual - Protocols

This is part of my Emacs configuration.

EmacsConf backstage: capturing submissions from e-mails

| emacsconf, emacs, org

2023-09-11: Updated code for recognizing fields.

People submit proposals for EmacsConf sessions via e-mail following this submission template. (You can still submit a proposal until Sept 14!) I mostly handle acceptance and scheduling, so I copy this information into our private conf.org file so that we can use it to plan the draft schedule, mail-merge speakers, and so on. I used to do this manually, but I'm experimenting with using functions to create the heading automatically so that it includes the date, talk title, and e-mail address from the e-mail, and it calculates the notification date for early acceptances as well. I use Notmuch for e-mail, so I can get the properties from (notmuch-show-get-message-properties).

2023-09-05_13-09-57.png
Figure 1: E-mail submission

emacsconf-mail-add-submission: Add the submission from the current e-mail.
(defun emacsconf-mail-add-submission (slug)
  "Add the submission from the current e-mail."
  (interactive "MTalk ID: ")
  (let* ((props (notmuch-show-get-message-properties))
         (from (or (plist-get (plist-get props :headers) :Reply-To)
                   (plist-get (plist-get props :headers) :From)))
         (body (plist-get
                (car
                 (plist-get props :body))
                :content))
         (date (format-time-string "%Y-%m-%d"
                                   (date-to-time (plist-get (plist-get props :headers) :Date))))
         (to-notify (format-time-string
                     "%Y-%m-%d"
                     (time-add
                      (days-to-time emacsconf-review-days)
                      (date-to-time (plist-get (plist-get props :headers) :Date)))))
         (data (emacsconf-mail-parse-submission body)))
    (when (string-match "<\\(.*\\)>" from)
      (setq from (match-string 1 from)))
    (with-current-buffer
        (find-file emacsconf-org-file)
      ;;  go to the submissions entry
      (goto-char (org-find-property "CUSTOM_ID" "submissions"))
      (when (org-find-property "CUSTOM_ID" slug)
        (error "Duplicate talk ID")))
    (find-file emacsconf-org-file)
    (delete-other-windows)
    (outline-next-heading)
    (org-insert-heading)
    (insert " " (or (plist-get data :title) "") "\n")
    (org-todo "TO_REVIEW")
    (org-entry-put (point) "CUSTOM_ID" slug)
    (org-entry-put (point) "SLUG" slug)
    (org-entry-put (point) "TRACK" "General")
    (org-entry-put (point) "EMAIL" from)
    (org-entry-put (point) "DATE_SUBMITTED" date)
    (org-entry-put (point) "DATE_TO_NOTIFY" to-notify)
    (when (plist-get data :time)
      (org-entry-put (point) "TIME" (plist-get data :time)))
    (when (plist-get data :availability)
      (org-entry-put (point) "AVAILABILITY"
                     (replace-regexp-in-string "\n+" " "
                                               (plist-get data :availability))))
    (when (plist-get data :public)
      (org-entry-put (point) "PUBLIC_CONTACT"
                     (replace-regexp-in-string "\n+" " "
                                               (plist-get data :public))))
    (when (plist-get data :private)
      (org-entry-put (point) "EMERGENCY"
                     (replace-regexp-in-string "\n+" " "
                                               (plist-get data :private))))
    (when (plist-get data :q-and-a)
      (org-entry-put (point) "Q_AND_A"
                     (replace-regexp-in-string "\n+" " "
                                               (plist-get data :q-and-a))))
    (save-excursion
      (insert (plist-get data :body)))
    (re-search-backward org-drawer-regexp)
    (org-fold-hide-drawer-toggle 'off)
    (org-end-of-meta-data)
    (split-window-below)))

emacsconf-mail-parse-submission: Extract data from EmacsConf 2023 submissions in BODY.
(defun emacsconf-mail-parse-submission (body)
  "Extract data from EmacsConf 2023 submissions in BODY."
  (when (listp body) (setq body (plist-get (car body) :content)))
  (let ((data (list :body body))
        (fields '((:title "^[* ]*Talk title")
                  (:description "^[* ]*Talk description")
                  (:format "^[* ]*Format")
                  (:intro "^[* ]*Introduction for you and your talk")
                  (:name "^[* ]*Speaker name")
                  (:availability "^[* ]*Speaker availability")
                  (:q-and-a "^[* ]*Preferred Q&A approach")
                  (:public "^[* ]*Public contact information")
                  (:private "^[* ]*Private emergency contact information")
                  (:release "^[* ]*Please include this speaker release"))))
    (with-temp-buffer
      (insert body)
      (goto-char (point-min))
      ;; Try to parse it
      (while fields
        ;; skip the field title
        (when (and (or (looking-at (cadar fields))
                       (re-search-forward (cadar fields) nil t))
                   (re-search-forward "\\(:[ \t\n]+\\|\n\n\\)" nil t))
          ;; get the text between this and the next field
          (setq data (plist-put data (caar fields)
                                (buffer-substring (point)
                                                  (or
                                                   (when (and (cdr fields)
                                                              (re-search-forward (cadr (cadr fields)) nil t))
                                                     (goto-char (match-beginning 0))
                                                     (point))
                                                   (point-max))))))
        (setq fields (cdr fields)))
      (if (string-match "[0-9]+" (or (plist-get data :format) ""))
          (plist-put data :time (match-string 0 (or (plist-get data :format) ""))))
      data)))

The functions above are in the emacsconf-el repository. When I call emacsconf-mail-parse-submission and give it the talk ID I want to use, it makes the Org entry.

2023-09-05_13-12-34.png
Figure 2: Creating the entry

We store structured data in Org Mode properties such as NAME, EMAIL and EMERGENCY. I tend to make mistakes when typing, so I have a short function that sets an Org property based on a region. This is the code from my personal config:

my-org-set-property: In the current entry, set PROPERTY to VALUE.
(defun my-org-set-property (property value)
  "In the current entry, set PROPERTY to VALUE.
Use the region if active."
  (interactive (list (org-read-property-name)
                     (when (region-active-p) (replace-regexp-in-string "[ \n\t]+" " " (buffer-substring (point) (mark))))))
  (org-set-property property value))

I've bound it to C-c C-x p. This is what it looks like when I use it:

output-2023-09-05-13:13:30.gif
Figure 3: Setting Org properties from the region

That helps me reduce errors in entering data. I sometimes forget details, so I ask other people to double-check my work, especially when it comes to speaker availability. That's how I copy the submission e-mails into our Org file.

Using Org Mode tables and Emacs Lisp to create Minecraft Java JSON command books

| minecraft, org, emacs, play
  • [2023-04-12 Wed]: Remove / from the beginning so that I can use this in a function. Split book function into JSON and command. Updated effects to hide particles.
  • [2023-04-10 Mon]: Separated trident into channeling and riptide.

A+ likes playing recent Minecraft snapshots because of the new features. The modding systems haven't been updated for the snaphots yet, so we couldn't use mods like JourneyMap to teleport around. I didn't want to be the keeper of coordinates and be in charge of teleporting people to various places.

It turns out that you can make clickable books using JSON. I used the Minecraft book editor to make a prototype book and figure out the syntax. Then I used a command block to give it to myself in order to work around the length limits on commands in chat. A+ loved being able to carry around a book that could teleport her to either of us or to specified places, change the time of day, clear the weather, and change game mode. That also meant that I no longer had to type all the commands to give her water breathing, night vision, or slow falling, or give her whatever tools she forgot to pack before she headed out. It was so handy, W- and I got our own copies too.

Manually creating the clickable targets was annoying, especially since we wanted the book to have slightly different content depending on the instance we were in. I wanted to be able to specify the contents using Org Mode tables and generate the JSON for the book using Emacs.

Here's a screenshot:

2023-04-09_10-09-48.png
Figure 1: Screenshot of command book

This is the code to make it:

(defun my-minecraft-remove-markup (s)
  (if (string-match "^[=~]\\(.+?\\)[=~]$" s)
      (match-string 1 s)
    s))

(defun my-minecraft-book-json (title author book)
  "Generate the JSON for TITLE AUTHOR BOOK.
BOOK should be a list of lists of the form (text click-command color)."
  (json-encode
   `((pages . 
            ,(apply 'vector
                    (mapcar
                     (lambda (page)
                       (json-encode
                        (apply 'vector 
                               (seq-mapcat
                                (lambda (command)
                                  (let ((text (my-minecraft-remove-markup (or (elt command 0) "")))
                                        (click (my-minecraft-remove-markup (or (elt command 1) "")))
                                        (color (or (elt command 2) "")))
                                    (unless (or (string-match "^<.*>$" text)
                                                (string-match "^<.*>$" click)
                                                (string-match "^<.*>$" color))
                                      (list
                                       (append
                                        (list (cons 'text text))
                                        (unless (string= click "")
                                          `((clickEvent 
                                             (action . "run_command")
                                             (value . ,(concat "/" click)))))                                    
                                        (unless (string= color "")
                                          (list (cons 'color
                                                      color))))
                                       (if (string= color "")
                                           '((text . "\n"))
                                         '((text . "\n")
                                           (color . "reset")))))))
                                page))))
                     (seq-partition book 14)
                     )))
     (author . ,author)
     (title . ,title))))

(defun my-minecraft-book (title author book)
  "Generate a command to put into a command block in order to get a book.
Label it with TITLE and AUTHOR.
BOOK should be a list of lists of the form (text click-command color).
Copy the command text to the kill ring for pasting into a command block."
  (let ((s (concat "item replace entity @p weapon.mainhand with written_book"
                   (my-minecraft-book-json title author book))))
    (kill-new s)
    s))

With this code, I can generate a simple book like this:

(my-minecraft-book "Simple book" "sachac"
                   '(("Daytime" "set time 0800")
                     ("Creative" "gamemode creative" "#0000cd")))
item replace entity @p weapon.mainhand with written_book{"pages":["[{\"text\":\"Daytime\",\"clickEvent\":{\"action\":\"run_command\",\"value\":\"/set time 0800\"}},{\"text\":\"\\n\"},{\"text\":\"Creative\",\"clickEvent\":{\"action\":\"run_command\",\"value\":\"/gamemode creative\"},\"color\":\"#0000cd\"},{\"text\":\"\\n\",\"color\":\"reset\"}]"],"author":"sachac","title":"Simple book"}

To place it in the world:

  1. I changed my server.properties to set enable-command-block=true.
  2. In the game, I used /gamemode creative to switch to creative mode.
  3. I used /give @p minecraft:command_block to give myself a command block.
  4. I right-clicked an empty place to set the block there.
  5. I right-clicked on the command block and pasted in the command.
  6. I added a button.

Then I clicked on the button and it replaced whatever I was holding with the book. I used item replace instead of give so that it's easy to replace old versions.

On the Org Mode side, it's much nicer to specify commands in a named table. For example, if I name the following table with #+name: mc-quick, I can refer to it with :var quick=mc-quick in the Emacs Lisp source block. (You can check the Org source for this post if that makes it easier to understand.)

Daytime time set 0800  
Clear weather weather clear  
Creative gamemode creative #0000cd
Survival gamemode survival #ff4500
Spectator gamemode spectator #228b22
(my-minecraft-book "Book from table" "sachac" quick)
item replace entity @p weapon.mainhand with written_book{"pages":["[{\"text\":\"Daytime\",\"clickEvent\":{\"action\":\"run_command\",\"value\":\"/time set 0800\"}},{\"text\":\"\\n\"},{\"text\":\"Clear weather\",\"clickEvent\":{\"action\":\"run_command\",\"value\":\"/weather clear\"}},{\"text\":\"\\n\"},{\"text\":\"Creative\",\"clickEvent\":{\"action\":\"run_command\",\"value\":\"/gamemode creative\"},\"color\":\"#0000cd\"},{\"text\":\"\\n\",\"color\":\"reset\"},{\"text\":\"Survival\",\"clickEvent\":{\"action\":\"run_command\",\"value\":\"/gamemode survival\"},\"color\":\"#ff4500\"},{\"text\":\"\\n\",\"color\":\"reset\"},{\"text\":\"Spectator\",\"clickEvent\":{\"action\":\"run_command\",\"value\":\"/gamemode spectator\"},\"color\":\"#228b22\"},{\"text\":\"\\n\",\"color\":\"reset\"}]"],"author":"sachac","title":"Book from table"}

Then I can define several named tables and append them together. Here's one for different effects:

Water breathing effect give @p minecraft:water_breathing infinite 255 true  
Night vision effect give @p minecraft:night_vision infinite 255 true  
Regeneration effect give @p minecraft:regeneration infinite 255 true  
Haste effect give @p minecraft:haste infinite 2 true  
Health boost effect give @p minecraft:health_boost infinite 255 true  
Slow falling effect give @p minecraft:slow_falling infinite 255 true  
Fire resist effect give @p minecraft:fire_resistance infinite 255 true  
Resistance effect give @p minecraft:resistance infinite 255 true  
Clear effects effect clear @p  

Some commands are pretty long. Specifying a width like <20> in the first row lets me use C-c TAB to toggle width.

Pickaxe give @p minecraft:diamond_pickaxe{Enchantments:[{id:"minecraft:fortune",lvl:4s},{id:"minecraft:mending",lvl:1s},{id:"minecraft:efficiency",lvl:4s}]}  
Silk touch pickaxe give @p minecraft:diamond_pickaxe{Enchantments:[{id:"minecraft:silk_touch",lvl:1s},{id:"minecraft:mending",lvl:1s}]}  
Sword give @p minecraft:diamond_sword{Enchantments:[{id:"minecraft:looting",lvl:4s},{id:"minecraft:mending",lvl:1s}]}  
Axe give @p minecraft:diamond_axe{Enchantments:[{id:"minecraft:looting",lvl:4s},{id:"minecraft:mending",lvl:1s}]}  
Shovel give @p minecraft:diamond_shovel{Enchantments:[{id:"minecraft:fortune",lvl:4s},{id:"minecraft:mending",lvl:1s},{id:"minecraft:efficiency",lvl:4s}]}  
Bow give @p minecraft:bow{Enchantments:[{id:"minecraft:infinity",lvl:1s},{id:"minecraft:mending",lvl:1s}]}  
Arrows give @p minecraft:arrow 64  
Torches give @p minecraft:torch 64  
Fishing give @p minecraft:fishing_rod{Enchantments:[{id:"minecraft:lure",lvl:4s},{id:"minecraft:luck_of_the_sea",lvl:4s},{id:"minecraft:mending",lvl:1s}]}  
Riptide trident give @p minecraft:trident{Enchantments:[{id:"minecraft:loyalty",lvl:4s},{id:"minecraft:mending",lvl:1s},{id:"minecraft:riptide",lvl:4s}]}  
Channeling trident give @p minecraft:trident{Enchantments:[{id:"minecraft:loyalty",lvl:4s},{id:"minecraft:mending",lvl:1s},{id:"minecraft:channeling",lvl:1s}]}  
Weather rain weather rain  
Weather thunder weather thunder  
Birch signs give @p minecraft:birch_sign 16  
Bucket of water give @p minecraft:water_bucket  
Bucket of milk give @p minecraft:milk_bucket  
Bucket of lava give @p minecraft:lava_bucket  
Water bottles give @p minecraft:potion{Potion:"minecraft:water"} 3  
Blaze powder give @p minecraft:blaze_powder 16  
Brewing stand give @p minecraft:brewing_stand  
Magma cream give @p minecraft:magma_cream  

Here's what that table looks like in Org Mode:

2023-04-10_09-55-57.png
Figure 2: With column width

Here's how to combine multiple tables:

#+begin_src emacs-lisp :results silent :var quick=mc-quick :var effects=mc-effects :var items=mc-items :exports code
(my-minecraft-book "Book from multiple tables" "sachac" (append quick effects items))
#+end_src

Now producing instance-specific books is just a matter of including the sections I want, like a table that has coordinates for different bases in that particular instance.

I thought about making an Org link type for click commands and some way of exporting that will convert to JSON and keep the whitespace. That way, I might be able to write longer notes and export them to Minecraft book JSON for in-game references, such as notes on villager blocks or potion ingredients. The table + Emacs Lisp approach is already quite useful for quick shortcuts, though, and it was easy to write. We'll see if we need more fanciness!

View org source for this post

Using rubik.el to make SVG last-layer diagrams from algorithms

| cubing, emacs, org

So I checked out emacs-cube, but I had a hard time figuring out how to work with the data model without getting into all the rendering because it figures "left" and "right" based on camera position. rubik.el seemed like an easier starting point. As far as I can tell, the rubik-cube-state local variable is an array with the faces specified as 6 groups of 9 integers in this order: top, front, right, back, left, bottom, with cells specified from left to right, top to bottom.

First, I wanted to recolour rubik so that it matched the setup of the Roofpig JS library I'm using for animations.

(defconst my-cubing-rubik-faces "YRGOBW")
;; make it match roofpig's default setup with yellow on top and red in front
(defconst rubik-faces [rubik-yellow
                       rubik-red
                       rubik-green
                       rubik-orange
                       rubik-blue
                       rubik-white])

Here are some functions to apply an algorithm (or actually, the inverse of the algorithm, which is useful for exploring a PLL case):

(defun my-cubing-normalize (alg)
  "Remove parentheses and clean up spaces in ALG."
  (string-trim
   (replace-regexp-in-string "[() ]+" " " alg)))

(defun my-cubing-reverse-alg (alg)
  "Reverse the given ALG."
  (mapconcat
   (lambda (step)
     (if (string-match "\\`\\([rludfsbRLUDFSBxyz]\\)\\(['i]\\)?\\'" step)
         (concat (match-string 1 step)
                 (if (match-string 2 step)
                     ""
                   "'"))
       step))
   (reverse
    (split-string (my-cubing-normalize alg) " "))
   " "))

(defun my-cubing-rubik-alg (alg)
  "Apply the reversed ALG to a solved cube.
Return the rubik.el cube state."
  (let ((reversed (my-cubing-reverse-alg alg)))
    (seq-reduce
     (lambda (cube o)
       (when (intern (format "rubik-%s"
                             (replace-regexp-in-string "'" "i" o)))
         (unless (string= o "")
           (rubik-apply-transformation
            cube
            (symbol-value
             (intern
              (format "rubik-%s"
                      (replace-regexp-in-string "'" "i" o)))))))
       cube)
     (split-string reversed " ")
     (rubik-make-initial-cube))))

Then I got the strings specifying the side colours and the top colours in the format that I needed for the SVG diagrams. I'm optimistically using number-sequence here instead of hard-coding the numbers so that I can figure out how to extend the idea for 4x4 someday.

(defun my-cubing-rubik-top-face-strings (&optional cube)
  ;; edges starting from back left
  (let ((cube (or cube rubik-cube-state)))
    (list
     (mapconcat
      (lambda (i)
        (char-to-string (elt my-cubing-rubik-faces (aref cube i))))
      (append
       (reverse (number-sequence (* 3 9) (+ 2 (* 3 9))))
       (reverse (number-sequence (* 2 9) (+ 2 (* 2 9))))
       (reverse (number-sequence (* 1 9) (+ 2 (* 1 9))))
       (reverse (number-sequence (* 4 9) (+ 2 (* 4 9))))))
     (mapconcat
      (lambda (i)
        (char-to-string (elt my-cubing-rubik-faces (aref cube i))))
      (number-sequence 0 8)))))

Then theoretically, it can make a diagram like this:

(defun my-cubing-rubik-last-layer-with-sides-from-alg (alg &optional arrows)
  (apply 'my-cubing-last-layer-with-sides
         (append
          (my-cubing-rubik-top-face-strings (my-cubing-rubik-alg alg))
          (list
           arrows))))

So I can invoke it with:

(my-cubing-rubik-last-layer-with-sides-from-alg
 "R U R' F' R U R' U' R' F R2 U' R' U'"
 '((1 7 t) (2 8 t)))
last-layer.svg

It's also nice to be able to interactively step through the algorithm. I prefer a more compact view of the undo/redo state.

;; Override undo information
(defun rubik-display-undo ()
  "Insert undo information at point."
  (cl-loop with line-str = "\nUndo: "
           for cmd in (reverse (cdr rubik-cube-undo))
           for i = 1 then (1+ i)
           do (progn
                (setq line-str (concat line-str (format "%s " (get cmd 'name))))
                (when (> (length line-str) fill-column)
                  (insert line-str)
                  (setq line-str (concat "\n" (make-string 6 ?\s)))))
           finally (insert line-str)))

;; Override redo information
(defun rubik-display-redo ()
  "Insert redo information at point."
  (cl-loop with line-str = "\nRedo: "
           for cmd in (cdr rubik-cube-redo)
           for i = 1 then (1+ i)
           do (progn
                (setq line-str (concat line-str (format "%s " (get cmd 'name))))
                (when (> (length line-str) fill-column)
                  (insert line-str)
                  (setq line-str (concat "\n" (make-string 6 ?\s)))))
           finally (insert line-str)))
  
(defun my-cubing-convert-alg-to-rubik-commands (alg)
  (mapcar
   (lambda (step)
     (intern
      (format "rubik-%s-command"
              (replace-regexp-in-string "'" "i" step))))
   (split-string (my-cubing-normalize alg) " ")))

(rubik-define-commands
  rubik-U "U" rubik-U2 "U2" rubik-Ui "U'"
  rubik-F "F" rubik-F2 "F2" rubik-Fi "F'"
  rubik-R "R" rubik-R2 "R2" rubik-Ri "R'"
  rubik-L "L" rubik-L2 "L" rubik-Li "L'"
  rubik-B "B" rubik-B2 "B" rubik-Bi "B'"
  rubik-D "D" rubik-D2 "D" rubik-Di "D'"
  rubik-x "x" rubik-x2 "x" rubik-xi "x'"
  rubik-y "y" rubik-y2 "y" rubik-yi "y'"
  rubik-z "z" rubik-z2 "z2" rubik-zi "z'")

(defun my-cubing-rubik-set-to-alg (alg)
  (interactive "MAlg: ")
  (rubik)
  (fit-window-to-buffer)
  (setq rubik-cube-state (my-cubing-rubik-alg alg))
  (setq rubik-cube-redo (append (list 'redo)
                                (my-cubing-convert-alg-to-rubik-commands
                                 alg)))
  (setq rubik-cube-undo '(undo))
  (rubik-draw-all)
  (display-buffer (current-buffer)))

And now I can combine all those pieces together in a custom Org link type that will allow me to interactively step through an algorithm if I open it within Emacs and that will export to a diagram and an animation.

(org-link-set-parameters
 "3x3"
 :follow #'my-cubing-rubik-open
 :export #'my-cubing-rubik-export)

(defun my-cubing-rubik-open (path &optional _)
  (my-cubing-rubik-set-to-alg (if (string-match "^\\(.*\\)\\?\\(.*\\)$" path)
                                  (match-string 1 path)
                                path))) 
  
(defun my-cubing-rubik-export (path _ format _)
  "Export PATH to FORMAT."
  (let (alg arrows params)
    (setq alg path)
    (when (string-match "^\\(.*\\)\\?\\(.*\\)$" path)
      (setq alg (match-string 1 path)
            params (org-protocol-convert-query-to-plist (match-string 2 path))
            arrows
            (mapcar (lambda (entry)
                      (mapcar 'string-to-number
                               (split-string entry "-"))) 
                    (split-string
                     (plist-get params :arrows) ","))))
    (concat
     (my-cubing-rubik-last-layer-with-sides-from-alg
      alg
      arrows)
     (format "<div class=\"roofpig\" data-config=\"base=PLL|alg=%s\"></div>"
             (my-cubing-normalize alg)))))

Let's try that with this F-perm, which I haven't memorized yet:

[[3x3:(R' U' F')(R U R' U')(R' F R2 U')(R' U' R U)(R' U R)?arrows=1-7,7-1,2-8,8-2]]

At some point, I'd like to change the display for rubik.el so that it uses SVGs. (Or the OpenGL hacks in https://github.com/Jimx-/emacs-gl, but that might be beyond my current ability.) In the meantime, this might be fun.

In rubik.el, M-r redoes a move and M-u undoes it. Here's what it looks like with my tweaked interface:

output-2023-02-09-15:25:42.gif
Figure 1: Animated GIF of rubik.el stepping through an F-perm
View org source for this post