Categories: geek » emacs

RSS - Atom - Subscribe via email

2023-10-16 Emacs news

| emacs, emacs-news

Links from reddit.com/r/emacs, r/orgmode, r/spacemacs, r/planetemacs, Hacker News, communick.news, lobste.rs, kbin, programming.dev, lemmy, planet.emacslife.com, YouTube, the Emacs NEWS file, Emacs Calendar, and emacs-devel. Thanks to Andrés Ramírez for emacs-devel links. Do you have an Emacs-related link or announcement? Please e-mail me at sacha@sachachua.com. Thank you!

#EmacsConf backstage: adding notes to Org logbook drawers from e-mails

| emacs, emacsconf, notmuch

Sometimes I want to work with all the talks associated with an email in my inbox. For example, maybe a speaker said that the draft schedules are fine, and I want to make a note of that in the conference Org file.

First we start with a function that gets the e-mail addresses for a talk. Some speakers have different e-mail addresses for public contact or private contact, and some e-mail us from other addresses.

emacsconf-mail-get-all-email-addresses: Return all the possible e-mail addresses for TALK.
(defun emacsconf-mail-get-all-email-addresses (talk)
  "Return all the possible e-mail addresses for TALK."
  (split-string
   (downcase
    (string-join
     (seq-uniq
      (seq-keep
       (lambda (field) (plist-get talk field))
       '(:email :public-email :email-alias)))
     ","))
   " *, *"))

Then we can use that to find the talks for a given e-mail address.

emacsconf-mail-talks: Return a list of talks matching EMAIL.
(defun emacsconf-mail-talks (email)
  "Return a list of talks matching EMAIL."
  (setq email (downcase (mail-strip-quoted-names email)))
  (seq-filter
   (lambda (o) (member email (emacsconf-mail-get-all-email-addresses o)))
   (emacsconf-get-talk-info)))

We can loop over that to add a note for the e-mail.

emacsconf-mail-add-to-logbook: Add to logbook for all matching talks from this speaker.
(defun emacsconf-mail-add-to-logbook (email note)
  "Add to logbook for all matching talks from this speaker."
  (interactive
   (let* ((email (mail-strip-quoted-names
                  (plist-get (plist-get (notmuch-show-get-message-properties) :headers)
                             :From)))
          (talks (emacsconf-mail-talks email)))
     (list
      email
      (read-string (format "Note for %s: "
                           (mapconcat (lambda (o) (plist-get o :slug))
                                      talks", "))))))
  (save-window-excursion
    (mapc
     (lambda (talk)
       (emacsconf-add-to-talk-logbook talk note))
     (emacsconf-mail-talks email))))

The actual addition of notes is handled by these functions.

emacsconf-add-to-logbook: Add NOTE as a logbook entry for the current subtree.
(defun emacsconf-add-to-logbook (note)
  "Add NOTE as a logbook entry for the current subtree."
  (move-marker org-log-note-return-to (point))
  (move-marker org-log-note-marker (point))
  (with-temp-buffer
    (insert note)
    (let ((org-log-note-purpose 'note))
      (org-store-log-note))))

Then we have a function that looks for the heading for a note and then adds a logbook entry to it.

emacsconf-add-to-talk-logbook: Add NOTE as a logbook entry for TALK.
(defun emacsconf-add-to-talk-logbook (talk note)
  "Add NOTE as a logbook entry for TALK."
  (interactive (list (emacsconf-complete-talk) (read-string "Note: ")))
  (save-excursion
    (emacsconf-with-talk-heading talk
      (emacsconf-add-to-logbook note))))

All together, that makes it easy to use Emacs as a very simple contact relationship management system where I can take notes based on the e-mails that come in.

output-2023-10-14-10:23:29.gif
Figure 1: Logging notes from e-mail

These functions are in emacsconf-mail.el.

#EmacsConf backstage: Using Spookfox to automate creating BigBlueButton rooms in Mozilla Firefox

| emacsconf, emacs, org

Naming conventions make it easier for other people to find things. Just like with file prefixes, I like to use a standard naming pattern for our BigBlueButton web conference rooms. For EmacsConf 2022, we used ec22-day-am-gen Speaker name (slugs). For EmacsConf 2023, I want to set up the BigBlueButton rooms before the schedule settles down, so I won't encode the time or track information into it. Instead, I'll use Speaker name (slugs) - emacsconf2023.

BigBlueButton does have an API for managing rooms, but that requires a shared secret that I don't know yet. I figured I'd just automate it through my browser. Over the last year, I've started using Spookfox to control the Firefox web browser from Emacs. It's been pretty handy for scrolling webpages up and down, so I wondered if I could replace my old xdotool-based automation. Here's what I came up with for this year.

First, I need a function that creates the BBB room for a group of talks and updates the Org entry with the URL. Adding a slight delay makes it a bit more reliable.

emacsconf-spookfox-create-bbb: Create a BBB room for this group of talks.
(defun emacsconf-spookfox-create-bbb (group)
  "Create a BBB room for this group of talks.
GROUP is (email . (talk talk talk)).
Needs a Spookfox connection."
  (let* ((bbb-name
          (format "%s (%s) - %s%s"
                  (mapconcat (lambda (o) (plist-get o :slug)) (cdr group) ", ")
                  (plist-get (cadr group) :speakers)
                  emacsconf-id
                  emacsconf-year))
         path
         (retrieve-command
          (format
           "window.location.origin + [...document.querySelectorAll('h4.room-name-text')].find((o) => o.textContent.trim() == '%s').closest('tr').querySelector('.delete-room').getAttribute('data-path')"
           bbb-name))
         (create-command (format "document.querySelector('#create-room-block').click();
document.querySelector('#create-room-name').value = \"%s\";
document.querySelector('#room_mute_on_join').click();
document.querySelector('.create-room-button').click();"
                                 bbb-name)))
    (setq path (spookfox-js-injection-eval-in-active-tab retrieve-command t))
    (unless path
      (dolist (cmd (split-string create-command ";"))
        (spookfox-js-injection-eval-in-active-tab cmd t)
        (sleep-for 2))
      (sleep-for 2)
      (setq path (spookfox-js-injection-eval-in-active-tab retrieve-command t)))
    (when path
      (dolist (talk (cdr group))
        (save-window-excursion
          (emacsconf-with-talk-heading talk
            (org-entry-put (point) "ROOM" path))))
      (cons bbb-name path))))

Then I need to iterate over the list of talks that have live Q&A sessions but don't have BBB rooms assigned yet so that I can create them.

emacsconf-spookfox-create-bbb-for-live-talks: Create BBB rooms for talks that don’t have them yet.
(defun emacsconf-spookfox-create-bbb-for-live-talks ()
  "Create BBB rooms for talks that don't have them yet."
  (let* ((talks (seq-filter
                 (lambda (o)
                   (and (string-match "live" (or (plist-get o :q-and-a) ""))
                        (not (string= (plist-get o :status) "CANCELLED"))
                        (not (plist-get o :bbb-room))))
                 (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
         (groups (and talks (emacsconf-mail-groups talks))))
    (dolist (group groups)
      (emacsconf-spookfox-create-bbb group))))

The result: a whole bunch of rooms ready for people to check in.

2023-10-14_09-24-34.png
Figure 1: BigBlueButton rooms

Using Spookfox to communicate with Firefox from Emacs Lisp made it easy to get data in and out of my browser. Handy!

This code is in emacsconf-spookfox.el.

#EmacsConf backstage: file prefixes

| emacs, org, emacsconf

Sometimes it makes sense to dynamically generate information related to a talk and then save it as an Org property so that I can manually edit it. For example, we like to name all the talk files using this pattern: "emacsconf-year-slug--title--speakers". That's a lot to type consistently! We can generate most of these prefixes automatically, but some might need tweaking, like when the talk title or speaker names have special characters.

Calculating the file prefix for a talk

First we need something that turns a string into an ID.

emacsconf-slugify: Turn S into an ID.
(defun emacsconf-slugify (s)
  "Turn S into an ID.
Replace spaces with dashes, remove non-alphanumeric characters,
and downcase the string."
  (replace-regexp-in-string
   " +" "-"
   (replace-regexp-in-string
    "[^a-z0-9 ]" ""
    (downcase s))))

Then we can use that to calculate the file prefix for a given talk.

emacsconf-file-prefix: Create the file prefix for TALK
(defun emacsconf-file-prefix (talk)
  "Create the file prefix for TALK."
  (concat emacsconf-id "-"
          emacsconf-year "-"
          (plist-get talk :slug) "--"
          (emacsconf-slugify (plist-get talk :title))
          (if (plist-get talk :speakers)
              (concat "--"
                     (emacsconf-slugify (plist-get talk :speakers)))
            "")))

Then we can map over all the talk entries that don't have FILE_PREFIX defined:

emacsconf-set-file-prefixes: Set the FILE_PREFIX property for each talk entry that needs it.
(defun emacsconf-set-file-prefixes ()
  "Set the FILE_PREFIX property for each talk entry that needs it."
  (interactive)
  (org-map-entries
   (lambda ()
     (org-entry-put
      (point) "FILE_PREFIX"
      (emacsconf-file-prefix (emacsconf-get-talk-info-for-subtree))))
   "SLUG={.}-FILE_PREFIX={.}"))

That stores the file prefix in an Org property, so we can edit it if it needs tweaking.

output-2023-10-10-15:17:17.gif
Figure 1: Setting the FILE_PREFIX for all talks that don't have that yet

Renaming files to match the file prefix

Now that we have that, how can we use it? One way is to rename files from within Emacs. I can mark multiple files with Dired's m command or work on them one at a time. If there are several files with the same extension, I can specify something to add to the filename to tell them apart.

emacsconf-rename-files: Rename the marked files or the current file to match TALK.
(defun emacsconf-rename-files (talk &optional filename)
  "Rename the marked files or the current file to match TALK.
If FILENAME is specified, use that as the extra part of the filename after the prefix.
This is useful for distinguishing files with the same extension.
Return the list of new filenames."
  (interactive (list (emacsconf-complete-talk-info)))
  (prog1
      (mapcar
       (lambda (file)
         (let* ((extra
                 (or filename
                     (read-string (format "Filename (%s): " (file-name-base file)))))
                (new-filename
                 (expand-file-name
                  (concat (plist-get talk :file-prefix)
                          (if (string= extra "")
                              ""
                            (concat "--" extra))
                          "."
                          (file-name-extension file))
                  (file-name-directory file))))
           (rename-file file new-filename t)
           new-filename))
       (or (dired-get-marked-files) (list (buffer-file-name))))
    (when (derived-mode-p 'dired-mode)
      (revert-buffer))))

output-2023-10-10-14:18:34.gif
Figure 2: Renaming multiple files

Working with files on other computers

Because Dired works over TRAMP, I can use that to rename files on a remote server without changing anything about the code. I can open the remote directory with Dired and everything just works.

TRAMP also makes it easy to copy a file to the backstage directory after it's renamed, which saves me having to do that as a separate step.

emacsconf-rename-and-upload-to-backstage: Rename marked files or the current file, then upload to backstage.
(defun emacsconf-rename-and-upload-to-backstage (talk &optional filename)
  "Rename marked files or the current file, then upload to backstage."
  (interactive (list (emacsconf-complete-talk-info)))
  (mapc
   (lambda (file)
     (copy-file
      file
      (expand-file-name
       (file-name-nondirectory file)
       emacsconf-backstage-dir)
      t))
   (emacsconf-rename-files talk)))

So if my emacsconf-backstage-dir is set to /ssh:orga@res:/var/www/res.emacsconf.org/2023/backstage, then it looks up the details for res in my ~/.ssh/config and copies the file there.

Renaming files using information from a JSON

What if I don't want to rename the files from Emacs? If I use Emacs's JSON support to export some information from the talks as a JSON file, then I can easily use that data from the command line.

Here's how I export the talk information:

emacsconf-talks-json: Return JSON format with a subset of talk information.
(defun emacsconf-publish-talks-json ()
  "Return JSON format with a subset of talk information."
  (json-encode
   (list
    :talks
    (mapcar
     (lambda (o)
       (apply
        'list
        (cons :start-time (format-time-string "%FT%T%z" (plist-get o :start-time) t))
        (cons :end-time (format-time-string "%FT%T%z" (plist-get o :end-time) t))
        (mapcar
         (lambda (field)
           (cons field (plist-get o field)))
         '(:slug :title :speakers :pronouns :pronunciation :url :track :file-prefix))))
     (emacsconf-filter-talks (emacsconf-get-talk-info))))))

emacsconf-publish-talks-json-to-files
(defun emacsconf-publish-talks-json-to-files ()
  "Export talk information as JSON so that we can use it in shell scripts."
  (interactive)
  (mapc (lambda (dir)
          (when (and dir (file-directory-p dir))
            (with-temp-file (expand-file-name "talks.json" dir)
              (insert (emacsconf-talks-json)))))
        (list emacsconf-res-dir emacsconf-ansible-directory)))

Then I can use jq to extract the information with

jq -r '.talks[] | select(.slug=="'$SLUG'")["file-prefix"]' < $TALKS_JSON

Here it is in the context of a shell script that renames the given file to match a talk's prefix.

#!/bin/bash
# 
# Usage: rename-original.sh $slug $file [$extra] [$talks-json]
SLUG=$1
FILE=$2
TALKS_JSON=${4:-~/current/talks.json}
EXTRA=""
if [ -z ${3-unset} ]; then
    EXTRA=""
elif [ -n "$3" ]; then
    EXTRA="--$3"
elif echo "$FILE" | grep -e '\(webm\|mp4\|mov\)'; then
    EXTRA="--original"
fi
filename=$(basename -- "$FILE")
extension="${filename##*.}"
filename="${filename%.*}"
FILE_PREFIX=$(jq -r '.talks[] | select(.slug=="'$SLUG'")["file-prefix"]' < $TALKS_JSON)
mv "$FILE" $FILE_PREFIX$EXTRA.$extension
echo $FILE_PREFIX$EXTRA.$extension
# Copy to original if needed
if [ -f $FILE_PREFIX--original.webm ] && [ ! -f $FILE_PREFIX--main.$extension ]; then
    cp $FILE_PREFIX--original.$extension $FILE_PREFIX--main.webm
fi

Then I can use something like rename-original.sh emacsconf video.webm to emacsconf-2023-emacsconf--emacsconforg-how-we-use-org-mode-and-tramp-to-organize-and-run-a-multitrack-conference--sacha-chua--original.webm.

Working with PsiTransfer-uploaded files

JSON support is useful for getting files into our system, too. For EmacsConf 2022, we used PsiTransfer as a password-protected web-based file upload service. That was much easier for speakers to deal with than FTP, especially for large files. PsiTransfer makes a JSON file for each batch of uploads, which is handy because the uploaded files are named based on the key instead of keeping their filenames and extensions. I wrote a function to copy an uploaded file from the PsiTransfer directory to the backstage directory, renaming it along the way. That meant that I could open the JSON for the uploaded files via TRAMP and then copy a file between two remote directories without manually downloading it to my computer.

emacsconf-upload-copy-from-json: Parse PsiTransfer JSON files and copy the uploaded file to the backstage directory.
(defun emacsconf-upload-copy-from-json (talk key filename)
  "Parse PsiTransfer JSON files and copy the uploaded file to the backstage directory.
The file is associated with TALK. KEY identifies the file in a multi-file upload.
FILENAME specifies an extra string to add to the file prefix if needed."
  (interactive (let-alist (json-parse-string (buffer-string) :object-type 'alist)
                 (list (emacsconf-complete-talk-info)
                       .metadata.key
                       (read-string (format "Filename: ")))))
  (let ((new-filename (concat (plist-get talk :file-prefix)
                              (if (string= filename "")
                                  filename
                                (concat "--" filename))
                              "."
                              (let-alist (json-parse-string (buffer-string) :object-type 'alist)
                                (file-name-extension .metadata.name)))))
    (copy-file
     key
     (expand-file-name new-filename emacsconf-backstage-dir)
     t)))

So that's how I can handle things that can be mostly automated but that might need a little human intervention: use Emacs Lisp to make a starting point, tweak it a little if needed, and then make it easy to use that value elsewhere. Renaming files can be tricky, so it's good to reduce the chance for typos!

2023-10-09 Emacs news

| emacs, emacs-news

Links from reddit.com/r/emacs, r/orgmode, r/spacemacs, r/planetemacs, Hacker News, communick.news, lobste.rs, kbin, programming.dev, lemmy, planet.emacslife.com, YouTube, the Emacs NEWS file, Emacs Calendar, and emacs-devel. Thanks to Andrés Ramírez for emacs-devel links. Do you have an Emacs-related link or announcement? Please e-mail me at sacha@sachachua.com. Thank you!

#EmacsConf backstage: reviewing the last message from a speaker

| emacs, emacsconf, notmuch

One of the things I keep an eye out for when organizing EmacsConf is the most recent time we heard from a speaker. Sometimes life happens and speakers get too busy to prepare a video, so we might offer to let them do it live. Sometimes e-mail delivery issues get in the way and we don't hear from speakers because some server in between has spam filters set too strong. So I made a function that lists the most recent e-mail we got from the speaker that includes "emacsconf" in it. That was a good excuse to learn more about tabulated-list-mode.

2023-10-07-13-18-04.svg
Figure 1: Redacted view of most recent e-mails from speakers

I started by figuring out how to get all the e-mail addresses associated with a talk.

emacsconf-mail-get-all-email-addresses: Return all the possible e-mail addresses for TALK.
(defun emacsconf-mail-get-all-email-addresses (talk)
  "Return all the possible e-mail addresses for TALK."
  (split-string
   (downcase
    (string-join
     (seq-uniq
      (seq-keep
       (lambda (field) (plist-get talk field))
       '(:email :public-email :email-alias)))
     ","))
   " *, *"))

Then I figured out the notmuch search to use to get all messages. Some people write a lot, so I limited it to just the ones that have emacsconf as well. Notmuch can return JSON, so that's easy to parse.

emacsconf-mail-notmuch-tag: Tag to use when searching the Notmuch database for mail.
(defvar emacsconf-mail-notmuch-tag "emacsconf" "Tag to use when searching the Notmuch database for mail.")

emacsconf-mail-notmuch-last-message-for-talk: Return the most recent message from the speakers for TALK.
(defun emacsconf-mail-notmuch-last-message-for-talk (talk &optional subject)
  "Return the most recent message from the speakers for TALK.
Limit to SUBJECT if specified."
  (let ((message (json-parse-string
                  (shell-command-to-string
                   (format "notmuch search --limit=1 --format=json \"%s%s\""
                           (mapconcat
                            (lambda (email) (concat "from:" (shell-quote-argument email)))
                            (emacsconf-mail-get-all-email-addresses talk)
                            " or ")
                           (emacsconf-surround
                            " and "
                            (and emacsconf-mail-notmuch-tag (shell-quote-argument emacsconf-mail-notmuch-tag))
                            "" "")
                           (emacsconf-surround
                            " and subject:"
                            (and subject (shell-quote-argument subject)) "" "")))
                  :object-type 'alist)))
    (cons `(email . ,(plist-get talk :email))
          (when (> (length message) 0)
            (elt message 0)))))

Then I could display all the groups of speakers so that it's easy to check if any of the speakers haven't e-mailed us in a while.

emacsconf-mail-notmuch-show-latest-messages-from-speakers: Verify that the email addresses in GROUPS have e-mailed recently.
(defun emacsconf-mail-notmuch-show-latest-messages-from-speakers (groups &optional subject)
  "Verify that the email addresses in GROUPS have e-mailed recently.
When called interactively, pop up a report buffer showing the e-mails
and messages by date, with oldest messages on top.
This minimizes the risk of mail delivery issues and radio silence."
  (interactive (list (emacsconf-mail-groups (seq-filter
                               (lambda (o) (not (string= (plist-get o :status) "CANCELLED")))
                               (emacsconf-get-talk-info)))))
  (let ((results
         (sort (mapcar
                (lambda (group)
                  (emacsconf-mail-notmuch-last-message-for-talk (cadr group) subject))
                groups)
               (lambda (a b)
                 (< (or (alist-get 'timestamp a) -1)
                    (or (alist-get 'timestamp b) -1))))))
    (when (called-interactively-p 'any)
      (with-current-buffer (get-buffer-create "*Mail report*")
        (let ((inhibit-read-only t))
          (erase-buffer))
        (tabulated-list-mode)
        (setq
         tabulated-list-entries
         (mapcar
          (lambda (row)
            (list
             (alist-get 'thread row)
             (vector
              (alist-get 'email row)
              (or (alist-get 'date_relative row) "")
              (or (alist-get 'subject row) ""))))
          results))
        (setq tabulated-list-format [("Email" 30 t)
                                     ("Date" 10 nil)
                                     ("Subject" 30 t)])
        (local-set-key (kbd "RET") #'emacsconf-mail-notmuch-visit-thread-from-summary)
        (tabulated-list-print)
        (tabulated-list-init-header)
        (pop-to-buffer (current-buffer))))
    results))

If I press RET on a line, I can open the most recent thread. This is handled by the emacsconf-mail-notmuch-visit-thread-from-summary, which is simplified by using the thread ID as the tabulated list ID.

2023-10-07-18-21-55.svg
Figure 2: Viewing a thread in a different window

emacsconf-mail-notmuch-visit-thread-from-summary: Display the thread from the summary.
(defun emacsconf-mail-notmuch-visit-thread-from-summary ()
  "Display the thread from the summary."
  (interactive)
  (let (message-buffer)
    (save-window-excursion
      (setq message-buffer (notmuch-show (tabulated-list-get-id))))
    (display-buffer message-buffer t)))

We haven't heard from a few speakers in a while, so I'll probably e-mail them this weekend to double-check that I'm not getting delivery issues with my e-mails to them. If that doesn't get a reply, I might try other communication methods. If they're just busy, that's cool.

It's a lot easier to spot missing or old entries in a table than it is to try to remember who we haven't heard from recently, so hooray for tabulated-list-mode!

This code is in emacsconf-mail.el.

Summarizing #EmacsConf's growth over 5 years by year, and making an animated GIF

| emacs, emacsconf, python

Of course, after I charted EmacsConf's growth in terms of number of submissions and minutes, I realized I also wanted to just sum everything up by year. So here it is:

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = df.drop('Weeks to CFP', axis=1).groupby(['Year']).sum()
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12,6))
fig1 = df['Count'].plot(kind="bar", ax=ax[0], title='Number of submissions')
fig2 = df['Minutes'].plot(kind="bar", ax=ax[1], title='Number of minutes')
fig.get_figure().savefig('emacsconf-by-year.png')
return df
Year Count Minutes
2019 28 429
2020 35 699
2021 44 578
2022 29 512
2023 39 730
emacsconf-by-year.png

I also wanted to make an animated GIF so that the cumulative graphs could be a little easier to understand.

import pandas as pd
import matplotlib.pyplot as plt
import imageio as io
df = pd.DataFrame(data[1:], columns=data[0])
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12,6))
count = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
minutes = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Minutes', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
ax[0].set_ylim([0, count.max().max()])
ax[1].set_ylim([0, minutes.max().max()])
with io.get_writer('emacsconf-combined.gif', mode='I', duration=[500, 500, 500, 500, 1000], loop=0) as writer:
    for year in range(2019, 2024):
        count[year].plot(ax=ax[0], title='Cumulative submissions')
        minutes[year].plot(ax=ax[1], title='Cumulative minutes')
        ax[0].legend(loc='upper left')
        ax[1].legend(loc='upper left')
        for axis in ax:
            for line in axis.get_lines():
                if line.get_label() == '2023':
                    line.set_linewidth(5)
            for line in axis.legend().get_lines():
                if line.get_label() == '2023':
                    line.set_linewidth(5)        
        filename = f'emacsconf-combined-${year}.png'
        fig.get_figure().savefig(filename)
        image = io.v3.imread(filename)
        writer.append_data(image)
emacsconf-combined.gif
Figure 1: Animated GIF showing the cumulative total submissions and minutes

I am not quite sure what kind of story this data tells (aside from the fact that there sure are a lot of great talks), but it was fun to learn how to make more kinds of graphs and animate them too. Could be useful someday. =)