EmacsConf.org
Sacha Chua
emacsconf.org/2023/talks/emacsconf
Starting from the next slide, you can press “play” once in the audio controls (or type “a”) to start the presentation, which advances automatically afterwards.
1. Overview
How we use Org Mode and TRAMP to organize and run a multi-track conference
1.1. Reasons for making this presentation
Revisit scrambled-together code and document things along the way
Show the process of thinking about a complex project and building it up
Point to more information
https://emacsconf.org/2023/talks/emacsconf
1.2. Map
2. Organizing information
E-mail: Notmuch + Emacs
Private conf.org
Public webpages
Mailing lists
Backstage area
Public media files
Configuration, etc.
Storing talk information as properties
2.1. Basic talk properties
* WAITING_FOR_PREREC EmacsConf.org: How we use Org Mode and TRAMP to organize and run a multi-track conference SCHEDULED: <2023-12-03 Sun 14:55-15:15> :PROPERTIES: :CUSTOM_ID: emacsconf :SLUG: emacsconf :NAME: Sacha Chua :NAME_SHORT: Sacha :END:
2.2. Capturing a talk proposal from an e-mail
2.3. Parsing a submission from e-mail
(defun emacsconf-mail-parse-submission (body) "Extract data from EmacsConf submissions in BODY." (when (listp body) (setq body (plist-get (car body) :content))) (let* ((data (list :body body)) (fields '((:title "^[* ]*Talk title") (:description "^[* ]*Talk description") (:format "^[* ]*Format") (:intro "^[* ]*Introduction for you and your talk") (:name "^[* ]*Speaker name") (:availability "^[* ]*Speaker availability") (:q-and-a "^[* ]*Preferred Q&A approach") (:public "^[* ]*Public contact information") (:private "^[* ]*Private emergency contact information") (:release "^[* ]*Please include this speaker release"))) field (field-regexp (mapconcat (lambda (o) (concat "\\(?:" (cadr o) "\\)")) fields "\\|"))) (with-temp-buffer (insert body) (goto-char (point-min)) ;; Try to parse it (catch 'done (while (not (eobp)) ;; skip the field title (unless (looking-at field-regexp) (unless (re-search-forward field-regexp nil t) (throw 'done nil))) (goto-char (match-beginning 0)) (setq field (seq-find (lambda (o) (looking-at (cadr o))) fields)) (when field ;; get the text between this and the next field (re-search-forward "\\(:[ \t\n]+\\|\n\n\\)" nil t) (setq data (plist-put data (car field) (buffer-substring (point) (or (and (re-search-forward field-regexp nil t) (goto-char (match-beginning 0)) (point)) (point-max)))))))) (if (string-match "[0-9]+" (or (plist-get data :format) "")) (plist-put data :time (match-string 0 (or (plist-get data :format) "")))) data)))
2.4. Setting the property to a region
(defun my-org-set-property (property value) "In the current entry, set PROPERTY to VALUE. Use the region if active." (interactive (list (org-read-property-name) (when (region-active-p) (replace-regexp-in-string "[ \n\t]+" " " (buffer-substring (point) (mark)))))) (org-set-property property value))
2.5. Setting properties
3. Calculating and then editing properties
3.1. Calculating and then editing properties
:FILE_PREFIX: emacsconf-year-slug--title--speakers
3.2. Setting properties that are unset
(defun emacsconf-set-file-prefixes () "Set the FILE_PREFIX property for each talk entry that needs it." (interactive) (org-map-entries (lambda () (org-entry-put (point) "FILE_PREFIX" (emacsconf-file-prefix (emacsconf-get-talk-info-for-subtree)))) "SLUG={.}-FILE_PREFIX={.}"))
4. Renaming files
4.1. Renaming files with Emacs Lisp
(defun emacsconf-rename-files (talk &optional filename) "Rename the marked files or the current file to match TALK. If FILENAME is specified, use that as the extra part of the filename after the prefix. This is useful for distinguishing files with the same extension. Return the list of new filenames." (interactive (list (emacsconf-complete-talk-info))) (prog1 (mapcar (lambda (file) (let* ((extra (or filename (read-string (format "Filename (%s): " (file-name-base file))))) (new-filename (expand-file-name (concat (plist-get talk :file-prefix) (if (string= extra "") "" (concat "--" extra)) "." (file-name-extension file)) (file-name-directory file)))) (rename-file file new-filename t) new-filename)) (or (dired-get-marked-files) (list (buffer-file-name)))) (when (derived-mode-p 'dired-mode) (revert-buffer))))
5. Renaming files with shell scripts
5.1. Exporting the information as JSON
(defun emacsconf-publish-talks-json () "Return JSON format with a subset of talk information." (let ((emacsconf-publishing-phase 'conference)) (json-encode (list :talks (mapcar (lambda (o) (apply 'list (cons :start-time (format-time-string "%FT%T%z" (plist-get o :start-time) t)) (cons :end-time (format-time-string "%FT%T%z" (plist-get o :end-time) t)) (mapcar (lambda (field) (cons field (plist-get o field))) '(:slug :title :speakers :pronouns :pronunciation :url :track :file-prefix :qa-url :qa-type :qa-backstage-url)))) (emacsconf-filter-talks (emacsconf-get-talk-info)))))))
5.2. Using the data with JQ
roles/prerec/templates/rename-original.sh
#!/bin/bash # {{ ansible_managed }} # Usage: rename-original.sh $slug $file [$extra] [$talks-json] SLUG=$1 FILE=$2 TALKS_JSON=${4:-~/current/talks.json} EXTRA="" if [ -z ${3-unset} ]; then EXTRA="" elif [ -n "$3" ]; then EXTRA="$3" elif echo "$FILE" | grep -e '\(webm\|mp4\|mov\)'; then EXTRA="--original" fi filename=$(basename -- "$FILE") extension="${filename##*.}" filename="${filename%.*}" FILE_PREFIX=$(jq -r '.talks[] | select(.slug=="'$SLUG'")["file-prefix"]' < $TALKS_JSON) if echo "$FILE" | grep -q \\. ; then mv "$FILE" $FILE_PREFIX$EXTRA.$extension echo $FILE_PREFIX$EXTRA.$extension else mv "$FILE" $FILE_PREFIX$EXTRA echo $FILE_PREFIX$EXTRA fi # Copy to original if needed if [ -f $FILE_PREFIX--original.webm ] && [ ! -f $FILE_PREFIX--main.$extension ]; then cp $FILE_PREFIX--original.$extension $FILE_PREFIX--main.webm fi
6. Parsing availability
6.1. Setting the timezone
6.2. The code
(defun emacsconf-timezone-set (timezone) "Set the timezone for the current Org entry." (interactive (list (progn (require 'tzc) (completing-read "Timezone: " tzc-time-zones)))) (org-entry-put (point) "TIMEZONE" timezone))
6.3. Converting timezones and encoding the availability
<= 11:00 EST
6.4. Validating errors
7. Scheduling talks
7.0.1. Schedule as table
7.0.2. Schedule as SVG
7.0.3. Testing different schedules
8. Translating times
8.1. Translating times
As of the time I write this e-mail, your tentative schedule is: Improving compiler diagnostics with Overlays Dec 2 Sat 1:00 pm EST translated to your local timezone US/Pacific: Dec 2 Sat 10:00 am PST I'm using ">= 11:00 -0500 (08:00 US/Pacific)" as the availability constraint for you when planning the talks, but since I sometimes mess up encoding these things, could you please doublecheck that this works for you?
format-time-string
9. Templates
9.1. A little function for templates
- tempo?
- s-lex-format:
(let ((x 1)) (s-lex-format "x is: ${x}")) - format: something %s something %s something %s … ?
- concat: hard to get overall view
9.2. The code
(defun emacsconf-replace-plist-in-string (attrs string) "Replace ${keyword} from ATTRS in STRING." (let ((a attrs) name val) (while a (setq name (pop a) val (pop a)) (when (stringp val) (setq string (replace-regexp-in-string (regexp-quote (concat "${" (substring (symbol-name name) 1) "}")) (or val "") string t t)))) string))
(emacsconf-replace-plist-in-string '(:test "Hello" :another-var "world") "Test 1: ${test} Test 2: ${another-var}") ;; Test 1: Hello Test 2: world
9.3. Getting all the talk information
(defun emacsconf-get-talk-info () (with-current-buffer (find-file-noselect emacsconf-org-file) (save-excursion (save-restriction (widen) (let (results) (org-map-entries (lambda () (when (or (org-entry-get (point) "TIME") (org-entry-get (point) "SLUG") (org-entry-get (point) "INCLUDE_IN_INFO")) (setq results (cons (emacsconf-get-talk-info-for-subtree) results))))) (nreverse results))))))
9.4. Talk info functions
(defun emacsconf-get-talk-info-for-subtree () "Run `emacsconf-talk-info-functions' to extract the info for this entry." (seq-reduce (lambda (prev val) (save-excursion (save-restriction (funcall val prev)))) emacsconf-talk-info-functions nil))
(defvar emacsconf-talk-info-functions '(emacsconf-get-talk-info-from-properties emacsconf-get-talk-categories emacsconf-get-talk-abstract-from-subtree emacsconf-get-talk-logbook emacsconf-add-talk-status emacsconf-add-checkin-time emacsconf-add-timezone-conversions emacsconf-add-speakers-with-pronouns emacsconf-add-live-info) "Functions to collect information.")
9.5. Getting info from the Org file
(defun emacsconf-get-talk-abstract-from-subtree (o) "Add the abstract from a subheading. The subheading should match `emacsconf-abstract-heading-regexp'." (plist-put o :abstract (substring-no-properties (or (emacsconf-get-subtree-entry "abstract") ""))))
9.6. Modifying talk info
(defun emacsconf-add-talk-status (o) "Add status label and public info." (plist-put o :status-label (or (assoc-default (plist-get o :status) emacsconf-status-types 'string= "") (plist-get o :status))) (when (or (member (plist-get o :status) (split-string "PLAYING CLOSED_Q OPEN_Q UNSTREAMED_Q TO_ARCHIVE TO_EXTRACT TO_FOLLOW_UP DONE")) (time-less-p (plist-get o :start-time) (current-time))) (plist-put o :public t)) o)
9.7. Performance
memoize for performance
10. Updating the conference wiki
10.1. The conference wiki
10.2. The code
;; from emacsconf-publish-format-talk-schedule-info (emacsconf-replace-plist-in-string (append o (list :irc-info (format "Discuss on IRC: [#%s](%s) \n" (plist-get o :channel) (plist-get o :webchat-url)) ; ... more entries )) "[[!toc ]] Format: ${format} ${pad-info}${irc-info}${status-info}${schedule-info}\n ${alternate-apac-info}\n")
10.3. Schedule
10.3.1. The Ikiwiki directive
The Ikiwiki markup looks like this:
[[!template id=sched time="""20""" q-and-a="""<a href="https://media.emacsconf.org/2023/current/bbb-emacsconf.html">BBB</a>""" startutc="""2023-12-02T19:50:00+0000""" endutc="""2023-12-02T20:10:00+0000""" start="""2:50""" end="""3:10""" title="""EmacsConf.org: How we use Org Mode and TRAMP to organize and run a multi-track conference""" url="""/2023/talks/emacsconf""" speakers="""Sacha Chua""" track="""Development""" watch="""https://emacsconf.org/2023/watch/dev""" slug="""emacsconf""" note=""""""]]
10.3.2. Emacs Lisp
emacsconf-publish-sched-directive in emacsconf-publish.el
(format "[[!template id=sched%s]]" (let ((result "") (attrs ;; ... some code to make a property list based on the talk ... )) (while attrs (let ((field (pop attrs)) (val (pop attrs))) (when val (setq result (concat result " " (substring (symbol-name field) 1) "=\"\"\"" val "\"\"\""))))) result))
10.4. Navigation
10.5. Emacs Lisp
(defun emacsconf-publish-nav-pages (&optional talks) "Generate links to the next and previous talks. During the schedule and conference phase, the talks are sorted by time. Otherwise, they're sorted by track and then schedule." (interactive (list (emacsconf-publish-prepare-for-display (or emacsconf-schedule-draft (emacsconf-get-talk-info))))) (let* ((next-talks (cdr talks)) (prev-talks (cons nil talks)) (label (if (member emacsconf-publishing-phase '(schedule conference)) "time" "track"))) (unless (file-directory-p (expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory))) (mkdir (expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory)))) (while talks (let* ((o (pop talks)) (next-talk (emacsconf-format-talk-link (pop next-talks))) (prev-talk (emacsconf-format-talk-link (pop prev-talks)))) (with-temp-file (expand-file-name (format "%s-nav.md" (plist-get o :slug)) (expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory))) (insert (concat "\n<div class=\"talk-nav\"> Back to the [[talks]] \n" (if prev-talk (format "Previous by %s: %s \n" label prev-talk) "") (if next-talk (format "Next by %s: %s \n" label next-talk) "") (if (plist-get o :track) ; tagging doesn't work here because ikiwiki will list the nav page (format "Track: <span class=\"sched-track %s\">%s</span> \n" (plist-get o :track) (plist-get o :track)) "") "</div> ")))))))
11. Etherpad
11.1. API
(defun emacsconf-pad-set-html (pad-id html) "Set PAD-ID contents to the given HTML." (interactive "MPad ID: \nMHTML: ") (let ((url-request-data (concat "html=" (url-hexify-string html))) (url-request-method "POST") (url-request-extra-headers '(("Content-Type" . "application/x-www-form-urlencoded")))) (emacsconf-pad-json-request (format "%sapi/1/setHTML?apikey=%s&padID=%s" emacsconf-pad-base (url-hexify-string emacsconf-pad-api-key) (url-hexify-string pad-id)) (called-interactively-p 'any))))
11.2. Pad template
<div>
<div>All talks: ${talks}</div>
<div><strong>${title}</strong></div>
<div>${base-url}${url} - ${speakers} - Track: ${track}</div>
<div>Watch/participate: ${watch}</div>
${bbb-info}
<div>IRC: ${irc-nick-details} https://chat.emacsconf.org/#/connect?join=emacsconf,emacsconf-${track-id} or #emacsconf-${track-id} on libera.chat network</div>
<div>Guidelines for conduct: ${base-url}conduct</div>
<div>See end of file for license (CC Attribution-ShareAlike 4.0 + GPLv3 or later)</div>
<div>----------------------------------------------------------------</div>
<div>Notes, discussions, links, feedback:</div>
<ul>${notes}</ul>
<div>----------------------------------------------------------------</div>
...
11.3. Checking the pad timestamp
(defun emacsconf-pad-prepopulate-talk-pad (o) "Fill in the pad for O." (interactive (list (let ((info (emacsconf-include-next-talks (emacsconf-get-talk-info) emacsconf-pad-number-of-next-talks))) (emacsconf-complete-talk-info info)))) (let ((pad-id (emacsconf-pad-id o))) (emacsconf-pad-create-pad pad-id) (when (or emacsconf-pad-force-all (not (emacsconf-pad-modified-p pad-id)) (progn (browse-url (emacsconf-pad-url o)) (y-or-n-p (format "%s might have been modified. Reset? " (plist-get o :slug))))) (emacsconf-pad-set-html pad-id (emacsconf-pad-initial-content o)) (save-window-excursion (emacsconf-with-talk-heading (plist-get o :slug) (let-alist (emacsconf-pad-get-last-edited pad-id) (org-entry-put (point) "PAD_RESET" (number-to-string .data.lastEdited))))))))
12. Mail merge
12.0.1. Mail merge
(defun emacsconf-mail-review (talk) "Let the speaker know we've received their proposal." (interactive (list (emacsconf-complete-talk-info))) (emacsconf-mail-prepare '(:subject "${conf-name} ${year} review: ${title}" :cc "emacsconf-submit@gnu.org" :reply-to "emacsconf-submit@gnu.org, ${email}, ${user-email}" :mail-followup-to "emacsconf-submit@gnu.org, ${email}, ${user-email}" :body " Hi, ${speakers-short}! Thanks for submitting your proposal! (ZZZ TODO: feedback) We'll wait a week (~ ${notification-date}) in case the other volunteers want to chime in regarding your talk. =) ${signature} ") (plist-get talk :email) (list :user-email user-mail-address :signature user-full-name :title (plist-get talk :title) :email (plist-get talk :email) :conf-name emacsconf-name :speakers-short (plist-get talk :speakers-short) :year emacsconf-year :notification-date (plist-get talk :date-to-notify))))
12.1. Mass e-mails
(defun emacsconf-mail-upload-and-backstage-info (group) "E-mail upload and backstage access information to GROUP." (interactive (list (emacsconf-mail-complete-email-group))) (emacsconf-mail-prepare (list :subject "${conf-name} ${year}: Upload instructions, backstage info" :reply-to "emacsconf-submit@gnu.org, ${email}, ${user-email}" :mail-followup-to "emacsconf-submit@gnu.org, ${email}, ${user-email}" :log-note "sent backstage and upload information" :body "${email-notes}Hi ${name}! Hope things are going well for you! I got the upload service up and running so you can upload your video${plural} and other talk resources (ex: speaker notes, Org files, slides, etc.). You can access it at ${upload-url} with the password \"${upload-password}\". Please let me know if you run into technical issues.${fill} If you can get your file(s) uploaded by ${video-target-date}, that would give us plenty of time to reencode it, edit captions, and so on. If life has gotten a bit busier than expected, that's okay. You can upload it when you can or we can plan to do your presentation live. (Late submissions and live presentations are a bit more stressful, but it'll probably work out.)${fill} We've also set up ${backstage} as the backstage area where you can view the videos and resources uploaded so far. You can access it with the username \"${backstage-user}\" and the password \"${backstage-password}\". Please keep the backstage password and other speakers' talk resources secret. This is a manual process, so your talk won't immediately show up there once you've upload it. Once we've processed your talk, you can use the backstage area to check if your talk has been correctly reencoded and see the progress on captions. You can also check out other people's talks. Enjoy!${fill} Thank you so much for contributing to ${conf-name} ${year}! ${signature}") (car group) (list :email-notes (emacsconf-surround "ZZZ: " (plist-get (cadr group) :email-notes) "\n\n" "") :backstage (emacsconf-replace-plist-in-string (list :year emacsconf-year) "https://media.emacsconf.org/${year}/backstage/") :plural (if (= 1 (length (cdr group))) "" "s") :backstage-user emacsconf-backstage-user :backstage-password emacsconf-backstage-password :upload-url (concat "https://ftp-upload.emacsconf.org/?sid=" emacsconf-upload-password "-" (mapconcat (lambda (o) (plist-get o :slug)) (cdr group) "-")) :upload-password emacsconf-upload-password :video-target-date emacsconf-video-target-date :name (plist-get (cadr group) :speakers-short) :email (car group) :user-email user-mail-address :signature user-full-name :conf-name emacsconf-name :year emacsconf-year)))
13. BigBlueButton
13.1. Creating BigBlueButton rooms from Emacs Lisp with Spookfox
(defun emacsconf-spookfox-create-bbb (group) "Create a BBB room for this group of talks. GROUP is (email . (talk talk talk)). Needs a Spookfox connection." (let* ((bbb-name (format "%s (%s) - %s%s" (mapconcat (lambda (o) (plist-get o :slug)) (cdr group) ", ") (plist-get (cadr group) :speakers) emacsconf-id emacsconf-year)) path (retrieve-command (format "window.location.origin + [...document.querySelectorAll('h4.room-name-text')].find((o) => o.textContent.trim() == '%s').closest('tr').querySelector('.delete-room').getAttribute('data-path')" bbb-name)) (create-command (format "document.querySelector('#create-room-block').click(); document.querySelector('#create-room-name').value = \"%s\"; document.querySelector('#room_mute_on_join').click(); document.querySelector('.create-room-button').click();" bbb-name))) (setq path (spookfox-js-injection-eval-in-active-tab retrieve-command t)) (unless path (dolist (cmd (split-string create-command ";")) (spookfox-js-injection-eval-in-active-tab cmd t) (sleep-for 2)) (sleep-for 2) (setq path (spookfox-js-injection-eval-in-active-tab retrieve-command t))) (when path (dolist (talk (cdr group)) (save-window-excursion (emacsconf-with-talk-heading talk (org-entry-put (point) "ROOM" path)))) (cons bbb-name path))))
14. Check-in instructions
14.1. Sending check-in instructions
(defun emacsconf-mail-checkin-instructions-to-all () "Draft check-in instructions for all speakers." (interactive) (let* ((talks (seq-filter (lambda (o) (and (plist-get o :email) (plist-get o :q-and-a))) (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info)))) (by-attendance (seq-group-by (lambda (o) (null (string-match "after" (plist-get o :q-and-a)))) talks))) (dolist (group (emacsconf-mail-groups (assoc-default nil by-attendance))) (emacsconf-mail-checkin-instructions-for-nonattending-speakers group)) (dolist (group (emacsconf-mail-groups (assoc-default t by-attendance))) (emacsconf-mail-checkin-instructions-for-attending-speakers group))))
15. Redirect
15.1. Dynamic redirects
(defun emacsconf-publish-bbb-redirect (talk &optional status) "Update the publicly-available redirect for TALK." (interactive (list (emacsconf-complete-talk-info))) (let ((bbb-filename (expand-file-name (format "bbb-%s.html" (plist-get talk :slug)) emacsconf-publish-current-dir)) (bbb-redirect-url (concat "https://media.emacsconf.org/" emacsconf-year "/current/bbb-" (plist-get talk :slug) ".html")) (status (or status (emacsconf-bbb-status (if (boundp 'org-state) (append (list :status org-state) talk) talk))))) (with-temp-file bbb-filename (insert (emacsconf-replace-plist-in-string (append talk (list :base-url emacsconf-base-url :bbb-redirect-url bbb-redirect-url)) (pcase status ('before "<html><head><meta http-equiv=\"refresh\" content=\"5; URL=${bbb-redirect-url}\"></head><body> The Q&A room for ${title} is not yet open. This page will refresh every 5 seconds until the BBB room is marked as open, or you can refresh it manually.</body></html>") ('open "<html><head><meta http-equiv=\"refresh\" content=\"0; URL=${bbb-room}\"></head><body> The live Q&A room for ${title} is now open. You should be redirected to <a href=\"${bbb-room}\">${bbb-room}</a> automatically, but if not, please visit the URL manually to join the Q&A.</body></html>") ('after "<html><head><body> The Q&A room for ${title} has finished. You can find more information about the talk at <a href=\"${base-url}${url}\">${base-url}${url}</a>.</body></html>") (_ "<html><head><body> There is no live Q&A room for ${title}. You can find more information about the talk at <a href=\"${base-url}${url}\">${base-url}${url}</a>.</body></html>" )))))))
15.2. Static redirects
(defun emacsconf-publish-bbb-static-redirects () "Create emergency redirects that can be copied over the right location." (interactive) (mapc (lambda (state) (let ((emacsconf-publish-current-dir (expand-file-name state (expand-file-name "redirects" (expand-file-name "assets" emacsconf-backstage-dir))))) (unless (file-directory-p emacsconf-publish-current-dir) (make-directory emacsconf-publish-current-dir t)) (mapc (lambda (talk) (emacsconf-publish-bbb-redirect talk (intern state))) (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))) '("before" "open" "after")))
16. Jumping to a talk
16.1. emacsconf-go-to-talk
16.2. Embark actions
17. Adding notes to the Org Mode logbook
17.1. Adding a note
17.2. The code
(defun emacsconf-mail-add-to-logbook (email note) "Add to logbook for all matching talks from this speaker." (interactive (let* ((email (mail-strip-quoted-names (plist-get (plist-get (notmuch-show-get-message-properties) :headers) :From))) (talks (emacsconf-mail-talks email))) (list email (read-string (format "Note for %s: " (mapconcat (lambda (o) (plist-get o :slug)) talks", ")))))) (save-window-excursion (mapc (lambda (talk) (emacsconf-add-to-talk-logbook talk note)) (emacsconf-mail-talks email))))
18. Captions
19. Backstage index with TRAMP