I have lots of different things in my Org Mode inbox. Following the PARA method, I want to file them under projects, areas, resources, or archive so that I can find related things later. Actually, no, I don't want to refile them. I do want to be able to:
find all the pieces related to something when I'm ready to start working on a task
find useful links again, especially if I can use my own words
Refiling is annoying on my phone, so I tend to wait until I'm back at
my computer. But even with org-refile-use-outline-path set to file
and the ability to specify substrings, there's still a bit of
friction.
Tagging is a little easier to do on my phone. I can add a few tags
when I share a webpage or create a task.
I thought it would be nice to have something that automatically
refiles my inbox headings tagged with various tags to other subtrees
where I've set a :TAG_TARGET: property or something like that. For
example, I can set the TAG_TARGET property to emacsconf to mean
that anything tagged with :emacsconf: should get filed under there.
So when I'm ready, I can call my-org-refile-to-tag-targets and have
lots of things disappear from my inbox.
Next step might be to write a function that will refile just the
current subtree (either going straight to the tag target or prompting
me for a destination if there isn't a matching one), so I can look at
stuff, decide if it needs to be scheduled first or something like
that, and then send it somewhere. There must be something I can pass a
property match to and it'll tell me if it matches the current
subtree - probably something along the lines of
org-make-tags-matcher…
announce it on IRC so that the talk details split up the chat log and make it easier to read
publish the media files to https://media.emacsconf.org/2023 (currently password-protected while testing) so that people can view them
update the talk wiki page with the media files and the captions
update the Youtube and Toobnix videos so that they're public: this
is manual for the moment, since I haven't put time into automating
it yet
I have code for most of this, and we've done it successfully for the
past couple of years. I just need to refamiliarize myself with how to
do it and how to set it up for testing during the dry run, and modify
it to work with the new crontab-based system.
Last year, I set up the server with the ability to act as the
controller, so that it wasn't limited to my laptop. The organizers
notebook says I put it in the orga@ user's account, and I probably
ran it under a screen session. I've submitted a talk for this year's
conference, so I can use a test video for that one. First, I need to
update the Ansible configuration for publishing and editing:
I needed to add a version attribute to the git repo checkout in the
Ansible playbook, since we'd switched from master to main. I also
needed to set emacs_version to 29.1 since I started using
seq-keep in my Emacs Lisp functions. For testing, I set
emacsconf-publishing-phase to conference.
Act on TODO state changes
org-after-todo-state-change-hook makes it easy to automatically run
functions when the TODO state changes. I add this hook that runs a
list of functions and passes the talk information so that I don't have
to parse the talk info in each function.
emacsconf-org-after-todo-state-change: Run all the hooks in ‘emacsconf-todo-hooks’.
(defunemacsconf-org-after-todo-state-change ()
"Run all the hooks in `emacsconf-todo-hooks'.If an `emacsconf-todo-hooks' entry is a list, run it only for thetracks with the ID in the cdr of that list."
(let* ((talk (emacsconf-get-talk-info-for-subtree))
(track (emacsconf-get-track (plist-get talk :track))))
(mapc
(lambda (hook-entry)
(cond
((symbolp hook-entry) (funcall hook-entry talk))
((member (plist-get track :id) (cdr hook-entry))
(funcall (car hook-entry) talk))))
emacsconf-todo-hooks)))
This can be enabled and disabled with the following functions.
emacsconf-add-org-after-todo-state-change-hook: Add FUNC to ‘org-after-todo-stage-change-hook’.
We used to scramble to upload all the videos in the days or weeks
after the conference, since the presentations were live. Since we
switched to encouraging speakers to upload videos before the
conference, we've been able to release the videos pretty much as soon
as the talk starts playing. This code in
emacsconf-publish.el takes care of copying the files from
the backstage to the public media directory, republishing the index,
and republishing the playlist. That way, people who come in late or
who want to refer to the video can easily get the full video right
away.
emacsconf-publish-media-files-on-change: Publish the files and update the index.
This is low-priority, but it might be nice to figure out. The easiest
way is probably to use open the Youtube/Toobnix URLs on my computer
and then use either Tampermonkey or Spookfox to set the talk to
public. Someday!
Update the talk status on the server
Last year, I experimented with having the shell scripts automatically
update the status of the talk from TO_STREAM to PLAYING and from
PLAYING to CLOSED_Q. Since I've moved the talk-running into
track-specific crontabs, now I need to sudo back to the orga user
and set XDG_RUNTIME_DIR in order to use emacsclient. I can call
this with sudo -u orga talk $slug $status in the
roles/obs/templates/handle-session script.
#!/bin/bash# # How to use: talk slug from-status-regexp to-status# or talk slug to-statusSLUG="$1"FROM_STATUS="$2"TO_STATUS="$3"XDG_RUNTIME_DIR=/run/user/
if [ "x$TO_STATUS" == "x" ]; thenFROM_STATUS=.
TO_STATUS="$2"ficd#echo "Pulling conf.org..."#git pullecho"Updating status..."XDG_RUNTIME_DIR=$XDG_RUNTIME_DIR emacsclient --eval "(emacsconf-with-todo-hooks (emacsconf-update-talk-status \"$SLUG\" \"$FROM_STATUS\" \"$TO_STATUS\"))" -a emacs
#echo "Committing and pushing in the background"#git commit -m "Update task status for $SLUG from $FROM_STATUS to $TO_STATUS" conf.org#git push &
Testing notes
Looks like everything works fine when I run it from the crontab: the
talk status is updated, the media files are published, the wiki is
updated, and the talks are announced on IRC. Backup plan A is to
manually control the talk status using Emacs on the server. Backup
plan B is to control the talk status using Emacs on my laptop. Backup
plan C is to call the individual functions instead of relying on the
todo state change functions. I think it'll all work out, although I'll
probably want to do another dry run at some point to make sure. Slowly
getting there…
I wanted to use Mermaid to make diagrams, but I ran into this issue when trying to run it:
Error: Could not find Chromium (rev. 1108766). This can occur if either
1. you did not perform an installation before running the script (e.g. `npm install`) or
2. your cache path is incorrectly configured (which is: /home/sacha/.cache/puppeteer).
For (2), check out our guide on configuring puppeteer at https://pptr.dev/guides/configuration.
It turns out that I needed to do the following:
sudo npm install -g puppeteer mermaid @mermaid-js/mermaid-cli --unsafe-perm
# Cache Chromium for my own user
node /usr/lib/node_modules/puppeteer/install.js --unsafe-perm
sudo npm install -g mermaid @mermaid-js/mermaid-cli
ln -s ~/.cache/puppeteer/chrome/linux-117.0.5938.149 ~/.cache/puppeteer/chrome/linux-1108766
ln -s ~/.cache/puppeteer/chrome/linux-117.0.5938.149/chrome-linux64 ~/.cache/puppeteer/chrome/linux-117.0.5938.149/chrome-linux
(The exact versions might be different for your installation.)
Then I could make a Mermaid file and try it out with mmdc -i input.mmd -o output.svg,
and then I could confirm that it works directly from Org with ob-mermaid:
Naming conventions make it easier for other people to find things.
Just like with file prefixes, I like to use a standard naming pattern
for our BigBlueButton web conference rooms. For EmacsConf 2022, we
used ec22-day-am-gen Speaker name (slugs). For
EmacsConf 2023, I want to set up the BigBlueButton rooms before the
schedule settles down, so I won't encode the time or track information
into it. Instead, I'll use Speaker name (slugs) - emacsconf2023.
BigBlueButton does have an API for managing rooms, but that requires a
shared secret that I don't know yet. I figured I'd just automate it
through my browser. Over the last year, I've started using Spookfox to
control the Firefox web browser from Emacs. It's been pretty handy for
scrolling webpages up and down, so I wondered if I could replace my
old xdotool-based automation. Here's what I came up with for this
year.
First, I need a function that creates the BBB room for a group of
talks and updates the Org entry with the URL. Adding a slight delay
makes it a bit more reliable.
emacsconf-spookfox-create-bbb: Create a BBB room for this group of talks.
Sometimes it makes sense to dynamically generate information related
to a talk and then save it as an Org property so that I can manually
edit it. For example, we like to name all the talk files using this
pattern: "emacsconf-year-slug--title--speakers". That's a lot to
type consistently! We can generate most of these prefixes
automatically, but some might need tweaking, like when the talk title
or speaker names have special characters.
First we need something that turns a string into an ID.
emacsconf-slugify: Turn S into an ID.
(defunemacsconf-slugify (s)
"Turn S into an ID.Replace spaces with dashes, remove non-alphanumeric characters,and downcase the string."
(replace-regexp-in-string
" +""-"
(replace-regexp-in-string
"[^a-z0-9 ]"""
(downcase s))))
Then we can use that to calculate the file prefix for a given talk.
emacsconf-file-prefix: Create the file prefix for TALK
Then we can map over all the talk entries that don't have FILE_PREFIX defined:
emacsconf-set-file-prefixes: Set the FILE_PREFIX property for each talk entry that needs it.
(defunemacsconf-set-file-prefixes ()
"Set the FILE_PREFIX property for each talk entry that needs it."
(interactive)
(org-map-entries
(lambda ()
(org-entry-put
(point) "FILE_PREFIX"
(emacsconf-file-prefix (emacsconf-get-talk-info-for-subtree))))
"SLUG={.}-FILE_PREFIX={.}"))
That stores the file prefix in an Org property, so we can edit it if
it needs tweaking.
Figure 1: Setting the FILE_PREFIX for all talks that don't have that yet
Renaming files to match the file prefix
Now that we have that, how can we use it? One way is to rename files
from within Emacs. I can mark multiple files with Dired's m command
or work on them one at a time. If there are several files with the
same extension, I can specify something to add to the filename to tell
them apart.
emacsconf-rename-files: Rename the marked files or the current file to match TALK.
(defunemacsconf-rename-files (talk &optional filename)
"Rename the marked files or the current file to match TALK.If FILENAME is specified, use that as the extra part of the filename after the prefix.This is useful for distinguishing files with the same extension.Return the list of new filenames."
(interactive (list (emacsconf-complete-talk-info)))
(prog1
(mapcar
(lambda (file)
(let* ((extra
(or filename
(read-string (format "Filename (%s): " (file-name-base file)))))
(new-filename
(expand-file-name
(concat (plist-get talk :file-prefix)
(if (string= extra "")
""
(concat "--" extra))
"."
(file-name-extension file))
(file-name-directory file))))
(rename-file file new-filename t)
new-filename))
(or (dired-get-marked-files) (list (buffer-file-name))))
(when (derived-mode-p 'dired-mode)
(revert-buffer))))
Figure 2: Renaming multiple files
Working with files on other computers
Because Dired works over TRAMP, I can use that to rename files on a
remote server without changing anything about the code. I can open the
remote directory with Dired and everything just works.
TRAMP also makes it easy to copy a file to the backstage directory
after it's renamed, which saves me having to do that as a separate
step.
emacsconf-rename-and-upload-to-backstage: Rename marked files or the current file, then upload to backstage.
(defunemacsconf-rename-and-upload-to-backstage (talk &optional filename)
"Rename marked files or the current file, then upload to backstage."
(interactive (list (emacsconf-complete-talk-info)))
(mapc
(lambda (file)
(copy-file
file
(expand-file-name
(file-name-nondirectory file)
emacsconf-backstage-dir)
t))
(emacsconf-rename-files talk)))
So if my emacsconf-backstage-dir is set to
/ssh:orga@res:/var/www/res.emacsconf.org/2023/backstage, then it
looks up the details for res in my ~/.ssh/config and copies the
file there.
Renaming files using information from a JSON
What if I don't want to rename the files from Emacs? If I use Emacs's
JSON support to export some information from the talks as a JSON file,
then I can easily use that data from the command line.
Here's how I export the talk information:
emacsconf-talks-json: Return JSON format with a subset of talk information.
(defunemacsconf-publish-talks-json ()
"Return JSON format with a subset of talk information."
(json-encode
(list
:talks
(mapcar
(lambda (o)
(apply
'list
(cons :start-time (format-time-string "%FT%T%z" (plist-get o :start-time) t))
(cons :end-time (format-time-string "%FT%T%z" (plist-get o :end-time) t))
(mapcar
(lambda (field)
(cons field (plist-get o field)))
'(:slug:title:speakers:pronouns:pronunciation:url:track:file-prefix))))
(emacsconf-filter-talks (emacsconf-get-talk-info))))))
emacsconf-publish-talks-json-to-files
(defunemacsconf-publish-talks-json-to-files ()
"Export talk information as JSON so that we can use it in shell scripts."
(interactive)
(mapc (lambda (dir)
(when (and dir (file-directory-p dir))
(with-temp-file (expand-file-name "talks.json" dir)
(insert (emacsconf-talks-json)))))
(list emacsconf-res-dir emacsconf-ansible-directory)))
Then I can use something like rename-original.sh emacsconf
video.webm to
emacsconf-2023-emacsconf--emacsconforg-how-we-use-org-mode-and-tramp-to-organize-and-run-a-multitrack-conference--sacha-chua--original.webm.
Working with PsiTransfer-uploaded files
JSON support is useful for getting files into our system, too. For
EmacsConf 2022, we used PsiTransfer as a password-protected web-based
file upload service. That was much easier for speakers to deal with
than FTP, especially for large files. PsiTransfer makes a JSON file
for each batch of uploads, which is handy because the uploaded files
are named based on the key instead of keeping their filenames and
extensions. I wrote a function to copy an uploaded file from the
PsiTransfer directory to the backstage directory, renaming it along
the way. That meant that I could open the JSON for the uploaded files
via TRAMP and then copy a file between two remote directories without
manually downloading it to my computer.
emacsconf-upload-copy-from-json: Parse PsiTransfer JSON files and copy the uploaded file to the backstage directory.
(defunemacsconf-upload-copy-from-json (talk key filename)
"Parse PsiTransfer JSON files and copy the uploaded file to the backstage directory.The file is associated with TALK. KEY identifies the file in a multi-file upload.FILENAME specifies an extra string to add to the file prefix if needed."
(interactive (let-alist (json-parse-string (buffer-string) :object-type'alist)
(list (emacsconf-complete-talk-info)
.metadata.key
(read-string (format "Filename: ")))))
(let ((new-filename (concat (plist-get talk :file-prefix)
(if (string= filename "")
filename
(concat "--" filename))
"."
(let-alist (json-parse-string (buffer-string) :object-type'alist)
(file-name-extension .metadata.name)))))
(copy-file
key
(expand-file-name new-filename emacsconf-backstage-dir)
t)))
So that's how I can handle things that can be mostly automated but
that might need a little human intervention: use Emacs Lisp to make a
starting point, tweak it a little if needed, and then make it easy to
use that value elsewhere. Renaming files can be tricky, so it's good
to reduce the chance for typos!
Having helped organize EmacsConf for a number of years now, I know
that I usually panic about whether we have submissions partway through
the call for participation. This causes us to extend the CFP deadline
and ask people to ask more people to submit things, and then we end up
with a wonderful deluge of talks that I then have to somehow squeeze
into a reasonable-looking schedule.
This year, I managed to not panic and I also resisted the urge to extend the
CFP deadline, trusting that there will actually be tons of cool stuff.
It helped that my schedule SVG code let me visualize what the
conference could feel like with the submissions so far, so we started
with a reasonably nice one-track conference and built up from there.
It also helped that I'd gone back to the submissions for 2022 and
plotted them by the number of weeks before the CFP deadline, and I
knew that there'd be a big spike from all those people whose Org
DEADLINE: properties would nudge them into finalizing their
proposals.
Out of curiosity, I wanted to see how the stats for this year compared
with previous years. I wrote a small function to collect the data that I wanted to summarize:
emacsconf-count-submissions-by-week: Count submissions in INFO by distance to CFP-DEADLINE.
(defunemacsconf-count-submissions-by-week (&optional info cfp-deadline)
"Count submissions in INFO by distance to CFP-DEADLINE."
(setq cfp-deadline (or cfp-deadline emacsconf-cfp-deadline))
(setq info (or info (emacsconf-get-talk-info)))
(cons '("Weeks to CFP end date""Count""Hours")
(mapcar (lambda (entry)
(list (car entry)
(length (cdr entry))
(apply '+ (mapcar 'cdr (cdr entry)))))
(seq-group-by
'car
(sort
(seq-keep
(lambda (o)
(and (emacsconf-publish-talk-p o)
(plist-get o :date-submitted)
(cons (floor (/ (days-between (plist-get o :date-submitted) cfp-deadline)
7.0))
(string-to-number
(or (plist-get o :video-duration)
(plist-get o :time)
"0")))))
info)
(lambda (a b) (< (car a) (car b))))))))
and then I ran it against the different files for each year, filling
in the previous years' data as needed. The resulting table is pretty
long, so I've put that in a collapsible section.
Some talks were proposed off-list and are not captured here, and
cancelled or withdrawn talks weren't included either. The times for
previous years use the actual video time, and the times for this year
use proposed times.
Off the top of my head, I didn't know of an easy way to make a pivot
table or cross-tab using just Org Mode or Emacs Lisp. I tried using
datamash, but I was having a hard time getting my output just the way
I wanted it. Fortunately, it was super-easy to get my data from an Org
table into Python so I could use pandas.pivot_table. Because I had
used #+NAME: submissions-by-week to label the table, I could use
:var data=submissions-by-week to refer to the data in my Python
program. Then I could summarize them by week.
Here's the number of submissions by the number of weeks to the
original CFP deadline, so we can see people generally like to target
the CFP date.
import pandas as pd
import matplotlib.pyplot as plt
df= pd.DataFrame(data[1:], columns=data[0])
df= pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True)
fig, ax= plt.subplots()
figure= df.plot(title='Number of submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
for line in plt.legend().get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
figure.get_figure().savefig('number-of-submissions.png')
return df
Weeks to CFP
2019
2020
2021
2022
2023
-12
0
0
0
0
4
-9
0
0
0
0
2
-8
0
0
4
2
0
-7
0
0
2
0
2
-6
0
0
1
0
0
-5
2
1
2
2
2
-4
1
1
2
0
2
-3
0
0
5
2
3
-2
6
1
1
2
5
-1
9
4
12
8
10
0
9
21
13
8
8
1
0
7
1
5
1
2
1
0
1
0
0
Calculating the cumulative number of submissions might be more useful.
Here, each row shows the number received so far.
import pandas as pd
import matplotlib.pyplot as plt
df= pd.DataFrame(data[1:], columns=data[0])
df= pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax= plt.subplots()
figure= df.plot(title='Cumulative submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
for line in plt.legend().get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
figure.get_figure().savefig('cumulative-submissions.png')
return df
Weeks to CFP
2019
2020
2021
2022
2023
-12
0
0
0
0
4
-9
0
0
0
0
6
-8
0
0
4
2
6
-7
0
0
6
2
8
-6
0
0
7
2
8
-5
2
1
9
4
10
-4
3
2
11
4
12
-3
3
2
16
6
15
-2
9
3
17
8
20
-1
18
7
29
16
30
0
27
28
42
24
38
1
27
35
43
29
39
2
28
35
44
29
39
Figure 1: Cumulative submissions by number of weeks to CFP end date
And here's the cumulative number of minutes based on the proposals.
import pandas as pd
import matplotlib.pyplot as plt
df= pd.DataFrame(data[1:], columns=data[0])
df= pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Minutes', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax= plt.subplots()
figure= df.plot(title='Cumulative minutes by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
for line in plt.legend().get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
figure.get_figure().savefig('cumulative-minutes.png')
return df
Weeks to CFP
2019
2020
2021
2022
2023
-12
0
0
0
0
70
-9
0
0
0
0
100
-8
0
0
50
25
100
-7
0
0
67
25
130
-6
0
0
74
25
130
-5
45
10
96
56
160
-4
66
25
115
56
220
-3
66
25
188
87
260
-2
192
55
198
104
390
-1
274
123
361
295
570
0
422
547
558
405
710
1
422
699
568
512
730
2
429
699
578
512
730
Figure 2: Cumulative minutes by number of weeks to the CFP end date
So… yeah… 730 minutes of talks for this year… I might've gotten
a little carried away. But I like all the talks! And I want them to be
captured in videos and maybe even transcribed by people who will take
the time to change misrecognized words like Emax into Emacs! And I
want people to be able to connect with other people who are interested
in the sorts of stuff they're doing! So we're going to make it happen.
The draft schedule's looking pretty full, but I think it'll work out,
especially if the speakers send in their videos on time. Let's see how
it all works out!
(…and look, I even got to learn how to do pivot tables and graphs with
Python!)
_xor had an interesting idea: can we use org-protocol to link to
things inside Emacs, so that we can have a webpage with bookmarks into
our Org files? Here's a quick hack that reuses org-store-link and
org-link-open.
(defunorg-protocol-open-link (info)
"Process an org-protocol://open style url with INFO."
(org-link-open (car (org-element-parse-secondary-string (plist-get info :link) '(link)))))
(defunorg-protocol-copy-open-link (arg)
(interactive"P")
(kill-new (concat "org-protocol://open?link=" (url-hexify-string (org-store-link arg)))))
(with-eval-after-load'org
(add-to-list 'org-protocol-protocol-alist'("org-open":protocol"open":function org-protocol-open-link)))
To make exporting and following easier, we also need a little code to
handle org-protocol links inside Org.
