Naming conventions make it easier for other people to find things.
Just like with file prefixes, I like to use a standard naming pattern
for our BigBlueButton web conference rooms. For EmacsConf 2022, we
used ec22-day-am-gen Speaker name (slugs). For
EmacsConf 2023, I want to set up the BigBlueButton rooms before the
schedule settles down, so I won't encode the time or track information
into it. Instead, I'll use Speaker name (slugs) - emacsconf2023.
BigBlueButton does have an API for managing rooms, but that requires a
shared secret that I don't know yet. I figured I'd just automate it
through my browser. Over the last year, I've started using Spookfox to
control the Firefox web browser from Emacs. It's been pretty handy for
scrolling webpages up and down, so I wondered if I could replace my
old xdotool-based automation. Here's what I came up with for this
year.
First, I need a function that creates the BBB room for a group of
talks and updates the Org entry with the URL. Adding a slight delay
makes it a bit more reliable.
emacsconf-spookfox-create-bbb: Create a BBB room for this group of talks.
Sometimes it makes sense to dynamically generate information related
to a talk and then save it as an Org property so that I can manually
edit it. For example, we like to name all the talk files using this
pattern: "emacsconf-year-slug--title--speakers". That's a lot to
type consistently! We can generate most of these prefixes
automatically, but some might need tweaking, like when the talk title
or speaker names have special characters.
First we need something that turns a string into an ID.
emacsconf-slugify: Turn S into an ID.
(defunemacsconf-slugify (s)
"Turn S into an ID.Replace spaces with dashes, remove non-alphanumeric characters,and downcase the string."
(replace-regexp-in-string
" +""-"
(replace-regexp-in-string
"[^a-z0-9 ]"""
(downcase s))))
Then we can use that to calculate the file prefix for a given talk.
emacsconf-file-prefix: Create the file prefix for TALK
Then we can map over all the talk entries that don't have FILE_PREFIX defined:
emacsconf-set-file-prefixes: Set the FILE_PREFIX property for each talk entry that needs it.
(defunemacsconf-set-file-prefixes ()
"Set the FILE_PREFIX property for each talk entry that needs it."
(interactive)
(org-map-entries
(lambda ()
(org-entry-put
(point) "FILE_PREFIX"
(emacsconf-file-prefix (emacsconf-get-talk-info-for-subtree))))
"SLUG={.}-FILE_PREFIX={.}"))
That stores the file prefix in an Org property, so we can edit it if
it needs tweaking.
Figure 1: Setting the FILE_PREFIX for all talks that don't have that yet
Renaming files to match the file prefix
Now that we have that, how can we use it? One way is to rename files
from within Emacs. I can mark multiple files with Dired's m command
or work on them one at a time. If there are several files with the
same extension, I can specify something to add to the filename to tell
them apart.
emacsconf-rename-files: Rename the marked files or the current file to match TALK.
(defunemacsconf-rename-files (talk &optional filename)
"Rename the marked files or the current file to match TALK.If FILENAME is specified, use that as the extra part of the filename after the prefix.This is useful for distinguishing files with the same extension.Return the list of new filenames."
(interactive (list (emacsconf-complete-talk-info)))
(prog1
(mapcar
(lambda (file)
(let* ((extra
(or filename
(read-string (format "Filename (%s): " (file-name-base file)))))
(new-filename
(expand-file-name
(concat (plist-get talk :file-prefix)
(if (string= extra "")
""
(concat "--" extra))
"."
(file-name-extension file))
(file-name-directory file))))
(rename-file file new-filename t)
new-filename))
(or (dired-get-marked-files) (list (buffer-file-name))))
(when (derived-mode-p 'dired-mode)
(revert-buffer))))
Figure 2: Renaming multiple files
Working with files on other computers
Because Dired works over TRAMP, I can use that to rename files on a
remote server without changing anything about the code. I can open the
remote directory with Dired and everything just works.
TRAMP also makes it easy to copy a file to the backstage directory
after it's renamed, which saves me having to do that as a separate
step.
emacsconf-rename-and-upload-to-backstage: Rename marked files or the current file, then upload to backstage.
(defunemacsconf-rename-and-upload-to-backstage (talk &optional filename)
"Rename marked files or the current file, then upload to backstage."
(interactive (list (emacsconf-complete-talk-info)))
(mapc
(lambda (file)
(copy-file
file
(expand-file-name
(file-name-nondirectory file)
emacsconf-backstage-dir)
t))
(emacsconf-rename-files talk)))
So if my emacsconf-backstage-dir is set to
/ssh:orga@res:/var/www/res.emacsconf.org/2023/backstage, then it
looks up the details for res in my ~/.ssh/config and copies the
file there.
Renaming files using information from a JSON
What if I don't want to rename the files from Emacs? If I use Emacs's
JSON support to export some information from the talks as a JSON file,
then I can easily use that data from the command line.
Here's how I export the talk information:
emacsconf-talks-json: Return JSON format with a subset of talk information.
(defunemacsconf-publish-talks-json ()
"Return JSON format with a subset of talk information."
(json-encode
(list
:talks
(mapcar
(lambda (o)
(apply
'list
(cons :start-time (format-time-string "%FT%T%z" (plist-get o :start-time) t))
(cons :end-time (format-time-string "%FT%T%z" (plist-get o :end-time) t))
(mapcar
(lambda (field)
(cons field (plist-get o field)))
'(:slug:title:speakers:pronouns:pronunciation:url:track:file-prefix))))
(emacsconf-filter-talks (emacsconf-get-talk-info))))))
emacsconf-publish-talks-json-to-files
(defunemacsconf-publish-talks-json-to-files ()
"Export talk information as JSON so that we can use it in shell scripts."
(interactive)
(mapc (lambda (dir)
(when (and dir (file-directory-p dir))
(with-temp-file (expand-file-name "talks.json" dir)
(insert (emacsconf-talks-json)))))
(list emacsconf-res-dir emacsconf-ansible-directory)))
Then I can use something like rename-original.sh emacsconf
video.webm to
emacsconf-2023-emacsconf--emacsconforg-how-we-use-org-mode-and-tramp-to-organize-and-run-a-multitrack-conference--sacha-chua--original.webm.
Working with PsiTransfer-uploaded files
JSON support is useful for getting files into our system, too. For
EmacsConf 2022, we used PsiTransfer as a password-protected web-based
file upload service. That was much easier for speakers to deal with
than FTP, especially for large files. PsiTransfer makes a JSON file
for each batch of uploads, which is handy because the uploaded files
are named based on the key instead of keeping their filenames and
extensions. I wrote a function to copy an uploaded file from the
PsiTransfer directory to the backstage directory, renaming it along
the way. That meant that I could open the JSON for the uploaded files
via TRAMP and then copy a file between two remote directories without
manually downloading it to my computer.
emacsconf-upload-copy-from-json: Parse PsiTransfer JSON files and copy the uploaded file to the backstage directory.
(defunemacsconf-upload-copy-from-json (talk key filename)
"Parse PsiTransfer JSON files and copy the uploaded file to the backstage directory.The file is associated with TALK. KEY identifies the file in a multi-file upload.FILENAME specifies an extra string to add to the file prefix if needed."
(interactive (let-alist (json-parse-string (buffer-string) :object-type'alist)
(list (emacsconf-complete-talk-info)
.metadata.key
(read-string (format "Filename: ")))))
(let ((new-filename (concat (plist-get talk :file-prefix)
(if (string= filename "")
filename
(concat "--" filename))
"."
(let-alist (json-parse-string (buffer-string) :object-type'alist)
(file-name-extension .metadata.name)))))
(copy-file
key
(expand-file-name new-filename emacsconf-backstage-dir)
t)))
So that's how I can handle things that can be mostly automated but
that might need a little human intervention: use Emacs Lisp to make a
starting point, tweak it a little if needed, and then make it easy to
use that value elsewhere. Renaming files can be tricky, so it's good
to reduce the chance for typos!
One of the things I keep an eye out for when organizing EmacsConf is
the most recent time we heard from a speaker. Sometimes life happens
and speakers get too busy to prepare a video, so we might offer to let
them do it live. Sometimes e-mail delivery issues get in the way and
we don't hear from speakers because some server in between has spam
filters set too strong. So I made a function that lists the most
recent e-mail we got from the speaker that includes "emacsconf" in it.
That was a good excuse to learn more about tabulated-list-mode.
Figure 1: Redacted view of most recent e-mails from speakers
I started by figuring out how to get all the e-mail addresses associated with a talk.
emacsconf-mail-get-all-email-addresses: Return all the possible e-mail addresses for TALK.
(defunemacsconf-mail-get-all-email-addresses (talk)
"Return all the possible e-mail addresses for TALK."
(split-string
(downcase
(string-join
(seq-uniq
(seq-keep
(lambda (field) (plist-get talk field))
'(:email:public-email:email-alias)))
","))
" *, *"))
Then I figured out the notmuch search to use to get all messages. Some
people write a lot, so I limited it to just the ones that have
emacsconf as well. Notmuch can return JSON, so that's easy to parse.
emacsconf-mail-notmuch-tag: Tag to use when searching the Notmuch database for mail.
(defvaremacsconf-mail-notmuch-tag"emacsconf""Tag to use when searching the Notmuch database for mail.")
emacsconf-mail-notmuch-last-message-for-talk: Return the most recent message from the speakers for TALK.
(defunemacsconf-mail-notmuch-last-message-for-talk (talk &optional subject)
"Return the most recent message from the speakers for TALK.Limit to SUBJECT if specified."
(let ((message (json-parse-string
(shell-command-to-string
(format "notmuch search --limit=1 --format=json \"%s%s\""
(mapconcat
(lambda (email) (concat "from:" (shell-quote-argument email)))
(emacsconf-mail-get-all-email-addresses talk)
" or ")
(emacsconf-surround
" and "
(and emacsconf-mail-notmuch-tag (shell-quote-argument emacsconf-mail-notmuch-tag))
"""")
(emacsconf-surround
" and subject:"
(and subject (shell-quote-argument subject)) """")))
:object-type'alist)))
(cons `(email . ,(plist-get talk :email))
(when (> (length message) 0)
(elt message 0)))))
Then I could display all the groups of speakers so that it's easy to
check if any of the speakers haven't e-mailed us in a while.
emacsconf-mail-notmuch-show-latest-messages-from-speakers: Verify that the email addresses in GROUPS have e-mailed recently.
(defunemacsconf-mail-notmuch-show-latest-messages-from-speakers (groups &optional subject)
"Verify that the email addresses in GROUPS have e-mailed recently.When called interactively, pop up a report buffer showing the e-mailsand messages by date, with oldest messages on top.This minimizes the risk of mail delivery issues and radio silence."
(interactive (list (emacsconf-mail-groups (seq-filter
(lambda (o) (not (string= (plist-get o :status) "CANCELLED")))
(emacsconf-get-talk-info)))))
(let ((results
(sort (mapcar
(lambda (group)
(emacsconf-mail-notmuch-last-message-for-talk (cadr group) subject))
groups)
(lambda (a b)
(< (or (alist-get 'timestamp a) -1)
(or (alist-get 'timestamp b) -1))))))
(when (called-interactively-p 'any)
(with-current-buffer (get-buffer-create "*Mail report*")
(let ((inhibit-read-only t))
(erase-buffer))
(tabulated-list-mode)
(setq
tabulated-list-entries
(mapcar
(lambda (row)
(list
(alist-get 'thread row)
(vector
(alist-get 'email row)
(or (alist-get 'date_relative row) "")
(or (alist-get 'subject row) ""))))
results))
(setq tabulated-list-format [("Email" 30 t)
("Date" 10 nil)
("Subject" 30 t)])
(local-set-key (kbd "RET") #'emacsconf-mail-notmuch-visit-thread-from-summary)
(tabulated-list-print)
(tabulated-list-init-header)
(pop-to-buffer (current-buffer))))
results))
If I press RET on a line, I can open the most recent thread. This is
handled by the emacsconf-mail-notmuch-visit-thread-from-summary,
which is simplified by using the thread ID as the tabulated list ID.
Figure 2: Viewing a thread in a different window
emacsconf-mail-notmuch-visit-thread-from-summary: Display the thread from the summary.
(defunemacsconf-mail-notmuch-visit-thread-from-summary ()
"Display the thread from the summary."
(interactive)
(let (message-buffer)
(save-window-excursion
(setq message-buffer (notmuch-show (tabulated-list-get-id))))
(display-buffer message-buffer t)))
We haven't heard from a few speakers in a while, so I'll probably
e-mail them this weekend to double-check that I'm not getting delivery
issues with my e-mails to them. If that doesn't get a reply, I might
try other communication methods. If they're just busy, that's cool.
It's a lot easier to spot missing or old entries in a table than it is
to try to remember who we haven't heard from recently, so hooray for
tabulated-list-mode!
import pandas as pd
import matplotlib.pyplot as plt
df= pd.DataFrame(data[1:], columns=data[0])
df= df.drop('Weeks to CFP', axis=1).groupby(['Year']).sum()
fig, ax= plt.subplots(nrows=1, ncols=2, figsize=(12,6))
fig1= df['Count'].plot(kind="bar", ax=ax[0], title='Number of submissions')
fig2= df['Minutes'].plot(kind="bar", ax=ax[1], title='Number of minutes')
fig.get_figure().savefig('emacsconf-by-year.png')
return df
Year
Count
Minutes
2019
28
429
2020
35
699
2021
44
578
2022
29
512
2023
39
730
I also wanted to make an animated GIF so that the cumulative graphs
could be a little easier to understand.
import pandas as pd
import matplotlib.pyplot as plt
import imageio as io
df= pd.DataFrame(data[1:], columns=data[0])
fig, ax= plt.subplots(nrows=1, ncols=2, figsize=(12,6))
count= pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
minutes= pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Minutes', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
ax[0].set_ylim([0, count.max().max()])
ax[1].set_ylim([0, minutes.max().max()])
with io.get_writer('emacsconf-combined.gif', mode='I', duration=[500, 500, 500, 500, 1000], loop=0) as writer:
for year inrange(2019, 2024):
count[year].plot(ax=ax[0], title='Cumulative submissions')
minutes[year].plot(ax=ax[1], title='Cumulative minutes')
ax[0].legend(loc='upper left')
ax[1].legend(loc='upper left')
for axis in ax:
for line in axis.get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
for line in axis.legend().get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
filename= f'emacsconf-combined-${year}.png' fig.get_figure().savefig(filename)
image= io.v3.imread(filename)
writer.append_data(image)
Figure 1: Animated GIF showing the cumulative total submissions and minutes
I am not quite sure what kind of story this data tells (aside from the
fact that there sure are a lot of great talks), but it was fun to
learn how to make more kinds of graphs and animate them too. Could be
useful someday. =)
Having helped organize EmacsConf for a number of years now, I know
that I usually panic about whether we have submissions partway through
the call for participation. This causes us to extend the CFP deadline
and ask people to ask more people to submit things, and then we end up
with a wonderful deluge of talks that I then have to somehow squeeze
into a reasonable-looking schedule.
This year, I managed to not panic and I also resisted the urge to extend the
CFP deadline, trusting that there will actually be tons of cool stuff.
It helped that my schedule SVG code let me visualize what the
conference could feel like with the submissions so far, so we started
with a reasonably nice one-track conference and built up from there.
It also helped that I'd gone back to the submissions for 2022 and
plotted them by the number of weeks before the CFP deadline, and I
knew that there'd be a big spike from all those people whose Org
DEADLINE: properties would nudge them into finalizing their
proposals.
Out of curiosity, I wanted to see how the stats for this year compared
with previous years. I wrote a small function to collect the data that I wanted to summarize:
emacsconf-count-submissions-by-week: Count submissions in INFO by distance to CFP-DEADLINE.
(defunemacsconf-count-submissions-by-week (&optional info cfp-deadline)
"Count submissions in INFO by distance to CFP-DEADLINE."
(setq cfp-deadline (or cfp-deadline emacsconf-cfp-deadline))
(setq info (or info (emacsconf-get-talk-info)))
(cons '("Weeks to CFP end date""Count""Hours")
(mapcar (lambda (entry)
(list (car entry)
(length (cdr entry))
(apply '+ (mapcar 'cdr (cdr entry)))))
(seq-group-by
'car
(sort
(seq-keep
(lambda (o)
(and (emacsconf-publish-talk-p o)
(plist-get o :date-submitted)
(cons (floor (/ (days-between (plist-get o :date-submitted) cfp-deadline)
7.0))
(string-to-number
(or (plist-get o :video-duration)
(plist-get o :time)
"0")))))
info)
(lambda (a b) (< (car a) (car b))))))))
and then I ran it against the different files for each year, filling
in the previous years' data as needed. The resulting table is pretty
long, so I've put that in a collapsible section.
Some talks were proposed off-list and are not captured here, and
cancelled or withdrawn talks weren't included either. The times for
previous years use the actual video time, and the times for this year
use proposed times.
Off the top of my head, I didn't know of an easy way to make a pivot
table or cross-tab using just Org Mode or Emacs Lisp. I tried using
datamash, but I was having a hard time getting my output just the way
I wanted it. Fortunately, it was super-easy to get my data from an Org
table into Python so I could use pandas.pivot_table. Because I had
used #+NAME: submissions-by-week to label the table, I could use
:var data=submissions-by-week to refer to the data in my Python
program. Then I could summarize them by week.
Here's the number of submissions by the number of weeks to the
original CFP deadline, so we can see people generally like to target
the CFP date.
import pandas as pd
import matplotlib.pyplot as plt
df= pd.DataFrame(data[1:], columns=data[0])
df= pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True)
fig, ax= plt.subplots()
figure= df.plot(title='Number of submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
for line in plt.legend().get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
figure.get_figure().savefig('number-of-submissions.png')
return df
Weeks to CFP
2019
2020
2021
2022
2023
-12
0
0
0
0
4
-9
0
0
0
0
2
-8
0
0
4
2
0
-7
0
0
2
0
2
-6
0
0
1
0
0
-5
2
1
2
2
2
-4
1
1
2
0
2
-3
0
0
5
2
3
-2
6
1
1
2
5
-1
9
4
12
8
10
0
9
21
13
8
8
1
0
7
1
5
1
2
1
0
1
0
0
Calculating the cumulative number of submissions might be more useful.
Here, each row shows the number received so far.
import pandas as pd
import matplotlib.pyplot as plt
df= pd.DataFrame(data[1:], columns=data[0])
df= pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax= plt.subplots()
figure= df.plot(title='Cumulative submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
for line in plt.legend().get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
figure.get_figure().savefig('cumulative-submissions.png')
return df
Weeks to CFP
2019
2020
2021
2022
2023
-12
0
0
0
0
4
-9
0
0
0
0
6
-8
0
0
4
2
6
-7
0
0
6
2
8
-6
0
0
7
2
8
-5
2
1
9
4
10
-4
3
2
11
4
12
-3
3
2
16
6
15
-2
9
3
17
8
20
-1
18
7
29
16
30
0
27
28
42
24
38
1
27
35
43
29
39
2
28
35
44
29
39
Figure 1: Cumulative submissions by number of weeks to CFP end date
And here's the cumulative number of minutes based on the proposals.
import pandas as pd
import matplotlib.pyplot as plt
df= pd.DataFrame(data[1:], columns=data[0])
df= pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Minutes', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax= plt.subplots()
figure= df.plot(title='Cumulative minutes by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
for line in plt.legend().get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
figure.get_figure().savefig('cumulative-minutes.png')
return df
Weeks to CFP
2019
2020
2021
2022
2023
-12
0
0
0
0
70
-9
0
0
0
0
100
-8
0
0
50
25
100
-7
0
0
67
25
130
-6
0
0
74
25
130
-5
45
10
96
56
160
-4
66
25
115
56
220
-3
66
25
188
87
260
-2
192
55
198
104
390
-1
274
123
361
295
570
0
422
547
558
405
710
1
422
699
568
512
730
2
429
699
578
512
730
Figure 2: Cumulative minutes by number of weeks to the CFP end date
So… yeah… 730 minutes of talks for this year… I might've gotten
a little carried away. But I like all the talks! And I want them to be
captured in videos and maybe even transcribed by people who will take
the time to change misrecognized words like Emax into Emacs! And I
want people to be able to connect with other people who are interested
in the sorts of stuff they're doing! So we're going to make it happen.
The draft schedule's looking pretty full, but I think it'll work out,
especially if the speakers send in their videos on time. Let's see how
it all works out!
(…and look, I even got to learn how to do pivot tables and graphs with
Python!)
