import pandas as pd
import matplotlib.pyplot as plt
df= pd.DataFrame(data[1:], columns=data[0])
df= df.drop('Weeks to CFP', axis=1).groupby(['Year']).sum()
fig, ax= plt.subplots(nrows=1, ncols=2, figsize=(12,6))
fig1= df['Count'].plot(kind="bar", ax=ax[0], title='Number of submissions')
fig2= df['Minutes'].plot(kind="bar", ax=ax[1], title='Number of minutes')
fig.get_figure().savefig('emacsconf-by-year.png')
return df
Year
Count
Minutes
2019
28
429
2020
35
699
2021
44
578
2022
29
512
2023
39
730
I also wanted to make an animated GIF so that the cumulative graphs
could be a little easier to understand.
import pandas as pd
import matplotlib.pyplot as plt
import imageio as io
df= pd.DataFrame(data[1:], columns=data[0])
fig, ax= plt.subplots(nrows=1, ncols=2, figsize=(12,6))
count= pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
minutes= pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Minutes', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
ax[0].set_ylim([0, count.max().max()])
ax[1].set_ylim([0, minutes.max().max()])
with io.get_writer('emacsconf-combined.gif', mode='I', duration=[500, 500, 500, 500, 1000], loop=0) as writer:
for year inrange(2019, 2024):
count[year].plot(ax=ax[0], title='Cumulative submissions')
minutes[year].plot(ax=ax[1], title='Cumulative minutes')
ax[0].legend(loc='upper left')
ax[1].legend(loc='upper left')
for axis in ax:
for line in axis.get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
for line in axis.legend().get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
filename= f'emacsconf-combined-${year}.png' fig.get_figure().savefig(filename)
image= io.v3.imread(filename)
writer.append_data(image)
Figure 1: Animated GIF showing the cumulative total submissions and minutes
I am not quite sure what kind of story this data tells (aside from the
fact that there sure are a lot of great talks), but it was fun to
learn how to make more kinds of graphs and animate them too. Could be
useful someday. =)
Having helped organize EmacsConf for a number of years now, I know
that I usually panic about whether we have submissions partway through
the call for participation. This causes us to extend the CFP deadline
and ask people to ask more people to submit things, and then we end up
with a wonderful deluge of talks that I then have to somehow squeeze
into a reasonable-looking schedule.
This year, I managed to not panic and I also resisted the urge to extend the
CFP deadline, trusting that there will actually be tons of cool stuff.
It helped that my schedule SVG code let me visualize what the
conference could feel like with the submissions so far, so we started
with a reasonably nice one-track conference and built up from there.
It also helped that I'd gone back to the submissions for 2022 and
plotted them by the number of weeks before the CFP deadline, and I
knew that there'd be a big spike from all those people whose Org
DEADLINE: properties would nudge them into finalizing their
proposals.
Out of curiosity, I wanted to see how the stats for this year compared
with previous years. I wrote a small function to collect the data that I wanted to summarize:
emacsconf-count-submissions-by-week: Count submissions in INFO by distance to CFP-DEADLINE.
(defunemacsconf-count-submissions-by-week (&optional info cfp-deadline)
"Count submissions in INFO by distance to CFP-DEADLINE."
(setq cfp-deadline (or cfp-deadline emacsconf-cfp-deadline))
(setq info (or info (emacsconf-get-talk-info)))
(cons '("Weeks to CFP end date""Count""Hours")
(mapcar (lambda (entry)
(list (car entry)
(length (cdr entry))
(apply '+ (mapcar 'cdr (cdr entry)))))
(seq-group-by
'car
(sort
(seq-keep
(lambda (o)
(and (emacsconf-publish-talk-p o)
(plist-get o :date-submitted)
(cons (floor (/ (days-between (plist-get o :date-submitted) cfp-deadline)
7.0))
(string-to-number
(or (plist-get o :video-duration)
(plist-get o :time)
"0")))))
info)
(lambda (a b) (< (car a) (car b))))))))
and then I ran it against the different files for each year, filling
in the previous years' data as needed. The resulting table is pretty
long, so I've put that in a collapsible section.
Some talks were proposed off-list and are not captured here, and
cancelled or withdrawn talks weren't included either. The times for
previous years use the actual video time, and the times for this year
use proposed times.
Off the top of my head, I didn't know of an easy way to make a pivot
table or cross-tab using just Org Mode or Emacs Lisp. I tried using
datamash, but I was having a hard time getting my output just the way
I wanted it. Fortunately, it was super-easy to get my data from an Org
table into Python so I could use pandas.pivot_table. Because I had
used #+NAME: submissions-by-week to label the table, I could use
:var data=submissions-by-week to refer to the data in my Python
program. Then I could summarize them by week.
Here's the number of submissions by the number of weeks to the
original CFP deadline, so we can see people generally like to target
the CFP date.
import pandas as pd
import matplotlib.pyplot as plt
df= pd.DataFrame(data[1:], columns=data[0])
df= pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True)
fig, ax= plt.subplots()
figure= df.plot(title='Number of submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
for line in plt.legend().get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
figure.get_figure().savefig('number-of-submissions.png')
return df
Weeks to CFP
2019
2020
2021
2022
2023
-12
0
0
0
0
4
-9
0
0
0
0
2
-8
0
0
4
2
0
-7
0
0
2
0
2
-6
0
0
1
0
0
-5
2
1
2
2
2
-4
1
1
2
0
2
-3
0
0
5
2
3
-2
6
1
1
2
5
-1
9
4
12
8
10
0
9
21
13
8
8
1
0
7
1
5
1
2
1
0
1
0
0
Calculating the cumulative number of submissions might be more useful.
Here, each row shows the number received so far.
import pandas as pd
import matplotlib.pyplot as plt
df= pd.DataFrame(data[1:], columns=data[0])
df= pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax= plt.subplots()
figure= df.plot(title='Cumulative submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
for line in plt.legend().get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
figure.get_figure().savefig('cumulative-submissions.png')
return df
Weeks to CFP
2019
2020
2021
2022
2023
-12
0
0
0
0
4
-9
0
0
0
0
6
-8
0
0
4
2
6
-7
0
0
6
2
8
-6
0
0
7
2
8
-5
2
1
9
4
10
-4
3
2
11
4
12
-3
3
2
16
6
15
-2
9
3
17
8
20
-1
18
7
29
16
30
0
27
28
42
24
38
1
27
35
43
29
39
2
28
35
44
29
39
Figure 1: Cumulative submissions by number of weeks to CFP end date
And here's the cumulative number of minutes based on the proposals.
import pandas as pd
import matplotlib.pyplot as plt
df= pd.DataFrame(data[1:], columns=data[0])
df= pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Minutes', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax= plt.subplots()
figure= df.plot(title='Cumulative minutes by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
for line in plt.legend().get_lines():
if line.get_label() =='2023':
line.set_linewidth(5)
figure.get_figure().savefig('cumulative-minutes.png')
return df
Weeks to CFP
2019
2020
2021
2022
2023
-12
0
0
0
0
70
-9
0
0
0
0
100
-8
0
0
50
25
100
-7
0
0
67
25
130
-6
0
0
74
25
130
-5
45
10
96
56
160
-4
66
25
115
56
220
-3
66
25
188
87
260
-2
192
55
198
104
390
-1
274
123
361
295
570
0
422
547
558
405
710
1
422
699
568
512
730
2
429
699
578
512
730
Figure 2: Cumulative minutes by number of weeks to the CFP end date
So… yeah… 730 minutes of talks for this year… I might've gotten
a little carried away. But I like all the talks! And I want them to be
captured in videos and maybe even transcribed by people who will take
the time to change misrecognized words like Emax into Emacs! And I
want people to be able to connect with other people who are interested
in the sorts of stuff they're doing! So we're going to make it happen.
The draft schedule's looking pretty full, but I think it'll work out,
especially if the speakers send in their videos on time. Let's see how
it all works out!
(…and look, I even got to learn how to do pivot tables and graphs with
Python!)
We send a number of e-mails to speakers while coordinating EmacsConf.
Here's a rough list of the standard e-mails:
confirmation that we've received their proposal and that we'll review it over the following week
acceptance
coordination with related talks
checking tentative schedule to see if people have any requests
instructions for uploading files
instructions for accessing the backstage area
confirmation that we've received their uploaded video
captions for review
miscellaneous todos
any large schedule changes
schedule confirmation and check-in instructions
thanks and resources
thanks and follow-up questions
Sending e-mails from within Emacs makes it super-easy to automate, of
course.
I started off with putting e-mail templates in our public organizers'
notebook because that made it easy to link to the drafts when asking
other volunteers for feedback. As the templates settle down, I've been
gradually moving some of them into our emacsconf-mail library so that
I don't need to copy the template into each year's notebook.
The e-mail templates use the same template functions we use to make
the wiki pages. Here's the function for confirming that we've received
a submission and letting the speaker know when to expect comments:
emacsconf-mail-review: Let the speaker know we’ve received their proposal.
(defunemacsconf-mail-review (talk)
"Let the speaker know we've received their proposal."
(interactive (list (emacsconf-complete-talk-info)))
(emacsconf-mail-prepare '(:subject"${conf-name} ${year} review: ${title}":cc"emacsconf-submit@gnu.org":reply-to"emacsconf-submit@gnu.org, ${email}, ${user-email}":mail-followup-to"emacsconf-submit@gnu.org, ${email}, ${user-email}":body"Hi, ${speakers-short}!Thanks for submitting your proposal! (ZZZ TODO: feedback)We'll wait a week (~ ${notification-date}) in case the other volunteers want to chime in regarding your talk. =)${signature}")
(plist-get talk :email)
(list
:user-email user-mail-address
:signature user-full-name
:title (plist-get talk :title)
:email (plist-get talk :email)
:conf-name emacsconf-name
:speakers-short (plist-get talk :speakers-short)
:year emacsconf-year
:notification-date (plist-get talk :date-to-notify))))
I recently extended emacsconf-mail-prepare so that it can insert the
template into a reply instead of always starting a new message. This
allows the messages to be properly threaded in the emacsconf-submit
list archives, which makes it easier to verify that all the
submissions have been acknowledged.
emacsconf-mail-prepare: Prepare the e-mail following TEMPLATE. Send it to EMAIL.
(defunemacsconf-mail-prepare (template email attrs)
"Prepare the e-mail following TEMPLATE. Send it to EMAIL.Use ATTRS to fill in the template.If `emacsconf-mail-batch-test' is non-nil, put the message in that buffer instead."
(let ((fields '((:reply-to"Reply-To")
(:mail-followup-to"Mail-Followup-To")
(:cc"Cc"))))
(if emacsconf-mail-batch-test
(emacsconf-mail-prepare-for-batch-test template email attrs fields)
;; prepare to send the mail
(if (and (derived-mode-p 'message-mode) (string-match "unsent mail" (buffer-name)))
;; add to headers
(emacsconf-mail-update-reply-headers template email attrs fields)
;; compose a new message
(compose-mail
email
(emacsconf-replace-plist-in-string attrs (or (plist-get template :subject) ""))
(seq-keep (lambda (field)
(when (plist-get template (car field))
(cons
(cadr field)
(emacsconf-replace-plist-in-string
attrs
(plist-get template (car field))))))
fields)))
(message-sort-headers)
(message-goto-body)
(save-excursion
(insert (string-trim (emacsconf-replace-plist-in-string attrs (plist-get template :body)))
"\n\n")
(goto-char (point-min))
(emacsconf-mail-merge-wrap))
(when (plist-get template :log-note)
(mapc (lambda (talk)
(emacsconf-mail-log-message-when-sent talk (plist-get template :log-note)))
(cdr group))))))
There's a little function that I can add to message-send-hook to
prevent me from sending a message if it still has ZZZ in it.
emacsconf-mail-check-for-zzz-before-sending: Throw an error if the ZZZ todo marker is still in the message.
(defunemacsconf-mail-check-for-zzz-before-sending ()
"Throw an error if the ZZZ todo marker is still in the message.Good for adding to `message-send-hook'."
(save-excursion
(goto-char (point-min))
(when (re-search-forward "ZZZ" nil t)
(unless (yes-or-no-p "ZZZ marker found. Send anyway? ")
(error"ZZZ marker found.")))))
Here's the function for sending an acceptance letter:
(defunemacsconf-mail-accept-talk (talk)
"Send acceptance letter."
(interactive (list (emacsconf-complete-talk-info)))
(emacsconf-mail-prepare '(:subject"${conf-name} ${year} acceptance: ${title}":cc"emacsconf-submit@gnu.org":slugs nil
:reply-to"emacsconf-submit@gnu.org, ${email}, ${user-email}":mail-followup-to"emacsconf-submit@gnu.org, ${email}, ${user-email}":body"Hi, ${speakers-short}!Looks like all systems are a go for your talk. Thanks forproposing it! Your talk page is now at ${url} . Please feel freeto update it or e-mail us if you'd like help with any changes.${fill}If you want to get started on your talk early, we have someinstructions at ${base}${year}/prepare/ that might help.We strongly encourage speakers to prepare a talk video by${video-target-date} in order to reduce technical risks and makethings flow more smoothly. Plus, we might be able to get it captionedby volunteers, just like the talks last year. We'll save ${time} minutesfor your talk, not including time for Q&A. Don't sweat it ifyou're a few minutes over or under. If it looks like a much shorter orlonger talk once you start getting into it, let us know and we mightbe able to adjust.${wrap}I'll follow up with the specific schedule for your talk once thingssettle down. In the meantime, please let us know if you have anyquestions or if there's anything we can do to help out!${signature}":function emacsconf-mail-accept-talk
:log-note"accepted talk")
(plist-get talk :email)
(list
:base emacsconf-base-url
:user-email user-mail-address
:year emacsconf-year
:signature user-full-name
:conf-name emacsconf-name
:title (plist-get talk :title)
:email (plist-get talk :email)
:time (plist-get talk :time)
:speakers-short (plist-get talk :speakers-short)
:url (concat emacsconf-base-url (plist-get talk :url))
:video-target-date emacsconf-video-target-date)))
We send confirmations and acceptances one at a time. Other e-mails are
sent to all the speakers and it's easier to draft them in a batch.
I'll cover that kind of mail merge in a separate post.
The EmacsConf 2023 call for participation has finished, hooray! We've
sent out acceptances and added talks to the wiki. We experimented with
early acceptances this year, which was really nice because it gives
people quick feedback and allows people to get started on their videos
early. That meant that I needed to be able to easily add talks to the
wiki throughout the call for participation. We use templates and an
Ikiwiki directive to make it easier to consistently format talk pages.
This post covers adding a talk to the wiki, and these functions are in
the emacsconf-el repository.
Amin Bandali picked Ikiwiki for the wiki for emacsconf.org. I think
it's because Ikiwiki works with plain text in a Git repository, which
fits nicely with our workflow. I can use Emacs Lisp to generate files
that are included in other files, and I can also use Emacs Lisp to
generate the starting point for files that may be manually edited
later on.
We organize conference pages by year. Under the directory for this
year (2023/), we have:
talks
talk descriptions that can be manually edited
talks/SLUG.md
the description for the talk. Includes
../info/SLUG-nav.md, ../info/SLUG-before.md, and
../info/SLUG-after.md
info
automatically-generated files that are included before and
after the talk description, and navigation links between talks
info/SLUG-nav.md
navigation links
info/SLUG-before.md
navigation links
info/SLUG-after.md
navigation links
schedule-details.md
list of talks
organizers-notebook/index.org
public planning notebook
The filenames and URLs for each talk are based on the ID for a talk. I
store that in the SLUG property to make it easy to differentiate
from CUSTOM_ID, since CUSTOM_ID is useful for lots of other things
in Org Mode. I usually assign the slugs when I add the talks to our
private conf.org file, although sometimes people suggest a specific
ID.
Templating
Publishing to wiki pages and replying to e-mails are easier if I can
substitute text into readable templates. There are a number of
templating functions for Emacs Lisp, like the built-in tempo.el or
s-lex-format from s.el. I ended up writing something that works
with plists instead, since we use property lists (plists) all over the
emacsconf-el library.
emacsconf-replace-plist-in-string: Replace ${keyword} from ATTRS in STRING.
(defunemacsconf-replace-plist-in-string (attrs string)
"Replace ${keyword} from ATTRS in STRING."
(let ((a attrs) name val)
(while a
(setq name (pop a) val (pop a))
(when (stringp val)
(setq string
(replace-regexp-in-string (regexp-quote (concat "${" (substring (symbol-name name) 1) "}"))
(or val "")
string t t))))
string))
It is also handy to be able to add text around another string only if
the string is non-nil, and to provide a different string to use if it
isn't specified..
emacsconf-surround: Concat BEFORE, TEXT, and AFTER if TEXT is specified, or return ALTERNATIVE.
(defunemacsconf-surround (before text after &optional alternative)
"Concat BEFORE, TEXT, and AFTER if TEXT is specified, or return ALTERNATIVE."
(if (and text (not (string= text "")))
(concat (or before "") text (or after ""))
alternative))
Getting the talk information
To get the data to fill in the template, we can run a bunch of
different functions. This lets us add or remove functions when we need
to. We pass the previous result to the next function in order to
accumulate properties or modify them. The result is a property list
for the current talk.
emacsconf-get-talk-info-for-subtree: Run ‘emacsconf-talk-info-functions’ to extract the info for this entry.
(defunemacsconf-get-talk-info-for-subtree ()
"Run `emacsconf-talk-info-functions' to extract the info for this entry."
(seq-reduce (lambda (prev val) (save-excursion (save-restriction (funcall val prev))))
emacsconf-talk-info-functions
nil))
emacsconf-talk-info-functions: Functions to collect information.
I add a *** Talk abstract subheading to the talk and put the rest of
the submission under a *** Talk details subheading. This allows me
to extract the text of the Talk abstract heading (or whatever
matches emacsconf-abstract-heading-regexp, which is set to "abstract".).
emacsconf-get-subtree-entry: Return the text for the subtree matching HEADING-REGEXP.
(defunemacsconf-get-subtree-entry (heading-regexp)
"Return the text for the subtree matching HEADING-REGEXP."
(car
(delq nil
(org-map-entries
(lambda ()
(when (string-match heading-regexp (org-entry-get (point) "ITEM"))
(org-get-entry)))
nil 'tree))))
emacsconf-get-talk-abstract-from-subtree: Add the abstract from a subheading.
(defunemacsconf-get-talk-abstract-from-subtree (o)
"Add the abstract from a subheading.The subheading should match `emacsconf-abstract-heading-regexp'."
(plist-put o :abstract (substring-no-properties (or (emacsconf-get-subtree-entry "abstract") ""))))
I include emacsconf-get-talk-abstract-from-subtree in
emacsconf-talk-info-functions so that it retrieves that information
when I call emacsconf-get-talk-info-for-subtree.
Publishing the talk page
We add accepted talks to the wiki so that people can see what kinds of
talks will be at EmacsConf 2023. To add the talk to the wiki, I use
emacsconf-publish-add-talk. It'll create the talk page without
overwriting anything that's already there and redo the
automatically-generated info pages that provide navigation, status,
and so on.
emacsconf-publish-add-talk: Add the current talk to the wiki.
(defunemacsconf-publish-add-talk ()
"Add the current talk to the wiki."
(interactive)
(emacsconf-publish-talk-page (emacsconf-get-talk-info-for-subtree))
(emacsconf-publish-info-pages)
(magit-status-setup-buffer emacsconf-directory))
The talk page includes the description and other resources.
emacsconf-publish-talk-page: Draft the talk page for O unless the page already exists or FORCE is non-nil.
Ikiwiki uses Markdown, so we can take advantage of Org's Markdown export.
emacsconf-convert-talk-abstract-to-markdown: Set the :abstract-md property to a Markdown version of the abstract.
(defunemacsconf-convert-talk-abstract-to-markdown (o)
"Set the :abstract-md property to a Markdown version of the abstract."
(plist-put o :abstract-md (org-export-string-as (or (plist-get o :abstract) "") 'md t)))
The emacsconf-format-main-schedule function displays the information
for the talks in each track. It's pretty straightforward, but I put it
in a function because I call it from a number of places.
emacsconf-publish-format-main-schedule: Include the schedule information for INFO.
(defunemacsconf-publish-format-main-schedule (info)
"Include the schedule information for INFO."
(mapconcat #'emacsconf-publish-sched-directive info "\n"))
We define an Ikiwiki sched directive that conditionally displays
things depending on what we specify, so it's easy to add more
information during the schedule or conference phase. This is
templates/sched.md in the EmacsConf wiki git repository:
This Emacs Lisp function converts a talk into that directive.
emacsconf-publish-sched-directive: Format the schedule directive with info for O.
(defunemacsconf-publish-sched-directive (o)
"Format the schedule directive with info for O."
(format "[[!template id=sched%s]]"
(let ((result "")
(attrs (append
(pcase emacsconf-publishing-phase
('program
(list
:time (plist-get o :time)))
((or'schedule'conference)
(list
:status (pcase (plist-get o :status)
("CAPTIONED""captioned")
("PREREC_RECEIVED""received")
("DONE""done")
("STARTED""now playing")
(_ nil))
:time (plist-get o :time)
:q-and-a (plist-get o :qa-link)
:pad (and emacsconf-publish-include-pads (plist-get o :pad-url))
:startutc (format-time-string "%FT%T%z" (plist-get o :start-time) t)
:endutc (format-time-string "%FT%T%z" (plist-get o :end-time) t)
:start (format-time-string "%-l:%M" (plist-get o :start-time) emacsconf-timezone)
:end (format-time-string "%-l:%M" (plist-get o :end-time) emacsconf-timezone)))
('resources
(list
:pad nil
:channel nil
:resources (mapconcat (lambda (s) (concat "<li>" s "</li>"))
(emacsconf-link-file-formats-as-list
(append o
(list :base-url (format "%s%s/" emacsconf-media-base-url emacsconf-year)))
(append emacsconf-main-extensions (list "--answers.webm""--answers.opus""--answers.vtt")))
""))))
(list
:title (plist-get o :title)
:url (concat "/" (plist-get o :url))
:speakers (plist-get o :speakers)
:track (if (member emacsconf-publishing-phase '(schedule conference)) (plist-get o :track))
:watch (plist-get o :watch-url)
:slug (plist-get o :slug)
:note
(string-join
(delq nil
(list
(when (plist-get o :captions-edited)
"captioned")
(when (and (plist-get o :public)
(or (plist-get o :toobnix-url)
(plist-get o :video-file)))
"video posted")))
", ")
)
)))
(while attrs
(let ((field (pop attrs))
(val (pop attrs)))
(when val
(setq result (concat result " " (substring (symbol-name field) 1) "=\"\"\"" val "\"\"\"")))))
result)))
Publishing auto-generated navigation
It's nice to be able to navigate between talks without going back to
the schedule page each time. This is handled by just keeping two extra
copies of the list: one with the first talk popped off, and one with
an extra element added to the beginning. Then we can use the heads of
those lists for next/previous links.
Figure 2: Navigation
emacsconf-publish-nav-pages: Generate links to the next and previous talks.
(defunemacsconf-publish-nav-pages (&optional talks)
"Generate links to the next and previous talks.During the schedule and conference phase, the talks are sorted by time.Otherwise, they're sorted by track and then schedule."
(interactive (list (emacsconf-publish-prepare-for-display (or emacsconf-schedule-draft (emacsconf-get-talk-info)))))
(let* ((next-talks (cdr talks))
(prev-talks (cons nil talks))
(label (if (member emacsconf-publishing-phase '(schedule conference))
"time""track")))
(unless (file-directory-p (expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory)))
(mkdir (expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory))))
(while talks
(let* ((o (pop talks))
(next-talk (emacsconf-format-talk-link (pop next-talks)))
(prev-talk (emacsconf-format-talk-link (pop prev-talks))))
(with-temp-file (expand-file-name (format "%s-nav.md" (plist-get o :slug))
(expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory)))
(insert (concat "\n<div class=\"talk-nav\">Back to the [[talks]] \n"
(if prev-talk (format "Previous by %s: %s \n" label prev-talk) "")
(if next-talk (format "Next by %s: %s \n" label next-talk) "")
(if (plist-get o :track) ; tagging doesn't work here because ikiwiki will list the nav page
(format "Track: <span class=\"sched-track %s\">%s</span> \n" (plist-get o :track) (plist-get o :track))
"")
"</div>")))))))
Before the talk description
We include some details about the schedule in the talk page, before the description.
Figure 3: Description
emacsconf-publish-before-page: Generate the page that has the info included before the abstract.
(defunemacsconf-publish-before-page (talk &optional info)
"Generate the page that has the info included before the abstract.This includes the intro note, the schedule, and talk resources."
(interactive (list (emacsconf-complete-talk-info)))
(setq info (or info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
(with-temp-file (expand-file-name (format "%s-before.md" (plist-get talk :slug))
(expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory)))
(insert "<!-- Automatically generated by emacsconf-publish-before-page -->\n")
(insert (emacsconf-surround "" (plist-get talk :intro-note) "\n\n"""))
(let ((is-live (emacsconf-talk-live-p talk)))
(when is-live (emacsconf-publish-captions-in-wiki talk))
(when (member emacsconf-publishing-phase '(schedule conference))
(insert (emacsconf-publish-format-talk-schedule-image talk info)))
(insert (emacsconf-publish-format-talk-schedule-info talk) "\n\n")
(insert
(if (plist-get talk :public) (emacsconf-wiki-talk-resources talk) "")
"\n# Description\n"))
(insert "<!-- End of emacsconf-publish-before-page -->")))
emacsconf-publish-format-talk-schedule-info: Format schedule information for O.
(defunemacsconf-publish-format-talk-schedule-info (o)
"Format schedule information for O."
(let ((friendly (concat "/" emacsconf-year "/talks/" (plist-get o :slug) ))
(timestamp (org-timestamp-from-string (plist-get o :scheduled))))
(emacsconf-replace-plist-in-string
(append o
(list
:format
(concat (or (plist-get o :video-time)
(plist-get o :time))
"-min talk"
(if (plist-get o :q-and-a)
(format " followed by %s Q&A%s"
(plist-get o :q-and-a)
(if (eq emacsconf-publishing-phase 'conference)
(format " (%s)"
(if (string-match "live" (plist-get o :q-and-a))
(if (eq 'after (emacsconf-bbb-status o))
"done"
(format "<https://emacsconf.org/current/%s/room>" (plist-get o :slug)))
(emacsconf-publish-webchat-link o)))
""))
""))
:pad-info
(if emacsconf-publish-include-pads
(format "Etherpad: <https://pad.emacsconf.org/%s-%s> \n" emacsconf-year (plist-get o :slug))
"")
:irc-info
(format "Discuss on IRC: [#%s](%s) \n" (plist-get o :channel)
(plist-get o :webchat-url))
:status-info
(if (member emacsconf-publishing-phase '(cfp program schedule conference)) (format "Status: %s \n" (plist-get o :status-label)) "")
:schedule-info
(if (and (member emacsconf-publishing-phase '(schedule conference))
(not (emacsconf-talk-all-done-p o))
(not (string= (plist-get o :status) "CANCELLED")))
(let ((start (org-timestamp-to-time (org-timestamp-split-range timestamp)))
(end (org-timestamp-to-time (org-timestamp-split-range timestamp t))))
(format
"<div>Times in different timezones:</div><div class=\"times\" start=\"%s\" end=\"%s\"><div class=\"conf-time\">%s</div><div class=\"others\"><div>which is the same as:</div>%s</div></div><div><a href=\"/%s/watch/%s/\">Find out how to watch and participate</a></div>"
(format-time-string "%Y-%m-%dT%H:%M:%SZ" start t)
(format-time-string "%Y-%m-%dT%H:%M:%SZ" end t)
(emacsconf-timezone-string o emacsconf-timezone)
(string-join (emacsconf-timezone-strings
o
(seq-remove (lambda (zone) (string= emacsconf-timezone zone))
emacsconf-timezones)) "<br />")
emacsconf-year
(plist-get (emacsconf-get-track (plist-get o :track)) :id)))
"")))
(concat
"[[!toc ]]Format: ${format} ${pad-info}${irc-info}${status-info}${schedule-info}\n"
(if (plist-get o :alternate-apac)
(format "[[!inline pages=\"internal(%s/inline-alternate)\" raw=\"yes\"]] \n" emacsconf-year)
"")
"\n"))))
After the talk description
After the talk description, we include a footer that makes it easier
for people to e-mail questions using either the PUBLIC_EMAIL
property of the talk or the emacsconf-org-private e-mail address.
emacsconf-publish-format-email-questions-and-comments: Invite people to e-mail either the public contact for TALK or the private list.
(defunemacsconf-publish-format-email-questions-and-comments (talk)
"Invite people to e-mail either the public contact for TALK or the private list."
(format "Questions or comments? Please e-mail %s"
(emacsconf-publish-format-public-email talk
(or
(and (string= (plist-get talk :public-email) "t")
(plist-get talk :email))
(plist-get talk :public-email)
"emacsconf-org-private@gnu.org"))))
emacsconf-publish-after-page: Generate the page with info included after the abstract.
(defunemacsconf-publish-after-page (talk &optional info)
"Generate the page with info included after the abstract.This includes captions, contact, and an invitation to participate."
(interactive (list (emacsconf-complete-talk-info)))
;; Contact information
(with-temp-file (expand-file-name (format "%s-after.md" (plist-get talk :slug))
(expand-file-name "info" (expand-file-name emacsconf-year emacsconf-directory)))
(insert
"<!-- Automatically generated by emacsconf-publish-after-page -->\n""\n\n";; main transcript
(if (plist-get talk :public) (emacsconf-publish-format-captions talk) "")
(emacsconf-publish-format-email-questions-and-comments talk) "\n"
(if (eq emacsconf-publishing-phase 'cfp)
(format "\n----\nGot an idea for an EmacsConf talk or session? We'd love to hear from you! Check out the [[Call for Participation|/%s/cfp]] for details.\n" emacsconf-year)
"")
"\n\n<!-- End of emacsconf-publish-after-page -->\n")))
Whenever the schedule changes
This function makes it easier to regenerate all those dynamic pages
that need to be updated whenever the schedule changes.
emacsconf-publish-info-pages: Populate year/info/*-nav, -before, and -after files.
(defunemacsconf-publish-info-pages (&optional info)
"Populate year/info/*-nav, -before, and -after files."
(interactive (list nil))
(setq info (or info (emacsconf-publish-prepare-for-display info)))
(emacsconf-publish-with-wiki-change
(emacsconf-publish-nav-pages info)
(emacsconf-publish-schedule info)
(mapc (lambda (o)
(emacsconf-publish-before-page o info)
(emacsconf-publish-after-page o info))
info)))
Summary
So once the review period has passed and we're ready to accept the
talk, I change the status to WAITING_FOR_PREREC and find room for it
in the schedule. Then I use emacsconf-publish-add-talk to add the
talk description to the wiki. I review the files it generated, tweak
hyperlinks as needed, add the pages to the Git repository, and push
the commit to the server. If I rearrange talks or change times, I just
need to run emacsconf-publish-info-pages and all the
dynamically-generated pages will be updated.
copy the talk overlay images and use them in the streaming software (OBS)
play videos
a recorded introduction if it exists
any extra videos we want to play
the main talk
and open up browser windows
the BigBlueButton web conference room for a live Q&A session
the talk's Etherpad collaborative document for questions
the Internet Relay Chat channel, if that's where the speaker wants to handle questions
To minimize the work involved in copying and pasting filenames and
URLs, I wanted to write scripts that could perform the right action
given the talk ID. I automated most of it so that it could work from
Emacs Lisp, and I also wrote shell scripts so that I (or someone else)
could run the appropriate commands from the terminal.
We display the conference logo, talk title, and speaker name on the
screen while the video is playing. This is handled with an OBS scene
that includes whatever image is at ~/video.png or ~/other.png,
since that results in a nicer display than using text in OBS. I'll go
into how we make the overlay images in a different blog post. This
post focuses on including the right image, which is just a matter of
copying the right file over ~/video.png.
FILE=$1if [[ ! -f $FILE ]]; thenLIST=(/data/emacsconf/assets/stream/emacsconf-[0-9][0-9][0-9][0-9]-$FILE*.webm)
FILE="${LIST[0]}"BY_SLUG=1
fishiftSLUG=$(echo"$FILE" | perl -ne 'if (/^emacsconf-[0-9]*-(.*?)--/) { print $1; } else { print; }')
if [[ -f /data/emacsconf/assets/overlays/$SLUG-other.png ]]; thenecho"Found other overlay for $SLUG, copying"
cp /data/emacsconf/assets/overlays/$SLUG-other.png ~/other.png
elseecho"Could not find /data/emacsconf/assets/overlays/$SLUG-other.png, please override ~/other.png manually"
cp /data/emacsconf/assets/overlays/blank-other.png ~/other.png
fiif [[ -f /data/emacsconf/assets/overlays/$SLUG-video.png ]]; thenecho"Found video overlay for $SLUG, copying"
cp /data/emacsconf/assets/overlays/$SLUG-video.png ~/video.png
elseecho"Could not find /data/emacsconf/assets/overlays/$SLUG-video.png, override ~/video.png manually"
cp /data/emacsconf/assets/overlays/blank-video.png ~/video.png
fi
set-overlay is called by the Emacs Lisp function emacsconf-stream-set-overlay:
emacsconf-stream-set-overlay: Reset the overlay for TALK, just in case.
(defunemacsconf-stream-set-overlay (talk)
"Reset the overlay for TALK, just in case.With a prefix argument (\\[universal-argument]), clear the overlay."
(interactive (list
(if current-prefix-arg
(emacsconf-complete-track)
(emacsconf-complete-talk-info))))
(emacsconf-stream-track-ssh
(emacsconf-get-track talk)
"overlay"
(if current-prefix-arg
"blank"
(plist-get talk :slug))))
Play the intro video or display the intro slide
Figure 3: Sample intro slide
We wanted to display the talk titles, speaker names, and URLs for both
the previous talk and the next talk. We generated all the intro
slides, and then as time permitted, we recorded introduction videos so
that we could practise saying people's names instead of panicking
during a full day. Actually generating the intro slide or video is a
topic for another blog post. This post just focuses on playing the
appropriate video or displaying the right image, which is handled by
the intro script.
#!/bin/bash# # Kill the background music if playingif screen -list | grep -q background; then
screen -S background -X quit
fi# Update the overlaySLUG=$1FILE=$1if [[ ! -f $FILE ]]; thenLIST=(/data/emacsconf/assets/stream/emacsconf--$FILE--*.webm)
FILE="${LIST[0]}"BY_SLUG=1
elseSLUG=$(echo"$FILE" | perl -ne 'if (/emacsconf-[0-9]*-(.*?)--/) { print $1; } else { print; }')
fishift
overlay $SLUGif [[ -f /data/emacsconf/assets/intros/$SLUG.webm ]]; then
mpv /data/emacsconf/assets/intros/$SLUG.webm
else
firefox /data/emacsconf/assets/in-between/$SLUG.png
fi
This is easy to call from Emacs Lisp.
emacsconf-stream-play-intro: Play the recorded intro or display the in-between slide for TALK.
(defunemacsconf-stream-play-intro (talk)
"Play the recorded intro or display the in-between slide for TALK."
(interactive (list (emacsconf-complete-talk-info)))
(setq talk (emacsconf-resolve-talk talk))
(emacsconf-stream-track-ssh talk "nohup""intro" (plist-get talk :slug)))
Play just the video
Sometimes we might need to restart a video without playing the
introduction again. The ready-to-stream videos are all in one
directory following the naming convention
emacsconf-year-slug--title--speakers--main.webm. We update the
--main.webm video as we go through the process of reencoding the
video, normalizing sound, and adding captions. We can play the latest
video by doing a wildcard match based on the slug.
#!/bin/bash# Play intro if recorded, then play files# # Kill the background music if playingif screen -list | grep -q background; then
screen -S background -X quit
fi# Update the overlayFILE=$1if [[ ! -f $FILE ]]; thenLIST=(/data/emacsconf/assets/stream/emacsconf--$FILE*--main.webm)
FILE="${LIST[0]}"BY_SLUG=1
fishiftSLUG=$(echo"$FILE" | perl -ne 'if (/emacsconf-[0-9]*-(.*?)--/) { print $1; } else { print; }')
overlay $SLUG
mpv $FILE $* &
emacsconf-stream-play-video: Play just the video for TALK.
(defunemacsconf-stream-play-video (talk)
"Play just the video for TALK."
(interactive (list (emacsconf-complete-talk-info)))
(setq talk (emacsconf-resolve-talk talk))
(emacsconf-stream-track-ssh
talk "nohup""play" (plist-get talk :slug)))
Play the intro and then the video
The easiest way to go through a talk is to play the introduction and
the video without further manual intervention. This shell script
updates the overlay, plays the intro if available, and then continues
with the video.
#!/bin/bash# Play intro if recorded, then play files# # Kill the background music if playingif screen -list | grep -q background; then
screen -S background -X quit
fi# Update the overlayFILE=$1if [[ ! -f $FILE ]]; thenLIST=(/data/emacsconf/assets/stream/emacsconf--$FILE*.webm)
FILE="${LIST[0]}"BY_SLUG=1
fishiftSLUG=$(echo"$FILE" | perl -ne 'if (/emacsconf-[0-9]*-(.*?)--/) { print $1; } else { print; }')
overlay $SLUG# Play the videoif [[ -f /data/emacsconf/assets/intros/$SLUG.webm ]]; then
intro $SLUGfi
mpv $FILE $* &
Along the lines of minimizing manual work, this more complex Emacs
Lisp function considers different combinations of intros and talks:
Recorded intro
Live intro
Recorded talk
automatically play both
show intro slide; remind host to play video
Live talk
play intro; host joins BBB
join BBB room automatically
emacsconf-stream-play-talk-on-change: Play the talk.
(defunemacsconf-stream-play-talk-on-change (talk)
"Play the talk."
(interactive (list (emacsconf-complete-talk-info)))
(setq talk (emacsconf-resolve-talk talk))
(when (or (not (boundp 'org-state)) (string= org-state "PLAYING"))
(if (plist-get talk :stream-files)
(progn
(emacsconf-stream-track-ssh
talk
"overlay"
(plist-get talk :slug))
(emacsconf-stream-track-ssh
talk
(append
(list
"nohup""mpv")
(split-string-and-unquote (plist-get talk :stream-files))
(list "&"))))
(emacsconf-stream-track-ssh
talk
(cons
"nohup"
(cond
((and
(plist-get talk :recorded-intro)
(plist-get talk :video-file)) ;; recorded intro and recorded talk
(message "should automatically play intro and recording")
(list "play-with-intro" (plist-get talk :slug))) ;; todo deal with stream files
((and
(plist-get talk :recorded-intro)
(null (plist-get talk :video-file))) ;; recorded intro and live talk; play the intro and join BBB
(message "should automatically play intro; join %s" (plist-get talk :bbb-backstage))
(list "intro" (plist-get talk :slug)))
((and
(null (plist-get talk :recorded-intro))
(plist-get talk :video-file)) ;; live intro and recorded talk, show slide and use Mumble; manually play talk
(message "should show intro slide; play %s afterwards" (plist-get talk :slug))
(list "intro" (plist-get talk :slug)))
((and
(null (plist-get talk :recorded-intro))
(null (plist-get talk :video-file))) ;; live intro and live talk, join the BBB
(message "join %s for live intro and talk" (plist-get talk :bbb-backstage))
(list "bbb" (plist-get talk :slug)))))))))
Open the Etherpad
We used Etherpad collaborative documents to collect people's questions
during the conference. I made an index page that linked to the
Etherpads for the different talks so that I could open it in the
browser used for streaming.
Figure 4: Backstage index
I also had an Emacs Lisp function that opened up the pad in the appropriate stream.
emacsconf-stream-open-pad: Open the Etherpad collaborative document for TALK.
#!/bin/bash# Display the Etherpad collaborative document# # Update the overlaySLUG=$1
overlay $SLUG
firefox https://pad.emacsconf.org/-$SLUG
Open the Big Blue Button web conference
Most Q&A sessions are done live through a BigBlueButton web
conference. We use redirects to make it easier to go to the talk URL.
Backstage redirects are protected by a username and password which is
shared with volunteers and saved in the browser used for streaming.
#!/bin/bash# Open the Big Blue Button room using the backstage link# # Kill the background music if playingif screen -list | grep -q background; then
screen -S background -X quit
fi# Update the overlaySLUG=$1
overlay $SLUG
firefox https://media.emacsconf.org//backstage/current/room/$SLUG
Public redirect URLs start off with a refresh loop and then are
overwritten with a redirect to the actual page when the host is okay
with opening up the Q&A for general participation. This is done by
changing the TODO status of the talk from CLOSED_Q to OPEN_Q.
emacsconf-publish-bbb-redirect: Update the publicly-available redirect for TALK.
(defunemacsconf-publish-bbb-redirect (talk &optional status)
"Update the publicly-available redirect for TALK."
(interactive (list (emacsconf-complete-talk-info)))
(let ((bbb-filename (expand-file-name (format "bbb-%s.html" (plist-get talk :slug))
emacsconf-publish-current-dir))
(bbb-redirect-url (concat "https://media.emacsconf.org/" emacsconf-year "/current/bbb-" (plist-get talk :slug) ".html"))
(status (or status (emacsconf-bbb-status (if (boundp 'org-state) (append (list :status org-state) talk) talk)))))
(with-temp-file bbb-filename
(insert
(emacsconf-replace-plist-in-string
(append talk (list :base-url emacsconf-base-url :bbb-redirect-url bbb-redirect-url))
(pcase status
('open"<html><head><meta http-equiv=\"refresh\" content=\"0; URL=${bbb-room}\"></head><body>The live Q&A room for ${title} is now open. You should be redirected to <a href=\"${bbb-room}\">${bbb-room}</a> automatically, but if not, please visit the URL manually to join the Q&A.</body></html>")
('before"<html><head><meta http-equiv=\"refresh\" content=\"5; URL=${bbb-redirect-url}\"></head><body>The Q&A room for ${title} is not yet open. This page will refresh every 5 seconds until the BBB room is marked as open, or you can refresh it manually.</body></html>")
('after"<html><head><body>The Q&A room for ${title} has finished. You can find more information about the talk at <a href=\"${base-url}${url}\">${base-url}${url}</a>.</body></html>")
(_
"<html><head><body>There is no live Q&A room for ${title}. You can find more information about the talk at <a href=\"${base-url}${url}\">${base-url}${url}</a>.</body></html>"
)))))))
Open up the stream chat
The IRC chat is the same for the whole track instead of changing for each talk. Since we might close the window, it's useful to be able to quickly open it again.
emacsconf-stream-join-chat: Join the IRC chat for TALK.
An Emacs Lisp function makes it easier to do the right thing depending on the type of Q&A planned for the talk.
emacsconf-stream-join-qa: Join the Q&A for TALK.
(defunemacsconf-stream-join-qa (talk)
"Join the Q&A for TALK.This uses the BBB room if available, or the IRC channel if not."
(interactive (list (emacsconf-complete-talk-info)))
(if (and (null (plist-get talk :video-file))
(string-match "live" (plist-get talk :q-and-a)))
(emacsconf-stream-track-ssh
talk
"nohup""firefox""-new-window"
(plist-get talk :pad-url))
(emacsconf-stream-track-ssh
talk
"nohup""firefox""-new-window"
(pcase (plist-get talk :q-and-a)
((or'nil"" (rx"Mumble"))
(plist-get talk :qa-slide-url))
((rx"live")
(plist-get talk :bbb-backstage))
((rx"IRC")
(plist-get talk :webchat-url))
((rx"pad")
(plist-get talk :pad-url))
(_ (plist-get talk :qa-slide-url))))))
Summary
Combining shell scripts (roles/obs/templates) and
Emacs Lisp functions (emacsconf-stream.el) help us
simplify the work of taking talk-specific actions that depend on the
kind of talk or Q&A session. Using simple identifiers and consistent
file name conventions means that we can refer to talks quickly and use
wildcards in shell scripts.
We started the conference with me jumping around and running most of
the commands, since I had hastily written them in the weeks leading up
to the conference and I was the most familiar with them. As the
conference went on, other organizers got the hang of the commands and
took over running their streams. Yay!
In the course of organizing and running EmacsConf, I often need to
jump to or act on specific talks. I have a function that jumps to the
talk heading so that I can look up additional information or add
notes.
Figure 1: Jumping to a talk
emacsconf-go-to-talk: Jump to the talk heading matching SEARCH.
Most of the work is done in a completion function that makes it easy
to specify a talk using the slug (talk ID), title, or speaker names.
emacsconf-complete-talk: Offer talks for completion.
(defunemacsconf-complete-talk (&optional info)
"Offer talks for completion.If INFO is specified, limit it to that list."
(let ((choices
(if (and (null info) emacsconf-complete-talk-cache)
emacsconf-complete-talk-cache
(mapcar (lambda (o)
(string-join
(delq nil
(mapcar (lambda (f) (plist-get o f))
'(:slug:title:speakers:irc)))
" - "))
(or info (emacsconf-get-talk-info))))))
(completing-read
"Talk: "
(lambda (string predicate action)
(if (eq action 'metadata)
'(metadata (category . emacsconf))
(complete-with-action action choices string predicate))))))
In addition to jumping to the Org heading for a talk, there are a
bunch of other things I might want to do. Embark lets me add a bunch
of shortcuts for working with a talk. I could open the caption file,
edit the talk's wiki page, change a talk's property, e-mail the
speaker, or more. Here's the Embark-related code from emacsconf.el:
Embark-related code
;;; Embark
(defunemacsconf-embark-finder ()
"Identify when we're on a talk subtree."
(when (and (derived-mode-p 'org-mode)
(org-entry-get-with-inheritance "SLUG"))
(cons 'emacsconf (org-entry-get-with-inheritance "SLUG"))))
(defunemacsconf-insert-talk-title (search)
"Insert the talk title matching SEARCH."
(interactive (list (emacsconf-complete-talk)))
(insert (plist-get (emacsconf-search-talk-info search) :title)))
(with-eval-after-load'embark
(add-to-list 'embark-target-finders'emacsconf-embark-finder)
(defvar-keymap embark-emacsconf-actions
:doc"Keymap for emacsconf-related things""a"#'emacsconf-announce"c"#'emacsconf-find-captions-from-slug"d"#'emacsconf-find-caption-directives-from-slug"p"#'emacsconf-set-property-from-slug"w"#'emacsconf-edit-wiki-page"s"#'emacsconf-set-start-time-for-slug"W"#'emacsconf-browse-wiki-page"u"#'emacsconf-update-talk"t"#'emacsconf-insert-talk-title"m"#'emacsconf-mail-speaker-from-slug"n"#'emacsconf-notmuch-search-mail-from-entry"f"#'org-forward-heading-same-level"b"#'org-backward-heading-same-level"RET"#'emacsconf-go-to-talk)
(add-to-list 'embark-keymap-alist'(emacsconf . embark-emacsconf-actions)))
;;; Status updates
For example, I sometimes need to open the wiki page for a talk in order to update the talk description.
emacsconf-edit-wiki-page: Open the wiki page for the talk matching SEARCH.
;;; Embark
(defunemacsconf-embark-finder ()
"Identify when we're on a talk subtree."
(when (and (derived-mode-p 'org-mode)
(org-entry-get-with-inheritance "SLUG"))
(cons 'emacsconf (org-entry-get-with-inheritance "SLUG"))))
Embark can also act on completion candidates, so I can call any of those actions from my C-c e t shortcut for emacsconf-go-to-talk. This is specified by the (metadata (category . emacsconf)) in emacsconf-complete-talk and the (add-to-list 'embark-keymap-alist '(emacsconf . embark-emacsconf-actions)) in my Embark configuration.
C-. is the embark-act shortcut in my configuration. When I need to
remember what the shortcuts are, I can use C-h
(embark-keymap-help) to list the keyboard shortcuts or select the command with completion.
Figure 2: Embark help for Emacsconf talks
The code above and related functions are in emacsconf.el or other files in the emacsconf-el repository.
Here's a quick overview of how I experiment with schedules for EmacsConf. I have all the talk details in Org subtrees. There's a SLUG property that has the talk ID, a TIME property that says how long a talk is, and a Q_AND_A property that says what kind of Q&A the speaker wants: live, IRC, Etherpad, or after the event. Some talks have fixed starting times, like the opening remarks. Others start when the Q&A for the previous session ends. I generate an SVG so that I can quickly see how the schedule looks. Are there big gaps? Did I follow everyone's availability constraints? Does the schedule flow reasonably logically?
Common functions are in emacsconf-schedule.el in the emacsconf-el repository, while conference-specific code is in our private conf.org file.
Motivation
I was working on the schedule for EmacsConf 2022 when I ran into a problem. I wanted more talks than could fit into the time that we had, even if I scheduled it as tightly as possible with just a few minutes of transition between talks. This had been the scheduling strategy we used in previous years, squishing all the talks together with just enough time to let people know where they could go to ask questions. We had experimented with an alternate track for Q&A during EmacsConf 2021, so that had given us a little more space for discussion, but it wasn't going to be enough to fit in all the talks we wanted for EmacsConf 2022.
Could I convince the other organizers to take on the extra work needed for a multi-track conference? I knew it would be a lot of extra technical risk, and we'd need another host and streamer too. Could I make the schedule easy to understand on the wiki if we had two tracks? The Org Mode table I'd been using for previous years was not going to be enough. I was getting lost in the text.
Figure 2: Scheduling using Org Mode tables
Visualizing the schedule as an SVG
I'd been playing around with SVGs in Emacs. It was pretty straightforward to write a function that used time for the X axis and displayed different tracks. That showed us how full the conference was going to be if I tried to pack everything into two days.
Figure 3: One full day
emacsconf-schedule-svg-track: Draw the actual rectangles and text for the talks.
(defunemacsconf-schedule-svg-track (svg base-x base-y width height start-time end-time info)
"Draw the actual rectangles and text for the talks."
(let ((scale (/ width (float-time (time-subtract end-time start-time)))))
(mapc
(lambda (o)
(let* ((offset (floor (* scale (float-time (time-subtract (plist-get o :start-time) start-time)))))
(size (floor (* scale (float-time (time-subtract (plist-get o :end-time) (plist-get o :start-time))))))
(x (+ base-x offset))
(y base-y)
(node (dom-node
'rect
(list
(cons 'x x)
(cons 'y y)
(cons 'opacity"0.8")
(cons 'width size)
(cons 'height (1- height))
(cons 'stroke"black")
(cons 'stroke-dasharray
(if (string-match "live" (or (plist-get o :q-and-a) "live"))
"""5,5,5"))
(cons 'fill
(cond
((string-match "BREAK\\|LUNCH" (plist-get o :title)) "white")
((plist-get o :invalid) "red")
((string-match "EST"
(or (plist-get o :availability) ""))
"lightgray")
(t "lightgreen"))))))
(parent (dom-node
'a
(list
(cons 'href
(concat
(if emacsconf-use-absolute-url
emacsconf-base-url
"/")
(plist-get o :url)))
(cons 'title (plist-get o :title))
(cons 'data-slug (plist-get o :slug)))
(dom-node 'title nil
(concat (format-time-string "%l:%M-" (plist-get o :start-time) emacsconf-timezone)
(format-time-string "%l:%M " (plist-get o :end-time) emacsconf-timezone)
(plist-get o :title)))
node
(dom-node
'g`((transform . ,(format "translate(%d,%d)"
(+ x size -2) (+ y height -2))))
(dom-node
'text
(list
(cons 'fill"black")
(cons 'x 0)
(cons 'y 0)
(cons 'font-size 10)
(cons 'transform"rotate(-90)"))
(svg--encode-text (or (plist-get o :slug) (plist-get o :title))))))))
(run-hook-with-args
'emacsconf-schedule-svg-modify-functions
o node parent)
(dom-append-child
svg
parent)))
info)))
emacsconf-schedule-svg-day: Add the time scale and the talks on a given day.
(defunemacsconf-schedule-svg-day (elem label width height start end tracks)
"Add the time scale and the talks on a given day."
(let* ((label-margin 15)
(track-height (/ (- height (* 2 label-margin)) (length tracks)))
(x 0) (y label-margin)
(scale (/ width (float-time (time-subtract end start))))
(time start))
(dom-append-child elem (dom-node 'title nil (concat "Schedule for " label)))
(svg-rectangle elem 0 0 width height :fill"white")
(svg-text elem label :x 3 :y (- label-margin 3) :fill"black":font-size"10")
(mapc (lambda (track)
(emacsconf-schedule-svg-track
elem x y width track-height
start end track)
(setq y (+ y track-height)))
tracks)
;; draw grid
(while (time-less-p time end)
(let ((x (* (float-time (time-subtract time start)) scale)))
(dom-append-child
elem
(dom-node
'g`((transform . ,(format "translate(%d,%d)" x label-margin)))
(dom-node
'line`((stroke . "darkgray")
(x1 . 0)
(y1 . 0)
(x2 . 0)
(y2 . ,(- height label-margin label-margin))))
(dom-node
'text`((fill . "black")
(x . 0)
(y . ,(- height 2 label-margin))
(font-size . 10)
(text-anchor . "left"))
(svg--encode-text (format-time-string "%-l %p" time emacsconf-timezone)))))
(setq time (time-add time (seconds-to-time 3600)))))
elem))
When the other organizers saw the two schedules, they were strongly in
favour of the two-track option. Yay!
Changing scheduling strategies
As I played around with the schedule, I wanted a quick way to test
different scheduling strategies, such as changing the length of Q&A
sessions for live web conferences versus IRC/Etherpad/email Q&A.
Putting those into variables allowed me to easily override them with a
let form, and I used a list of functions to calculate or modify the
schedule from the Org trees.
Figure 5: Changing the default Q&A times
emacsconf-schedule-allocate-buffer-time: Allocate buffer time based on whether INFO has live Q&A.
(defunemacsconf-schedule-allocate-buffer-time (info)
"Allocate buffer time based on whether INFO has live Q&A.Uses `emacsconf-schedule-default-buffer-minutes' and`emacsconf-schedule-default-buffer-minutes-for-live-q-and-a'."
(mapcar (lambda (o)
(when (plist-get o :slug)
(unless (plist-get o :buffer)
(plist-put o :buffer
(number-to-string
(if (string-match "live" (or (plist-get o :q-and-a) "live"))
emacsconf-schedule-default-buffer-minutes-for-live-q-and-a
emacsconf-schedule-default-buffer-minutes)))))
o)
info))
This is actually applied by emacsconf-schedule-prepare, which runs through the list of strategies defined in emacsconf-schedule-strategies.
emacsconf-schedule-prepare: Apply ‘emacsconf-schedule-strategies’ to INFO to determine the schedule.
(defunemacsconf-schedule-prepare (&optional info)
"Apply `emacsconf-schedule-strategies' to INFO to determine the schedule."
(emacsconf-schedule-based-on-info
(seq-reduce (lambda (prev val) (funcall val prev))
emacsconf-schedule-strategies
(or info (emacsconf-get-talk-info)))))
Type of Q&A
I wanted to see which sessions had live Q&A via web conference and
which ones had IRC/Etherpad/asynchronous Q&A. I set the talk outlines
so that dashed lines mean asynchronous Q&A and solid lines mean live.
Live Q&As take a little more work on our end because the host starts
it up and reads questions, but they're more interactive. This is handled by
I wanted to be able to quickly reorganize talks by moving their IDs around in a list instead of just using the order of the subtrees in my Org Mode file. I wrote a function that took a list of symbols, looked up each of the talks, and returned a list of talk info property lists. I also added the ability to override some things about talks, such as whether something started at a fixed time.
emacsconf-schedule-inflate-sexp: Takes a list of talk IDs and returns a list that includes the scheduling info.
(defunemacsconf-schedule-inflate-sexp (sequence &optional info include-time)
"Takes a list of talk IDs and returns a list that includes the scheduling info.Pairs with `emacsconf-schedule-dump-sexp'."
(setq info (or info (emacsconf-get-talk-info)))
(let ((by-assoc (mapcar (lambda (o) (cons (intern (plist-get o :slug)) o))
(emacsconf-filter-talks info)))
date)
(mapcar
(lambda (seq)
(unless (listp seq) (setq seq (list seq)))
(if include-time
(error"Not yet implemented")
(let ((start-prop (or (plist-get (cdr seq) :start)
(and (stringp (cdr seq)) (cdr seq))))
(time-prop (or (plist-get (cdr seq) :time) ; this is duration in minutes
(and (numberp (cdr seq)) (cdr seq))))
(track-prop (plist-get (cdr seq) :track)))
(append
;; overriding
(when start-prop
(if (string-match "-" start-prop)
(setq date (format-time-string "%Y-%m-%d" (date-to-time start-prop)))
(setq start-prop (concat date " " start-prop)))
(list
:scheduled (format-time-string (cdr org-time-stamp-formats) (date-to-time start-prop)
emacsconf-timezone)
:start-time (date-to-time start-prop)
:fixed-time t))
(when track-prop
(list :track track-prop))
(when time-prop
(list :time (if (numberp time-prop) (number-to-string time-prop) time-prop)))
;; base entity
(cond
((eq (car seq) 'lunch)
(list :title"LUNCH":time (number-to-string emacsconf-schedule-lunch-time)))
((eq (car seq) 'break)
(list :title"BREAK":time (number-to-string emacsconf-schedule-break-time)))
((symbolp (car seq))
(assoc-default (car seq) by-assoc))
((stringp (car seq))
(or (seq-find (lambda (o) (string= (plist-get o :title) (car seq))) info)
(list :title (car seq))))
(t (error"Unknown %s" (prin1-to-string seq))))))))
sequence)))
That allowed me to specify a test schedule like this:
(setq emacsconf-schedule-plan
'(("GEN Saturday, Dec 2":start"2023-12-02 09:00")
sat-open
adventure writing one uni lunch hyperdrive ref mentor flat
(hn :start"15:30")
(web :start"16:00")
("GEN Sunday, Dec 3":start"2023-12-03 09:00")
sun-open
windows extending lunch lspocaml sharing emacsen
voice llm))
(setq emacsconf-schedule-draft
(emacsconf-schedule-prepare
(emacsconf-schedule-inflate-sexp emacsconf-schedule-plan)))
Validating the schedule
Now that I could see the schedule visually, it made sense to add some validation. As mentioned in my post about timezones, I wanted to validate live Q&A against the speaker's availability and colour sessions red if they were outside the times I'd written down.
I also wanted to arrange the schedule so that live Q&A sessions didn't start at the same time, giving me a little time to switch between sessions in case I needed to help out. I wrote a function that checked if the Q&A for a session started within five minutes of the previous one. This turned out to be pretty useful, since I ended up mostly taking care of playing the videos and opening the browsers for both streams. With that in place, I could just move the talks around until everything fit well together.
emacsconf-schedule-validate-live-q-and-a-sessions-are-staggered: Try to avoid overlapping the start of live Q&A sessions.
(defunemacsconf-schedule-validate-live-q-and-a-sessions-are-staggered (schedule)
"Try to avoid overlapping the start of live Q&A sessions.Return nil if there are no errors."
(when emacsconf-schedule-validate-live-q-and-a-sessions-buffer
(let (last-end)
(delq nil
(mapcar (lambda (o)
(prog1
(when (and last-end
(time-less-p
(plist-get o :end-time)
(time-add last-end (seconds-to-time (* emacsconf-schedule-validate-live-q-and-a-sessions-buffer 60)))))
(plist-put o :invalid (format "%s live Q&A starts at %s within %d minutes of previous live Q&A at %s"
(plist-get o :slug)
(format-time-string "%m-%d %-l:%M"
(plist-get o :end-time))
emacsconf-schedule-validate-live-q-and-a-sessions-buffer
(format-time-string "%m-%d %-l:%M"
last-end)))
(plist-get o :invalid))
(setq last-end (plist-get o :end-time))))
(sort
(seq-filter (lambda (o) (string-match "live" (or (plist-get o :q-and-a) "")))
schedule)
(lambda (a b)
(time-less-p (plist-get a :end-time) (plist-get b :end-time)))
))))))
Publishing SVGs in the wiki
When the schedule settled down, it made perfect sense to include the image on the schedule page as well as on each talk page. I wanted people to be able to click on a rectangle and load the talk page. That meant including the SVG in the markup and allowing the attributes in ikiwiki's HTML sanitizer. In our htmlscrubber.pm, I needed to add svg rect text g title line to allow, and add version xmlns x y x1 y1 x2 y2 fill font-size font-weight stroke stroke-width stroke-dasharray transform opacity to the default attributes.
Figure 6: Opening a talk from the SVG
For the public-facing pages, I wanted to colour the talks based on
track. I specified the colours in emacsconf-tracks (peachpuff and
skyblue) and used them in emacsconf-schedule-svg-color-by-track.
emacsconf-schedule-svg-color-by-track: Color sessions based on track.
(defunemacsconf-schedule-svg-color-by-track (o node &optional parent)
"Color sessions based on track."
(let ((track (emacsconf-get-track (plist-get o :track))))
(when track
(dom-set-attribute node 'fill (plist-get track :color)))))
I wanted talk pages to highlight the talk on the schedule so that
people could easily find other sessions that conflict. Because a
number of people in the Emacs community browse with Javascript turned
off, I used Emacs Lisp to generate a copy of the SVG with the current
talk highlighted.
emacsconf-publish-format-talk-page-schedule: Add the schedule image for TALK based on INFO.
(defunemacsconf-publish-format-talk-page-schedule (talk info)
"Add the schedule image for TALK based on INFO."
(concat
"\nThe following image shows where the talk is in the schedule for "
(format-time-string "%a %Y-%m-%d" (plist-get talk :start-time) emacsconf-timezone) ". Solid lines show talks with Q&A via BigBlueButton. Dashed lines show talks with Q&A via IRC or Etherpad."
(format "<div class=\"schedule-in-context schedule-svg-container\" data-slug=\"%s\">\n" (plist-get talk :slug))
(let* ((width 800) (height 150)
(talk-date (format-time-string "%Y-%m-%d" (plist-get talk :start-time) emacsconf-timezone))
(start (date-to-time (concat talk-date "T" emacsconf-schedule-start-time emacsconf-timezone-offset)))
(end (date-to-time (concat talk-date "T" emacsconf-schedule-end-time emacsconf-timezone-offset)))
svg)
(with-temp-buffer
(setq svg (emacsconf-schedule-svg-day
(svg-create width height)
(format-time-string "%A" (plist-get talk :start-time) emacsconf-timezone)
width height
start end
(emacsconf-by-track
(seq-filter (lambda (o) (string= (format-time-string "%Y-%m-%d" (plist-get o :start-time) emacsconf-timezone)
talk-date))
info))))
(mapc (lambda (node)
(let ((rect (car (dom-by-tag node 'rect))))
(if (string= (dom-attr node 'data-slug) (plist-get talk :slug))
(progn
(dom-set-attribute rect 'opacity"0.8")
(dom-set-attribute rect 'stroke-width"3")
(dom-set-attribute (car (dom-by-tag node 'text)) 'font-weight"bold"))
(dom-set-attribute rect 'opacity"0.5"))))
(dom-by-tag svg 'a))
(svg-print svg)
(buffer-string)))
"\n</div>\n""\n"
(emacsconf-format-talk-schedule-info talk) "\n\n"))
Colouring talks based on status
As the conference approached, I wanted to colour talks based on their status, so I could see how much of our schedule already had videos and even how many had been captioned. That was just a matter of adding a modifier function to change the SVG rectangle colour depending on the TODO status.
Figure 7: Changing talk colour based on status
emacsconf-schedule-svg-color-by-status: Set talk color based on status.
(defunemacsconf-schedule-svg-color-by-status (o node &optional _)
"Set talk color based on status.Processing: palegoldenrod,Waiting to be assigned a captioner: yellow,Captioning in progress: lightgreen,Ready to stream: green,Other status: gray"
(unless (plist-get o :invalid)
(dom-set-attribute node 'fill
(pcase (plist-get o :status)
((rx (or"TO_PROCESS""PROCESSING""TO_AUTOCAP"))
"palegoldenrod")
((rx (or"TO_ASSIGN"))
"yellow")
((rx (or"TO_CAPTION"))
"lightgreen")
((rx (or"TO_STREAM"))
"green")
(_ "gray")))))
Summary
Common functions are in emacsconf-schedule.el in the emacsconf-el repository, while conference-specific code is in our private conf.org file.
Some information is much easier to work with graphically than in plain text. Seeing the schedules as images made it easier to spot gaps or errors, tinker with parameters, and communicate with other people. Writing Emacs Lisp functions to modify my data made it easier to try out different things without rearranging the text in my Org file. Because Emacs can display images inside the editor, it was easy for me to make changes and see the images update right away. Using SVG also made it possible to export the image and make it interactive. Handy technique!
