I ran into quota limits when uploading videos to YouTube with a command-line tool, so I uploaded videos by selecting up to 15 videos at a time using the web-based interface. Each video was a draft, though, and I was having a hard time updating its visibility through the API. I think it eventually worked, but in the meantime, I used this very hacky hack to look for the "Edit Draft" button and click through the screens to publish them.
emacsconf-extract-youtube-publish-video-drafts-with-spookfox: Look for drafts and publish them.
Another example of a hacky Spookfox workaround was publishing the
unlisted videos. I couldn't figure out how to properly authenticate
with the Toobnix (Peertube) API to change the visibility of videos.
Peertube uses AngularJS components in the front end, so using
.click() on the input elements didn't seem to trigger anything. I
found out that I needed to use .dispatchEvent(new Event('input')) to
tell the dropdown for the visibility to display the options. source
emacsconf-extract-toobnix-publish-video-from-edit-page: Messy hack to set a video to public and store the URL.
(defunemacsconf-extract-toobnix-publish-video-from-edit-page ()
"Messy hack to set a video to public and store the URL."
(interactive)
(spookfox-js-injection-eval-in-active-tab "document.querySelector('label[for=privacy]').scrollIntoView(); document.querySelector('label[for=privacy]').closest('.form-group').querySelector('input').dispatchEvent(new Event('input'));" t)
(sit-for 1)
(spookfox-js-injection-eval-in-active-tab "document.querySelector('span[title=\"Anyone can see this video\"]').click()" t)
(sit-for 1)
(spookfox-js-injection-eval-in-active-tab "document.querySelector('button.orange-button').click()" t)(sit-for 3)
(emacsconf-extract-store-url)
(shell-command "xdotool key Alt+Tab sleep 1 key Ctrl+w Alt+Tab"))
It's a little nicer using Spookfox to automate browser interactions
than using xdotool, since I can get data out of it too. I could also
have used Puppeteer from either Python or NodeJS, but it's nice
staying with Emacs Lisp. Spookfox has some Javascript limitations
(can't close windows, etc.), so I might still use bits of xdotool or
Puppeteer to work around that. Still, it's nice to now have an idea of
how to talk to AngularJS components.
I wanted the EmacsConf 2023 Youtube and
Toobnix playlists
to mostly reflect the schedule of the conference by track, with talks
followed by their Q&A sessions (if recorded).
The list
I used Emacs Lisp to generate a list of videos in the order I wanted.
That Sunday closing remarks aren't actually in the playlists because
they're combined with the Q&A for my session on how we run Emacsconf.
emacsconf-extract-check-playlists: Return a table for checking playlist order.
(defunemacsconf-extract-check-playlists ()
"Return a table for checking playlist order."
(let ((pos 0))
(seq-mapcat (lambda (o)
(delq
nil
(list
(when (emacsconf-talk-file o "--main.webm")
(cl-incf pos)
(list pos
(plist-get o :title)
(org-link-make-string
(plist-get o :youtube-url)
"YouTube")
(org-link-make-string
(plist-get o :toobnix-url)
"Toobnix")))
(when (emacsconf-talk-file o "--answers.webm")
(cl-incf pos)
(list pos (concat "Q&A: " (plist-get o :title))
(org-link-make-string
(plist-get o :qa-youtube-url)
"YouTube")
(org-link-make-string
(plist-get o :qa-toobnix-url)
"Toobnix"))))))
(emacsconf-publish-prepare-for-display (emacsconf-get-talk-info)))))
1
An Org-Mode based text adventure game for learning the basics of Emacs, inside Emacs, written in Emacs Lisp
I bulk-added the Youtube videos to the playlist. The videos were not
in order because I uploaded some late submissions and forgotten
videos, which then got added to the end of the list.
I tried using the API to sort the playlist. This got it most of the
way there, and then I sorted the rest by hand.
emacsconf-extract-youtube-api-sort-playlist: Try to roughly sort the playlist.
(defunemacsconf-extract-youtube-api-sort-playlist (&optional dry-run-only)
"Try to roughly sort the playlist."
(interactive)
(setq emacsconf-extract-youtube-api-playlist (seq-find (lambda (o) (let-alist o (string= .snippet.title (concat emacsconf-name " " emacsconf-year))))
(assoc-default 'items emacsconf-extract-youtube-api-playlists)))
(setq emacsconf-extract-youtube-api-playlist-items
(emacsconf-extract-youtube-api-paginated-request (concat "https://youtube.googleapis.com/youtube/v3/playlistItems?part=snippet,contentDetails,status&forMine=true&order=date&maxResults=100&playlistId="
(url-hexify-string (assoc-default 'id emacsconf-extract-youtube-api-playlist)))))
(let* ((playlist-info emacsconf-extract-youtube-api-playlists)
(playlist-items emacsconf-extract-youtube-api-playlist-items)
(info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info)))
(slugs (seq-map (lambda (o) (plist-get o :slug)) info))
(position (1- (length playlist-items)))
result)
;; sort items
(mapc (lambda (talk)
(when (plist-get talk :qa-youtube-id)
;; move the q & a
(let ((video-object (emacsconf-extract-youtube-find-url-video-in-list
(plist-get talk :qa-youtube-url)
playlist-items)))
(let-alist video-object
(cond
((null video-object)
(message "Could not find video for %s" (plist-get talk :slug)))
;; not in the right position, try to move it
((< .snippet.position position)
(let ((video-id .id)
(playlist-id .snippet.playlistId)
(resource-id .snippet.resourceId))
(message "Trying to move %s Q&A to %d from %d" (plist-get talk :slug) position .snippet.position)
(add-to-list 'result (list (plist-get talk :slug) "answers" .snippet.position position))
(unless dry-run-only
(plz 'put"https://www.googleapis.com/youtube/v3/playlistItems?part=snippet":headers`(("Authorization" . ,(url-oauth-auth "https://youtube.googleapis.com/youtube/v3/"))
("Accept" . "application/json")
("Content-Type" . "application/json"))
:body (json-encode
`((id . ,video-id)
(snippet
(playlistId . ,playlist-id)
(resourceId . ,resource-id)
(position . ,position))))))))))
(setq position (1- position))))
;; move the talk if needed
(let ((video-object
(emacsconf-extract-youtube-find-url-video-in-list
(plist-get talk :youtube-url)
playlist-items)))
(let-alist video-object
(cond
((null video-object)
(message "Could not find video for %s" (plist-get talk :slug)))
;; not in the right position, try to move it
((< .snippet.position position)
(let ((video-id .id)
(playlist-id .snippet.playlistId)
(resource-id .snippet.resourceId))
(message "Trying to move %s to %d from %d" (plist-get talk :slug) position .snippet.position)
(add-to-list 'result (list (plist-get talk :slug) "main" .snippet.position position))
(unless dry-run-only
(plz 'put"https://www.googleapis.com/youtube/v3/playlistItems?part=snippet":headers`(("Authorization" . ,(url-oauth-auth "https://youtube.googleapis.com/youtube/v3/"))
("Accept" . "application/json")
("Content-Type" . "application/json"))
:body (json-encode
`((id . ,video-id)
(snippet
(playlistId . ,playlist-id)
(resourceId . ,resource-id)
(position . ,position))))))
))))
(setq position (1- position))))
(nreverse info))
result))
I needed to sort some of the videos manually. Trying to scroll by
dragging items to the top of the currently-displayed section of the
list was slow, and dropping the item near the top of the list so that
I could pick it up again after paging up was a little disorienting.
Fortunately, keyboard scrolling with page-up and page-down worked even
while dragging an item, so that was what I ended up doing: select the
item and then page-up while dragging.
YouTube doesn't display numbers for the playlist positions, but this
will add them. The numbers don't dynamically update when the list is
reordered, so I just re-ran the code after moving things around.
emacsconf-extract-youtube-spookfox-add-playlist-numbers: Number the playlist for easier checking.
(defunemacsconf-extract-youtube-spookfox-add-playlist-numbers ()
"Number the playlist for easier checking.Related: `emacsconf-extract-check-playlists'."
(interactive)
(spookfox-js-injection-eval-in-active-tab "[...document.querySelectorAll('ytd-playlist-video-renderer')].forEach((o, i) => { o.querySelector('.number')?.remove(); let div = document.createElement('div'); div.classList.add('number'); div.textContent = i; o.prepend(div) }))" t))
Figure 1: Adding numbers to the Youtube playlist
In retrospect, I could
probably have just cleared the playlist and then added the videos using the
in the right order instead of fiddling with inserting things.
Toobnix (Peertube)
Toobnix (Peertube) doesn't seem to have a way to bulk-add videos to a
playlist (or even to bulk-set their visibility). I started trying to
figure out how to use the API, but I got stuck because my token didn't
seem to let me access unlisted videos or do other things that required
proper authentication. Anyway, I came up with this messy hack to open
the videos in sequence and add them to the playlist using Spookfox.
(defunemacsconf-extract-toobnix-set-up-playlist ()
(interactive)
(mapcar
(lambda (o)
(when (plist-get o :toobnix-url)
(browse-url (plist-get o :toobnix-url))
(read-key "press a key when page is loaded")
(spookfox-js-injection-eval-in-active-tab "document.querySelector('.action-button-save').click()" t)
(spookfox-js-injection-eval-in-active-tab "document.querySelector('my-peertube-checkbox').click()" t)
(read-key "press a key when saved to playlist"))
(when (plist-get o :qa-toobnix-url)
(browse-url (plist-get o :qa-toobnix-url))
(read-key "press a key when page is loaded")
(spookfox-js-injection-eval-in-active-tab "document.querySelector('.action-button-save').click()" t)
(spookfox-js-injection-eval-in-active-tab "document.querySelector('my-peertube-checkbox').click()" t)
(read-key "press a key when saved to playlist")))
(emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
Maybe next year, I might be able to figure out how to use the APIs to
do this stuff automatically.
We upload EmacsConf videos to both YouTube and Toobnix, which is a
PeerTube instance. This makes it easier for people to come across them
after the conference.
I can upload to Toobnix and set titles and descriptions using the
peertube-cli tool. I tried a Python script for uploading to YouTube,
but it was a bit annoying due to quota restrictions. Instead, I
uploaded the videos by dragging and dropping them into YouTube Studio.
This allowed me to upload 15 at a time.
The videos on YouTube had just the filenames. I wanted to rename the
videos and set the descriptions. In 2022, I used xdotool, simulating
mouse clicks and pasting in text for larger text blocks.
Using xdotool wasn't very elegant, since I needed to figure out the
coordinates for each click. I tried using Spookfox to control Mozilla
Firefox from Emacs, but Youtube's editing interface didn't seem to
have any textboxes that I could set. I decided to use EmacsConf 2023
as an excuse to learn how to talk to the Youtube Data API, which
required figuring out OAuth. Even though it was easy to find examples
in Python and NodeJS, I wanted to see if I could stick with using
Emacs Lisp so that I could add the code to the emacsconf-el repository.
After a quick search, I picked url-http-oauth as the library that I'd
try first. I used the url-http-oauth-demo.el included in the package
to figure out what to set for the YouTube Data API. I wrote a function
to make getting the redirect URL easier
(emacsconf-extract-oauth-browse-and-prompt). Once I authenticated
successfully, I explored using alphapapa's plz library. It can handle
finding the JSON object and parsing it out for me. With it, I updated
videos to include titles and descriptions from my Emacs code, and I
copied the video IDs into my Org properties.
emacsconf-extract.el code for Youtube renaming
;;; YouTube;; When the token needs refreshing, delete the associated lines from;; ~/.authinfo This code just sets the title and description. Still;; need to figure out how to properly set the license, visibility,;; recording date, and captions.;;;; To avoid being prompted for the client secret, it's helpful to have a line in ~/.authinfo or ~/.authinfo.gpg with;; machine https://oauth2.googleapis.com/token username CLIENT_ID password CLIENT_SECRET
(defvaremacsconf-extract-google-client-identifier nil)
(defvaremacsconf-extract-youtube-api-channels nil)
(defvaremacsconf-extract-youtube-api-categories nil)
(defunemacsconf-extract-oauth-browse-and-prompt (url)
"Open URL and wait for the redirected code URL."
(browse-url url)
(read-from-minibuffer "Paste the redirected code URL: "))
(defunemacsconf-extract-youtube-api-setup ()
(interactive)
(require'plz)
(require'url-http-oauth)
(when (getenv "GOOGLE_APPLICATION_CREDENTIALS")
(let-alist (json-read-file (getenv "GOOGLE_APPLICATION_CREDENTIALS"))
(setq emacsconf-extract-google-client-identifier .web.client_id)))
(unless (url-http-oauth-interposed-p "https://youtube.googleapis.com/youtube/v3/")
(url-http-oauth-interpose
`(("client-identifier" . ,emacsconf-extract-google-client-identifier)
("resource-url" . "https://youtube.googleapis.com/youtube/v3/")
("authorization-code-function" . emacsconf-extract-oauth-browse-and-prompt)
("authorization-endpoint" . "https://accounts.google.com/o/oauth2/v2/auth")
("authorization-extra-arguments" .
(("redirect_uri" . "http://localhost:8080")))
("access-token-endpoint" . "https://oauth2.googleapis.com/token")
("scope" . "https://www.googleapis.com/auth/youtube")
("client-secret-method" . prompt))))
(setq emacsconf-extract-youtube-api-channels
(plz 'get"https://youtube.googleapis.com/youtube/v3/channels?part=contentDetails&mine=true":headers`(("Authorization" . ,(url-oauth-auth "https://youtube.googleapis.com/youtube/v3/")))
:as#'json-read))
(setq emacsconf-extract-youtube-api-categories
(plz 'get"https://youtube.googleapis.com/youtube/v3/videoCategories?part=snippet®ionCode=CA":headers`(("Authorization" . ,(url-oauth-auth "https://youtube.googleapis.com/youtube/v3/")))
:as#'json-read))
(setq emacsconf-extract-youtube-api-videos
(plz 'get (concat "https://youtube.googleapis.com/youtube/v3/playlistItems?part=snippet,contentDetails,status&forMine=true&order=date&maxResults=50&playlistId="
(url-hexify-string
(let-alist (elt (assoc-default 'items emacsconf-extract-youtube-api-channels) 0)
.contentDetails.relatedPlaylists.uploads)
))
:headers`(("Authorization" . ,(url-oauth-auth "https://youtube.googleapis.com/youtube/v3/")))
:as#'json-read)))
(defvaremacsconf-extract-youtube-tags'("emacs""emacsconf"))
(defunemacsconf-extract-youtube-object (video-id talk &optional privacy-status)
"Format the video object for VIDEO-ID using TALK details."
(setq privacy-status (or privacy-status "unlisted"))
(let ((properties (emacsconf-publish-talk-video-properties talk 'youtube)))
`((id . ,video-id)
(kind . "youtube#video")
(snippet
(categoryId . "28")
(title . ,(plist-get properties :title))
(tags . ,emacsconf-extract-youtube-tags)
(description . ,(plist-get properties :description))
;; Even though I set recordingDetails and status, it doesn't seem to stick.;; I'll leave this in here in case someone else can figure it out.
(recordingDetails (recordingDate . ,(format-time-string "%Y-%m-%dT%TZ" (plist-get talk :start-time) t))))
(status (privacyStatus . "unlisted")
(license . "creativeCommon")))))
(defunemacsconf-extract-youtube-api-update-video (video-object)
"Update VIDEO-OBJECT."
(let-alist video-object
(let* ((slug (cond;; not yet renamed
((string-match (rx (literal emacsconf-id) " " (literal emacsconf-year) " "
(group (1+ (or (syntax word) "-")))
" ")
.snippet.title)
(match-string 1 .snippet.title))
;; renamed, match the description instead
((string-match (rx (literal emacsconf-base-url) (literal emacsconf-year) "/talks/"
(group (1+ (or (syntax word) "-"))))
.snippet.description)
(match-string 1 .snippet.description))
;; can't find, prompt
(t
(when (string-match (rx (literal emacsconf-id) " " (literal emacsconf-year))
.snippet.title)
(completing-read (format "Slug for %s: "
.snippet.title)
(seq-map (lambda (o) (plist-get o :slug))
(emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))))))
(video-id .snippet.resourceId.videoId)
(id .id)
result)
(when slug
;; set the YOUTUBE_URL property
(emacsconf-with-talk-heading slug
(org-entry-put (point) "YOUTUBE_URL" (concat "https://www.youtube.com/watch?v=" video-id))
(org-entry-put (point) "YOUTUBE_ID" id))
(plz 'put"https://www.googleapis.com/youtube/v3/videos?part=snippet,recordingDetails,status":headers`(("Authorization" . ,(url-oauth-auth "https://youtube.googleapis.com/youtube/v3/"))
("Accept" . "application/json")
("Content-Type" . "application/json"))
:body (json-encode (emacsconf-extract-youtube-object video-id (emacsconf-resolve-talk slug))))))))
(defunemacsconf-extract-youtube-rename-videos (&optional videos)
"Rename videos and set the YOUTUBE_URL property in the Org heading."
(let ((info (emacsconf-get-talk-info)))
(mapc
(lambda (video)
(when (string-match (rx (literal emacsconf-id) " " (literal emacsconf-year)))
(emacsconf-extract-youtube-api-update-video video)))
(assoc-default 'items (or videos emacsconf-extract-youtube-api-videos)))))
(provide'emacsconf-extract)
I haven't quite figured out how to set status and recordingDetails
properly. The code sets them, but they don't stick. That's okay. I
think I can set those as a batch operation. It looks like I need to
change visibility one by one, though, which might be a good
opportunity to check the end of the video for anything that needs to
be trimmed off.
I also want to figure out how to upload captions. I'm not entirely
sure how to do multipart form data yet with the url library or
plz. It might be nice to someday set up an HTTP server so that Emacs
can handle OAuth redirects itself. I'll save that for another blog
post and share my notes for now.
EmacsConf 2023 is less than a month away. Speakers have been uploading
videos, captioning volunteers have been editing away, and I thiiiiink
I've gotten most of the infrastructure dusted off. Exciting!
Here's where we are with regard to talk status:
Waiting for 26 talks (~550 minutes) out of 42 total. Talks received so
far:
Speakers have been really nice about keeping in touch, so I'm not too
stressed about gaps in the schedule. Captioning volunteers have been
chugging through the talks and OpenAI Whisper's gotten a bit better at
spelling things, so that's terrific too. It's so exciting!
zaeph and bandali will probably host the general track and the
development track respectively. They've done it for a number of years
now, so it'll probably be fine even if we don't have a dry run all
together since they've got limited availability. (And we can take on
new volunteers if people want to help read questions!)
My stress level is pretty manageable at this point. I can even spend
evenings playing video games with the kiddo and weekends going on
little bike adventures, so that's awesome. I'm still a little worried
about tech hiccups, but we'll probably be able to figure things out.
Next steps are:
keep processing videos and captions
make the intro videos available so that speakers can correct my
pronunciation of their names
smoothen out and document the process for last-minute submissions
One of the benefits of volunteering for or speaking at EmacsConf is
that you get early access to the talks. We upload the files to a
password-protected web server. It's nothing fancy, just a directory
that has basic authentication and a static index.html generated by
Emacs Lisp.
I organize the talks by TODO status so that people can easily see
which talks are ready to be captioned. Since captioning can take a
little while, we reserve the talks we want to caption. That way,
people don't accidentally work on a talk that someone else is already
captioning. When people e-mail me to reserve a talk, I move the talk
from TO_ASSIGN to TO_CAPTION and add their name in the
:CAPTIONER: property.
Publishing the backstage index is done by an Emacs Lisp function that
smooshes the information together and then writes the files directly
to the server using TRAMP.
emacsconf-publish-backstage-index
(defunemacsconf-publish-backstage-index (&optional filename)
(interactive)
(setq filename (or filename (expand-file-name "index.html" emacsconf-backstage-dir)))
(let ((info (or emacsconf-schedule-draft (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
(emacsconf-main-extensions (append emacsconf-main-extensions emacsconf-publish-backstage-extensions)))
(with-temp-file filename
(let* ((talks
(mapcar
(lambda (o) (append
(list :captions-edited t) o))
(seq-filter (lambda (o) (plist-get o :speakers))
(emacsconf-active-talks (emacsconf-filter-talks info)))))
(by-status (seq-group-by (lambda (o) (plist-get o :status)) talks))
(files (directory-files emacsconf-backstage-dir)))
(insert
"<html><head><meta charset=\"UTF-8\"><link rel=\"stylesheet\" href=\"/style.css\" /></head><body>"
(if (file-exists-p (expand-file-name "include-in-index.html" emacsconf-cache-dir))
(with-temp-buffer (insert-file-contents (expand-file-name "include-in-index.html" emacsconf-cache-dir)) (buffer-string))
"")
"<p>Schedule by status: (gray: waiting, light yellow: processing, yellow: to assign, light green: captioning, green: captioned and ready)<br />Updated by conf.org and the wiki repository</br />"
(let* ((emacsconf-schedule-svg-modify-functions '(emacsconf-schedule-svg-color-by-status))
(img (emacsconf-schedule-svg 800 200 info)))
(with-temp-buffer
(mapc (lambda (node)
(dom-set-attribute
node 'href
(concat "#" (dom-attr node 'data-slug))))
(dom-by-tag img 'a))
(svg-print img)
(buffer-string)))
"</p>"
(if (eq emacsconf-backstage-phase 'prerec)
(format "<p>Waiting for %d talks (~%d minutes) out of %d total</p>"
(length (assoc-default "WAITING_FOR_PREREC" by-status))
(emacsconf-sum :time (assoc-default "WAITING_FOR_PREREC" by-status))
(length talks))
"")
"<ul>"
(mapconcat
(lambda (status)
(concat "<li>"
(if (string= status "TO_ASSIGN")
"TO_ASSIGN (waiting for volunteers)"
status)
": "
(mapconcat (lambda (o) (format "<a href=\"#%s\">%s</a>"
(plist-get o :slug)
(plist-get o :slug)))
(assoc-default status by-status)
", ")
"</li>"))
(pcase emacsconf-backstage-phase
('prerec'("TO_PROCESS""PROCESSING""TO_ASSIGN""TO_CAPTION""TO_STREAM"))
('harvest'("TO_ARCHIVE""TO_REVIEW_QA""TO_INDEX_QA""TO_CAPTION_QA")))
"")
"</ul>"
(pcase emacsconf-backstage-phase
('prerec
(concat
(emacsconf-publish-backstage-processing by-status files)
(emacsconf-publish-backstage-to-assign by-status files)
(emacsconf-publish-backstage-to-caption by-status files)
(emacsconf-publish-backstage-to-stream by-status files)))
('harvest
(let ((stages
'(("TO_REVIEW_QA" .
"Please review the --bbb-webcams.webm file and/or the --bbb-webcams.vtt and tell us (emacsconf-submit@gnu.org) if a Q&A session can be published or if it needs to be trimmed (lots of silence at the end of the recording, accidentally included sensitive information, etc.).")
("TO_INDEX_QA" .
"Please review the --answers.webm and --answers.vtt files to make chapter markers so that people can jump to specific parts of the Q&A session. The <a href=\"https://emacsconf.org/harvesting/\">harvesting page on the wiki</a> has some notes on the process. That way, it's easier for people to see the questions andanswers without needing to listen to everything again. You can see <a href=\"https://emacsconf.org/2022/talks/asmblox\">asmblox</a> for an example of the Q&A chapter markers.")
("TO_CAPTION_QA" .
"Please edit the --answers.vtt for the Q&A talk you're interested in, correcting misrecognized words and cleaning it up so that it's nice to use as closed captions. All talks should now have large-model VTTs to make it easier to edit."))))
(mapconcat
(lambda (stage)
(let ((status (car stage)))
(format
"<h1>%s: %d talk(s) (%d minutes)</h1>%s<ol class=\"videos\">%s</ol>"
status
(length (assoc-default status by-status))
(emacsconf-sum :video-time (assoc-default status by-status))
(cdr stage)
(mapconcat
(lambda (f)
(format "<li><a name=\"%s\"></a><strong><a href=\"%s%s\">%s</a></strong><br />%s (id:%s)<br />%s</li>"
(plist-get f :slug)
emacsconf-base-url
(plist-get f :url)
(plist-get f :title)
(plist-get f :speakers-with-pronouns)
(plist-get f :slug)
(emacsconf-publish-index-card
(append (list
:video-note
(unless (file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--bbb-webcams.webm") emacsconf-cache-dir))
"<div>No Q&A video for this talk</div>")
:video-file
(cond
((file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--answers.webm") emacsconf-cache-dir))
(concat (plist-get f :file-prefix) "--answers.webm"))
((file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--bbb-webcams.webm") emacsconf-cache-dir))
(concat (plist-get f :file-prefix) "--bbb-webcams.webm"))
(t t)) ;; omit video:video-id"-qanda":extra
(concat
(emacsconf-surround "QA note: " (plist-get f :qa-note) "<br />")
(format "Q&A archiving: <a href=\"%s-%s.txt\">IRC: %s-%s</a>"
(format-time-string "%Y-%m-%d" (plist-get f :start-time))
(plist-get (emacsconf-get-track f) :channel)
(format-time-string "%Y-%m-%d" (plist-get f :start-time))
(plist-get (emacsconf-get-track f) :channel))
(emacsconf-surround ", <a href=\""
(if (file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--pad.txt")
emacsconf-cache-dir))
(concat (plist-get f :file-prefix) "--pad.txt"))
"\">Etherpad (Markdown)</a>""")
(emacsconf-surround ", <a href=\"" (plist-get f :bbb-playback) "\">BBB playback</a>""")
(emacsconf-surround ", <a href=\""
(if (file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--bbb.txt")
emacsconf-cache-dir))
(concat (plist-get f :file-prefix) "--bbb.txt"))
"\">BBB text chat</a>""")
(emacsconf-surround ", <a href=\""
(if (file-exists-p (expand-file-name (concat (plist-get f :file-prefix) "--bbb-webcams.opus")
emacsconf-cache-dir))
(concat (plist-get f :file-prefix) "--bbb-webcams.opus"))
"\">BBB audio only</a>"""))
:files (emacsconf-publish-talk-files f files))
f)
emacsconf-main-extensions)))
(assoc-default status by-status) "\n"))))
stages
"\n"))))
(if (file-exists-p (expand-file-name "include-in-index-footer.html" emacsconf-cache-dir))
(with-temp-buffer (insert-file-contents (expand-file-name "include-in-index-footer.html" emacsconf-cache-dir)) (buffer-string))
"")
"</body></html>")))))
For example, the section for talks that are waiting for volunteers is handled by the function emacsconf-publish-backstage-to-assign.
emacsconf-publish-backstage-to-assign
(defunemacsconf-publish-backstage-to-assign (by-status files)
(let ((list (assoc-default "TO_ASSIGN" by-status)))
(format "<h1>%s talk(s) to be captioned, waiting for volunteers (%d minutes)</h1><p>You can e-mail <a href=\"mailto:sacha@sachachua.com\">sacha@sachachua.com</a> to call dibs on editing the captions for one of these talks. We use OpenAI Whisper to provide auto-generated VTT that you can use as a starting point, but you can also write the captions from scratch if you like. If you're writing them from scratch, you can choose to include timing information, or we can probably figure them out afterwards with a forced alignment tool. More info: <a href=\"https://emacsconf.org/captioning/\">captioning tips</a></p><ul class=\"videos\">%s</ul>"
(length list)
(emacsconf-sum :video-time list)
(mapconcat
(lambda (f)
(setq f (append
f
(list :extra
(if (plist-get f :caption-note) (concat "<div class=\"caption-note\">" (plist-get f :caption-note) "</div>") "")
:files
(emacsconf-publish-talk-files f files))))
(format "<li><a name=\"%s\"></a><strong><a href=\"%s\">%s</a></strong><br />%s (id:%s)<br />%s</li>"
(plist-get f :slug)
(plist-get f :absolute-url)
(plist-get f :title)
(plist-get f :speakers)
(plist-get f :slug)
(emacsconf-publish-index-card f)))
list
"\n"))))
Each talk has a little card that includes its video and links to files.
emacsconf-publish-index-card: Format an HTML card for TALK, linking the files in EXTENSIONS.
(defunemacsconf-publish-index-card (talk &optional extensions)
"Format an HTML card for TALK, linking the files in EXTENSIONS."
(let* ((file-prefix (plist-get talk :file-prefix))
(video-file (plist-get talk :video-file))
(video (and file-prefix
(emacsconf-publish-index-card-video
(or (plist-get talk :video-id)
(concat (plist-get talk :slug) "-mainVideo"))
video-file talk extensions))))
;; Add extra information to the talk
(setq talk
(append
talk
(list
:time-info (emacsconf-surround "Duration: " (plist-get talk :video-duration) " minutes""")
:video-html (or (plist-get video :video) "")
:audio-html (or (plist-get video :audio) "")
:chapter-list (or (plist-get video :chapter-list) "")
:resources (or (plist-get video :resources) "")
:extra (or (plist-get talk :extra) "")
:speaker-info (or (plist-get talk :speakers-with-pronouns) ""))))
(emacsconf-replace-plist-in-string
talk
"<div class=\"vid\">${video-html}${audio-html}<div>${extra}</div>${time-info}${resources}${chapter-list}</div>")))
So it's all built on templates and a little bit of code to make the
appropriate lists. You can find this code in
emacsconf-publish.el.
announce it on IRC so that the talk details split up the chat log and make it easier to read
publish the media files to https://media.emacsconf.org/2023 (currently password-protected while testing) so that people can view them
update the talk wiki page with the media files and the captions
update the Youtube and Toobnix videos so that they're public: this
is manual for the moment, since I haven't put time into automating
it yet
I have code for most of this, and we've done it successfully for the
past couple of years. I just need to refamiliarize myself with how to
do it and how to set it up for testing during the dry run, and modify
it to work with the new crontab-based system.
Last year, I set up the server with the ability to act as the
controller, so that it wasn't limited to my laptop. The organizers
notebook says I put it in the orga@ user's account, and I probably
ran it under a screen session. I've submitted a talk for this year's
conference, so I can use a test video for that one. First, I need to
update the Ansible configuration for publishing and editing:
I needed to add a version attribute to the git repo checkout in the
Ansible playbook, since we'd switched from master to main. I also
needed to set emacs_version to 29.1 since I started using
seq-keep in my Emacs Lisp functions. For testing, I set
emacsconf-publishing-phase to conference.
Act on TODO state changes
org-after-todo-state-change-hook makes it easy to automatically run
functions when the TODO state changes. I add this hook that runs a
list of functions and passes the talk information so that I don't have
to parse the talk info in each function.
emacsconf-org-after-todo-state-change: Run all the hooks in ‘emacsconf-todo-hooks’.
(defunemacsconf-org-after-todo-state-change ()
"Run all the hooks in `emacsconf-todo-hooks'.If an `emacsconf-todo-hooks' entry is a list, run it only for thetracks with the ID in the cdr of that list."
(let* ((talk (emacsconf-get-talk-info-for-subtree))
(track (emacsconf-get-track (plist-get talk :track))))
(mapc
(lambda (hook-entry)
(cond
((symbolp hook-entry) (funcall hook-entry talk))
((member (plist-get track :id) (cdr hook-entry))
(funcall (car hook-entry) talk))))
emacsconf-todo-hooks)))
This can be enabled and disabled with the following functions.
emacsconf-add-org-after-todo-state-change-hook: Add FUNC to ‘org-after-todo-stage-change-hook’.
We used to scramble to upload all the videos in the days or weeks
after the conference, since the presentations were live. Since we
switched to encouraging speakers to upload videos before the
conference, we've been able to release the videos pretty much as soon
as the talk starts playing. This code in
emacsconf-publish.el takes care of copying the files from
the backstage to the public media directory, republishing the index,
and republishing the playlist. That way, people who come in late or
who want to refer to the video can easily get the full video right
away.
emacsconf-publish-media-files-on-change: Publish the files and update the index.
This is low-priority, but it might be nice to figure out. The easiest
way is probably to use open the Youtube/Toobnix URLs on my computer
and then use either Tampermonkey or Spookfox to set the talk to
public. Someday!
Update the talk status on the server
Last year, I experimented with having the shell scripts automatically
update the status of the talk from TO_STREAM to PLAYING and from
PLAYING to CLOSED_Q. Since I've moved the talk-running into
track-specific crontabs, now I need to sudo back to the orga user
and set XDG_RUNTIME_DIR in order to use emacsclient. I can call
this with sudo -u orga talk $slug $status in the
roles/obs/templates/handle-session script.
#!/bin/bash# # How to use: talk slug from-status-regexp to-status# or talk slug to-statusSLUG="$1"FROM_STATUS="$2"TO_STATUS="$3"XDG_RUNTIME_DIR=/run/user/
if [ "x$TO_STATUS" == "x" ]; thenFROM_STATUS=.
TO_STATUS="$2"ficd#echo "Pulling conf.org..."#git pullecho"Updating status..."XDG_RUNTIME_DIR=$XDG_RUNTIME_DIR emacsclient --eval "(emacsconf-with-todo-hooks (emacsconf-update-talk-status \"$SLUG\" \"$FROM_STATUS\" \"$TO_STATUS\"))" -a emacs
#echo "Committing and pushing in the background"#git commit -m "Update task status for $SLUG from $FROM_STATUS to $TO_STATUS" conf.org#git push &
Testing notes
Looks like everything works fine when I run it from the crontab: the
talk status is updated, the media files are published, the wiki is
updated, and the talks are announced on IRC. Backup plan A is to
manually control the talk status using Emacs on the server. Backup
plan B is to control the talk status using Emacs on my laptop. Backup
plan C is to call the individual functions instead of relying on the
todo state change functions. I think it'll all work out, although I'll
probably want to do another dry run at some point to make sure. Slowly
getting there…
[2023-10-26 Thu]: updated handle-session and added talk
I figured out multi-track streaming so close to EmacsConf 2022 that
there wasn't enough time to get other volunteers used to working with
the setup, especially since I was still scrambling to figure out more
infrastructure as the conference approached. We decided I'd run both
streams myself, which meant I needed to make things as automatic as
possible so that I wouldn't go crazy. I wanted a lot of things to
happen automatically: playing recorded intros and videos, browsing to
the right URLs depending on the type of Q&A, publishing updates to the
wiki, and so on.
I used timers and TODO state changes to execute commands via TRAMP,
which was pretty cool for the most part. But it turned out TRAMP
doesn't like being called when it's already running, like when it's
being called from two timers going off at the same time. It gives a
"Forbidden reentrant call of TRAMP". We found a couple of quick
workarounds: I could reschedule the talks to be a minute apart, or I
could cancel the conflicting timer and just start them with the shell
scripts.
Last year, we had a shell script that played the intro and the main
talk, and other scripts to handle the Q&A by opening BigBlueButton,
Etherpad, or the IRC channel. Much of the logic was in Emacs Lisp
because it was easy to write it that way. For this year, I wanted to
write a script that handled the intro, video, and Q&A portions. This
is now in roles/obs/templates/handle-session.
handle-session
#!/bin/bash# ## Handle the intro/talk/Q&A for a session# Usage: handle-session $SLUGYEAR=""BASE_DIR=""FIREFOX_NAME=firefox-esr
SLUG=$1# Kill background music if playingif screen -list | grep -q background; then
screen -S background -X quit
fi# Update the status
sudo -u talk $SLUG PLAYING &
# Update the overlay
overlay $SLUG# Play the intro if it exists. If it doesn't exist, switch to the intro slide and stop processing.if [[ -f $BASE_DIR/assets/intros/$SLUG.webm ]]; then
killall -s TERM $FIREFOX_NAME
mpv $BASE_DIR/assets/intros/$SLUG.webm
else
firefox --kiosk $BASE_DIR/assets/in-between/$SLUG.png
exit 0
fi# Play the video if it exists. If it doesn't exist, switch to the BBB room and stop processing.if [ "x$TEST_MODE" = "x" ]; thenLIST=($BASE_DIR/assets/stream/--$SLUG*--main.webm)
elseLIST=($BASE_DIR/assets/test/--$SLUG*--main.webm)
fiFILE="${LIST[0]}"if [ ! -f "$FILE" ]; then# Is there an original file?LIST=($BASE_DIR/assets/stream/--$SLUG*--original.{webm,mp4,mov})
FILE="${LIST[0]}"fiif [[ -f $FILE ]]; then
killall -s TERM $FIREFOX_NAME
mpv $FILEelse
/usr/local/bin/bbb $SLUGexit 0
fi
sudo -u talk $SLUG CLOSED_Q &
# Open the appropriate Q&A URLQA=$(jq -r '.talks[] | select(.slug=="'$SLUG'")["qa-backstage-url"]' < $BASE_DIR/talks.json)
QA_TYPE=$(jq -r '.talks[] | select(.slug=="'$SLUG'")["qa-type"]' < $BASE_DIR/talks.json)
echo"QA_TYPE $QA_TYPE QA $QA"if [ "$QA_TYPE" = "live" ]; then
/usr/local/bin/bbb $SLUGelif [ "$QA" != "null" ]; then
/usr/local/bin/music &
/usr/bin/firefox $QA# i3-msg 'layout splith'fiwait
Now that we have a script that handles all the different things
related to a session, it's easier to schedule the execution of that
script. Instead of using Emacs timers and running into that problem
with tramp, I want to try using cron. Cron is a standard UNIX and
Linux tool for scheduling things to run at certain times. You make a
plain text file in a particular format: minute, hour, day of month,
month, day of week, and then the command, and then you tell cron to
use that file with something like crontab your-file. Since it's
plain text, we can generate it with Emacs Lisp and
format-time-string, save with TRAMP, and install with ssh. Each
track has its own user account for streaming, so each track can have
its own file.
emacsconf-stream-format-crontab: Return crontab entries for TALKS.
(defunemacsconf-stream-format-crontab (track talks &optional test-mode)
"Return crontab entries for TALKS.Use the display specified in TRACK.If TEST-MODE is non-nil, load the videos from the test directory."
(concat
(format
"PATH=/usr/local/bin:/usr/binMAILTO=\"\"XDG_RUNTIME_DIR=\"/run/user/%d\"" (plist-get track :uid))
(mapconcat
(lambda (talk)
(format "%s /usr/bin/screen -dmS play-%s bash -c \"DISPLAY=%s TEST_MODE=%s /usr/local/bin/handle-session %s | tee -a ~/track.log\"\n";; cron times are UTC
(format-time-string "%-M %-H %-d %m *" (plist-get talk :start-time))
(plist-get talk :slug)
(plist-get track :vnc-display)
(if test-mode "1""")
(plist-get talk :slug)))
(emacsconf-filter-talks talks))))
emacsconf-stream-crontabs: Write the streaming users’ crontab files.
(defunemacsconf-stream-crontabs (&optional test-mode info)
"Write the streaming users' crontab files.If TEST-MODE is non-nil, use the videos in the test directory.If INFO is non-nil, use that as the schedule instead."
(interactive)
(let ((emacsconf-publishing-phase 'conference))
(setq info (or info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))
(dolist (track emacsconf-tracks)
(let ((talks (seq-filter (lambda (talk)
(string= (plist-get talk :track)
(plist-get track :name)))
info))
(crontab (expand-file-name (concat (plist-get track :id) ".crontab")
(concat (plist-get track :tramp) "~"))))
(with-temp-file crontab
(when (plist-get track :autopilot)
(insert (emacsconf-stream-format-crontab track talks test-mode))))
(emacsconf-stream-track-ssh track (concat "crontab ~/" (plist-get track :id) ".crontab"))))))
I want to test the whole setup before the conference, of course.
First, I needed test videos. This generates test videos and subtitles
following our naming convention.
Then I needed to write a crontab based on a different schedule. This
code sets up a series of test videos to start about a minute after I
run the code, with the dev stream set up to start a minute after the
gen stream.
I needed to use timedatectl set-timezone America/Toronto to change
the server's timezone to America/Toronto so that the crontab would
run at the right time.
I also needed to specify the PATH so that I didn't need to add the
absolute paths in all the other shell scripts, XDG_RUNTIME_DIR to
get audio working, and DISPLAY so that windows showed up in the
right place.
I think this will let me run both tracks for EmacsConf with more ease
and less frantic juggling. We'll see!
