Re-encoding the EmacsConf videos with FFmpeg and GNU Parallel
| geek, linux, emacsconf, ffmpeg, video
It turns out that using -crf 56 compressed the EmacsConf a little
too aggressively, losing too much information in the video. We wanted
to reencode everything, maybe going back to the default value of -crf
32. My laptop would have taken a long time to do all of those videos.
Fortunately, one of the other volunteers shared a VM on a machine with
12 cores, and I had access to a few other systems. It was a good
opportunity to learn how to use GNU Parallel to send jobs to different
machines and retrieve the results.
First, I updated the compression script, compress-video-low.sh:
Q=$1 WIDTH=1280 HEIGHT=720 AUDIO_RATE=48000 VIDEO_FILTER="scale=w=${WIDTH}:h=${HEIGHT}:force_original_aspect_ratio=1,pad=${WIDTH}:${HEIGHT}:(ow-iw)/2:(oh-ih)/2,fps=25,colorspace=all=bt709:iall=bt601-6-625:fast=1" FILE=$2 SUFFIX=$Q shift shift ffmpeg -y -i "$FILE" -pixel_format yuv420p -vf $VIDEO_FILTER -colorspace 1 -color_primaries 1 -color_trc 1 -c:v libvpx-vp9 -b:v 0 -crf $Q -aq-mode 2 -tile-columns 0 -tile-rows 0 -frame-parallel 0 -cpu-used 8 -auto-alt-ref 1 -lag-in-frames 25 -g 240 -pass 1 -f webm -an -threads 8 /dev/null && if [[ $FILE =~ "webm" ]]; then ffmpeg -y -i "$FILE" $* -pixel_format yuv420p -vf $VIDEO_FILTER -colorspace 1 -color_primaries 1 -color_trc 1 -c:v libvpx-vp9 -b:v 0 -crf $Q -tile-columns 2 -tile-rows 2 -frame-parallel 0 -cpu-used -5 -auto-alt-ref 1 -lag-in-frames 25 -pass 2 -g 240 -ac 2 -threads 8 -c:a copy "${FILE%.*}--compressed$SUFFIX.webm" else ffmpeg -y -i "$FILE" $* -pixel_format yuv420p -vf $VIDEO_FILTER -colorspace 1 -color_primaries 1 -color_trc 1 -c:v libvpx-vp9 -b:v 0 -crf $Q -tile-columns 2 -tile-rows 2 -frame-parallel 0 -cpu-used -5 -auto-alt-ref 1 -lag-in-frames 25 -pass 2 -g 240 -ac 2 -threads 8 -c:a libvorbis "${FILE%.*}--compressed$SUFFIX.webm" fi
I made an originals.txt file with all the original filenames. It looked like this:
emacsconf-2020-frownies--the-true-frownies-are-the-friends-we-made-along-the-way-an-anecdote-of-emacs-s-malleability--case-duckworth.mkv emacsconf-2021-montessori--emacs-and-montessori-philosophy--grant-shangreaux.webm emacsconf-2021-pattern--emacs-as-design-pattern-learning--greta-goetz.mp4 ...
I set up a ~/.parallel/emacsconf profile with something like this so
that I could use three computers and my laptop, sending one job each
and displaying progress:
--sshlogin computer1 --sshlogin computer2 --sshlogin computer3 --sshlogin : -j 1 --progress --verbose --joblog parallel.log
I already had SSH key-based authentication set up so that I could connect to the three remote computers.
Then I spread the jobs over four computers with the following command:
cat originals.txt | parallel -J emacsconf \ --transferfile {} \ --return '{=$_ =~ s/\..*?$/--compressed32.webm/=}' \ --cleanup \ --basefile compress-video-low.sh \ bash compress-video-low.sh 32 {}
It copied each file over to the computer it was assigned to, processed the file, and then copied the file back.
It was also helpful to occasionally do echo 'killall -9 ffmpeg' |
parallel -J emacsconf -j 1 --onall if I cancelled a run.
It still took a long time, but less than it would have if any one computer had to crunch through everything on its own.
This was much better than my previous way of doing things, which involved copying the files over, running ffmpeg commands, copying the files back, and getting somewhat confused about which directory I was in and which file I assigned where and what to do about incompletely-encoded files.
I sometimes ran into problems with incompletely-encoded files because
I'd cancelled the FFmpeg process. Even though ffprobe said the files
were long, they were missing a large chunk of video at the end. I
added a compile-media-verify-video-frames function to
compile-media.el so that I could get the last few seconds of frames,
compare them against the duration, and report an error if there was a
big gap.
Then I changed emacsconf-publish.el to use the new filenames, and I regenerated all the pages. For EmacsConf 2020, I used some Emacs Lisp to update the files. I'm not particularly fond of wrangling video files (lots of waiting, high chance of error), but I'm glad I got the computers to work together.
