Sacha Chua - category - subed

Comparing pronunciation recordings across time

Sacha Chua — Fri, 13 Mar 2026 01:11:45 GMT

[2026-03-27 Fri]: Added reconstructions from today and moved more of the code to sachac/learn-lang.
[2026-03-24 Tue]: Added updates from today.
[2026-03-15 Sun]: Added reference audio for the second set.
[2026-03-14 Sat]: I added pronunciation segments for the new set of tongue-twisters I got on Mar 13.
[2026-03-12 Thu]: I added a column for Feb 20, the first session with the sentences. I also added keyboard shortcuts (1..n) for playing the audio of the row that the mouse is on.

2026-02-20: First set: Maman peint un grand lapin blanc, etc.

My French tutor gave me a list of sentences to help me practise pronunciation.

I can fuzzy-match these with the word timing JSON from WhisperX, like this.

Extract all approximately matching phrases

Improving subed-vtt parsing; using dedicated windows in Emacs; training my intuition

Sacha Chua — Tue, 28 Jan 2025 18:07:54 GMT

While putting together some notes on how to use subed.el with auto-generated YouTube captions, I decided to get subed-word-data working with the Youtube VTT format. The sample file I downloaded from one of my YouTube videos had a cue whose text started with a blank line, so I ended up redoing the way subed-vtt.el parsed cues. Twice, actually. The first time, I changed it to handle multiple blocks in cue text (separated by blank lines). Then I came across this test suite for WebVTT parsing and found out that the timing line for a cue doesn't have to be preceded by a blank line, so then I needed to change the VTT parsing again.

Fortunately, I inherited a large Buttercup test suite from subed.el's original author. I've been adding to it over the years. As I learned more about how I wanted subed.el to behave, I added more cases. Then I worked on shifting the code to behave the way I wanted it to. At first, I didn't quite understand what I wanted the code to do, but as I pinned down more test cases, I was able to figure it out.

This was the first time I used toggle-window-dedicated (C-x w d) extensively. This function keeps the same buffer displayed in the window instead of letting Emacs Lisp functions replace it. First, I set up a large window for my subed-vtt.el. I split the other side into a smaller window for my test-subed-vtt.el, another window for a temporary VTT file, and a window for output from whatever command I'm running. I set most of the windows to dedicated except for my temporary output window. I also enabled winner-mode just in case I messed things up so that I could restore with winner-undo. Dedicating the windows meant I didn't have to keep fussing around with my windows and buffers to get things back to where I wanted them to be. I did use C-x o (other-window) a lot, so maybe it'll be worth getting the hang of ace-window.

Figure 1: Dedicated windows for the code, the test file, and a VTT buffer for interactive testing; a spare window for other output

Sometimes I wanted to focus on one of those small windows. prot/window-single-toggle was helpful for maximizing a window and then returning to the previous configuration.

I mostly evaluated or edebugged my code and then used my-buttercup-run-dwim to run a suite in my test-subed-vtt.el. sp-backward-up-sexp and sp-narrow-to-sexp were also helpful for navigating the suites and focusing on whatever I was working on.

I'm looking forward to exploring the other test cases from that repository. It feels good to get better as a coder.

I just finished reading Mathematica: a Secret World of Imagination and Curiosity, by David Bessis. The idea of training your intuition echoed in my mind as I wrote test cases and changed the code. I started to look forward to the gaps between my understanding of the spec and my understanding of the test cases, and the gaps between the test cases and my implementation. It felt almost like a conversation. Sometimes it was hard to translate an idea into code. I felt myself getting muddled and turned around. Whenever I noticed that, I just had to back up and start from something I understood, figure out what I was uncertain about, and then go from there.

I like the way that subed.el gives me tools for thinking about text, times, and metadata together. That abstraction has been helpful for editing audio and making videos. The more solid I can make it, the easier it will be to imagine other things that use those ideas.

View org source for this post

You can e-mail me at sacha@sachachua.com.

Automatically correcting phrasing and misrecognized words in speech-to-text captions by using a script

Sacha Chua — Sat, 11 Jan 2025 14:11:40 GMT

I usually write my scripts with phrases that could be turned into the subtitles. I figured I might as well combine that information with the WhisperX transcripts which I use to cut out my false starts and oopses. To do that, I use the string-distance function, which calculates how similar strings are, based on the Levenshtein [distance] algorithm. If I take each line of the script and compare it with the list of words in the transcription, I can add one transcribed word at a time, until I find the number with the minimum distance from my current script phrase. This lets me approximately match strings despite misrecognized words. I use oopses to signal mistakes. When I detect those, I look for the previous script line that is closest to the words I restart with. I can then skip the previous lines automatically. When the script and the transcript are close, I can automatically correct the words. If not, I can use comments to easily compare them at that point. Even though I haven't optimized anything, it runs well enough for my short videos. With these subtitles as a base, I can get timestamps with subed-align and then there's just the matter of tweaking the times and adding the visuals.

2025-01-09-01 Matching a script with a transcript – subed emacs

Text from sketch

#YayEmacs 9: Trimming/adding silences to get to a target; subed-record-sum-time

Sacha Chua — Thu, 09 Jan 2025 15:24:29 GMT

New in this video: subed-record-sum-time, #+PAD_LEFT and #+PAD_RIGHT

Video not supported. Thumbnail:

Download the video

I like the constraints of a one-minute video, so I added a subed-record-sum-time command. That way, when I edit the video using Emacs, I can check how long the result will be. First, I split the subtitles, align it with the audio to fix the timestamps, and double check the times. Then I can skip my oopses. Sometimes WhisperX doesn't catch them, so I also look at waveforms and characters per second. I already talk quickly, so I'm not going to speed that up but I can trim the pauses in between phrases which is easy to do with waveforms. Sometimes, after reviewing a draft, I realize I need a little more time. If the original audio has some silence, I can just copy and paste it. If not, I can pad left or pad right to add some silence. I can try the flow of some sections and compile the video when I'm ready. Emacs can do almost anything. Yay Emacs!

You can watch this on YouTube, download the video, or download the audio.

Play by play:

I like the constraints of a one-minute video, so I added a subed-record-sum-time command. That way, when I edit the video using Emacs, I can check how long the result will be.
- subed-record uses subtitles and directives in comments in a VTT subtitle file to edit audio and video. subed-record-sum-time calculates the resulting duration and displays it in the minibuffer.
First, I split the subtitles, align it with the audio to fix the timestamps, and double check the times.
- I'm experimenting with an algorithmic way to combine the breaks from my script with the text from the transcript. subed-align calls the aeneas forced alignment tool to match up the text with the timestamps. I use subed-waveform-show-all to show all the waveforms.
Then I can skip my oopses.
- Adding a NOTE #+SKIP comment before a subtitle makes subed-record-compile-video and subed-record-compile-flow skip that part of the audio.
Sometimes WhisperX doesn't catch them,
- WhisperX sometimes doesn't transcribe my false starts if I repeat things quickly.
so I also look at waveforms
- subed-waveform-show-all adds waveforms for all the subtitles. If I notice there's a pause or a repeated shape in the waveform, or if I listen and notice the repetition, I can confirm by middle-clicking on the waveform to sample part of it.
and characters per second.
- Low characters per second is sometimes a sign that the timestamps are incorrect or there's a repetition that wasn't transcribed.
I already talk quickly, so I'm not going to speed that up
- Also, I already sound like a chipmunk; mechanically speeding up my recording to fit in a certain time will make that worse =)
but I can trim the pauses in between phrases which is easy to do with waveforms.
- left-click to set the start, right-click to set the stop. If I want to adjust the previous/next one at the same time, I would use shift-left-click or shift-right-click, but here I want to skip the gaps between phrases, so I adjust the current subtitle without making the previous/next one longer.
Sometimes, after reviewing a draft, I realize I need a little more time.
- I can specify visuals like a video, animated GIF, or an image by adding a [[file:...]] link in the comment for a subtitle. That visual will be used until the next visual is specified in a comment on a different subtitle. subed-record-compile-video can automatically speed up video clips to fit in the time for the current audio segment, which is the set of subtitles before the next visual is defined. After I compile and review the video, sometimes I notice that something goes by too quickly.
If the original audio has some silence, I can just copy and paste it.
- This can sometimes feel more natural than adding in complete silence.
If not, I can pad left or pad right to add some silence.
- I added a new feature so that I could specify something like #+PAD_RIGHT: 1.5 in a comment to add 1.5 seconds of silence after the audio specified by that subtitle.
I can try the flow of some sections
- I can select a region and then use M-x subed-record-compile-try-flow to play the audio or C-u M-x subed-record-compile-try-flow to play the audio+video for that region.
and compile the video when I'm ready.
- subed-record-compile-video compiles the video to the file specified in #+OUTPUT: filename. ffmpeg is very arcane, so I'm glad I can simplify my use of it with Emacs Lisp.
Emacs can do almost anything. Yay Emacs!
- Non-linear audio and video editing is actually pretty fun in a text editor, especially when I can just use M-x vundo to navigate my undo history.

Links:

View org source for this post

You can e-mail me at sacha@sachachua.com.

Editing videos with Emacs and subed-record.el

Sacha Chua — Wed, 01 Jan 2025 19:23:04 GMT

I want to document more of my Minecraft adventures with A+. Video is a natural way to do this. It builds on her familiarity with the tutorials and streams she enjoys watching. I set up OBS on her laptop and plugged in my Blue Yeti microphone. We did our first interview yesterday. I edited and subtitled it (because why not!), uploaded it as an unlisted YouTube video, and shared it with her dad, sister, and cousins.

I did the video editing in Emacs with subed-record. First, I used WhisperX to transcribe the video, and I used subed-align to fix the timestamps with aeneas. I normalized the audio with Audacity and I exported the .opus file for use in subed-record.el. Then I added NOTE #+SKIP before times I wanted to remove, like when she asked for a retake. Here's what that subtitle markup looks like:

WEBVTT

NOTE #+SKIP

00:00:00.000 --> 00:00:16.679
And then I'll record in my side also
and we'll just put it in somehow.
Somehow. Okay. We can edit that, right?
Yeah, we'll learn how to edit things.
It'll be great.

NOTE
Introduction
#+AUDIO: cuberventures-001.opus
[[file:intro.webm]]
#+OUTPUT: cuberventures-001-fxnt-create-2-windmill-home-cafe-trains-hotel-half-underwater.webm

00:00:16.680 --> 00:00:19.399
Okay, so now we're here with .

00:00:19.400 --> 00:00:23.039
I want to find out what you like about Minecraft and

00:00:23.040 --> 00:00:26.079
all the cool things that you have been building lately.

This was a little different from my usual video creation workflow, where I record the audio and the video separately. When I wrote subed-record.el, I assumed I'd edit the audio first, choose images/GIFs/videos that were already ready to go, and then combine those visuals with sections of audio, speeding things up or slowing things down as needed. Now I wanted to apply the same edits to the video as I did to the audio. A+ did a great job of looking at stuff in Minecraft while talking about them, so I wanted to keep her narration in sync. I added some code to allow me to specify a same-edits keyword for the visuals. That meant that I would use the same selection list that I used for cutting the audio. Here's what that subtitle markup looks like:

NOTE
[[file:2024-12-31 10-35-14.mkv]]
#+OPTIONS: same-edits

00:00:43.860 --> 00:00:45.941
Shall we take a tour of my world?

00:00:45.942 --> 00:00:50.079
Sure, let's tell people which mod pack this is.

00:00:50.080 --> 00:00:55.639
This is FXNT Create 2, also known as FoxyNoTail Create 2.

NOTE Windmill

00:00:55.640 --> 00:00:58.239
I've got this little bit of path leading to the interview

00:00:58.240 --> 00:01:01.839
room. This is my unfinished windmill. I've been meaning to

This workflow lets me cut out segments in the middle of the video, like this:

00:17:30.200 --> 00:17:33.119
great start for a tour. I'm looking forward to seeing what

00:17:33.120 --> 00:17:34.112
you will build next.

NOTE #+SKIP

00:17:34.113 --> 00:18:02.379
Do you have any last words before
we try to figure out this video editing thing?
Yeah. We'll cut that last part out.
Let's just do a retake on that last part.
Someday. Out here. Okay. There you go.
This is a beautiful view.

00:18:02.380 --> 00:18:08.119
The last things I want to say about this world is there'll be

I also wanted to start the video with a segment from my recording, so we could see her avatar on screen during the introduction. She kept her computer on first-person POV instead of changing the camera. I used mpv to figure out the timestamps for the start and end of the part that I wanted to use, then I used ffmpeg to cut that clip. I added a comment with a link to that video in order to use it before the main video. That's the [[file:intro.webm]] in the first section's comments.

After testing a small section of the transcript by selecting a region and using subed-record-compile-video, I deselected the region and used subed-record-compile-video to produce the whole video.

I also modified subed-record-compile-subtitles to include the other non-directive comments, so I can include the section headings in the raw VTT file and have them turn up in the exported version. Then I can use the new subed-section-comments-as-chapters command to copy those as chapters for the YouTube description.

We're not going to share that particular video yet, but I'm looking forward to trying that technique with videos about stuff I'm figuring out in Minecraft or Emacs. It's also tempting me to think about ways to specify transitions like crossfades and other fancy effects like overlays.

I like using the transcript as the starting point for video editing. It just makes sense to me to work with it as text. I also like this experiment with documenting more of our Minecraft experimentation. It seems to get her talking and encourages her to build more. I'm looking forward to learning more about Minecraft and making videos too.

We did another video today using the new shortcuts I've just set up for toggling OBS recording. This time we didn't even need to do any editing. I used Org Export to make her a little HTML file that had the two videos on it, so she can review it any time. Onward!

View org source for this post

You can e-mail me at sacha@sachachua.com.

subed.el: Tweaking subtitle times

Sacha Chua — Thu, 19 Dec 2024 14:51:17 GMT

When subtitle times are too far off from the video or audio, people start worrying if their video has frozen or jumped ahead. It's good to keep subtitles roughly in time with the audio.

For EmacsConf, we can get timing information from two places. WhisperX produces a JSON file with word data in the process of doing the speech recognition, and the aeneas forced alignment tool can use synthesized text-to-speech to figure out the timestamps for each line of text compared to a media file.

Aeneas timestamps are more helpful once we start editing, but it can be confused by long silences, extraneous noises, multiple speakers, and inaccurate transcripts (words added or removed).

When I combine the WhisperX word data with subtitles, I can see where the times might need a closer look because matching words weren't found.

Figure 1: Screenshot with word data loaded

Loading word data requires a pretty close match at the moment, but since we change only about 4% of the subtitle text when editing, those cues are still helpful. (I measured this by the Levenshtein distance between the combined cue texts of edited subtitles versus the original WhisperX transcripts, using string-distance to approximate the editing percentage.)

Calculating how much we edited

Checking caption timing by skimming with Emacs Lisp or JS

Sacha Chua — Sun, 17 Nov 2024 12:29:02 GMT

Sometimes automatic subtitle timing tools like Aeneas can get confused by silences, extraneous sounds, filler words, mis-starts, and text that I've edited out of the raw captions for easier readability. It's good to quickly check each caption. I used to listen to captions at 1.5x speed, watching carefully as each caption displayed. This took a fair bit of time and focus, so… it usually didn't happen. Sampling the first second of each caption is faster and requires a little less attention.

Skimming with subed.el

Here's a function that I wrote to play the first second of each subtitle.

(defvar my-subed-skim-msecs 1000 "Number of milliseconds to play when skimming.")
(defun my-subed-skim-starts ()
  (interactive)
  (subed-mpv-unpause)
  (subed-disable-loop-over-current-subtitle)
  (catch 'done
    (while (not (eobp))
      (subed-mpv-jump-to-current-subtitle)
      (let ((ch
             (read-char "(q)uit? " nil (/ my-subed-skim-msecs 1000.0))))
        (when ch
          (throw 'done t)))
      (subed-forward-subtitle-time-start)
      (when (and subed-waveform-minor-mode
                 (not subed-waveform-show-all))
        (subed-waveform-refresh))
      (recenter)))
  (subed-mpv-pause))

Now I can read the lines as the subtitles play, and I can press any key to stop so that I can fix timestamps.

Skimming with Javascript

I also want to check the times on the Web in case there have been caching issues. Here's some Javascript to skim the first second of each cue in the first text track for a video, with some code to make it easy to process the first video in the visible area.

function getVisibleVideo() {
  const videos = document.querySelectorAll('video');
  for (const video of videos) {
    const rect = video.getBoundingClientRect();
    if (
      rect.top >= 0 &&
      rect.left >= 0 &&
      rect.bottom <= (window.innerHeight || document.documentElement.clientHeight) &&
      rect.right <= (window.innerWidth || document.documentElement.clientWidth)
    ) {
      return video;
    }
  }
  return null;
}

async function skimVideo(video=getVisibleVideo(), msecs=1000) {
  // Get the first text track (assumed to be captions/subtitles)
  const textTrack = video.textTracks[0];
  if (!textTrack) return;
  const remaining = [...textTrack.cues].filter((cue) => cue.endTime >= video.currentTime);
  video.play();
  // Play the first 1 second of each visible subtitle
  for (let i = 0; i < remaining.length && !video.paused; i++) {
    video.currentTime = remaining[i].startTime;
    await new Promise((resolve) => setTimeout(resolve, msecs));
  }
}

Then I can call it with skimVideo();. Actually, in our backstage area, it might be useful to add a Skim button so that I can skim things from my phone.

function handleSkimButton(event) {
   const vid = event.target.closest('.vid').querySelector('video');
   skimVideo(vid);
 }

document.querySelectorAll('video').forEach((vid) => {
   const div = document.createElement('div');
   const skim = document.createElement('button');
   skim.textContent = 'Skim';
   div.appendChild(skim);
   vid.parentNode.insertBefore(div, vid.nextSibling);
   skim.addEventListener('click', handleSkimButton);
});

Results

How much faster is it this way?

Some code to help figure out the speedup