Sacha Chua - category - audio

subed.el and quoting snippets of YouTube videos from the auto captions

2026-10-26T02:11:17Z

I'm preparing for a conversation with linkarzu. It would be fun to quote snippets from some of the other videos he has. I like the way he does a lot of reaction videos that build on what people have shared. He tends to listen to things at 1.25 or 1.5x speed. I'm not sure if I have the patience even for that, so it'd be nice to be able to just quickly skim the text. He also clips a lot of his own videos. It might be interesting to play around with my workflows for listening to snippets and clipping them.

I've grabbed YouTube captions and reformatted them before:

This time around, I want to get the VTT format so that I can possibly use them with subed-record to clip audio or video.

From a YouTube link, I want to run a single command that gets me a VTT file with that as the media source and with the captions. If there are any chapters in the description, it would be lovely to get them too. I can get them from the JSON, yay!

Now I can use M-x sacha-subed-youtube to open the VTT file, which is cached locally, and I can use M-x sacha-subed-youtube-region-as-quote after selecting a section. Well, actually, I have an Embark command s (for "subed") that I can run on a YouTube video link and a selected-mode shortcut for y that I can run when the region is selected. Results:

I Asked Prot If I Should Switch to Emacs

00:09:46: I think the value of Org is that it is in the context of Emacs, which can now do other things. It can be your email client. It can be your agenda. Of course, it can be your programming environment. All these sorts of things, like your RSS feed reader and so on. And in the context of all that, you also have Org. So for me, that's kind of the value. But otherwise, having Org elsewhere, it's great still, because the paradigm of Org, I think, is a portable paradigm.

Org Mode in Neovim Is Finally Here - Full org.nvim Demo

00:00:00: I think that I achieved like 100% of parity with the original Org Mode version on Emacs.

Code for subed + YouTube

Using Emacs to visualize speaking and target words/phrases in real-ish time

2026-10-07T11:47:04Z

Figure 1: Screenshot showing my list of target phrases and which ones have been recognized

When I'm working with a tutor to improve my French, I sometimes need to push myself to provide more details when I'm sharing and ask more follow-up questions in response to what they're saying, instead of falling back to « D'accord » and « C'est cool. » Visualizing the timeline in real-time lets me see if the turn-taking looks good. I might even need to take a more active role in controlling the conversation if I happen to be meeting a tutor who tends to go on monologues.

When I'm talking out loud on my own for practice, this might also nudge me to reduce the pauses and enrich my vocabulary by practising saying things often enough that they become more automatic.

1. Voice activity detection and speech recognition

This code uses this server vad-file.py, which I generated with Google Gemini. It emits JSONL for voice activity detection events (start/stop) and uses Orukeet to transcribe the speech once the voice activity detection notices it's finished. I also use spacy to look up token information because I want to someday colour the nouns by gender, which might allow me to correct myself in the next sentence instead of hesitating in the previous one.

code for sacha-vad-tutor

Expanding yasnippets by voice in Emacs and other applications

2026-03-04T16:17:40Z

Yasnippet is a template system for Emacs. I want to use it by voice. I'd like to be able to say things like "Okay, define interactive function" and have that expand to a matching snippet in Emacs or other applications. Here's a quick demonstration of expanding simple snippets:

Download the video

Screencast of expanding snippets by voice in Emacs and in other applications

Transcript

Using speech recognition for on-the-fly translations in Emacs and faking in-buffer completion for the results

2026-03-31T16:31:15Z

[2026-03-31 Tue]: I added the recognized text so that I can confirm what was translated. I also moved my-type-with-hint to learn-lang-type-with-hint.

When I'm writing a journal entry in French, I sometimes want to translate a phrase that I can't look up word by word using a dictionary. Instead of switching to a browser, I can use an Emacs function to prompt me for text and either insert or display the translation. The plz library makes HTTP requests slightly neater.

(defun my-lang-en-to-fr (text &optional display-only)
  (interactive (list (read-string "Text: ") current-prefix-arg))
  (let* ((url "https://translation.googleapis.com/language/translate/v2")
         (params `(("key" . ,(getenv "GOOGLE_API_KEY"))
                   ("q" . ,text)
                   ("source" . "en")
                   ("target" . "fr")
                   ("format" . "text")))
         (query-string (mapconcat
                        (lambda (pair)
                          (format "%s=%s"
                                  (url-hexify-string (car pair))
                                  (url-hexify-string (cdr pair))))
                        params
                        "&"))
         (full-url (concat url "?" query-string)))
    (let* ((response (plz 'get full-url :as #'json-read))
           (data (alist-get 'data response))
           (translations (alist-get 'translations data))
           (first-translation (car translations))
           (translated-text (alist-get 'translatedText first-translation)))
      (when (called-interactively-p 'any)
        (if display-only
            (message "%s" translated-text)
          (insert translated-text)))
      translated-text)))

I think it would be even nicer if I could use speech synthesis, so I can keep it a little more separate from my typing thoughts. I want to be able to say "Okay, translate …" or "Okay, … in French" to get a translation. I've been using my fork of natrys/whisper.el for speech recognition in English, and I like it a lot. By adding a function to whisper-after-transcription-hook, I can modify the intermediate results before they're inserted into the buffer.

(defun my-whisper-translate ()
  (goto-char (point-min))
  (let ((case-fold-search t))
    (when (re-search-forward "okay[,\\.]? translate[,\\.]? \\(.+\\)\\|okay[,\\.]? \\(.+?\\) in French" nil t)
      (let* ((s (or (match-string 1) (match-string 2)))
             (translation (save-match-data (my-lang-en-to-fr s))))
        (replace-match
         (propertize translation
                     'type-hint translation
                     'type-original s
                     'help-echo s))))))

(with-eval-after-load 'whisper
  (add-hook 'whisper-after-transcription-hook 'my-whisper-translate 70))

But that's too easy. I want to actually type things myself so that I get more practice. Something like an autocomplete suggestion would be handy as a way of showing me a hint at the cursor. The usual completion-at-point functions are too eager to insert things if there's only one candidate, so we'll just fake it with an overlay. This code works only with my whisper.el fork because it supports using a list of functions for whisper-insert-text-at-point.

(defun my-whisper-maybe-type-with-hints (text)
  "Add this function to `whisper-insert-text-at-point'."
  (let* ((hint (and text (org-find-text-property-in-string 'type-hint text)))
         (original (and text (org-find-text-property-in-string 'type-original text))))
    (if hint
        (progn
          (learn-lang-type-with-hint hint original)
          nil)
      text)))

Here's a demonstration of me saying "Okay, this is a test, in French.":

Download the video

Screencast of using speech recognition to translate into French and provide a hint when typing

Since we're faking in-buffer completion here, maybe we can still get away with considering this as an entry for Emacs Carnival February 2026: Completion ? =)

This is part of my Emacs configuration.

View Org source for this post

You can e-mail me at sacha@sachachua.com.

Using Silero voice activity detection to automatically queue multiple transcriptions with natrys/whisper.el

2026-03-27T23:03:01Z

[2026-03-27 Fri]: Moved code to sachac/speech-input.

I can queue multiple transcriptions with whisper.el so that they get processed sequentially with backup audio. It catches up when I pause to think. Now I want to use Silero voice activity detection to do that kind of segmentation for me automatically.

First, I need a Python server that can print out events when it notices the start or stop of a speech segment. If I print out the timestamps, I might be able to cross-reference it someday with interestingthings. For now, even just paying attention to the end of a segment is enough for what I want to do.

Python script for printing out events

Queuing multiple transcriptions with whisper.el speech recognition

2026-01-23T19:50:37Z

I want to be able to talk out loud and have the ideas go into Emacs. I can do this in a number of different ways:

I briefly demonstrated a step-by-step approach with natrys/whisper.el with a single file. I press a keyboard shortcut to start the recording, another shortcut to stop the recording, and it transcribes it in the background. But the way whisper.el is set up is that if I press the keyboard shortcut to start recording again it will offer to interrupt the transcription process, which is not what I want. I want to just keep talking and have it process results as things come in.
I'm also experimenting with Google Chrome's web speech API to do continuous speech recognition, which I can get into Emacs using a web socket.
What I've just figured out is how to layer a semi-continuous interface for speech recognition on top of whisper.el so that while it's processing in the background, I can just press a keyboard shortcut (I'm using numpad 9 to call my-whisper-continue) to stop the previous recording, queue it for processing, and start the next recording. If I use this keyboard shortcut to separate my thoughts, then Whisper has a much easier time making sense of the whole sentence or paragraph or whatever, instead of trying to use the sliding 30 second context window that many streaming approaches to speech recognition try to use.

Question: Did you fix the keyboard delay you've got while speech catches what you're saying?

Sometimes, when the speed recognition kicks in, my computer gets busy. When my computer gets really busy, it doesn't process my keystrokes in the right order, which is very annoying because then I have to delete the previous word and retype it. I haven't sorted that out yet, but it seems like I probably have to lower the priority on different processes. On the plus side, as I mentioned, if I dictate things instead of typing them, then I don't run into that problem at all.

Also, other notes on delays: The continuous speech recognition via Google Chrome shows up fairly quickly, but it's not very precise, and it doesn't have punctuation. Even if there's a little bit of a delay, as long as I press the my-whisper-continue shortcut after each thought, then I can get that text into my Emacs buffer using the nicer transcription from my selected model. There is going to be a bit of a delay for that one because it gets processed at the end of the thought. Also, I need to start thinking in complete sentences instead of just adding one cause after the other as my brain goes on all of these tangents. I think it's pretty promising. There's the continuous speech recognition via Google Chrome if I don't mind the lower accuracy and lack of punctuation, and I can still get the pretty version on the other side.

Why talk out loud? I liked the Bookclub Tapas presentation that Maddie Sullivan did at EmacsConf 2025. Talking out loud helps me be a lot more verbose about what I'm saying, compared to typing things out or even like having to switch to my notes or interrupting my screen with an Org capture buffer. Of course I want to clean that up for putting into a blog post, but given that my life still sometimes has random interruptions from a kiddo who must have my attention at that very minute, having that kind of record that I can at least try to reread afterwards to reconstruct what I was thinking about sounds like it might be helpful.

Still, making sense out loud is hard. I'm not actually used to talking to people that much now. This is probably a good reason for me to experiment with streaming more. Then I get the practice in talking out loud, there are backup recordings, and people can ask questions when things are unclear.

Of course, sometimes the text doesn't quite make sense because of the speech recognition errors. I can usually figure it out from the context. I save the audio as well so that I can go back and listen to it again if I really need to.

Anyway, here's the code for sending the current recording to whisper in the background and starting another recording. It assumes a lot about how things are set up. For example, I'm only testing this with a local speaches server instead of whisper.cpp. You might need to look at my other speech related configuration blog posts and sections in order to make sense of it.

Code for queuing whisper.el requests to a local server

Emacs and whisper.el: Trying out different speech-to-text backends and models

2026-01-20T19:21:46Z

Update [2026-10-06 Tue]: I'm currently using Orukeet instead of Whisper/Speaches. Experimenting with Orukeet for speech recognition

I was curious about parakeet because I heard that it was faster than Whisper on the HuggingFace leaderboard. When I installed it and got it running on my laptop (CPU only, no GPU), it seemed like my results were a little faster than whisper.cpp with the large model, but much slower than whisper.cpp with the base model. The base model is decent for quick dictation, so I got curious about other backends and other models.

In order to try natrys/whisper.el with other backends, I needed to work around how whisper.el validates the model names and sends requests to the servers. Here's the quick and dirty code for doing so, in case you want to try it out for yourself.

(defvar sacha-whisper-url-format "http://%s:%d/transcribe")
;;;###autoload
(defun sacha-whisper--transcribe-via-local-server ()
  "Transcribe audio using the local whisper server."
  (message "[-] Transcribing via local server")
  (whisper--setup-mode-line :show 'transcribing)
  (whisper--ensure-server)
  (setq whisper--transcribing-process
        (whisper--process-curl-request
         (format sacha-whisper-url-format whisper-server-host whisper-server-port)
         (list "Content-Type: multipart/form-data")
         (list (concat "file=@" whisper--temp-file)
               "temperature=0.0"
               "temperature_inc=0.2"
               "response_format=json"
               (concat "model=" whisper-model)
               (concat "language=" whisper-language)))))
;;;###autoload
(defun sacha-whisper--check-model-consistency () t)

I moved my server code to speech-input-speaches.el.

(with-eval-after-load 'whisper
  (advice-add 'whisper--transcribe-via-local-server :override #'sacha-whisper--transcribe-via-local-server)
  (advice-add 'whisper--check-model-consistency :override #'sacha-whisper--check-model-consistency)
  ;;(advice-add 'whisper--ensure-server :override #'speech-input-speaches-server-start)
  )

Then I have this function for trying things out.

(defun sacha-test-whisper-api (url &optional args)
  (with-temp-buffer
    (apply #'call-process "curl" nil t nil "-s"
           url
         (append (mapcan
                  (lambda (h) (list "-H" h))
                  (list "Content-Type: multipart/form-data"))
                 (mapcan
                  (lambda (h) (list "-F" h))
                  (list (concat "file=@" whisper--temp-file)
                        "temperature=0.0"
                        "temperature_inc=0.2"
                        "response_format=verbose_json"
                        (concat "language=" whisper-language)))
                 args))
    (message "%s %s" (buffer-string) url)))

Here's the audio file. It is around 10 seconds long. I run the benchmark 3 times and report the average time.

Download the audio

Download

Code for running the benchmarks