This is handled by the following code:
(defun my-whisper-maybe-expand-snippet (text)
"Add to `whisper-insert-text-at-point'."
(if (and text
(string-match
"^ok\\(?:ay\\)?[,\\.]? \\(.+\\)" text))
(let* ((name
(downcase
(string-trim
(replace-regexp-in-string "[,\\.]" "" (match-string 1 text)))))
(matching
(seq-find (lambda (o)
(subed-word-data-compare-normalized-string-distance
name
(downcase (yas--template-name o))))
(yas--all-templates (yas--get-snippet-tables)))))
(if matching
(progn
(if (frame-focus-state)
(progn
(yas-expand-snippet matching)
nil)
;; In another application
(with-temp-buffer
(yas-minor-mode)
(yas-expand-snippet matching)
(buffer-string))))
text))
text))
This code relies on my fork of whisper.el, which lets me specify a list of functions for whisper-insert-text-at-point. (I haven't asked for upstream review yet because I'm still testing things, and I don't know if it actually works for anyone else yet.) It does approximate matching on the snippet name using a function from subed-word-data.el which just uses string-distance. I could probably duplicate the function in my config, but then I'd have to update it in two places if I come up with more ideas.
The code for inserting into other functions is defined in my-whisper-maybe-type, which is very simple:
(defun my-whisper-maybe-type (text)
"If Emacs is not the focused app, simulate typing TEXT.
Add this function to `whisper-insert-text-at-point'."
(when text
(if (frame-focus-state)
text
(make-process :name "xdotool" :command
(list "xdotool" "type"
text))
nil)))
Someday I'd like to provide alternative names for snippets. I also want to make it easy to fill in snippet fields by voice. I'd love to be able to answer minibuffer questions from yas-choose-value, yas-completing-read, and other functions by voice too. Could be fun!
Related:
You can e-mail me at sacha@sachachua.com.
]]>: I added the recognized text so that I can confirm what was translated. I also moved my-type-with-hint to learn-lang-type-with-hint.
When I'm writing a journal entry in French, I sometimes want to translate a phrase that I can't look up word by word using a dictionary. Instead of switching to a browser, I can use an Emacs function to prompt me for text and either insert or display the translation. The plz library makes HTTP requests slightly neater.
(defun my-lang-en-to-fr (text &optional display-only)
(interactive (list (read-string "Text: ") current-prefix-arg))
(let* ((url "https://translation.googleapis.com/language/translate/v2")
(params `(("key" . ,(getenv "GOOGLE_API_KEY"))
("q" . ,text)
("source" . "en")
("target" . "fr")
("format" . "text")))
(query-string (mapconcat
(lambda (pair)
(format "%s=%s"
(url-hexify-string (car pair))
(url-hexify-string (cdr pair))))
params
"&"))
(full-url (concat url "?" query-string)))
(let* ((response (plz 'get full-url :as #'json-read))
(data (alist-get 'data response))
(translations (alist-get 'translations data))
(first-translation (car translations))
(translated-text (alist-get 'translatedText first-translation)))
(when (called-interactively-p 'any)
(if display-only
(message "%s" translated-text)
(insert translated-text)))
translated-text)))
I think it would be even nicer if I could use speech synthesis, so I can keep it a little more separate from my typing thoughts. I want to be able to say "Okay, translate …" or "Okay, … in French" to get a translation. I've been using my fork of natrys/whisper.el for speech recognition in English, and I like it a lot. By adding a function to whisper-after-transcription-hook, I can modify the intermediate results before they're inserted into the buffer.
(defun my-whisper-translate ()
(goto-char (point-min))
(let ((case-fold-search t))
(when (re-search-forward "okay[,\\.]? translate[,\\.]? \\(.+\\)\\|okay[,\\.]? \\(.+?\\) in French" nil t)
(let* ((s (or (match-string 1) (match-string 2)))
(translation (save-match-data (my-lang-en-to-fr s))))
(replace-match
(propertize translation
'type-hint translation
'type-original s
'help-echo s))))))
(with-eval-after-load 'whisper
(add-hook 'whisper-after-transcription-hook 'my-whisper-translate 70))
But that's too easy. I want to actually type things myself so that I get more practice. Something like an autocomplete suggestion would be handy as a way of showing me a hint at the cursor. The usual completion-at-point functions are too eager to insert things if there's only one candidate, so we'll just fake it with an overlay. This code works only with my whisper.el fork because it supports using a list of functions for whisper-insert-text-at-point.
(defun my-whisper-maybe-type-with-hints (text)
"Add this function to `whisper-insert-text-at-point'."
(let* ((hint (and text (org-find-text-property-in-string 'type-hint text)))
(original (and text (org-find-text-property-in-string 'type-original text))))
(if hint
(progn
(learn-lang-type-with-hint hint original)
nil)
text)))
Here's a demonstration of me saying "Okay, this is a test, in French.":
Since we're faking in-buffer completion here, maybe we can still get away with considering this as an entry for Emacs Carnival February 2026: Completion ? =)
You can e-mail me at sacha@sachachua.com.
]]>: Moved code to sachac/speech-input.
I can queue multiple transcriptions with whisper.el so that they get processed sequentially with backup audio. It catches up when I pause to think. Now I want to use Silero voice activity detection to do that kind of segmentation for me automatically.
First, I need a Python server that can print out events when it notices the start or stop of a speech segment. If I print out the timestamps, I might be able to cross-reference it someday with interestingthings. For now, even just paying attention to the end of a segment is enough for what I want to do.
import sounddevice as sd
import numpy as np
import torch
import sys
from datetime import datetime, timedelta
SILENCE_DURATION = 500
SAMPLING_RATE = 16000
CHUNK_SIZE = 512
model, utils = torch.hub.load(repo_or_dir='snakers4/silero-vad',
model='silero_vad',
force_reload=False)
(get_speech_timestamps, save_audio, read_audio, VADIterator, collect_chunks) = utils
vad_iterator = VADIterator(model, threshold=0.5, min_silence_duration_ms=SILENCE_DURATION)
stream_start_time = None
def format_iso_with_offset(offset_seconds):
if stream_start_time is None:
return "PENDING"
event_time = stream_start_time + timedelta(seconds=offset_seconds)
return event_time.astimezone().isoformat(timespec='milliseconds')
def audio_callback(indata, frames, time, status):
global stream_start_time
if status:
print(status, file=sys.stderr)
if stream_start_time is None:
stream_start_time = datetime.now()
tensor_input = torch.from_numpy(indata.copy()).flatten()
speech_dict = vad_iterator(tensor_input, return_seconds=True)
if speech_dict:
if "start" in speech_dict:
print(f"START {format_iso_with_offset(speech_dict['start'])}", flush=True)
if "end" in speech_dict:
print(f"END {format_iso_with_offset(speech_dict['end'])}", flush=True)
try:
with sd.InputStream(samplerate=SAMPLING_RATE,
channels=1,
callback=audio_callback,
blocksize=CHUNK_SIZE):
while True:
pass
except KeyboardInterrupt:
print("\nStopping...")
I've mode the code to speech-input-vad.el, so now I can use speech-input-vad-start to start monitoring speech events.
(use-package speech-input :load-path "~/proj/speech-input/")
Because I added Pulse properties to the process environment, I can easily use epwgraph to rewire the input so that it gets the input from my VirtualMicSink instead of the default system audio device. (Someday I'll figure out how to specify that as the input automatically.)
Then I can start this process from Emacs:
(with-eval-after-load 'speech-input-vad
(add-hook 'speech-input-vad-on-end-functions #'my-whisper-maybe-continue))
(defun my-whisper-maybe-continue ()
(when (process-live-p whisper--recording-process)
(my-whisper-continue)))
Now I can press my shortcut for my-whisper-continue to start the process. As I keep talking, it will continue to record. When I pause for more than a second between sentences, then it will send that chunk to the server for transcription without me having to press another button, while still listening for more speech.
How is this different from the streaming approach that many real-time speech recognition services offer? I think this gives me a bit more visibility into and control of the process. For my personal use, I don't need to have everything processed as quickly as possible, and I'm not trying to replicate live captions. I just want to be able to look back over the last five minutes to try to remember what I was talking about. I usually have a lot of quiet time as I think through my next steps, and it's fine to have it catch up then. I also like that I can save time-stamped audio files for later processing, divided according to the speech segments. Those might be a little bit easier to work with when I get around to compositing them into a video.
You can e-mail me at sacha@sachachua.com.
]]>my-whisper-continue) to stop the previous recording, queue it for processing, and start the next recording. If I use this keyboard shortcut to separate my thoughts, then Whisper has a much easier time making sense of the whole sentence or paragraph or whatever, instead of trying to use the sliding 30 second context window that many streaming approaches to speech recognition try to use.Question: Did you fix the keyboard delay you've got while speech catches what you're saying?
Sometimes, when the speed recognition kicks in, my computer gets busy. When my computer gets really busy, it doesn't process my keystrokes in the right order, which is very annoying because then I have to delete the previous word and retype it. I haven't sorted that out yet, but it seems like I probably have to lower the priority on different processes. On the plus side, as I mentioned, if I dictate things instead of typing them, then I don't run into that problem at all.
Also, other notes on delays: The continuous speech recognition via Google Chrome shows up fairly quickly, but it's not very precise, and it doesn't have punctuation. Even if there's a little bit of a delay, as long as I press the my-whisper-continue shortcut after each thought, then I can get that text into my Emacs buffer using the nicer transcription from my selected model. There is going to be a bit of a delay for that one because it gets processed at the end of the thought. Also, I need to start thinking in complete sentences instead of just adding one cause after the other as my brain goes on all of these tangents. I think it's pretty promising. There's the continuous speech recognition via Google Chrome if I don't mind the lower accuracy and lack of punctuation, and I can still get the pretty version on the other side.
Why talk out loud? I liked the Bookclub Tapas presentation that Maddie Sullivan did at EmacsConf 2025. Talking out loud helps me be a lot more verbose about what I'm saying, compared to typing things out or even like having to switch to my notes or interrupting my screen with an Org capture buffer. Of course I want to clean that up for putting into a blog post, but given that my life still sometimes has random interruptions from a kiddo who must have my attention at that very minute, having that kind of record that I can at least try to reread afterwards to reconstruct what I was thinking about sounds like it might be helpful.
Still, making sense out loud is hard. I'm not actually used to talking to people that much now. This is probably a good reason for me to experiment with streaming more. Then I get the practice in talking out loud, there are backup recordings, and people can ask questions when things are unclear.
Of course, sometimes the text doesn't quite make sense because of the speech recognition errors. I can usually figure it out from the context. I save the audio as well so that I can go back and listen to it again if I really need to.
Anyway, here's the code for sending the current recording to whisper in the background and starting another recording. It assumes a lot about how things are set up. For example, I'm only testing this with a local speaches server instead of whisper.cpp. You might need to look at my other speech related configuration blog posts and sections in order to make sense of it.
(defvar my-whisper--queue nil)
(defun my-whisper-continue (&optional arg)
"Send what we've got so far for transcription and then continue recording.
Call with \\[universal-argument] to signal that we can stop."
(interactive "P")
(require 'whisper)
(if arg
(my-whisper-done)
(setq whisper--marker (point-marker) whisper--point-buffer (current-buffer))
(when (process-live-p whisper--recording-process)
;; queue only if the last one is not asking for the same file
(unless
(string=
(plist-get
(car
(last my-whisper--queue))
:file)
whisper--temp-file)
(add-to-list
'my-whisper--queue
(list :file whisper--temp-file
:buffer
(format "*result: %s*" (file-name-base whisper--temp-file)))
t))
;; Remove the sentinel; handle results ourselves
(set-process-sentinel whisper--recording-process
(lambda (process event)
(my-whisper-process-queue)))
(interrupt-process whisper--recording-process))
(run-hooks 'whisper-before-transcription-hook)
(whisper--setup-mode-line :show 'recording)
(whisper--record-audio)))
(defun my-whisper-discard ()
"Ignore the previous recording."
(interactive)
(when (process-live-p whisper--recording-process)
;; Remove the sentinel; handle results ourselves
(set-process-sentinel whisper--recording-process
(lambda (process event)
(when (file-exists-p whisper--temp-file)
(delete-file whisper--temp-file))
(my-whisper-process-queue)))
(interrupt-process whisper--recording-process)))
(defun my-whisper-discard-and-continue ()
"Ignore the previous recording and continue."
(interactive)
(if (process-live-p whisper--recording-process)
(progn
;; Remove the sentinel; handle results ourselves
(set-process-sentinel whisper--recording-process
(lambda (process event)
(my-whisper-process-queue)
(my-whisper-continue)))
(interrupt-process whisper--recording-process))
(my-whisper-continue)))
(defun my-whisper-done ()
(interactive)
(when (process-live-p whisper--recording-process)
(add-to-list
'my-whisper--queue
(list :file whisper--temp-file
:buffer
(format "*result: %s*" (file-name-base whisper--temp-file)))
t)
;; Remove the sentinel; handle results ourselves
(set-process-sentinel whisper--recording-process
(lambda (process event)
(my-whisper-process-queue)))
(whisper--setup-mode-line :hide 'recording)
(interrupt-process whisper--recording-process)))
(defun my-whisper-process-queue-result ()
"Process the first part of the queue that already has results."
(while (plist-get (car my-whisper--queue) :results)
(let ((o (pop my-whisper--queue)))
(unless my-whisper-target-markers
(setq whisper--marker (point-marker)
whisper--point-buffer (current-buffer)))
(with-current-buffer (plist-get o :buffer)
(erase-buffer)
(insert (plist-get o :results)))
;; Only works with my fork: https://github.com/sachac/whisper.el/tree/whisper-insert-text-at-point-function
(whisper--handle-transcription-output nil (plist-get o :buffer)))))
(defun my-whisper-process-queue ()
(let (o)
(while (setq o (seq-find (lambda (o) (and (plist-get o :file)
(not (plist-get o :process))
(not (plist-get o :results))))
my-whisper--queue))
(let* ((headers (list "Content-Type: multipart/form-data"))
(params (list (concat "file=@"
(plist-get o :file))
"temperature=0.0"
"temperature_inc=0.2"
"response_format=json"
(concat "model=" whisper-model)
(concat "language=" whisper-language)))
(url (format my-whisper-url-format whisper-server-host whisper-server-port))
(command `("curl" "-s"
,url
,@(mapcan (lambda (h) (list "-H" h)) headers)
,@(mapcan (lambda (p) (list "-F" p)) params))))
(with-current-buffer (get-buffer-create (plist-get o :buffer))
(erase-buffer))
(plist-put
o :process
(make-process
:name "whisper-curl"
:command command
:buffer (plist-get o :buffer)
:coding 'utf-8
:sentinel
(lambda (process event)
(with-current-buffer (process-buffer process)
(let ((current my-whisper--queue-item))
(when (and (get-buffer (plist-get current :buffer))
(string-equal "finished\n" event))
(with-current-buffer (plist-get current :buffer)
(goto-char (point-min))
(plist-put current :results
(or
(condition-case nil
(gethash "text" (json-parse-buffer))
(error ""))
"(error)"))))))
(my-whisper-process-queue-result))))
(plist-put o :command (string-join command " "))
(with-current-buffer (process-buffer (plist-get o :process))
(setq-local my-whisper--queue-item o))))))
(defvar-local my-whisper--queue-item nil)
(defun my-whisper-reprocess-queue ()
(interactive)
(setq whisper--marker (point-marker) whisper--point-buffer (current-buffer))
(mapc (lambda (o)
(when (process-live-p (plist-get o :process))
(kill-process (plist-get o :process)))
(when (get-buffer (plist-get o :buffer))
(kill-buffer (plist-get o :buffer)))
(plist-put o :process nil)
(plist-put o :results nil))
my-whisper--queue)
(my-whisper-process-queue))
(defun my-whisper-clear-queue ()
(interactive)
(mapc (lambda (o)
(when (process-live-p (plist-get o :process))
(kill-process (plist-get o :process)))
(when (get-buffer (plist-get o :buffer))
(kill-buffer (plist-get o :buffer)))
(plist-put o :process nil)
(plist-put o :results nil))
my-whisper--queue)
(setq my-whisper--queue nil))
(keymap-global-set "<kp-9>" #'my-whisper-continue)
(keymap-global-set "<kp-8>" #'my-whisper-discard-and-continue)
(keymap-global-set "C-<kp-9>" #'my-whisper-done)
You can comment on Mastodon or e-mail me at sacha@sachachua.com.
]]>In order to try natrys/whisper.el with other backends, I needed to work around how whisper.el validates the model names and sends requests to the servers. Here's the quick and dirty code for doing so, in case you want to try it out for yourself.
(defvar my-whisper-url-format "http://%s:%d/transcribe")
;;;###autoload
(defun my-whisper--transcribe-via-local-server ()
"Transcribe audio using the local whisper server."
(message "[-] Transcribing via local server")
(whisper--setup-mode-line :show 'transcribing)
(whisper--ensure-server)
(setq whisper--transcribing-process
(whisper--process-curl-request
(format my-whisper-url-format whisper-server-host whisper-server-port)
(list "Content-Type: multipart/form-data")
(list (concat "file=@" whisper--temp-file)
"temperature=0.0"
"temperature_inc=0.2"
"response_format=json"
(concat "model=" whisper-model)
(concat "language=" whisper-language)))))
;;;###autoload
(defun my-whisper--check-model-consistency () t)
I moved my server code to speech-input-speaches.el.
(with-eval-after-load 'whisper
(advice-add 'whisper--transcribe-via-local-server :override #'my-whisper--transcribe-via-local-server)
(advice-add 'whisper--check-model-consistency :override #'my-whisper--check-model-consistency)
(advice-add 'whisper--ensure-server :override #'speech-input-speaches-server-start)
)
Then I have this function for trying things out.
(defun my-test-whisper-api (url &optional args)
(with-temp-buffer
(apply #'call-process "curl" nil t nil "-s"
url
(append (mapcan
(lambda (h) (list "-H" h))
(list "Content-Type: multipart/form-data"))
(mapcan
(lambda (h) (list "-F" h))
(list (concat "file=@" whisper--temp-file)
"temperature=0.0"
"temperature_inc=0.2"
"response_format=verbose_json"
(concat "language=" whisper-language)))
args))
(message "%s %s" (buffer-string) url)))
Here's the audio file. It is around 10 seconds long. I run the benchmark 3 times and report the average time.
Download
(let ((times '3))
(mapcar
(lambda (group)
(let ((whisper--temp-file "/home/sacha/recordings/whisper/2026-01-19-14-17-53.wav"))
;; warm up the model
(eval (cadr group))
(list
(format "%.3f"
(/ (car
(benchmark-call (lambda () (eval (cadr group))) times))
times))
(car group))))
'(
("parakeet"
(my-test-whisper-api
(format "http://%s:%d/v1/audio/transcriptions" whisper-server-host 5092)))
("whisper.cpp base-q4_0"
(my-test-whisper-api
(format "http://%s:%d/inference" whisper-server-host 8642)))
("speaches whisper-base"
(my-test-whisper-api
(format "http://%s:%d/v1/audio/transcriptions" whisper-server-host 8001)
(list "-F" "model=Systran/faster-whisper-base")))
("speaches whisper-base.en"
(my-test-whisper-api
(format "http://%s:%d/v1/audio/transcriptions" whisper-server-host 8001)
(list "-F" "model=Systran/faster-whisper-base.en")))
("speaches whisper-small"
(my-test-whisper-api
(format "http://%s:%d/v1/audio/transcriptions" whisper-server-host 8001)
(list "-F" "model=Systran/faster-whisper-small")))
("speaches whisper-small.en"
(my-test-whisper-api
(format "http://%s:%d/v1/audio/transcriptions" whisper-server-host 8001)
(list "-F" "model=Systran/faster-whisper-small.en")))
("speaches lorneluo/whisper-small-ct2-int8"
(my-test-whisper-api
(format "http://%s:%d/v1/audio/transcriptions" whisper-server-host 8001)
(list "-F" "model=lorneluo/whisper-small-ct2-int8")))
;; needed export TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1
("whisperx-server Systran/faster-whisper-small"
(my-test-whisper-api
(format "http://%s:%d/transcribe" whisper-server-host 8002)))))
)
| 3.694 | parakeet |
| 2.484 | whisper.cpp base-q4_0 |
| 1.547 | speaches whisper-base |
| 1.425 | speaches whisper-base.en |
| 4.076 | speaches whisper-small |
| 3.735 | speaches whisper-small.en |
| 2.870 | speaches lorneluo/whisper-small-ct2-int8 |
| 4.537 | whisperx-server Systran/faster-whisper-small |
All the results above were CPU-only, no GPU acceleration.
I tried it with:
Update: After a lot of rebooting, I figured out how to get the Nvidia P1000 GPU on my Lenovo P52 to work for up to the Whisper medium models. Large models don't fit in the 4GB of VRAM it has. Here are the benchmarking results:
(let ((times '3))
(mapcar
(lambda (group)
(let ((whisper--temp-file "/tmp/2026-01-19-14-17-53.wav"))
;; warm up the model
(eval (cadr group))
(list
(format "%.3f"
(/ (car
(benchmark-call (lambda () (eval (cadr group))) times))
times))
(car group))
))
'(
("speaches whisper-tiny"
(my-test-whisper-api
(format "http://%s:%d/v1/audio/transcriptions" "localhost" 8000)
(list "-F" "model=Systran/faster-whisper-tiny.en")))
("speaches whisper-base"
(my-test-whisper-api
(format "http://%s:%d/v1/audio/transcriptions" "localhost" 8000)
(list "-F" "model=Systran/faster-whisper-base.en")))
("speaches whisper-small"
(my-test-whisper-api
(format "http://%s:%d/v1/audio/transcriptions" "localhost" 8000)
(list "-F" "model=Systran/faster-whisper-small.en")))
("speaches whisper-medium"
(my-test-whisper-api
(format "http://%s:%d/v1/audio/transcriptions" "localhost" 8000)
(list "-F" "model=Systran/faster-whisper-medium.en"))))))
| 0.596 | speaches whisper-tiny |
| 0.940 | speaches whisper-base |
| 2.909 | speaches whisper-small |
| 8.740 | speaches whisper-medium |
That probably means I can upgrade to using the small model on a regular basis. For large models, I can use the command-line tool, since the overhead of loading the model is probably small compared to the processing time. I could also run two Speaches servers (one on CPU and one on GPU), but I don't think I'll need that level yet.
I like how speaches lets me switch models on the fly, so maybe I can use small.en generally and switch to a different model when I want to try dictating in French. Here's how I've set it up to use the server.
(setq whisper-server-port 8000
whisper-model "Systran/faster-whisper-small.en"
my-whisper-url-format "http://%s:%d/v1/audio/transcriptions")
Benchmark notes: I have a Lenovo P52 laptop (released 2018) with an Intel Core i7-8850H (6 cores, 12 threads; 2.6 GHz base / 4.3 GHz turbo) with 64GB RAM and an SSD. It has a soldered-on Nvidia P1000 with 4GB of VRAM.
You can comment on Mastodon or e-mail me at sacha@sachachua.com.
]]>my-whisper-run, use seq-reduce to go through the functions.I want to get my thoughts into the computer quickly, and talking might be a good way to do some of that. OpenAI Whisper is reasonably good at recognizing my speech now and whisper.el gives me a convenient way to call whisper.cpp from Emacs with a single keybinding. (Note: This is not the same whisper package as the one on MELPA.) Here is how I have it set up for reasonable performance on my Lenovo P52 with just the CPU, no GPU.
I've bound <f9> to the command whisper-run. I press <f9> to start recording, talk, and then press <f9> to stop recording. By default, it inserts the text into the buffer at the current point. I've set whisper-return-cursor-to-start to nil so that I can keep going.
(use-package whisper
:vc (:url "https://github.com/natrys/whisper.el")
:load-path "~/vendor/whisper.el"
:config
(setq whisper--mode-line-recording-indicator "⏺")
(setq whisper-quantize "q4_0")
(setq whisper-install-directory "~/vendor")
(setq whisper--install-path (concat
(expand-file-name (file-name-as-directory whisper-install-directory))
"whisper.cpp/"))
;; Get it running with whisper-server-mode set to nil first before you switch to 'local.
;; If you change models,
;; (whisper-install-whispercpp (whisper--check-install-and-run nil "whisper-start"))
(setq whisper-server-mode 'local)
(setq whisper-model "base")
(setq whisper-return-cursor-to-start nil)
;(setq whisper--ffmpeg-input-device "alsa_input.usb-Blue_Microphones_Yeti_Stereo_Microphone_REV8-00.analog-stereo")
(setq whisper--ffmpeg-input-device "VirtualMicSink.monitor")
(setq whisper-language "en")
(setq whisper-recording-timeout 3000)
(setq whisper-before-transcription-hook nil)
(setq whisper-use-threads (1- (num-processors)))
(setq whisper-transcription-buffer-name-function 'whisper--simple-transcription-buffer-name)
(add-hook 'whisper-after-transcription-hook 'my-subed-fix-common-errors-from-start -100)
:bind
(("<f9>" . whisper-run)
("C-<f9>" . my-whisper-run)
("S-<f9>" . my-whisper-replay)
("M-<f9>" . my-whisper-toggle-language)))
Let's see if we can process "Computer remind me to…":
(defvar my-whisper-org-reminder-template "t")
(defun my-whisper-org-process-reminder ()
(let ((text (buffer-string))
reminder)
(when (string-match "computer[,\.]? reminds? me to \\(.+\\)" text)
(setq reminder (match-string 1 text))
(save-window-excursion
(with-current-buffer (if (markerp whisper--marker) (marker-buffer whisper--marker) (current-buffer))
(when (markerp whisper--marker) (goto-char whisper--marker))
(org-capture nil my-whisper-org-reminder-template)
(insert reminder)
(org-capture-finalize)))
(erase-buffer))))
(with-eval-after-load 'whisper
(add-hook 'whisper-after-transcription-hook 'my-whisper-org-process-reminder 50))
Disk space is inexpensive and backups are great, so let's save each file using the timestamp.
(defvar my-whisper-dir "~/recordings/whisper/")
(defun my-whisper-set-temp-filename ()
(setq whisper--temp-file (expand-file-name
(format-time-string "%Y-%m-%d-%H-%M-%S.wav")
my-whisper-dir)))
(with-eval-after-load 'whisper
(add-hook 'whisper-before-transcription-hook #'my-whisper-set-temp-filename))
The technology isn't quite there yet to do real-time audio transcription so that I can see what it understands while I'm saying things, but that might be distracting anyway. If I do it in short segments, it might still be okay. I can replay the most recently recorded snippet in case it's missed something and I've forgotten what I just said.
(defun my-whisper-replay (&optional file)
"Replay the last temporary recording."
(interactive (list
(when current-prefix-arg
(read-file-name "File: " my-whisper-dir))))
(setq whisper--temp-file (or file whisper--temp-file))
(mpv-play whisper--temp-file))
(defun my-whisper-insert-retry (&optional file)
(interactive (list
(when current-prefix-arg
(read-file-name "File: " my-whisper-dir))))
(whisper--cleanup-transcription)
(setq whisper--marker (point-marker)
whisper--temp-file (or file whisper--temp-file))
(whisper--transcribe-audio))
Il peut aussi comprendre le français.
(defun my-whisper-toggle-language ()
"Set the language explicitly, since sometimes auto doesn't figure out the right one."
(interactive)
(setq whisper-language (if (string= whisper-language "en") "fr" "en"))
;; If using a server, we need to restart for the language
(when (process-live-p whisper--server-process) (kill-process whisper--server-process))
(message "%s" whisper-language))
I could use this with org-capture, but that's a lot of keystrokes. My shortcut for org-capture is C-c r. I need to press at least one key to set the template, <f9> to start recording, <f9> to stop recording, and C-c C-c to save it. I want to be able to capture notes to my currently clocked in task without having an Org capture buffer interrupt my display.
To clock in, I can use C-c C-x i or my ! speed command. Bonus: the modeline displays the current task to keep me on track, and I can use org-clock-goto (which I've bound to C-c j) to jump to it.
Then, when I'm looking at something else and I want to record a note, I can press <f9> to start the recording, and then C-<f9> to save it to my currently clocked task along with a link to whatever I'm looking at. (Update: Ooh, now I can save a screenshot too.)
(defun my-whisper-reset (text)
(setq my-whisper-skip-annotation nil)
(remove-hook 'whisper-insert-text-at-point #'my-whisper-org-save-to-clocked-task)
text)
;; Only works with my tweaks to whisper.el
;; https://github.com/sachac/whisper.el/tree/whisper-insert-text-at-point-function
(with-eval-after-load 'whisper
(setq whisper-insert-text-at-point
'(my-whisper-handle-commands
my-whisper-save-text
my-whisper-save-to-file
my-whisper-maybe-expand-snippet
my-whisper-maybe-type
my-whisper-maybe-type-with-hints
my-whisper-insert
my-whisper-reset)))
(defvar my-whisper-last-annotation nil "Last annotation so we can skip duplicates.")
(defvar my-whisper-skip-annotation nil)
(defvar my-whisper-target-markers nil "List of markers to send text to.")
(defun my-whisper-insert (text)
(let ((markers
(cond
((null my-whisper-target-markers)
(list whisper--marker)) ; current point where whisper was started
((listp my-whisper-target-markers)
my-whisper-target-markers)
((markerp my-whisper-target-markers)
(list my-whisper-target-markers))))
(orig-point (point))
(orig-buffer (current-buffer)))
(when text
(mapcar (lambda (marker)
(with-current-buffer (marker-buffer marker)
(save-restriction
(widen)
(when (markerp marker) (goto-char marker))
(when (and (derived-mode-p 'org-mode) (org-at-drawer-p))
(insert "\n"))
(whisper--insert-text
(concat
(if (looking-back "[ \t\n]\\|^")
""
" ")
(string-trim text)))
;; Move the marker forward here
(move-marker marker (point)))))
markers)
(when my-whisper-target-markers
(goto-char orig-point))
nil)))
(defun my-whisper-maybe-type (text)
(when text
(if (frame-focus-state)
text
(make-process :name "xdotool" :command
(list "xdotool" "type"
text))
nil)))
(defun my-whisper-clear-markers ()
(interactive)
(setq my-whisper-target-markers nil))
(defun my-whisper-use-current-point (&optional add)
(interactive (list current-prefix-arg))
(if add
(push (point-marker) my-whisper-target-markers)
(setq my-whisper-target-markers (list (point-marker)))))
(defun my-whisper-run-at-point (&optional add)
(interactive (list current-prefix-arg))
(my-whisper-clear-markers)
(whisper-run))
(keymap-global-set "<f9>" #'my-whisper-run-at-point)
(keymap-global-set "<kp-1>" #'whisper-run)
(defun my-whisper-jump-to-marker ()
(interactive)
(with-current-buffer (marker-buffer (car my-whisper-target-markers))
(goto-char (car my-whisper-target-markers))))
(defun my-whisper-use-currently-clocked-task (&optional add)
(interactive (list current-prefix-arg))
(save-window-excursion
(save-restriction
(save-excursion
(org-clock-goto)
(org-end-of-meta-data)
(org-end-of-subtree)
(if add
(push (point-marker) my-whisper-target-markers)
(setq my-whisper-target-markers (list (point-marker))))))))
(defun my-whisper-run (&optional skip-annotation)
(interactive (list current-prefix-arg))
(require 'whisper)
(add-hook 'whisper-insert-text-at-point #'my-whisper-org-save-to-clocked-task -10)
(whisper-run)
(when skip-annotation
(setq my-whisper-skip-annotation t)))
(defun my-whisper-save-text (text)
"Save TEXT beside `whisper--temp-file'."
(when text
(let ((link (org-store-link nil)))
(with-temp-file (concat (file-name-sans-extension whisper--temp-file) ".txt")
(when link
(insert link "\n"))
(insert text)))
text))
(defun my-whisper-org-save-to-clocked-task (text)
(when text
(save-window-excursion
(with-current-buffer (if (markerp whisper--marker) (marker-buffer whisper--marker) (current-buffer))
(when (markerp whisper--marker) (goto-char whisper--marker))
;; Take a screenshot maybe
(let* ((link (and (not my-whisper-skip-annotation)
(org-store-link nil)))
(region (and (region-active-p) (buffer-substring (region-beginning) (region-end))))
(screenshot-filename
(when (or
(null link)
(not (string= my-whisper-last-annotation link))
(not (frame-focus-state))) ; not in focus, take a screenshot
(my-screenshot-current-screen (concat (file-name-sans-extension whisper--temp-file) ".png")))))
(if (org-clocking-p)
(save-window-excursion
(save-restriction
(save-excursion
(org-clock-goto)
(org-end-of-subtree)
(unless (bolp)
(insert "\n"))
(insert "\n")
(if (and link (not (string= my-whisper-last-annotation link)))
(insert
(if screenshot-filename
(concat "(" (org-link-make-string
(concat "file:" screenshot-filename)
"screenshot") ") ")
"")
link
"\n")
(when screenshot-filename
(insert (org-link-make-string
(concat "file:" screenshot-filename)
"screenshot")
"\n")))
(when region
(insert "#+begin_example\n" region "\n#+end_example\n"))
(insert text "\n")
(setq my-whisper-last-annotation link)))
(run-at-time 0.5 nil (lambda (text) (message "Added clock note: %s" text)) text))
;; No clocked task, prompt for a place to capture it
(kill-new text)
(setq org-capture-initial text)
(call-interactively 'org-capture)
;; Delay the window configuration
(let ((config (current-window-configuration)))
(run-at-time 0.5 nil
(lambda (text config)
(set-window-configuration config)
(message "Copied: %s" text))
text config))))))))
(with-eval-after-load 'org
(add-hook 'org-clock-in-hook #'my-whisper-org-clear-saved-annotation))
(defun my-whisper-org-clear-saved-annotation ()
(setq my-whisper-org-last-annotation nil))
Here's an idea for a function that saves the recognized text with a timestamp.
(defvar my-whisper-notes "~/sync/stream/narration.org")
(defun my-whisper-save-to-file (text)
(when text
(let ((link (org-store-link nil)))
(with-current-buffer (find-file-noselect my-whisper-notes)
(goto-char (point-max))
(insert "\n\n" (format-time-string "%H:%M ") text "\n" (if link (concat link "\n") ""))
(save-buffer)
(run-at-time 0.5 nil (lambda (text) (message "Saved to file: %s" text)) text)))
text))
And now I can redo things if needed:
(defun my-whisper-redo ()
(interactive)
(setq whisper--marker (point-marker))
(whisper--transcribe-audio))
I think I've just figured out my Pipewire setup so that I can record audio in OBS while also being able to do speech to text, without the audio stuttering. qpwgraph was super helpful for visualizing the Pipewire connections and fixing them.
systemctl --user restart pipewire
sleep 2
pactl load-module module-null-sink \
sink_name="VirtualMicSink" sink_properties=device.description=VirtualMicSink
pactl load-module module-null-sink \
sink_name="CombinedSink" sink_properties=device.description=CombinedSink
if pactl list short sources | grep -i pci-0000; then
pactl load-module module-loopback \
source="alsa_input.pci-0000_00_1f.3.analog-stereo" \
sink="VirtualMicSink" \
latency_msec=100 \
adjust_time=1 \
source_output_properties="node.description='SysToVMic' node.name='SysToVMic' media.name='SysVToMic'" \
sink_input_properties="node.description='SysToVMic' node.name='SysToVMic' media.role='filter'"
sink_input_properties=media.role=filter
pactl load-module module-loopback \ source="alsa_output.pci-0000_00_1f.3.analog-stereo.monitor" \
sink="CombinedSink" \
node_name="SystemOutToCombined" \
source_output_properties="node.description='SysOutToCombined' node.name='SysOutToCombined'" \
sink_input_properties="node.description='SysOutToCombined' node.name='SysOutToCombined' media.role='filter'" \
latency_msec=100 adjust_time=1
fi
if pactl list short sources | grep -i yeti; then
pactl load-module module-loopback \
source="alsa_input.usb-Blue_Microphones_Yeti_Stereo_Microphone_REV8-00.analog-stereo" \
sink="VirtualMicSink" \
latency_msec=100 \
adjust_time=1 \
source_output_properties="node.description='YetiToVMic' node.name='YetiToVMic' media.name='YetiToVMic'" \
sink_input_properties="node.description='YetiToVMic' node.name='YetiToVMic' media.role='filter'"
pactl load-module module-loopback \ source="alsa_output.usb-Blue_Microphones_Yeti_Stereo_Microphone_REV8-00.analog-stereo.monitor" \
sink="CombinedSink" \
source_output_properties="node.description='YetiOutToCombined' node.name='YetiOutToCombined' media.name='YetiOutToCombined' " \
sink_input_properties="node.description='YetiOutToCombined' node.name='YetiOutToCombined' media.role='filter'" \
latency_msec=100 adjust_time=1
fi
pactl load-module module-loopback \
source="VirtualMicSink.monitor" \
sink="CombinedSink" \
source_output_properties="node.description='VMicToCombined' node.name='VMicToCombined' media.name='VMicToCombined'" \
sink_input_properties="node.description='VMicToCombined' node.name='VMicToCombined' media.role='filter'" \
latency_msec=100 adjust_time=1
pactl load-module module-null-sink \
sink_name="ExtraSink1" sink_properties=device.description=ExtraSink1
pactl load-module module-loopback \
source="ExtraSink1.monitor" \
sink="CombinedSink" \
source_output_properties="node.description='ExtraSink1ToCombined' node.name='ExtraSink1ToCombined' media.name='ExtraSink1ToCombined'" \
sink_input_properties="node.description='ExtraSink1ToCombined' node.name='ExtraSink1ToCombined' media.role='filter'" \
latency_msec=100 adjust_time=1
Here's a demo:
And then I define a global shortcut in KDE that runs:
/home/sacha/bin/xdotool-emacs key --clearmodifiers F9
So now I can dictate into other applications or save into Emacs. Which suggests of course that I should get it working with C-f9 as well, if I can avoid the keyboard shortcut loop…
You can comment on Mastodon or e-mail me at sacha@sachachua.com.
]]>#+PAD_LEFT and #+PAD_RIGHT
I like the constraints of a one-minute video, so I added a subed-record-sum-time command. That way, when I edit the video using Emacs, I can check how long the result will be. First, I split the subtitles, align it with the audio to fix the timestamps, and double check the times. Then I can skip my oopses. Sometimes WhisperX doesn't catch them, so I also look at waveforms and characters per second. I already talk quickly, so I'm not going to speed that up but I can trim the pauses in between phrases which is easy to do with waveforms. Sometimes, after reviewing a draft, I realize I need a little more time. If the original audio has some silence, I can just copy and paste it. If not, I can pad left or pad right to add some silence. I can try the flow of some sections and compile the video when I'm ready. Emacs can do almost anything. Yay Emacs!
You can watch this on YouTube, download the video, or download the audio.
Play by play:
subed-record-sum-time command. That way, when I edit the video using Emacs, I can check how long the result will be.
subed-record-sum-time calculates
the resulting duration and displays it in the
minibuffer.subed-align calls
the aeneas forced alignment tool to match up
the text with the timestamps. I use
subed-waveform-show-all to show all the
waveforms.NOTE #+SKIP comment before a
subtitle makes subed-record-compile-video
and subed-record-compile-flow skip that part
of the audio.subed-waveform-show-all adds waveforms for
all the subtitles. If I notice there's a pause
or a repeated shape in the waveform, or if I
listen and notice the repetition, I can
confirm by middle-clicking on the waveform to
sample part of it.[[file:...]]
link in the comment for a subtitle. That
visual will be used until the next visual is
specified in a comment on a different
subtitle. subed-record-compile-video can
automatically speed up video clips to fit in
the time for the current audio segment, which
is the set of subtitles before the next visual
is defined. After I compile and review the
video, sometimes I notice that something goes by too quickly.#+PAD_RIGHT: 1.5 in a comment
to add 1.5 seconds of silence after the audio
specified by that subtitle.M-x
subed-record-compile-try-flow to play the
audio or C-u M-x
subed-record-compile-try-flow to play the
audio+video for that region.subed-record-compile-video compiles the
video to the file specified in #+OUTPUT:
filename. ffmpeg is very arcane, so I'm glad
I can simplify my use of it with Emacs Lisp.M-x vundo to navigate my undo
history.Links:
Related:
You can e-mail me at sacha@sachachua.com.
]]>