Sacha Chua - category - audio https://sachachua.com/blog/category/audio/feed/index.xml Emacs, sketches, and life Wed, 07 Oct 2026 19:52:45 GMT en-US daily 1 11ty Using Emacs to visualize speaking and target words/phrases in real-ish time https://sachachua.com/blog/2026/10/using-emacs-to-visualize-speaking-and-target-words-phrases-in-real-ish-time/ Wed, 07 Oct 2026 11:47:04 GMT audio speech-recognition emacs learning visualization https://sachachua.com/blog/2026/10/using-emacs-to-visualize-speaking-and-target-words-phrases-in-real-ish-time/
2026-10-07_07-33-03.png
Figure 1: Screenshot showing my list of target phrases and which ones have been recognized

When I'm working with a tutor to improve my French, I sometimes need to push myself to provide more details when I'm sharing and ask more follow-up questions in response to what they're saying, instead of falling back to « D'accord » and « C'est cool. » Visualizing the timeline in real-time lets me see if the turn-taking looks good. I might even need to take a more active role in controlling the conversation if I happen to be meeting a tutor who tends to go on monologues.

When I'm talking out loud on my own for practice, this might also nudge me to reduce the pauses and enrich my vocabulary by practising saying things often enough that they become more automatic.

1. Voice activity detection and speech recognition

This code uses this server vad-file.py, which I generated with Google Gemini. It emits JSONL for voice activity detection events (start/stop) and uses Orukeet to transcribe the speech once the voice activity detection notices it's finished. I also use spacy to look up token information because I want to someday colour the nouns by gender, which might allow me to correct myself in the next sentence instead of hesitating in the previous one. 

;; Report on the balance between my tutor and me
(defvar sacha-vad-tutor-process nil)
(defvar sacha-vad-tutor-dir "~/vendor/orukeet")
(defvar sacha-vad-tutor-command `(,(expand-file-name ".venv/bin/python" sacha-vad-tutor-dir)
                                  "asr.py"
                                  "--spacy-model"
                                  "fr_dep_news_trf"
                                  "-s" "300" "-t"
                                  "VirtualMicSink.monitor=sacha" "alsa_output.usb-AT_AT2040USB_202011110001-00.analog-stereo.monitor=tutor"))
(defvar sacha-vad-tutor-events nil "List of events as an list of alists (label start stop duration).")
(defvar sacha-vad-tutor-labels '("sacha" "tutor"))
(defvar sacha-vad-tutor-summary '(("sacha" . 0)
                                  ("tutor" . 0)))
(defvar sacha-vad-tutor-session-length (* 60 60) "Number of seconds to use for scale.")

(defun sacha-vad-tutor-update-view ()
  "Visualize the last 3 minutes and the overview of the session.
Also keep track of target phrases."
  (interactive)
  (with-current-buffer (get-buffer-create "*tutor stats*")
    (erase-buffer)
    (let* ((total (apply '+ (mapcar 'cdr sacha-vad-tutor-summary)))
           (height 30)
           (width (window-width nil t))
           (max-seconds
            (if sacha-vad-tutor-events
                (apply 'max (mapcar (lambda (o) (alist-get 'end o))
                                    sacha-vad-tutor-events))
              0))
           (threshold 180)              ; last three minutes
           (threshold-seconds (- max-seconds threshold))
           (recent
            (seq-keep
             (lambda (o)
               (when (>= (alist-get 'start o)
                         threshold-seconds)
                 (let-alist o
                   `((label . ,.label)
                     (start . ,(- .start threshold-seconds))
                     (end . ,(- .end threshold-seconds))))))
             (copy-sequence sacha-vad-tutor-events)))
           (recent-total (apply '+ (mapcar (lambda (o)
                                             (-
                                              (alist-get 'end o)
                                              (alist-get 'start o)))
                                           recent))))
      (save-excursion
        (insert
         (format "Last %d minute%s:\n"
                 (/ threshold 60)
                 (if (> (/ threshold 60) 1)
                     "s"
                   ""))
         (mapconcat
          (lambda (o)
            (format "%d%% %s\n"
                    (/ (* 100.0 (cdr o)) recent-total)
                    (car o)))
          (seq-map
           (lambda (group)
             (cons (car group)
                   (apply '+ (mapcar (lambda (o)
                                       (-
                                        (alist-get 'end o)
                                        (alist-get 'start o)))
                                     (cdr group)))))
           (seq-group-by (lambda (o) (alist-get 'label o)) recent)))
         "\n")
        (svg-insert-image
         (sacha-vad-visualize
          sacha-vad-tutor-labels
          recent
          t
          width height))
        (insert
         "\nTotal:\n"

         (mapconcat
          (lambda (o)
            (format "%d%% %s\n"
                    (/ (* 100.0 (cdr o)) total)
                    (car o)))
          sacha-vad-tutor-summary))
        (svg-insert-image
         (sacha-vad-visualize sacha-vad-tutor-labels sacha-vad-tutor-events t
                              width height
                              sacha-vad-tutor-session-length))
        (insert "\n\n"  (sacha-learn-lang-bingo-summary))))))

(defun sacha-vad-handle-end-event (data)
  "Handle the end event."
  (let ((event (list
                (cons 'label (alist-get 'label data))
                (cons 'start (- (alist-get 'timestamp data)
                                (alist-get 'duration data)
                                (or sacha-vad-tutor-start-seconds 0)))
                (cons 'end (- (alist-get 'timestamp data)
                              (or sacha-vad-tutor-start-seconds 0))))))
    (push event sacha-vad-tutor-events)
    (let-alist data
      (incf (alist-get (alist-get 'label data) sacha-vad-tutor-summary 0 nil 'string=)
            .duration))
    (sacha-vad-tutor-update-view)))

(defun sacha-vad-tutor-filter (proc string)
  "Add and parse."
  (when (buffer-live-p (process-buffer proc))
    ;; split the string by newline?
    (with-current-buffer (process-buffer proc)
      (goto-char (point-max))
      (save-excursion
        (insert string))
      (catch 'waiting
        (unless (or (eobp) (looking-at "\n*$"))
          (let ((data (json-read)))
            (when (string= (alist-get 'event data) "end")
              (sacha-vad-handle-end-event data))
            (when (string= (alist-get 'event data) "final")
              (sacha-vad-handle-text-event data))))))))

(defvar sacha-vad-last-speaker nil "Previous speaker.")

(defun sacha-vad-handle-text-event (data)
  (with-current-buffer (get-buffer-create "*Transcript*")
    (set (make-local-variable 'window-point-insertion-type) t)
    (save-excursion
      (goto-char (point-max))
      ;; insert break
      (insert
       (if (string= sacha-vad-last-speaker (alist-get 'label data))
           " "
         (concat (if (bobp) "" "\n\n") (alist-get 'label data) ": "))
       (alist-get 'text data))
      (setq sacha-vad-last-speaker (alist-get 'label data))
      (when (string= (alist-get 'label data) "sacha")
        (sacha-learn-lang-notice-bingo-words (alist-get 'text data))))))

(defun sacha-vad-tutor-start ()
  "Start the process."
  (interactive)
  (sacha-vad-tutor-ensure)
  (setq sacha-vad-tutor-summary '(("sacha" . 0)
                                  ("tutor" . 0))
        sacha-vad-tutor-events nil)
  (with-current-buffer (get-buffer-create "*Transcript*")
    (erase-buffer)
    (display-buffer (current-buffer)))
  (display-buffer (get-buffer-create "*tutor stats*")))

(defun sacha-vad-tutor-ensure ()
  "Start the process if it's not already running."
  (interactive)
  (unless (process-live-p sacha-vad-tutor-process)
    (setq sacha-vad-tutor-process-start-time (current-time))
    (setq sacha-vad-tutor-start-seconds 0)
    (let ((default-directory sacha-vad-tutor-dir))
      (setq sacha-vad-tutor-process
            (make-process
             :name "vad-tutor"
             :command sacha-vad-tutor-command
             :connection-type 'pipe
             :buffer (get-buffer-create "*vad-tutor*")
             :stderr (get-buffer-create "*vad-tutor-err*")
             :filter #'sacha-vad-tutor-filter)))))

(defun sacha-vad-tutor-stop ()
  "Stop the process."
  (interactive)
  (interrupt-process sacha-vad-tutor-process))

;;;###autoload
(defun sacha-js-parse-jsonl-file (file)
  "Parse jsonl as a list"
  (let (results)
    (with-temp-buffer
      (insert-file-contents file)
      (goto-char (point-min))
      (while (not (eobp))
        (condition-case nil
            (push (json-read) results)
          (error (goto-char (point-max))))))
    (nreverse results)))

(defvar sacha-vad-tutor-start-seconds 0 "Number of seconds to consider as the start.")
(defvar sacha-vad-tutor-process-start-time nil "Time that the process started.")

;;;###autoload
(defun sacha-vad-tutor-reset-clock ()
  "Reset the start of the session."
  (interactive)
  (setq sacha-vad-tutor-start-seconds
        (time-to-seconds (time-subtract (current-time)
                                        sacha-vad-tutor-process-start-time)))
  (mapcar (lambda (o) (setcdr o 0)) sacha-vad-tutor-summary)
  (setq sacha-vad-tutor-events nil))

(defun sacha-vad-visualize (labels list &optional output-file width height max-seconds)
  (setq height (or height 25))
  (setq width (or width (window-width nil t)))
  (let* ((svg (svg-create width height))
         (max-seconds (max
                       (or max-seconds 0)
                       (if list
                           (apply 'max (mapcar (lambda (o) (alist-get 'end o))
                                               list))
                         0)))
         (pixels-per-second (/ (* 1.0 width) max-seconds))
         (row-height (/ height (length labels))))
    (dolist (o list)
      (let-alist o
        (let ((y (* row-height (or (seq-position labels .label 'string=) -1))))
          (when (>= y 0)
            (svg-rectangle
             svg
             (* .start pixels-per-second)
             y
             (* (- .end .start) pixels-per-second)
             row-height :fill "green")))))
    (cond
     ((stringp output-file)
      (with-temp-file output-file
        (svg-print svg)))
     (output-file
      svg)
     (t
      (with-current-buffer (get-buffer-create "*vad*")
        (goto-char (point-min))
        (display-buffer (current-buffer))
        (erase-buffer)
        (svg-insert-image svg))))))

(defun sacha-vad-stats (labels list)
  "Return the balance of time talking as (label total percentage)."
  (let* ((max-seconds (apply '+ (mapcar
                                 (lambda (o) (let-alist o (- .end .start)))
                                 list)))
         (durations (mapcar (lambda (o)
                            (cons
                             (car o)
                             (mapcar (lambda (e) (let-alist e (- .end .start)))
                                     (cdr o))))
                          (seq-group-by (lambda (o) (alist-get 'label o))
                                        list))))
    (seq-map
     (lambda (label)
       (let* ((row (cdr (assoc label durations)))
              (total (apply '+ row)))
         `((label . ,label)
           (total . ,total)
           (count . ,(length row))
           (percentage . ,(and total max-seconds (> max-seconds 0) (* 100.0 (/ total max-seconds)))))))
     labels)))

;; (sacha-vad-stats '("sacha" "tutor") (seq-take (sacha-js-parse-jsonl-file "~/proj/speech/output2.log") 5))
;; (sacha-vad-stats '("sacha" "tutor") (sacha-js-parse-jsonl-file "~/proj/speech/output.log"))
;; (sacha-vad-visualize '("sacha" "tutor") (sacha-js-parse-jsonl-file "~/proj/speech/output.log"))
;; (sacha-vad-visualize '("sacha" "tutor") (seq-take (sacha-js-parse-jsonl-file "~/proj/speech/output2.log") 30))

(defun sacha-vad-tutor-group-output (files)
  (let ((grouped-files (seq-group-by
                        (lambda (o)
                          (substring (file-name-base o) 0 10))
                        files)))
    (mapcar
     (lambda (group)
       (let ((start 0))
         (cons (car (cdr group))
               (seq-reduce
                (lambda (prev file)
                  (append prev
                          (let ((max-seconds 0))
                            (prog1 (mapcar (lambda (event)
                                             (incf (alist-get 'start event) start)
                                             (incf (alist-get 'end event) start)
                                             (setq max-seconds (max (alist-get 'end event) max-seconds))
                                             event)
                                           (sacha-js-parse-jsonl-file file))
                              (setq start max-seconds)))))
                (cdr group)
                nil))))
     grouped-files)))

(defun sacha-vad-tutor-make-table (files &optional width height)
  "Make a summary table."
  (let ((labels sacha-vad-tutor-labels)
        (width (or width 500))
        (height (or height 30)))
    ;; combine data as needed
    (append
     '(("Date" "Tutor" "Minutes" "%" "Visualization") hline)
     (seq-map
      (lambda (group)
        (let* ((list (cdr group))
               (base (file-name-base (car group))))
          (if list
              (let-alist (car (sacha-vad-stats labels list))
                (let ((svg-name
                       (expand-file-name
                        (concat (file-name-base (car group)) "-" .label ".svg")
                        "~/proj/french/conversation")))
                  (sacha-vad-visualize
                   (list .label)
                   list
                   svg-name
                   width
                   height)
                  (list
                   (format-time-string "%Y-%m-%d"
                                       (seconds-to-time
                                        (sacha-filename-timestamp (car group))))
                   ;; (org-link-make-string (concat "file:" o) "file")
                   (if (string-match "-conversation-\\([^\\.]+\\)" base)
                       (s-capitalize (substring (match-string 1 base) 0 1))
                     "")

                   (round (/ .total 60))
                   (format "%d%%" .percentage)
                   (org-link-make-string (concat "file:" svg-name)
                                         (concat "file:" svg-name))
                   )))
            (list "?"))))
      (sacha-vad-tutor-group-output files))
     nil)))

2. Noticing bingo phrases

Here's the code that reminds me to use the words or expressions on my to-do list, and then keeps track of how often I've used them, prioritizes the ones that I haven't used yet, and displays these. It can work with either my Whisper dictation workflow or the VAD events server described above.

I keep the list of expressions in file:///home/sacha/proj/french/bingo.txt. 

(defvar sacha-learn-lang-bingo-word-file "~/proj/french/bingo.txt")
(defvar sacha-learn-lang-bingo-words nil "List of words.")
(defvar sacha-learn-lang-bingo-words-hash nil "Data for the counts.")

;;;###autoload
(defun sacha-learn-lang-bingo-words ()
  "Load the list of words from the file."
  (interactive)
  (setq sacha-learn-lang-bingo-words
        (seq-map (lambda (o)
                   (when (= (length o) 1)
                     (setq o (list (car o) 0)))
                   (setq o (append o (list
                                      (string-trim (replace-regexp-in-string
                                                    " *\\[.+?\\] *" " "
                                                    (car o))))))
                   o)
                 (pcsv-parse-file sacha-learn-lang-bingo-word-file)))
  sacha-learn-lang-bingo-words)

;;;###autoload
(defun sacha-learn-lang-bingo-handler (event)
  "Manually register the event."
  (interactive "e")
    (let* ((pos (posn-point (event-end event)))
           (val (get-text-property pos 'bingo)))
      (when val
        (sacha-learn-lang-notice-bingo-words val))))

(defvar-keymap sacha-learn-lang-bingo-map
  "<mouse-1>" #'sacha-learn-lang-bingo-handler)

;;;###autoload
(defun sacha-learn-lang-bingo-summary ()
  "Summarize the words to say."
  (let ((target (seq-filter (lambda (row) (= (elt row 1) 0))
                            sacha-learn-lang-bingo-words))
        (yay (seq-filter (lambda (row) (> (elt row 1) 0))
                         sacha-learn-lang-bingo-words)))
    (concat
     (if target
         (concat "Target: "
                 (mapconcat (lambda (row)
                              (propertize
                               (car row)
                               'keymap sacha-learn-lang-bingo-map
                               'bingo (car row)))
                            target
                            "; ")
                 "\n\n")
       "")
     (if yay
         (concat
          "Yay! "
          (mapconcat (lambda (row)
                       (propertize (format "%s (%s)" (car row) (elt row 1))
                                   'keymap sacha-learn-lang-bingo-map
                                   'bingo (car row)))
                     (seq-filter (lambda (row) (> (elt row 1) 0))
                                 sacha-learn-lang-bingo-words)
                     "; ")
          "\n\n")
       ""))))

;;;###autoload
(defun sacha-learn-lang-notice-bingo-words (text)
  "Look for the bingo words in TEXT and celebrate when I use them.
Add to `whisper-insert-text-at-point'."
  (when text
    (unless sacha-learn-lang-bingo-words (sacha-learn-lang-bingo-words))
    (let ((found (seq-find
                  (lambda (o)
                    (and (elt o 2)
                         (string-match (concat "\\<" (elt o 2)) text)))
                  sacha-learn-lang-bingo-words)))
      (when found
        (incf (elt found 1))
        (sacha-vad-tutor-update-view)
        (with-current-buffer (get-buffer-create "*Bingo*")
          (erase-buffer)
          (insert (sacha-learn-lang-bingo-summary))))))
  text)
This is part of my EmacsConfiguration.

You can e-mail me at sacha@sachachua.com.

]]> Expanding yasnippets by voice in Emacs and other applications https://sachachua.com/blog/2026/03/expanding-yasnippet-by-voice/ Wed, 04 Mar 2026 16:17:40 GMT emacs audio speech-recognition https://sachachua.com/blog/2026/03/expanding-yasnippet-by-voice/ Yasnippet is a template system for Emacs. I want to use it by voice. I'd like to be able to say things like "Okay, define interactive function" and have that expand to a matching snippet in Emacs or other applications. Here's a quick demonstration of expanding simple snippets:

Screencast of expanding snippets by voice in Emacs and in other applications

Transcript
  • 00:00 So I've defined some yasnippets with names that I can say. Here, for example, in this menu, you can see I've got "define interactive function" and "with a buffer that I'll display." And in fundamental mode, I have some other things too. Let's give it a try.
  • 00:19 I press my shortcut. "Okay, define an interactive function." You can see that this is a yasnippet. Tab navigation still works.
  • 00:33 I can say, "OK, with a buffer that I'll display," and it expands that also.
  • 00:45 I can expand snippets in other applications as well, thanks to a global keyboard shortcut.
  • 00:50 Here, for example, I can say, "OK, my email." It inserts my email address.
  • 01:02 Yasnippet definitions can also execute Emacs Lisp. So I can say, "OK, date today," and have that evaluated to the actual date.
  • 01:21 So that's an example of using voice to expand snippets.

This is handled by the following code: 

(defun my-whisper-maybe-expand-snippet (text)
  "Add to `whisper-insert-text-at-point'."
  (if (and text
           (string-match
            "^ok\\(?:ay\\)?[,\\.]? \\(.+\\)" text))
    (let* ((name
            (downcase
             (string-trim
              (replace-regexp-in-string "[,\\.]" "" (match-string 1 text)))))
           (matching
            (seq-find (lambda (o)
                        (subed-word-data-compare-normalized-string-distance
                         name
                         (downcase (yas--template-name o))))
                      (yas--all-templates (yas--get-snippet-tables)))))
      (if matching
          (progn
            (if (frame-focus-state)
                (progn
                  (yas-expand-snippet matching)
                  nil)
              ;; In another application
              (with-temp-buffer
                (yas-minor-mode)
                (yas-expand-snippet matching)
                (buffer-string))))
        text))
    text))

This code relies on my fork of whisper.el, which lets me specify a list of functions for whisper-insert-text-at-point. (I haven't asked for upstream review yet because I'm still testing things, and I don't know if it actually works for anyone else yet.) It does approximate matching on the snippet name using a function from subed-word-data.el which just uses string-distance. I could probably duplicate the function in my config, but then I'd have to update it in two places if I come up with more ideas.

The code for inserting into other functions is defined in my-whisper-maybe-type, which is very simple: 

(defun my-whisper-maybe-type (text)
  "If Emacs is not the focused app, simulate typing TEXT.
Add this function to `whisper-insert-text-at-point'."
  (when text
    (if (frame-focus-state)
        text
      (make-process :name "xdotool" :command
                    (list "xdotool" "type"
                          text))
      nil)))

Someday I'd like to provide alternative names for snippets. I also want to make it easy to fill in snippet fields by voice. I'd love to be able to answer minibuffer questions from yas-choose-value, yas-completing-read, and other functions by voice too. Could be fun!

Related:

This is part of my Emacs configuration.

You can e-mail me at sacha@sachachua.com.

]]> Using speech recognition for on-the-fly translations in Emacs and faking in-buffer completion for the results https://sachachua.com/blog/2026/02/using-speech-recognition-for-on-the-fly-translations-in-emacs-and-faking-in-buffer-completion-for-the-results/ Fri, 27 Feb 2026 20:11:58 GMT audio speech-recognition emacs speech https://sachachua.com/blog/2026/02/using-speech-recognition-for-on-the-fly-translations-in-emacs-and-faking-in-buffer-completion-for-the-results/

: I added the recognized text so that I can confirm what was translated. I also moved my-type-with-hint to learn-lang-type-with-hint.

When I'm writing a journal entry in French, I sometimes want to translate a phrase that I can't look up word by word using a dictionary. Instead of switching to a browser, I can use an Emacs function to prompt me for text and either insert or display the translation. The plz library makes HTTP requests slightly neater. 

(defun my-lang-en-to-fr (text &optional display-only)
  (interactive (list (read-string "Text: ") current-prefix-arg))
  (let* ((url "https://translation.googleapis.com/language/translate/v2")
         (params `(("key" . ,(getenv "GOOGLE_API_KEY"))
                   ("q" . ,text)
                   ("source" . "en")
                   ("target" . "fr")
                   ("format" . "text")))
         (query-string (mapconcat
                        (lambda (pair)
                          (format "%s=%s"
                                  (url-hexify-string (car pair))
                                  (url-hexify-string (cdr pair))))
                        params
                        "&"))
         (full-url (concat url "?" query-string)))
    (let* ((response (plz 'get full-url :as #'json-read))
           (data (alist-get 'data response))
           (translations (alist-get 'translations data))
           (first-translation (car translations))
           (translated-text (alist-get 'translatedText first-translation)))
      (when (called-interactively-p 'any)
        (if display-only
            (message "%s" translated-text)
          (insert translated-text)))
      translated-text)))

I think it would be even nicer if I could use speech synthesis, so I can keep it a little more separate from my typing thoughts. I want to be able to say "Okay, translate …" or "Okay, … in French" to get a translation. I've been using my fork of natrys/whisper.el for speech recognition in English, and I like it a lot. By adding a function to whisper-after-transcription-hook, I can modify the intermediate results before they're inserted into the buffer. 

(defun my-whisper-translate ()
  (goto-char (point-min))
  (let ((case-fold-search t))
    (when (re-search-forward "okay[,\\.]? translate[,\\.]? \\(.+\\)\\|okay[,\\.]? \\(.+?\\) in French" nil t)
      (let* ((s (or (match-string 1) (match-string 2)))
             (translation (save-match-data (my-lang-en-to-fr s))))
        (replace-match
         (propertize translation
                     'type-hint translation
                     'type-original s
                     'help-echo s))))))

(with-eval-after-load 'whisper
  (add-hook 'whisper-after-transcription-hook 'my-whisper-translate 70))

But that's too easy. I want to actually type things myself so that I get more practice. Something like an autocomplete suggestion would be handy as a way of showing me a hint at the cursor. The usual completion-at-point functions are too eager to insert things if there's only one candidate, so we'll just fake it with an overlay. This code works only with my whisper.el fork because it supports using a list of functions for whisper-insert-text-at-point. 

(defun my-whisper-maybe-type-with-hints (text)
  "Add this function to `whisper-insert-text-at-point'."
  (let* ((hint (and text (org-find-text-property-in-string 'type-hint text)))
         (original (and text (org-find-text-property-in-string 'type-original text))))
    (if hint
        (progn
          (learn-lang-type-with-hint hint original)
          nil)
      text)))

Here's a demonstration of me saying "Okay, this is a test, in French.":

Screencast of using speech recognition to translate into French and provide a hint when typing

Since we're faking in-buffer completion here, maybe we can still get away with considering this as an entry for Emacs Carnival February 2026: Completion ? =)

This is part of my Emacs configuration.

You can e-mail me at sacha@sachachua.com.

]]> Using Silero voice activity detection to automatically queue multiple transcriptions with natrys/whisper.el https://sachachua.com/blog/2026/01/using-silero-voice-activity-detection-to-automatically-queue-multiple-transcriptions-with-natrys-whisper-el/ Thu, 29 Jan 2026 16:24:46 GMT audio speech-recognition emacs https://sachachua.com/blog/2026/01/using-silero-voice-activity-detection-to-automatically-queue-multiple-transcriptions-with-natrys-whisper-el/

: Moved code to sachac/speech-input.

I can queue multiple transcriptions with whisper.el so that they get processed sequentially with backup audio. It catches up when I pause to think. Now I want to use Silero voice activity detection to do that kind of segmentation for me automatically.

First, I need a Python server that can print out events when it notices the start or stop of a speech segment. If I print out the timestamps, I might be able to cross-reference it someday with interestingthings. For now, even just paying attention to the end of a segment is enough for what I want to do.

Python script for printing out events 
import sounddevice as sd
import numpy as np
import torch
import sys
from datetime import datetime, timedelta

SILENCE_DURATION = 500
SAMPLING_RATE = 16000
CHUNK_SIZE = 512
model, utils = torch.hub.load(repo_or_dir='snakers4/silero-vad',
                              model='silero_vad',
                              force_reload=False)

(get_speech_timestamps, save_audio, read_audio, VADIterator, collect_chunks) = utils
vad_iterator = VADIterator(model, threshold=0.5, min_silence_duration_ms=SILENCE_DURATION)

stream_start_time = None

def format_iso_with_offset(offset_seconds):
    if stream_start_time is None:
        return "PENDING"
    event_time = stream_start_time + timedelta(seconds=offset_seconds)
    return event_time.astimezone().isoformat(timespec='milliseconds')

def audio_callback(indata, frames, time, status):
    global stream_start_time
    if status:
        print(status, file=sys.stderr)
    if stream_start_time is None:
        stream_start_time = datetime.now()
    tensor_input = torch.from_numpy(indata.copy()).flatten()
    speech_dict = vad_iterator(tensor_input, return_seconds=True)
    if speech_dict:
        if "start" in speech_dict:
            print(f"START {format_iso_with_offset(speech_dict['start'])}", flush=True)
        if "end" in speech_dict:
            print(f"END {format_iso_with_offset(speech_dict['end'])}", flush=True)
try:
    with sd.InputStream(samplerate=SAMPLING_RATE,
                        channels=1,
                        callback=audio_callback,
                        blocksize=CHUNK_SIZE):
        while True:
            pass
except KeyboardInterrupt:
    print("\nStopping...")

I've mode the code to speech-input-vad.el, so now I can use speech-input-vad-start to start monitoring speech events. 

(use-package speech-input :load-path "~/proj/speech-input/")

Because I added Pulse properties to the process environment, I can easily use epwgraph to rewire the input so that it gets the input from my VirtualMicSink instead of the default system audio device. (Someday I'll figure out how to specify that as the input automatically.)

Then I can start this process from Emacs: 

(with-eval-after-load 'speech-input-vad
  (add-hook 'speech-input-vad-on-end-functions #'my-whisper-maybe-continue))

(defun my-whisper-maybe-continue ()
  (when (process-live-p whisper--recording-process)
    (my-whisper-continue)))

Now I can press my shortcut for my-whisper-continue to start the process. As I keep talking, it will continue to record. When I pause for more than a second between sentences, then it will send that chunk to the server for transcription without me having to press another button, while still listening for more speech.

How is this different from the streaming approach that many real-time speech recognition services offer? I think this gives me a bit more visibility into and control of the process. For my personal use, I don't need to have everything processed as quickly as possible, and I'm not trying to replicate live captions. I just want to be able to look back over the last five minutes to try to remember what I was talking about. I usually have a lot of quiet time as I think through my next steps, and it's fine to have it catch up then. I also like that I can save time-stamped audio files for later processing, divided according to the speech segments. Those might be a little bit easier to work with when I get around to compositing them into a video.

This is part of my Emacs configuration.

You can e-mail me at sacha@sachachua.com.

]]> Queuing multiple transcriptions with whisper.el speech recognition https://sachachua.com/blog/2026/01/queue-multiple-transcriptions-with-whisper-el-speech-recognition/ Fri, 23 Jan 2026 19:50:37 GMT audio speech-recognition emacs https://sachachua.com/blog/2026/01/queue-multiple-transcriptions-with-whisper-el-speech-recognition/ I want to be able to talk out loud and have the ideas go into Emacs. I can do this in a number of different ways:

  1. I briefly demonstrated a step-by-step approach with natrys/whisper.el with a single file. I press a keyboard shortcut to start the recording, another shortcut to stop the recording, and it transcribes it in the background. But the way whisper.el is set up is that if I press the keyboard shortcut to start recording again it will offer to interrupt the transcription process, which is not what I want. I want to just keep talking and have it process results as things come in.
  2. I'm also experimenting with Google Chrome's web speech API to do continuous speech recognition, which I can get into Emacs using a web socket.
  3. What I've just figured out is how to layer a semi-continuous interface for speech recognition on top of whisper.el so that while it's processing in the background, I can just press a keyboard shortcut (I'm using numpad 9 to call my-whisper-continue) to stop the previous recording, queue it for processing, and start the next recording. If I use this keyboard shortcut to separate my thoughts, then Whisper has a much easier time making sense of the whole sentence or paragraph or whatever, instead of trying to use the sliding 30 second context window that many streaming approaches to speech recognition try to use.

Question: Did you fix the keyboard delay you've got while speech catches what you're saying?

Sometimes, when the speed recognition kicks in, my computer gets busy. When my computer gets really busy, it doesn't process my keystrokes in the right order, which is very annoying because then I have to delete the previous word and retype it. I haven't sorted that out yet, but it seems like I probably have to lower the priority on different processes. On the plus side, as I mentioned, if I dictate things instead of typing them, then I don't run into that problem at all.

Also, other notes on delays: The continuous speech recognition via Google Chrome shows up fairly quickly, but it's not very precise, and it doesn't have punctuation. Even if there's a little bit of a delay, as long as I press the my-whisper-continue shortcut after each thought, then I can get that text into my Emacs buffer using the nicer transcription from my selected model. There is going to be a bit of a delay for that one because it gets processed at the end of the thought. Also, I need to start thinking in complete sentences instead of just adding one cause after the other as my brain goes on all of these tangents. I think it's pretty promising. There's the continuous speech recognition via Google Chrome if I don't mind the lower accuracy and lack of punctuation, and I can still get the pretty version on the other side.

Why talk out loud? I liked the Bookclub Tapas presentation that Maddie Sullivan did at EmacsConf 2025. Talking out loud helps me be a lot more verbose about what I'm saying, compared to typing things out or even like having to switch to my notes or interrupting my screen with an Org capture buffer. Of course I want to clean that up for putting into a blog post, but given that my life still sometimes has random interruptions from a kiddo who must have my attention at that very minute, having that kind of record that I can at least try to reread afterwards to reconstruct what I was thinking about sounds like it might be helpful.

Still, making sense out loud is hard. I'm not actually used to talking to people that much now. This is probably a good reason for me to experiment with streaming more. Then I get the practice in talking out loud, there are backup recordings, and people can ask questions when things are unclear.

Of course, sometimes the text doesn't quite make sense because of the speech recognition errors. I can usually figure it out from the context. I save the audio as well so that I can go back and listen to it again if I really need to.

Anyway, here's the code for sending the current recording to whisper in the background and starting another recording. It assumes a lot about how things are set up. For example, I'm only testing this with a local speaches server instead of whisper.cpp. You might need to look at my other speech related configuration blog posts and sections in order to make sense of it.

Code for queuing whisper.el requests to a local server 
(defvar my-whisper--queue nil)
(defun my-whisper-continue (&optional arg)
  "Send what we've got so far for transcription and then continue recording.
Call with \\[universal-argument] to signal that we can stop."
  (interactive "P")
  (require 'whisper)
  (if arg
      (my-whisper-done)
    (setq whisper--marker (point-marker) whisper--point-buffer (current-buffer))
    (when (process-live-p whisper--recording-process)
      ;; queue only if the last one is not asking for the same file
      (unless
          (string=
           (plist-get
            (car
             (last my-whisper--queue))
            :file)
           whisper--temp-file)
        (add-to-list
         'my-whisper--queue
         (list :file whisper--temp-file
               :buffer
               (format "*result: %s*" (file-name-base whisper--temp-file)))
         t))
      ;; Remove the sentinel; handle results ourselves
      (set-process-sentinel whisper--recording-process
                            (lambda (process event)
                              (my-whisper-process-queue)))
      (interrupt-process whisper--recording-process))
    (run-hooks 'whisper-before-transcription-hook)
    (whisper--setup-mode-line :show 'recording)
    (whisper--record-audio)))

(defun my-whisper-discard ()
 "Ignore the previous recording."
  (interactive)
  (when (process-live-p whisper--recording-process)
    ;; Remove the sentinel; handle results ourselves
    (set-process-sentinel whisper--recording-process
                          (lambda (process event)
                            (when (file-exists-p whisper--temp-file)
                              (delete-file whisper--temp-file))
                            (my-whisper-process-queue)))
    (interrupt-process whisper--recording-process)))

(defun my-whisper-discard-and-continue ()
 "Ignore the previous recording and continue."
  (interactive)
  (if (process-live-p whisper--recording-process)
      (progn
        ;; Remove the sentinel; handle results ourselves
        (set-process-sentinel whisper--recording-process
                              (lambda (process event)
                                (my-whisper-process-queue)
                                (my-whisper-continue)))
        (interrupt-process whisper--recording-process))
    (my-whisper-continue)))

(defun my-whisper-done ()
  (interactive)
  (when (process-live-p whisper--recording-process)
    (add-to-list
     'my-whisper--queue
     (list :file whisper--temp-file
           :buffer
           (format "*result: %s*" (file-name-base whisper--temp-file)))
     t)
    ;; Remove the sentinel; handle results ourselves
    (set-process-sentinel whisper--recording-process
                          (lambda (process event)
                            (my-whisper-process-queue)))
    (whisper--setup-mode-line :hide 'recording)
    (interrupt-process whisper--recording-process)))

(defun my-whisper-process-queue-result ()
  "Process the first part of the queue that already has results."
  (while (plist-get (car my-whisper--queue) :results)
    (let ((o (pop my-whisper--queue)))
      (unless my-whisper-target-markers
        (setq whisper--marker (point-marker)
              whisper--point-buffer (current-buffer)))
      (with-current-buffer (plist-get o :buffer)
        (erase-buffer)
        (insert (plist-get o :results)))
      ;; Only works with my fork: https://github.com/sachac/whisper.el/tree/whisper-insert-text-at-point-function
      (whisper--handle-transcription-output nil (plist-get o :buffer)))))

(defun my-whisper-process-queue ()
  (let (o)
    (while (setq o (seq-find (lambda (o) (and (plist-get o :file)
                                              (not (plist-get o :process))
                                              (not (plist-get o :results))))
                             my-whisper--queue))
      (let* ((headers (list "Content-Type: multipart/form-data"))
             (params (list (concat "file=@"
                                   (plist-get o :file))
                           "temperature=0.0"
                           "temperature_inc=0.2"
                           "response_format=json"
                           (concat "model=" whisper-model)
                           (concat "language=" whisper-language)))
             (url (format my-whisper-url-format whisper-server-host whisper-server-port))
             (command `("curl" "-s"
                        ,url
                        ,@(mapcan (lambda (h) (list "-H" h)) headers)
                        ,@(mapcan (lambda (p) (list "-F" p)) params))))
        (with-current-buffer (get-buffer-create (plist-get o :buffer))
          (erase-buffer))
        (plist-put
         o :process
         (make-process
          :name "whisper-curl"
          :command command
          :buffer (plist-get o :buffer)
          :coding 'utf-8
          :sentinel
          (lambda (process event)
            (with-current-buffer (process-buffer process)
              (let ((current my-whisper--queue-item))
                (when (and (get-buffer (plist-get current :buffer))
                           (string-equal "finished\n" event))
                  (with-current-buffer (plist-get current :buffer)
                    (goto-char (point-min))
                    (plist-put current :results
                               (or
                                (condition-case nil
                                    (gethash "text" (json-parse-buffer))
                                  (error ""))
                                "(error)"))))))
            (my-whisper-process-queue-result))))
        (plist-put o :command (string-join command " "))
        (with-current-buffer (process-buffer (plist-get o :process))
          (setq-local my-whisper--queue-item o))))))
(defvar-local my-whisper--queue-item nil)

(defun my-whisper-reprocess-queue ()
  (interactive)
  (setq whisper--marker (point-marker) whisper--point-buffer (current-buffer))
  (mapc (lambda (o)
          (when (process-live-p (plist-get o :process))
            (kill-process (plist-get o :process)))
          (when (get-buffer (plist-get o :buffer))
            (kill-buffer (plist-get o :buffer)))
          (plist-put o :process nil)
          (plist-put o :results nil))
        my-whisper--queue)
  (my-whisper-process-queue))

(defun my-whisper-clear-queue ()
  (interactive)
  (mapc (lambda (o)
          (when (process-live-p (plist-get o :process))
            (kill-process (plist-get o :process)))
          (when (get-buffer (plist-get o :buffer))
            (kill-buffer (plist-get o :buffer)))
          (plist-put o :process nil)
          (plist-put o :results nil))
        my-whisper--queue)
  (setq my-whisper--queue nil))

(keymap-global-set "<kp-9>" #'my-whisper-continue)
(keymap-global-set "<kp-8>" #'my-whisper-discard-and-continue)
(keymap-global-set "C-<kp-9>" #'my-whisper-done)
This is part of my Emacs configuration.

You can comment on Mastodon or e-mail me at sacha@sachachua.com.

]]> Emacs and whisper.el: Trying out different speech-to-text backends and models https://sachachua.com/blog/2026/01/emacs-and-whisper-el-trying-out-different-speech-to-text-backends-and-models/ Tue, 20 Jan 2026 19:21:46 GMT audio emacs speech-recognition https://sachachua.com/blog/2026/01/emacs-and-whisper-el-trying-out-different-speech-to-text-backends-and-models/ Update : I'm currently using Orukeet instead of Whisper/Speaches. Experimenting with Orukeet for speech recognition

I was curious about parakeet because I heard that it was faster than Whisper on the HuggingFace leaderboard. When I installed it and got it running on my laptop (CPU only, no GPU), it seemed like my results were a little faster than whisper.cpp with the large model, but much slower than whisper.cpp with the base model. The base model is decent for quick dictation, so I got curious about other backends and other models.

In order to try natrys/whisper.el with other backends, I needed to work around how whisper.el validates the model names and sends requests to the servers. Here's the quick and dirty code for doing so, in case you want to try it out for yourself. 

(defvar sacha-whisper-url-format "http://%s:%d/transcribe")
;;;###autoload
(defun sacha-whisper--transcribe-via-local-server ()
  "Transcribe audio using the local whisper server."
  (message "[-] Transcribing via local server")
  (whisper--setup-mode-line :show 'transcribing)
  (whisper--ensure-server)
  (setq whisper--transcribing-process
        (whisper--process-curl-request
         (format sacha-whisper-url-format whisper-server-host whisper-server-port)
         (list "Content-Type: multipart/form-data")
         (list (concat "file=@" whisper--temp-file)
               "temperature=0.0"
               "temperature_inc=0.2"
               "response_format=json"
               (concat "model=" whisper-model)
               (concat "language=" whisper-language)))))
;;;###autoload
(defun sacha-whisper--check-model-consistency () t)

I moved my server code to speech-input-speaches.el. 

(with-eval-after-load 'whisper
  (advice-add 'whisper--transcribe-via-local-server :override #'sacha-whisper--transcribe-via-local-server)
  (advice-add 'whisper--check-model-consistency :override #'sacha-whisper--check-model-consistency)
  ;;(advice-add 'whisper--ensure-server :override #'speech-input-speaches-server-start)
  )

Then I have this function for trying things out. 

(defun sacha-test-whisper-api (url &optional args)
  (with-temp-buffer
    (apply #'call-process "curl" nil t nil "-s"
           url
         (append (mapcan
                  (lambda (h) (list "-H" h))
                  (list "Content-Type: multipart/form-data"))
                 (mapcan
                  (lambda (h) (list "-F" h))
                  (list (concat "file=@" whisper--temp-file)
                        "temperature=0.0"
                        "temperature_inc=0.2"
                        "response_format=verbose_json"
                        (concat "language=" whisper-language)))
                 args))
    (message "%s %s" (buffer-string) url)))

Here's the audio file. It is around 10 seconds long. I run the benchmark 3 times and report the average time.

Download

Code for running the benchmarks 
(let ((times '3))
(mapcar
 (lambda (group)
   (let ((whisper--temp-file "/home/sacha/recordings/whisper/2026-01-19-14-17-53.wav"))
     ;; warm up the model
     (eval (cadr group))
     (list
      (format "%.3f"
              (/ (car
                  (benchmark-call (lambda () (eval (cadr group))) times))
                 times))
      (car group))))
 '(
   ("parakeet"
    (sacha-test-whisper-api
     (format "http://%s:%d/v1/audio/transcriptions" whisper-server-host 5092)))
   ("whisper.cpp base-q4_0"
    (sacha-test-whisper-api
     (format "http://%s:%d/inference" whisper-server-host 8642)))
   ("speaches whisper-base"
    (sacha-test-whisper-api
     (format "http://%s:%d/v1/audio/transcriptions" whisper-server-host 8001)
     (list "-F" "model=Systran/faster-whisper-base")))
   ("speaches whisper-base.en"
    (sacha-test-whisper-api
     (format "http://%s:%d/v1/audio/transcriptions" whisper-server-host 8001)
     (list "-F" "model=Systran/faster-whisper-base.en")))
   ("speaches whisper-small"
    (sacha-test-whisper-api
     (format "http://%s:%d/v1/audio/transcriptions" whisper-server-host 8001)
     (list "-F" "model=Systran/faster-whisper-small")))
   ("speaches whisper-small.en"
    (sacha-test-whisper-api
     (format "http://%s:%d/v1/audio/transcriptions" whisper-server-host 8001)
     (list "-F" "model=Systran/faster-whisper-small.en")))
   ("speaches lorneluo/whisper-small-ct2-int8"
    (sacha-test-whisper-api
     (format "http://%s:%d/v1/audio/transcriptions" whisper-server-host 8001)
     (list "-F" "model=lorneluo/whisper-small-ct2-int8")))
   ;; needed export TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1
   ("whisperx-server Systran/faster-whisper-small"
    (sacha-test-whisper-api
     (format "http://%s:%d/transcribe" whisper-server-host 8002)))))
)
3.694 parakeet
2.484 whisper.cpp base-q4_0
1.547 speaches whisper-base
1.425 speaches whisper-base.en
4.076 speaches whisper-small
3.735 speaches whisper-small.en
2.870 speaches lorneluo/whisper-small-ct2-int8
4.537 whisperx-server Systran/faster-whisper-small

All the results above were CPU-only, no GPU acceleration.

I tried it with:

Update: After a lot of rebooting, I figured out how to get the Nvidia P1000 GPU on my Lenovo P52 to work for up to the Whisper medium models. Large models don't fit in the 4GB of VRAM it has. Here are the benchmarking results:

Benchmarking code 
(let ((times '3))
(mapcar
 (lambda (group)
   (let ((whisper--temp-file "/tmp/2026-01-19-14-17-53.wav"))
     ;; warm up the model
     (eval (cadr group))
     (list
      (format "%.3f"
              (/ (car
                  (benchmark-call (lambda () (eval (cadr group))) times))
                 times))
      (car group))
     ))
 '(
   ("speaches whisper-tiny"
    (sacha-test-whisper-api
     (format "http://%s:%d/v1/audio/transcriptions" "localhost" 8000)
     (list "-F" "model=Systran/faster-whisper-tiny.en")))
   ("speaches whisper-base"
    (sacha-test-whisper-api
     (format "http://%s:%d/v1/audio/transcriptions" "localhost" 8000)
     (list "-F" "model=Systran/faster-whisper-base.en")))
   ("speaches whisper-small"
    (sacha-test-whisper-api
     (format "http://%s:%d/v1/audio/transcriptions" "localhost" 8000)
     (list "-F" "model=Systran/faster-whisper-small.en")))
   ("speaches whisper-medium"
    (sacha-test-whisper-api
     (format "http://%s:%d/v1/audio/transcriptions" "localhost" 8000)
     (list "-F" "model=Systran/faster-whisper-medium.en"))))))
0.596 speaches whisper-tiny
0.940 speaches whisper-base
2.909 speaches whisper-small
8.740 speaches whisper-medium

That probably means I can upgrade to using the small model on a regular basis. For large models, I can use the command-line tool, since the overhead of loading the model is probably small compared to the processing time. I could also run two Speaches servers (one on CPU and one on GPU), but I don't think I'll need that level yet.

I like how speaches lets me switch models on the fly, so maybe I can use small.en generally and switch to a different model when I want to try dictating in French. Here's how I've set it up to use the server. 

(defvar sacha-speech-input-model-aliases
  '(("small" . "Systran/faster-whisper-small.en")
    ("medium" . "Systran/faster-whisper-medium.en")
    ("base" . "Systran/faster-whisper-base.en")
    ("tiny" . "Systran/faster-whisper-tiny.en")
    ("large" . "Systran/faster-whisper-large-v2")))

(defun sacha-speech-input-set-model (model-name)
  "Change the speech recognition model to MODEL-NAME.
Use `sacha-speech-input-model-aliases' for aliases."
  (interactive (list (speech-input-speaches-read-model-name)))
  (when (assoc-default model-name sacha-speech-input-model-aliases #'string=)
    (setq model-name (assoc-default model-name sacha-speech-input-model-aliases #'string=)))
  (setq whisper-model model-name)
  (setq speech-input-model model-name
        speech-input-transcribe-model model-name))

(setq whisper-model "Systran/faster-whisper-small.en"
      sacha-whisper-url-format "http://%s:%d/v1/audio/transcriptions")

Benchmark notes: I have a Lenovo P52 laptop (released 2018) with an Intel Core i7-8850H (6 cores, 12 threads; 2.6 GHz base / 4.3 GHz turbo) with 64GB RAM and an SSD. It has a soldered-on Nvidia P1000 with 4GB of VRAM.

This is part of my EmacsConfiguration.

You can comment on Mastodon or e-mail me at sacha@sachachua.com.

]]> Using whisper.el to convert speech to text and save it to the currently clocked task in Org Mode or elsewhere https://sachachua.com/blog/2026/01/using-whisper-el-to-capture-text-to-speech-in-emacs/ Sun, 04 Jan 2026 01:23:57 GMT emacs audio speech-recognition speech https://sachachua.com/blog/2026/01/using-whisper-el-to-capture-text-to-speech-in-emacs/
  • : I now use Orukeet as the speech recognition model because it's faster.
  • : Major change: I switched to my fork of natrys/whisper.el so that I can specify functions that change the window configuration etc.
  • : Change main function to sacha-whisper-run, use seq-reduce to go through the functions.
  • : Added code for automatically capturing screenshots, saving text, working with a list of functions.
  • : Added demo, fixed some bugs.
  • : Added note about difference from MELPA package, fixed :vc

I want to get my thoughts into the computer quickly, and talking might be a good way to do some of that. whisper.el gives me a convenient way to interact with a server compatible with OpenAI Whisper's API. (Experimenting with Orukeet for speech recognition) Note: This is not the same whisper package as the one on MELPA. Here is how I have it set up for reasonable performance on my Lenovo P52 with just the CPU, no GPU.

I've bound <f9> to the command whisper-run in Emacs, but I use my window manager's custom shortcuts to map <kp-0> to execute /home/sacha/bin/xdotool-emacs key --clearmodifiers F9, which sends <f9> to Emacs. (file:///home/sacha/bin/xdotool-emacs) That makes it a shortcut that works even in other applications.

I press <kp-0> to start recording, talk, and then press <kp-0> to stop recording. By default, it inserts the text into the buffer at the current point. I've set whisper-return-cursor-to-start to nil so that I can keep going. 

(use-package whisper
  :vc (:url "https://github.com/natrys/whisper.el")
  :load-path "~/vendor/whisper.el"
  :config
  (setq whisper--mode-line-recording-indicator "⏺")
  (setq whisper-quantize "q4_0")
  (setq whisper-install-directory "~/vendor")
  (setq whisper--install-path (concat
     (expand-file-name (file-name-as-directory whisper-install-directory))
     "whisper.cpp/"))
  (setq whisper-return-cursor-to-start nil)
  (setq whisper--ffmpeg-input-device "VirtualMicSink.monitor")
  (setq whisper-language "en")
  (setq whisper-recording-timeout 3000)
  (setq whisper-before-transcription-hook nil)
  (setq whisper-use-threads (1- (num-processors)))
  (setq whisper-transcription-buffer-name-function 'whisper--simple-transcription-buffer-name)
  (add-hook 'whisper-after-transcription-hook 'sacha-subed-fix-common-errors-from-start -100)
  :bind
  (("<f9>" . whisper-run)
   ("C-<f9>" . sacha-whisper-run)
   ("S-<f2>" . whisper-run)
   ("S-<f9>" . sacha-whisper-replay)
   ("M-<f9>" . sacha-whisper-toggle-language)))

: "Remind me to…" is now handled by https://codeberg.org/sachac/.emacs.d/src/branch/gh-pages/lisp/.user-lisp-autoloads.el.

Disk space is inexpensive and backups are great, so let's save each file using the timestamp. 

(defvar sacha-whisper-dir "~/recordings/whisper/")
;;;###autoload
(defun sacha-whisper-set-temp-filename ()
  (setq whisper--temp-file (expand-file-name
                            (format-time-string "%Y-%m-%d-%H-%M-%S.wav")
                            sacha-whisper-dir)))


(with-eval-after-load 'whisper
  (add-hook 'whisper-before-transcription-hook #'sacha-whisper-set-temp-filename))

I can replay the most recently recorded snippet in case it's missed something and I've forgotten what I just said. 

;;;###autoload
(defun sacha-whisper-replay (&optional file)
  "Replay the last temporary recording."
  (interactive (list
                (when current-prefix-arg
                  (read-file-name "File: " sacha-whisper-dir))))
  (setq whisper--temp-file (or file whisper--temp-file))
  (mpv-play whisper--temp-file))

;;;###autoload
(defun sacha-whisper-insert-retry (&optional file)
  (interactive (list
                (when current-prefix-arg
                  (read-file-name "File: " sacha-whisper-dir))))
  (whisper--cleanup-transcription)
  (setq whisper--marker (point-marker)
        whisper--temp-file (or file whisper--temp-file))
  (whisper--transcribe-audio))

Update : Il peut aussi comprendre le français. I used to have code to set the language in case the auto-recognition doesn't get it right, but now I'm using Orukeet as the speech recognition engine now, so it doesn't take a language setting. 

;;;###autoload
(defun sacha-whisper-reset (text)
  (setq sacha-whisper-skip-annotation nil)
  (remove-hook 'whisper-insert-text-at-point #'sacha-whisper-org-save-to-clocked-task)
  text)

;; Only works with my tweaks to whisper.el
;; https://github.com/sachac/whisper.el/tree/whisper-insert-text-at-point-function
(with-eval-after-load 'whisper
  (setq whisper-insert-text-at-point
        '(sacha-whisper-scratch-that
          sacha-whisper-handle-commands
          sacha-whisper-save-text
          sacha-learn-lang-notice-bingo-words
;          sacha-whisper-save-to-file
          sacha-save-to-kill-ring-after-current
          sacha-whisper-maybe-expand-snippet
          sacha-speech-input-quantified-track
          sacha-whisper-maybe-type
          sacha-whisper-maybe-type-with-hints
          sacha-whisper-insert
          sacha-whisper-reset)))

(defvar sacha-whisper-last-annotation nil "Last annotation so we can skip duplicates.")
(defvar sacha-whisper-skip-annotation nil)
(defvar sacha-whisper-target-markers nil "List of markers to send text to.")

;;;###autoload
(defun sacha-whisper-insert (text)
  (let ((markers
         (cond
          ((null sacha-whisper-target-markers)
           (list whisper--marker)) ; current point where whisper was started
          ((listp sacha-whisper-target-markers)
           sacha-whisper-target-markers)
          ((markerp sacha-whisper-target-markers)
           (list sacha-whisper-target-markers))))
        (orig-point (point))
        (orig-buffer (current-buffer)))
    (when text
      (mapcar (lambda (marker)
                (with-current-buffer (marker-buffer marker)
                  (save-restriction
                    (widen)
                    (when (markerp marker) (goto-char marker))
                    (when (and (derived-mode-p 'org-mode) (org-at-drawer-p))
                      (insert "\n"))
                    (whisper--insert-text
                     (concat
                      (if (looking-back "[ \t\n]\\|^")
                          ""
                        " ")
                      (string-trim text)))
                    ;; Move the marker forward here
                    (move-marker marker (point)))))
              markers)
      (when sacha-whisper-target-markers
        (goto-char orig-point))
      nil)))

;;;###autoload
(defun sacha-whisper-maybe-type (text)
  "If Emacs is not the focused app, simulate typing TEXT.
Add this function to `whisper-insert-text-at-point'."
  (when text
    (if (frame-focus-state)
        text
      (make-process :name "xdotool" :command
                    (list "xdotool" "type"
                          text))
      nil)))

;;;###autoload
(defun sacha-whisper-clear-markers ()
  (interactive)
  (setq sacha-whisper-target-markers nil))

;;;###autoload
(defun sacha-whisper-use-current-point (&optional add)
  (interactive (list current-prefix-arg))
  (if add
      (push (point-marker) sacha-whisper-target-markers)
    (setq sacha-whisper-target-markers (list (point-marker)))))

;;;###autoload
(defun sacha-whisper-run-at-point (&optional add)
  (interactive (list current-prefix-arg))
  (sacha-whisper-clear-markers)
  (whisper-run))


(keymap-global-set "<f9>" #'sacha-whisper-run-at-point)
(keymap-global-set "<kp-1>" #'whisper-run)

;;;###autoload
(defun sacha-whisper-jump-to-marker ()
  (interactive)
  (with-current-buffer (marker-buffer (car sacha-whisper-target-markers))
    (goto-char (car sacha-whisper-target-markers))))

;;;###autoload
(defun sacha-whisper-use-currently-clocked-task (&optional add)
  (interactive (list current-prefix-arg))
  (save-window-excursion
    (save-restriction
      (save-excursion
        (org-clock-goto)
        (org-end-of-meta-data)
        (org-end-of-subtree)
        (if add
            (push (point-marker) sacha-whisper-target-markers)
          (setq sacha-whisper-target-markers (list (point-marker))))))))

;;;###autoload
(defun sacha-whisper-run (&optional skip-annotation)
  (interactive (list current-prefix-arg))
  (require 'whisper)
  (add-hook 'whisper-insert-text-at-point #'sacha-whisper-org-save-to-clocked-task -10)
  (whisper-run)
  (when skip-annotation
    (setq sacha-whisper-skip-annotation t)))

;;;###autoload
(defun sacha-whisper-save-text (text)
  "Save TEXT beside `whisper--temp-file'."
  (when text
    (let ((link (org-store-link nil)))
      (with-temp-file (concat (file-name-sans-extension whisper--temp-file) ".txt")
        (when link
          (insert link "\n"))
        (insert text)))
    text))

;;;###autoload
(defun sacha-whisper-org-save-to-clocked-task (text)
  (when text
    (save-window-excursion
      (with-current-buffer (if (markerp whisper--marker) (marker-buffer whisper--marker) (current-buffer))
        (when (markerp whisper--marker) (goto-char whisper--marker))
        ;; Take a screenshot maybe
        (let* ((link (and (not sacha-whisper-skip-annotation)
                          (org-store-link nil)))
               (region (and (region-active-p) (buffer-substring (region-beginning) (region-end))))
               (screenshot-filename
                (when (or
                       (null link)
                       (not (string= sacha-whisper-last-annotation link))
                       (not (frame-focus-state))) ; not in focus, take a screenshot
                  (sacha-screenshot-current-screen (concat (file-name-sans-extension whisper--temp-file) ".png")))))
          (if (org-clocking-p)
              (save-window-excursion
                (save-restriction
                  (save-excursion
                    (org-clock-goto)
                    (org-end-of-subtree)
                    (unless (bolp)
                      (insert "\n"))
                    (insert "\n")
                    (if (and link (not (string= sacha-whisper-last-annotation link)))
                        (insert
                         (if screenshot-filename
                             (concat "(" (org-link-make-string
                                          (concat "file:" screenshot-filename)
                                          "screenshot") ") ")
                           "")
                         link
                         "\n")
                      (when screenshot-filename
                        (insert (org-link-make-string
                                 (concat "file:" screenshot-filename)
                                 "screenshot")
                                "\n")))
                    (when region
                      (insert "#+begin_example\n" region "\n#+end_example\n"))
                    (insert text "\n")
                    (setq sacha-whisper-last-annotation link)))
                (run-at-time 0.5 nil (lambda (text) (message "Added clock note: %s" text)) text))
            ;; No clocked task, prompt for a place to capture it
            (kill-new text)
            (setq org-capture-initial text)
            (call-interactively 'org-capture)
            ;; Delay the window configuration
            (let ((config (current-window-configuration)))
              (run-at-time 0.5 nil
                           (lambda (text config)
                             (set-window-configuration config)
                             (message "Copied: %s" text))
                           text config))))))))


(with-eval-after-load 'org
  (add-hook 'org-clock-in-hook #'sacha-whisper-org-clear-saved-annotation))

;;;###autoload
(defun sacha-whisper-org-clear-saved-annotation ()
  (setq sacha-whisper-org-last-annotation nil))

Here's an idea for a function that saves the recognized text with a timestamp. 

(defvar sacha-whisper-notes "~/sync/stream/narration.org")
;;;###autoload
(defun sacha-whisper-save-to-file (text)
  (when text
    (let ((link (org-store-link nil)))
      (with-current-buffer (find-file-noselect sacha-whisper-notes)
        (goto-char (point-max))
        (insert "\n\n" (format-time-string "%H:%M ") text "\n" (if link (concat link "\n") ""))
        (save-buffer)
        (run-at-time 0.5 nil (lambda (text) (message "Saved to file: %s" text)) text)))
    text))

Let's add it to the kill ring, but not at the most recent slot. That way, it's easy for me to get to, but it doesn't overwrite anything that I've copied. 

;;;###autoload
(defun sacha-save-to-kill-ring-after-current (text)
  "Save TEXT to the kill ring, but not at the top spot."
  (when text
    (if kill-ring
        (let ((temp (pop kill-ring)))
          (kill-new text)
          (push temp kill-ring))
      (kill-new text)))
  text)

And now I can redo things if needed: 

;;;###autoload
(defun sacha-whisper-redo ()
  (interactive)
  (setq whisper--marker (point-marker))
  (whisper--transcribe-audio))

I think I've just figured out my Pipewire setup so that I can record audio in OBS while also being able to do speech to text, without the audio stuttering. qpwgraph was super helpful for visualizing the Pipewire connections and fixing them. 


flatpak run com.github.wwmm.easyeffects -q 2>/dev/null
systemctl --user restart pipewire pipewire-pulse wireplumber
sleep 5
nohup flatpak run com.github.wwmm.easyeffects --gapplication-service >/dev/null 2>&1 &

pactl unload-module module-null-sink 2>/dev/null
pactl unload-module module-loopback 2>/dev/null
pactl unload-module module-remap-source 2>/dev/null

pactl load-module module-null-sink \
  sink_name="VirtualMicSink" \
  sink_properties=device.description="VirtualMic_backend"

pactl load-module module-remap-source \
  source_name="VirtualMic" \
  master="VirtualMicSink.monitor" \
  source_properties="device.description='Actual_Virtual_Mic' device.icon_name='audio-input-microphone' device.class='filter'"

pactl load-module module-null-sink \
  sink_name="CombinedSink" \
  sink_properties=device.description=CombinedSink


if pactl list short sources | grep -i pci-0000; then
  pactl load-module module-loopback \
    source="alsa_input.pci-0000_00_1f.3.analog-stereo" \
    sink="VirtualMicSink" \
    latency_msec=200 \
    adjust_time=3 \
    source_output_properties="node.description='SysToVMic' node.name='SysToVMic' media.name='SysVToMic'" \
    sink_input_properties="node.description='SysToVMic' node.name='SysToVMic' media.role='filter'"
  pactl load-module module-loopback \
    source="alsa_input.pci-0000_00_1f.3.analog-stereo" \
    sink="CombinedSink" \
    latency_msec=200 \
    adjust_time=3 \
    source_output_properties="node.description='SysToCombined' node.name='SysToCombined' media.name='SysVToMic'" \
    sink_input_properties="node.description='SysToCombined' node.name='SysToCombined' media.role='filter'"
fi

if pactl list short sources | grep -i USB_Advanced; then
    # YETI=alsa_input.usb-C-Media_Electronics_Inc._USB_Advanced_Audio_Device-00.analog-stereo
    YETI_OUT=alsa_output.usb-C-Media_Electronics_Inc._USB_Advanced_Audio_Device-00.analog-stereo
    YETI=easyeffects_sink.monitor
  # pactl load-module module-loopback \
  #   source="$YETI" \
  #   sink="VirtualMicSink" \
  #   latency_msec=200 \
  #   adjust_time=3 \
  #   source_output_properties="node.description='YetiToVMic' node.name='YetiToVMic' media.name='YetiToVMic'" \
  #   sink_input_properties="node.description='YetiToVMic' node.name='YetiToVMic' media.role='filter'"
  pactl load-module module-loopback \
    source="$YETI" \
    sink="VirtualMicSink" \
    latency_msec=200 \
    adjust_time=3 \
    source_output_properties="node.description='YetiToVMic' node.name='YetiToVMic' media.name='YetiToVMic'" \
    sink_input_properties="node.description='YetiToVMic' node.name='YetiToVMic' media.role='filter'"
  pactl load-module module-loopback \
    source="$YETI" \
    sink="CombinedSink" \
    latency_msec=200 \
    adjust_time=3 \
    source_output_properties="node.description='SysToCombined' node.name='SysToCombined' media.name='SysVToMic'" \
    sink_input_properties="node.description='SysToCombined' node.name='SysToCombined' media.role='filter'"
  pactl load-module module-null-sink sink_name=MeetSink sink_properties=device.description="Google_Meet_Audio"
  pactl load-module module-loopback source="$YETI" sink=MeetSink

  pactl load-module module-combine-sink sink_name=SnippetBridge slaves="MeetSink,$YETI_OUT" label="Snippet Bridge"
fi

pactl load-module module-null-sink \
  sink_name="ExtraSink1" sink_properties=device.description=ExtraSink1

pactl load-module module-loopback \
  source="ExtraSink1.monitor" \
  sink="CombinedSink" \
  source_output_properties="node.description='ExtraSink1ToCombined' node.name='ExtraSink1ToCombined' media.name='ExtraSink1ToCombined'" \
  sink_input_properties="node.description='ExtraSink1ToCombined' node.name='ExtraSink1ToCombined' media.role='filter'" \
  latency_msec=200 adjust_time=3

pactl set-default-source VirtualMic

Here's a demo:

Screencast of using whisper.el to do speech-to-text into the current buffer, clocked-in task, or other function

Transcript

0:00 Inserting into the current buffer

Here's a quick demonstration of using whisper.el to log notes.

0:13 Inserting text and moving on

I can insert text into the current buffer one after the other.

0:31 Clocking in

If I clock into a task, I can add to the end of that clocked in task using my custom code by pressing C-<f9> or whatever my shortcut was. I can do that multiple times.

1:05 Logging a note from a different file

I can do that while looking at a different file.

1:15 I can look at an info page

I can do it looking at an info page, for example, and annotations will include a link back to whatever I was looking at.

1:33 Adding without an annotation (C-u)

I just added an optional argument so that I can also capture a note without saving an annotation. That way, if I'm going to say a lot of things about the same buffer, I don't have to have a lot of links that I need to edit out.

2:42 Saving to a different function

I can also have it save to a different function.

And then I define a global shortcut in KDE that runs xdotool-emacs: 

/home/sacha/bin/xdotool-emacs key --clearmodifiers F9

Here's xdotool-emacs: 

#!/bin/bash
ORIGINAL_WINDOW=$(xdotool getactivewindow)
xdotool search --onlyvisible --class emacs windowactivate --sync
sleep 0.01
xdotool $*
xdotool windowactivate "$ORIGINAL_WINDOW"

So now I can dictate into other applications or save into Emacs. Which suggests of course that I should get it working with C-f9 as well, if I can avoid the keyboard shortcut loop…

This is part of my Emacs configuration.

You can comment on Mastodon or e-mail me at sacha@sachachua.com.

]]>