Using whisper.el to convert speech to text and save it to the currently clocked task in Org Mode or elsewhere

Posted: - Modified: | emacs, audio, speech-recognition, speech
  • : I now use Orukeet as the speech recognition model because it's faster.
  • : Major change: I switched to my fork of natrys/whisper.el so that I can specify functions that change the window configuration etc.
  • : Change main function to sacha-whisper-run, use seq-reduce to go through the functions.
  • : Added code for automatically capturing screenshots, saving text, working with a list of functions.
  • : Added demo, fixed some bugs.
  • : Added note about difference from MELPA package, fixed :vc

I want to get my thoughts into the computer quickly, and talking might be a good way to do some of that. whisper.el gives me a convenient way to interact with a server compatible with OpenAI Whisper's API. (Experimenting with Orukeet for speech recognition) Note: This is not the same whisper package as the one on MELPA. Here is how I have it set up for reasonable performance on my Lenovo P52 with just the CPU, no GPU.

I've bound <f9> to the command whisper-run in Emacs, but I use my window manager's custom shortcuts to map <kp-0> to execute /home/sacha/bin/xdotool-emacs key --clearmodifiers F9, which sends <f9> to Emacs. (file:///home/sacha/bin/xdotool-emacs) That makes it a shortcut that works even in other applications.

I press <kp-0> to start recording, talk, and then press <kp-0> to stop recording. By default, it inserts the text into the buffer at the current point. I've set whisper-return-cursor-to-start to nil so that I can keep going. 

(use-package whisper
  :vc (:url "https://github.com/natrys/whisper.el")
  :load-path "~/vendor/whisper.el"
  :config
  (setq whisper--mode-line-recording-indicator "⏺")
  (setq whisper-quantize "q4_0")
  (setq whisper-install-directory "~/vendor")
  (setq whisper--install-path (concat
     (expand-file-name (file-name-as-directory whisper-install-directory))
     "whisper.cpp/"))
  (setq whisper-return-cursor-to-start nil)
  (setq whisper--ffmpeg-input-device "VirtualMicSink.monitor")
  (setq whisper-language "en")
  (setq whisper-recording-timeout 3000)
  (setq whisper-before-transcription-hook nil)
  (setq whisper-use-threads (1- (num-processors)))
  (setq whisper-transcription-buffer-name-function 'whisper--simple-transcription-buffer-name)
  (add-hook 'whisper-after-transcription-hook 'sacha-subed-fix-common-errors-from-start -100)
  :bind
  (("<f9>" . whisper-run)
   ("C-<f9>" . sacha-whisper-run)
   ("S-<f2>" . whisper-run)
   ("S-<f9>" . sacha-whisper-replay)
   ("M-<f9>" . sacha-whisper-toggle-language)))

: "Remind me to…" is now handled by https://codeberg.org/sachac/.emacs.d/src/branch/gh-pages/lisp/.user-lisp-autoloads.el.

Disk space is inexpensive and backups are great, so let's save each file using the timestamp. 

(defvar sacha-whisper-dir "~/recordings/whisper/")
;;;###autoload
(defun sacha-whisper-set-temp-filename ()
  (setq whisper--temp-file (expand-file-name
                            (format-time-string "%Y-%m-%d-%H-%M-%S.wav")
                            sacha-whisper-dir)))


(with-eval-after-load 'whisper
  (add-hook 'whisper-before-transcription-hook #'sacha-whisper-set-temp-filename))

I can replay the most recently recorded snippet in case it's missed something and I've forgotten what I just said. 

;;;###autoload
(defun sacha-whisper-replay (&optional file)
  "Replay the last temporary recording."
  (interactive (list
                (when current-prefix-arg
                  (read-file-name "File: " sacha-whisper-dir))))
  (setq whisper--temp-file (or file whisper--temp-file))
  (mpv-play whisper--temp-file))

;;;###autoload
(defun sacha-whisper-insert-retry (&optional file)
  (interactive (list
                (when current-prefix-arg
                  (read-file-name "File: " sacha-whisper-dir))))
  (whisper--cleanup-transcription)
  (setq whisper--marker (point-marker)
        whisper--temp-file (or file whisper--temp-file))
  (whisper--transcribe-audio))

Update : Il peut aussi comprendre le français. I used to have code to set the language in case the auto-recognition doesn't get it right, but now I'm using Orukeet as the speech recognition engine now, so it doesn't take a language setting. 

;;;###autoload
(defun sacha-whisper-reset (text)
  (setq sacha-whisper-skip-annotation nil)
  (remove-hook 'whisper-insert-text-at-point #'sacha-whisper-org-save-to-clocked-task)
  text)

;; Only works with my tweaks to whisper.el
;; https://github.com/sachac/whisper.el/tree/whisper-insert-text-at-point-function
(with-eval-after-load 'whisper
  (setq whisper-insert-text-at-point
        '(sacha-whisper-scratch-that
          sacha-whisper-handle-commands
          sacha-whisper-save-text
          sacha-learn-lang-notice-bingo-words
;          sacha-whisper-save-to-file
          sacha-save-to-kill-ring-after-current
          sacha-whisper-maybe-expand-snippet
          sacha-speech-input-quantified-track
          sacha-whisper-maybe-type
          sacha-whisper-maybe-type-with-hints
          sacha-whisper-insert
          sacha-whisper-reset)))

(defvar sacha-whisper-last-annotation nil "Last annotation so we can skip duplicates.")
(defvar sacha-whisper-skip-annotation nil)
(defvar sacha-whisper-target-markers nil "List of markers to send text to.")

;;;###autoload
(defun sacha-whisper-insert (text)
  (let ((markers
         (cond
          ((null sacha-whisper-target-markers)
           (list whisper--marker)) ; current point where whisper was started
          ((listp sacha-whisper-target-markers)
           sacha-whisper-target-markers)
          ((markerp sacha-whisper-target-markers)
           (list sacha-whisper-target-markers))))
        (orig-point (point))
        (orig-buffer (current-buffer)))
    (when text
      (mapcar (lambda (marker)
                (with-current-buffer (marker-buffer marker)
                  (save-restriction
                    (widen)
                    (when (markerp marker) (goto-char marker))
                    (when (and (derived-mode-p 'org-mode) (org-at-drawer-p))
                      (insert "\n"))
                    (whisper--insert-text
                     (concat
                      (if (looking-back "[ \t\n]\\|^")
                          ""
                        " ")
                      (string-trim text)))
                    ;; Move the marker forward here
                    (move-marker marker (point)))))
              markers)
      (when sacha-whisper-target-markers
        (goto-char orig-point))
      nil)))

;;;###autoload
(defun sacha-whisper-maybe-type (text)
  "If Emacs is not the focused app, simulate typing TEXT.
Add this function to `whisper-insert-text-at-point'."
  (when text
    (if (frame-focus-state)
        text
      (make-process :name "xdotool" :command
                    (list "xdotool" "type"
                          text))
      nil)))

;;;###autoload
(defun sacha-whisper-clear-markers ()
  (interactive)
  (setq sacha-whisper-target-markers nil))

;;;###autoload
(defun sacha-whisper-use-current-point (&optional add)
  (interactive (list current-prefix-arg))
  (if add
      (push (point-marker) sacha-whisper-target-markers)
    (setq sacha-whisper-target-markers (list (point-marker)))))

;;;###autoload
(defun sacha-whisper-run-at-point (&optional add)
  (interactive (list current-prefix-arg))
  (sacha-whisper-clear-markers)
  (whisper-run))


(keymap-global-set "<f9>" #'sacha-whisper-run-at-point)
(keymap-global-set "<kp-1>" #'whisper-run)

;;;###autoload
(defun sacha-whisper-jump-to-marker ()
  (interactive)
  (with-current-buffer (marker-buffer (car sacha-whisper-target-markers))
    (goto-char (car sacha-whisper-target-markers))))

;;;###autoload
(defun sacha-whisper-use-currently-clocked-task (&optional add)
  (interactive (list current-prefix-arg))
  (save-window-excursion
    (save-restriction
      (save-excursion
        (org-clock-goto)
        (org-end-of-meta-data)
        (org-end-of-subtree)
        (if add
            (push (point-marker) sacha-whisper-target-markers)
          (setq sacha-whisper-target-markers (list (point-marker))))))))

;;;###autoload
(defun sacha-whisper-run (&optional skip-annotation)
  (interactive (list current-prefix-arg))
  (require 'whisper)
  (add-hook 'whisper-insert-text-at-point #'sacha-whisper-org-save-to-clocked-task -10)
  (whisper-run)
  (when skip-annotation
    (setq sacha-whisper-skip-annotation t)))

;;;###autoload
(defun sacha-whisper-save-text (text)
  "Save TEXT beside `whisper--temp-file'."
  (when text
    (let ((link (org-store-link nil)))
      (with-temp-file (concat (file-name-sans-extension whisper--temp-file) ".txt")
        (when link
          (insert link "\n"))
        (insert text)))
    text))

;;;###autoload
(defun sacha-whisper-org-save-to-clocked-task (text)
  (when text
    (save-window-excursion
      (with-current-buffer (if (markerp whisper--marker) (marker-buffer whisper--marker) (current-buffer))
        (when (markerp whisper--marker) (goto-char whisper--marker))
        ;; Take a screenshot maybe
        (let* ((link (and (not sacha-whisper-skip-annotation)
                          (org-store-link nil)))
               (region (and (region-active-p) (buffer-substring (region-beginning) (region-end))))
               (screenshot-filename
                (when (or
                       (null link)
                       (not (string= sacha-whisper-last-annotation link))
                       (not (frame-focus-state))) ; not in focus, take a screenshot
                  (sacha-screenshot-current-screen (concat (file-name-sans-extension whisper--temp-file) ".png")))))
          (if (org-clocking-p)
              (save-window-excursion
                (save-restriction
                  (save-excursion
                    (org-clock-goto)
                    (org-end-of-subtree)
                    (unless (bolp)
                      (insert "\n"))
                    (insert "\n")
                    (if (and link (not (string= sacha-whisper-last-annotation link)))
                        (insert
                         (if screenshot-filename
                             (concat "(" (org-link-make-string
                                          (concat "file:" screenshot-filename)
                                          "screenshot") ") ")
                           "")
                         link
                         "\n")
                      (when screenshot-filename
                        (insert (org-link-make-string
                                 (concat "file:" screenshot-filename)
                                 "screenshot")
                                "\n")))
                    (when region
                      (insert "#+begin_example\n" region "\n#+end_example\n"))
                    (insert text "\n")
                    (setq sacha-whisper-last-annotation link)))
                (run-at-time 0.5 nil (lambda (text) (message "Added clock note: %s" text)) text))
            ;; No clocked task, prompt for a place to capture it
            (kill-new text)
            (setq org-capture-initial text)
            (call-interactively 'org-capture)
            ;; Delay the window configuration
            (let ((config (current-window-configuration)))
              (run-at-time 0.5 nil
                           (lambda (text config)
                             (set-window-configuration config)
                             (message "Copied: %s" text))
                           text config))))))))


(with-eval-after-load 'org
  (add-hook 'org-clock-in-hook #'sacha-whisper-org-clear-saved-annotation))

;;;###autoload
(defun sacha-whisper-org-clear-saved-annotation ()
  (setq sacha-whisper-org-last-annotation nil))

Here's an idea for a function that saves the recognized text with a timestamp. 

(defvar sacha-whisper-notes "~/sync/stream/narration.org")
;;;###autoload
(defun sacha-whisper-save-to-file (text)
  (when text
    (let ((link (org-store-link nil)))
      (with-current-buffer (find-file-noselect sacha-whisper-notes)
        (goto-char (point-max))
        (insert "\n\n" (format-time-string "%H:%M ") text "\n" (if link (concat link "\n") ""))
        (save-buffer)
        (run-at-time 0.5 nil (lambda (text) (message "Saved to file: %s" text)) text)))
    text))

Let's add it to the kill ring, but not at the most recent slot. That way, it's easy for me to get to, but it doesn't overwrite anything that I've copied. 

;;;###autoload
(defun sacha-save-to-kill-ring-after-current (text)
  "Save TEXT to the kill ring, but not at the top spot."
  (when text
    (if kill-ring
        (let ((temp (pop kill-ring)))
          (kill-new text)
          (push temp kill-ring))
      (kill-new text)))
  text)

And now I can redo things if needed: 

;;;###autoload
(defun sacha-whisper-redo ()
  (interactive)
  (setq whisper--marker (point-marker))
  (whisper--transcribe-audio))

I think I've just figured out my Pipewire setup so that I can record audio in OBS while also being able to do speech to text, without the audio stuttering. qpwgraph was super helpful for visualizing the Pipewire connections and fixing them. 


flatpak run com.github.wwmm.easyeffects -q 2>/dev/null
systemctl --user restart pipewire pipewire-pulse wireplumber
sleep 5
nohup flatpak run com.github.wwmm.easyeffects --gapplication-service >/dev/null 2>&1 &

pactl unload-module module-null-sink 2>/dev/null
pactl unload-module module-loopback 2>/dev/null
pactl unload-module module-remap-source 2>/dev/null

pactl load-module module-null-sink \
  sink_name="VirtualMicSink" \
  sink_properties=device.description="VirtualMic_backend"

pactl load-module module-remap-source \
  source_name="VirtualMic" \
  master="VirtualMicSink.monitor" \
  source_properties="device.description='Actual_Virtual_Mic' device.icon_name='audio-input-microphone' device.class='filter'"

pactl load-module module-null-sink \
  sink_name="CombinedSink" \
  sink_properties=device.description=CombinedSink


if pactl list short sources | grep -i pci-0000; then
  pactl load-module module-loopback \
    source="alsa_input.pci-0000_00_1f.3.analog-stereo" \
    sink="VirtualMicSink" \
    latency_msec=200 \
    adjust_time=3 \
    source_output_properties="node.description='SysToVMic' node.name='SysToVMic' media.name='SysVToMic'" \
    sink_input_properties="node.description='SysToVMic' node.name='SysToVMic' media.role='filter'"
  pactl load-module module-loopback \
    source="alsa_input.pci-0000_00_1f.3.analog-stereo" \
    sink="CombinedSink" \
    latency_msec=200 \
    adjust_time=3 \
    source_output_properties="node.description='SysToCombined' node.name='SysToCombined' media.name='SysVToMic'" \
    sink_input_properties="node.description='SysToCombined' node.name='SysToCombined' media.role='filter'"
fi

if pactl list short sources | grep -i USB_Advanced; then
    # YETI=alsa_input.usb-C-Media_Electronics_Inc._USB_Advanced_Audio_Device-00.analog-stereo
    YETI_OUT=alsa_output.usb-C-Media_Electronics_Inc._USB_Advanced_Audio_Device-00.analog-stereo
    YETI=easyeffects_sink.monitor
  # pactl load-module module-loopback \
  #   source="$YETI" \
  #   sink="VirtualMicSink" \
  #   latency_msec=200 \
  #   adjust_time=3 \
  #   source_output_properties="node.description='YetiToVMic' node.name='YetiToVMic' media.name='YetiToVMic'" \
  #   sink_input_properties="node.description='YetiToVMic' node.name='YetiToVMic' media.role='filter'"
  pactl load-module module-loopback \
    source="$YETI" \
    sink="VirtualMicSink" \
    latency_msec=200 \
    adjust_time=3 \
    source_output_properties="node.description='YetiToVMic' node.name='YetiToVMic' media.name='YetiToVMic'" \
    sink_input_properties="node.description='YetiToVMic' node.name='YetiToVMic' media.role='filter'"
  pactl load-module module-loopback \
    source="$YETI" \
    sink="CombinedSink" \
    latency_msec=200 \
    adjust_time=3 \
    source_output_properties="node.description='SysToCombined' node.name='SysToCombined' media.name='SysVToMic'" \
    sink_input_properties="node.description='SysToCombined' node.name='SysToCombined' media.role='filter'"
  pactl load-module module-null-sink sink_name=MeetSink sink_properties=device.description="Google_Meet_Audio"
  pactl load-module module-loopback source="$YETI" sink=MeetSink

  pactl load-module module-combine-sink sink_name=SnippetBridge slaves="MeetSink,$YETI_OUT" label="Snippet Bridge"
fi

pactl load-module module-null-sink \
  sink_name="ExtraSink1" sink_properties=device.description=ExtraSink1

pactl load-module module-loopback \
  source="ExtraSink1.monitor" \
  sink="CombinedSink" \
  source_output_properties="node.description='ExtraSink1ToCombined' node.name='ExtraSink1ToCombined' media.name='ExtraSink1ToCombined'" \
  sink_input_properties="node.description='ExtraSink1ToCombined' node.name='ExtraSink1ToCombined' media.role='filter'" \
  latency_msec=200 adjust_time=3

pactl set-default-source VirtualMic

Here's a demo:

Screencast of using whisper.el to do speech-to-text into the current buffer, clocked-in task, or other function

Transcript

0:00 Inserting into the current buffer

Here's a quick demonstration of using whisper.el to log notes.

0:13 Inserting text and moving on

I can insert text into the current buffer one after the other.

0:31 Clocking in

If I clock into a task, I can add to the end of that clocked in task using my custom code by pressing C-<f9> or whatever my shortcut was. I can do that multiple times.

1:05 Logging a note from a different file

I can do that while looking at a different file.

1:15 I can look at an info page

I can do it looking at an info page, for example, and annotations will include a link back to whatever I was looking at.

1:33 Adding without an annotation (C-u)

I just added an optional argument so that I can also capture a note without saving an annotation. That way, if I'm going to say a lot of things about the same buffer, I don't have to have a lot of links that I need to edit out.

2:42 Saving to a different function

I can also have it save to a different function.

And then I define a global shortcut in KDE that runs xdotool-emacs: 

/home/sacha/bin/xdotool-emacs key --clearmodifiers F9

Here's xdotool-emacs: 

#!/bin/bash
ORIGINAL_WINDOW=$(xdotool getactivewindow)
xdotool search --onlyvisible --class emacs windowactivate --sync
sleep 0.01
xdotool $*
xdotool windowactivate "$ORIGINAL_WINDOW"

So now I can dictate into other applications or save into Emacs. Which suggests of course that I should get it working with C-f9 as well, if I can avoid the keyboard shortcut loop…

This is part of my Emacs configuration.
View Org source for this post
You can comment on Mastodon or e-mail me at sacha@sachachua.com.
More posts about emacs, audio, speech-recognition, speech