Categories: spookfox

RSS - Atom - Subscribe via email

Running the current Org Mode Babel Javascript block from Emacs using Spookfox

| emacs, org, spookfox

I often want to send Javascript from Emacs to the web browser. It's handy for testing code snippets or working with data on pages that require Javascript or authentication. I could start Google Chrome or Mozilla Firefox with their remote debugging protocols, copy the websocket URLs, and talk to the browser through something like Puppeteer, but it's so much easier to use the Spookfox extension for Mozilla to execute code in the active tab. spookfox-js-injection-eval-in-active-tab lets you evaluate Javascript and get the results back in Emacs Lisp.

I wanted to be able to execute code even more easily. This code lets me add a :spookfox t parameter to Org Babel Javascript blocks so that I can run the block in my Firefox active tab. For example, if I have (spookfox-init) set up, Spookfox connected, and https://planet.emacslife.com in my active tab, I can use it with the following code:

#+begin_src js :eval never-export :spookfox t :exports results
[...document.querySelectorAll('.post > h2')].slice(0,5).map((o) => '- ' + o.textContent.trim().replace(/[ \n]+/g, ' ') + '\n').join('')
#+end_src
  • Mario Jason Braganza: Updated to Emacs 29.2
  • Irreal: Zamansky: Learning Elisp #16
  • Tim Heaney: Lisp syntax
  • Erik L. Arneson: Many Posts of Interest for January 2024
  • William Denton: Basic citations in Org (Part 4)

Evaluating a Javascript block with :spookfox t

To do this, we wrap some advice around the org-babel-execute:js function that's called by org-babel-execute-src-block.

(defun my-org-babel-execute:js-spookfox (old-fn body params)
  "Maybe execute Spookfox."
  (if (assq :spookfox params)
      (spookfox-js-injection-eval-in-active-tab
       body t)
    (funcall old-fn body params)))
(with-eval-after-load 'ob-js
  (advice-add 'org-babel-execute:js :around #'my-org-babel-execute:js-spookfox))

I can also run the block in Spookfox without adding the parameter if I make an interactive function:

(defun my-spookfox-eval-org-block ()
  (interactive)
  (let ((block (org-element-context)))
    (when (and (eq (org-element-type block) 'src-block)
               (string= (org-element-property :language block) "js"))
      (spookfox-js-injection-eval-in-active-tab
       (nth 2 (org-src--contents-area block))
       t))))

I can add that as an Embark context action:

(with-eval-after-load 'embark-org
  (define-key embark-org-src-block-map "f" #'my-spookfox-eval-org-block))

In Javascript buffers, I want the ability to send the current line, region, or buffer too, just like nodejs-repl does.

(defun my-spookfox-send-region (start end)
  (interactive "r")
  (spookfox-js-injection-eval-in-active-tab (buffer-substring start end) t))

(defun my-spookfox-send-buffer ()
  (interactive)
  (my-spookfox-send-region (point-min) (point-max)))

(defun my-spookfox-send-line ()
  (interactive)
  (my-spookfox-send-region (line-beginning-position) (line-end-position)))

(defun my-spookfox-send-last-expression ()
  (interactive)
  (my-spookfox-send-region (save-excursion (nodejs-repl--beginning-of-expression)) (point)))

(defvar-keymap my-js-spookfox-minor-mode-map
  :doc "Send parts of the buffer to Spookfox."
  "C-x C-e" 'my-spookfox-send-last-expression
  "C-c C-j" 'my-spookfox-send-line
  "C-c C-r" 'my-spookfox-send-region
  "C-c C-c" 'my-spookfox-send-buffer)

(define-minor-mode my-js-spookfox-minor-mode "Send code to Spookfox.")

I usually edit Javascript files with js2-mode, so I can use my-js-spookfox-minor-mode in addition to that.

I can turn the minor mode on automatically for :spookfox t source blocks. There's no org-babel-edit-prep:js yet, I think, so we need to define it instead of advising it.

(defun org-babel-edit-prep:js (info)
  (when (assq :spookfox (nth 2 info))
    (my-js-spookfox-minor-mode 1)))

Let's try it out by sending the last line repeatedly:

Sending the current line

I used to do this kind of interaction with Skewer, which also has some extra stuff for evaluating CSS and HTML. Skewer hasn't been updated in a while, but maybe I should also check that out again to see if I can get it working.

Anyway, now it's just a little bit easier to tinker with Javascript!

View org source for this post
This is part of my Emacs configuration.

EmacsConf backstage: Using Spookfox to publish YouTube and Toobnix video drafts

| emacsconf, emacs, spookfox, youtube, video

I ran into quota limits when uploading videos to YouTube with a command-line tool, so I uploaded videos by selecting up to 15 videos at a time using the web-based interface. Each video was a draft, though, and I was having a hard time updating its visibility through the API. I think it eventually worked, but in the meantime, I used this very hacky hack to look for the "Edit Draft" button and click through the screens to publish them.

emacsconf-extract-youtube-publish-video-drafts-with-spookfox: Look for drafts and publish them.
(defun emacsconf-extract-youtube-publish-video-drafts-with-spookfox ()
  "Look for drafts and publish them."
  (while (not (eq (spookfox-js-injection-eval-in-active-tab
                   "document.querySelector('.edit-draft-button div') != null" t) :false))
    (progn
      (spookfox-js-injection-eval-in-active-tab
       "document.querySelector('.edit-draft-button div').click()" t)
      (sleep-for 2)
      (spookfox-js-injection-eval-in-active-tab
       "document.querySelector('#step-title-3').click()" t)
      (when (spookfox-js-injection-eval-in-active-tab
             "document.querySelector('tp-yt-paper-radio-button[name=\"PUBLIC\"] #radioLabel').click()" t)
        (spookfox-js-injection-eval-in-active-tab
         "document.querySelector('#done-button').click()" t)
        (while (not (eq  (spookfox-js-injection-eval-in-active-tab
                          "document.querySelector('#close-button .label') == null" t)
                         :false))
          (sleep-for 1))

        (spookfox-js-injection-eval-in-active-tab
         "document.querySelector('#close-button .label').click()" t)
        (sleep-for 1)))))

Another example of a hacky Spookfox workaround was publishing the unlisted videos. I couldn't figure out how to properly authenticate with the Toobnix (Peertube) API to change the visibility of videos. Peertube uses AngularJS components in the front end, so using .click() on the input elements didn't seem to trigger anything. I found out that I needed to use .dispatchEvent(new Event('input')) to tell the dropdown for the visibility to display the options. source

emacsconf-extract-toobnix-publish-video-from-edit-page: Messy hack to set a video to public and store the URL.
(defun emacsconf-extract-toobnix-publish-video-from-edit-page ()
  "Messy hack to set a video to public and store the URL."
  (interactive)
  (spookfox-js-injection-eval-in-active-tab "document.querySelector('label[for=privacy]').scrollIntoView(); document.querySelector('label[for=privacy]').closest('.form-group').querySelector('input').dispatchEvent(new Event('input'));" t)
  (sit-for 1)
  (spookfox-js-injection-eval-in-active-tab "document.querySelector('span[title=\"Anyone can see this video\"]').click()" t)
  (sit-for 1)
  (spookfox-js-injection-eval-in-active-tab "document.querySelector('button.orange-button').click()" t)(sit-for 3)
  (emacsconf-extract-store-url)
  (shell-command "xdotool key Alt+Tab sleep 1 key Ctrl+w Alt+Tab"))

It's a little nicer using Spookfox to automate browser interactions than using xdotool, since I can get data out of it too. I could also have used Puppeteer from either Python or NodeJS, but it's nice staying with Emacs Lisp. Spookfox has some Javascript limitations (can't close windows, etc.), so I might still use bits of xdotool or Puppeteer to work around that. Still, it's nice to now have an idea of how to talk to AngularJS components.

EmacsConf backstage: Making a (play)list, checking it twice

| emacs, emacsconf, spookfox, youtube, video

I wanted the EmacsConf 2023 Youtube and Toobnix playlists to mostly reflect the schedule of the conference by track, with talks followed by their Q&A sessions (if recorded).

The list

I used Emacs Lisp to generate a list of videos in the order I wanted. That Sunday closing remarks aren't actually in the playlists because they're combined with the Q&A for my session on how we run Emacsconf.

emacsconf-extract-check-playlists: Return a table for checking playlist order.
(defun emacsconf-extract-check-playlists ()
  "Return a table for checking playlist order."
  (let ((pos 0))
    (seq-mapcat (lambda (o)
                  (delq
                   nil
                   (list
                    (when (emacsconf-talk-file o "--main.webm")
                      (cl-incf pos)
                      (list pos
                            (plist-get o :title)
                            (org-link-make-string
                             (plist-get o :youtube-url)
                             "YouTube")
                            (org-link-make-string
                             (plist-get o :toobnix-url)
                             "Toobnix")))
                    (when (emacsconf-talk-file o "--answers.webm")
                      (cl-incf pos)
                      (list pos (concat "Q&A: " (plist-get o :title))
                            (org-link-make-string
                             (plist-get o :qa-youtube-url)
                             "YouTube")
                            (org-link-make-string
                             (plist-get o :qa-toobnix-url)
                             "Toobnix"))))))
                (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info)))))

1 An Org-Mode based text adventure game for learning the basics of Emacs, inside Emacs, written in Emacs Lisp YouTube Toobnix
2 Authoring and presenting university courses with Emacs and a full libre software stack YouTube Toobnix
3 Q&A: Authoring and presenting university courses with Emacs and a full libre software stack YouTube Toobnix
4 Teaching computer and data science with literate programming tools YouTube Toobnix
5 Q&A: Teaching computer and data science with literate programming tools YouTube Toobnix
6 Who needs Excel? Managing your students qualifications with org-table YouTube Toobnix
7 one.el: the static site generator for Emacs Lisp Programmers YouTube Toobnix
8 Q&A: one.el: the static site generator for Emacs Lisp Programmers YouTube Toobnix
9 Emacs turbo-charges my writing YouTube Toobnix
10 Q&A: Emacs turbo-charges my writing YouTube Toobnix
11 Why Nabokov would use Org-Mode if he were writing today YouTube Toobnix
12 Q&A: Why Nabokov would use Org-Mode if he were writing today YouTube Toobnix
13 Collaborative data processing and documenting using org-babel YouTube Toobnix
14 How I play TTRPGs in Emacs YouTube Toobnix
15 Q&A: How I play TTRPGs in Emacs YouTube Toobnix
16 Org-Mode workflow: informal reference tracking YouTube Toobnix
17 (Un)entangling projects and repos YouTube Toobnix
18 Emacs development updates YouTube Toobnix
19 Emacs core development: how it works YouTube Toobnix
20 Top 10 ways Hyperbole amps up Emacs YouTube Toobnix
21 Using Koutline for stream of thought journaling YouTube Toobnix
22 Parallel text replacement YouTube Toobnix
23 Q&A: Parallel text replacement YouTube Toobnix
24 Eat and Eat powered Eshell, fast featureful terminal inside Emacs YouTube Toobnix
25 The browser in a buffer YouTube Toobnix
26 Speedcubing in Emacs YouTube Toobnix
27 Emacs MultiMedia System (EMMS) YouTube Toobnix
28 Q&A: Emacs MultiMedia System (EMMS) YouTube Toobnix
29 Programming with steno YouTube Toobnix
30 Mentoring VS-Coders as an Emacsian (or How to show not tell people about the wonders of Emacs) YouTube Toobnix
31 Q&A: Mentoring VS-Coders as an Emacsian (or How to show not tell people about the wonders of Emacs) YouTube Toobnix
32 Emacs saves the Web (maybe) YouTube Toobnix
33 Q&A: Emacs saves the Web (maybe) YouTube Toobnix
34 Sharing Emacs is Caring Emacs: Emacs education and why I embraced video YouTube Toobnix
35 Q&A: Sharing Emacs is Caring Emacs: Emacs education and why I embraced video YouTube Toobnix
36 MatplotLLM, iterative natural language data visualization in org-babel YouTube Toobnix
37 Enhancing productivity with voice computing YouTube Toobnix
38 Q&A: Enhancing productivity with voice computing YouTube Toobnix
39 LLM clients in Emacs, functionality and standardization YouTube Toobnix
40 Q&A: LLM clients in Emacs, functionality and standardization YouTube Toobnix
41 Improving compiler diagnostics with overlays YouTube Toobnix
42 Q&A: Improving compiler diagnostics with overlays YouTube Toobnix
43 Editor Integrated REPL Driven Development for all languages YouTube Toobnix
44 REPLs in strange places: Lua, LaTeX, LPeg, LPegRex, TikZ YouTube Toobnix
45 Literate Documentation with Emacs and Org Mode YouTube Toobnix
46 Q&A: Literate Documentation with Emacs and Org Mode YouTube Toobnix
47 Windows into Freedom YouTube Toobnix
48 Bringing joy to Scheme programming YouTube Toobnix
49 Q&A: Bringing joy to Scheme programming YouTube Toobnix
50 GNU Emacs: A World of Possibilities YouTube Toobnix
51 Q&A: GNU Emacs: A World of Possibilities YouTube Toobnix
52 A modern Emacs look-and-feel without pain YouTube Toobnix
53 The Emacsen family, the design of an Emacs and the importance of Lisp YouTube Toobnix
54 Q&A: The Emacsen family, the design of an Emacs and the importance of Lisp YouTube Toobnix
55 emacs-gc-stats: Does garbage collection actually slow down Emacs? YouTube Toobnix
56 Q&A: emacs-gc-stats: Does garbage collection actually slow down Emacs? YouTube Toobnix
57 hyperdrive.el: Peer-to-peer filesystem in Emacs YouTube Toobnix
58 Q&A: hyperdrive.el: Peer-to-peer filesystem in Emacs YouTube Toobnix
59 Writing a language server in OCaml for Emacs, fun, and profit YouTube Toobnix
60 Q&A: Writing a language server in OCaml for Emacs, fun, and profit YouTube Toobnix
61 What I learned by writing test cases for GNU Hyperbole YouTube Toobnix
62 Q&A: What I learned by writing test cases for GNU Hyperbole YouTube Toobnix
63 EmacsConf.org: How we use Org Mode and TRAMP to organize and run a multi-track conference YouTube Toobnix
64 Q&A: EmacsConf.org: How we use Org Mode and TRAMP to organize and run a multi-track conference YouTube Toobnix
65 Saturday opening remarks YouTube Toobnix
66 Saturday closing remarks YouTube Toobnix
67 Sunday opening remarks YouTube Toobnix
68 Sunday closing remarks YouTube Toobnix

YouTube

I bulk-added the Youtube videos to the playlist. The videos were not in order because I uploaded some late submissions and forgotten videos, which then got added to the end of the list.

I tried using the API to sort the playlist. This got it most of the way there, and then I sorted the rest by hand.

emacsconf-extract-youtube-api-sort-playlist: Try to roughly sort the playlist.
(defun emacsconf-extract-youtube-api-sort-playlist (&optional dry-run-only)
  "Try to roughly sort the playlist."
  (interactive)
  (setq emacsconf-extract-youtube-api-playlist (seq-find (lambda (o) (let-alist o (string= .snippet.title (concat emacsconf-name " " emacsconf-year))))
                                        (assoc-default 'items emacsconf-extract-youtube-api-playlists)))
  (setq emacsconf-extract-youtube-api-playlist-items
        (emacsconf-extract-youtube-api-paginated-request (concat "https://youtube.googleapis.com/youtube/v3/playlistItems?part=snippet,contentDetails,status&forMine=true&order=date&maxResults=100&playlistId="
                                                (url-hexify-string (assoc-default 'id emacsconf-extract-youtube-api-playlist)))))
  (let* ((playlist-info emacsconf-extract-youtube-api-playlists)
         (playlist-items emacsconf-extract-youtube-api-playlist-items)
         (info (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info)))
         (slugs (seq-map (lambda (o) (plist-get o :slug)) info))
         (position (1- (length playlist-items)))
         result)
    ;; sort items
    (mapc (lambda (talk)
            (when (plist-get talk :qa-youtube-id)
              ;; move the q & a
              (let ((video-object (emacsconf-extract-youtube-find-url-video-in-list
                                   (plist-get talk :qa-youtube-url)
                                   playlist-items)))
                (let-alist video-object
                  (cond
                   ((null video-object)
                    (message "Could not find video for %s" (plist-get talk :slug)))
                   ;; not in the right position, try to move it
                   ((< .snippet.position position)
                    (let ((video-id .id)
                          (playlist-id .snippet.playlistId)
                          (resource-id .snippet.resourceId))
                      (message "Trying to move %s Q&A to %d from %d" (plist-get talk :slug) position .snippet.position)
                      (add-to-list 'result (list (plist-get talk :slug) "answers" .snippet.position position))
                      (unless dry-run-only
                        (plz 'put "https://www.googleapis.com/youtube/v3/playlistItems?part=snippet"
                          :headers `(("Authorization" . ,(url-oauth-auth "https://youtube.googleapis.com/youtube/v3/"))
                                     ("Accept" . "application/json")
                                     ("Content-Type" . "application/json"))
                          :body (json-encode
                                 `((id . ,video-id)
                                   (snippet
                                    (playlistId . ,playlist-id)
                                    (resourceId . ,resource-id)
                                    (position . ,position))))))))))
                (setq position (1- position))))
            ;; move the talk if needed
            (let ((video-object
                   (emacsconf-extract-youtube-find-url-video-in-list
                    (plist-get talk :youtube-url)
                    playlist-items)))
              (let-alist video-object
                (cond
                 ((null video-object)
                  (message "Could not find video for %s" (plist-get talk :slug)))
                 ;; not in the right position, try to move it
                 ((< .snippet.position position)
                  (let ((video-id .id)
                        (playlist-id .snippet.playlistId)
                        (resource-id .snippet.resourceId))
                    (message "Trying to move %s to %d from %d" (plist-get talk :slug) position .snippet.position)
                    (add-to-list 'result (list (plist-get talk :slug) "main" .snippet.position position))
                    (unless dry-run-only
                      (plz 'put "https://www.googleapis.com/youtube/v3/playlistItems?part=snippet"
                        :headers `(("Authorization" . ,(url-oauth-auth "https://youtube.googleapis.com/youtube/v3/"))
                                   ("Accept" . "application/json")
                                   ("Content-Type" . "application/json"))
                        :body (json-encode
                               `((id . ,video-id)
                                 (snippet
                                  (playlistId . ,playlist-id)
                                  (resourceId . ,resource-id)
                                  (position . ,position))))))
                    ))))
              (setq position (1- position))))
          (nreverse info))
    result))

I needed to sort some of the videos manually. Trying to scroll by dragging items to the top of the currently-displayed section of the list was slow, and dropping the item near the top of the list so that I could pick it up again after paging up was a little disorienting. Fortunately, keyboard scrolling with page-up and page-down worked even while dragging an item, so that was what I ended up doing: select the item and then page-up while dragging.

YouTube doesn't display numbers for the playlist positions, but this will add them. The numbers don't dynamically update when the list is reordered, so I just re-ran the code after moving things around.

emacsconf-extract-youtube-spookfox-add-playlist-numbers: Number the playlist for easier checking.
(defun emacsconf-extract-youtube-spookfox-add-playlist-numbers ()
  "Number the playlist for easier checking.
Related: `emacsconf-extract-check-playlists'."
  (interactive)
  (spookfox-js-injection-eval-in-active-tab "[...document.querySelectorAll('ytd-playlist-video-renderer')].forEach((o, i) => { o.querySelector('.number')?.remove(); let div = document.createElement('div'); div.classList.add('number'); div.textContent = i; o.prepend(div) }))" t))

2023-12-11_12-57-25.png
Figure 1: Adding numbers to the Youtube playlist

In retrospect, I could probably have just cleared the playlist and then added the videos using the in the right order instead of fiddling with inserting things.

Toobnix (Peertube)

Toobnix (Peertube) doesn't seem to have a way to bulk-add videos to a playlist (or even to bulk-set their visibility). I started trying to figure out how to use the API, but I got stuck because my token didn't seem to let me access unlisted videos or do other things that required proper authentication. Anyway, I came up with this messy hack to open the videos in sequence and add them to the playlist using Spookfox.

(defun emacsconf-extract-toobnix-set-up-playlist ()
  (interactive)
  (mapcar
   (lambda (o)
     (when (plist-get o :toobnix-url)
       (browse-url (plist-get o :toobnix-url))
       (read-key "press a key when page is loaded")
       (spookfox-js-injection-eval-in-active-tab "document.querySelector('.action-button-save').click()" t)
       (spookfox-js-injection-eval-in-active-tab "document.querySelector('my-peertube-checkbox').click()" t)
       (read-key "press a key when saved to playlist"))
     (when (plist-get o :qa-toobnix-url)
       (browse-url (plist-get o :qa-toobnix-url))
       (read-key "press a key when page is loaded")
       (spookfox-js-injection-eval-in-active-tab "document.querySelector('.action-button-save').click()" t)
       (spookfox-js-injection-eval-in-active-tab "document.querySelector('my-peertube-checkbox').click()" t)
       (read-key "press a key when saved to playlist")))
   (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))))

Maybe next year, I might be able to figure out how to use the APIs to do this stuff automatically.

This code is in emacsconf-extract.el.