Combining Mastodon timelines using mastodon.el

| emacs

I like checking out the #emacs hashtag when I put together Emacs News. In the past, I usually browsed the hashtag timeline on emacs.ch, which also picked up updates from other people that emacs.ch was following. Now that I've moved to @sacha@social.sachachua.com and emacs.ch is winding down, I wanted to see if there was a way for me to see a combined view using mastodon.social's API feed (paging by max_id as needed). I haven't enabled public timeline feeds on my server, so I also need to reuse the OAuth mechanics from mastodon.el.

First, let's start by making a unified timeline. By digging around in mastodon-tl.el, I found that I could easily create a timeline view by passing it a vector of toot JSONs.

(defun my-mastodon-fetch-posts-after (base-url after-date)
  "Page backwards through BASE-URL using max_id for all the posts after AFTER-DATE."
  (require 'plz)
  (require 'mastodon-http)
  (let ((results [])
        (url base-url)
        (use-mastodon-el (not (string-match "^http" base-url)))
        page filtered)
    (while url
      (setq page (if use-mastodon-el
                     (mastodon-http--get-json (mastodon-http--api url) nil :silent)
                   (seq-map (lambda (o)
                              (cons (cons 'external t) o))
                            (plz 'get url :as #'json-read)))
            filtered (seq-filter (lambda (o) (string< after-date (assoc-default 'created_at o)))
                                 page))
      (if filtered
          (progn
            (setq results (seq-concatenate 'vector filtered results)
                  url (concat base-url (if (string-match "\\?" base-url) "&" "?")
                              "max_id=" (number-to-string (1- (string-to-number (assoc-default 'id (elt (last page) 0)))))))
            (message "%s %s" (assoc-default 'created_at (elt (last page) 0)) url))
        (setq url nil)))
    results))

(defun my-mastodon-combined-tag-timeline (later-than tag servers)
  "Display items after LATER-THAN about TAG from SERVERS and the current mastodon.el account."
  (interactive (list
                (org-read-date nil nil nil nil nil "-Mon")
                "#emacs"
                '("mastodon.social" "emacs.ch")))
  (require 'mastodon)
  (require 'mastodon-tl)
  (require 'mastodon-toot)
  (let* ((limit 40)
         (sources (cons (format "timelines/tag/emacs?count=%d" limit)
                        (mapcar (lambda (s)
                                  (format "https://%s/api/v1/timelines/tag/emacs?count=%d" s limit))
                                servers)))
         (combined
          (sort
           (seq-reduce (lambda (prev val)
                         (seq-union prev
                                    (my-mastodon-fetch-posts-after val later-than)
                                    (lambda (a b) (string= (assoc-default 'uri a)
                                                           (assoc-default 'uri b)))))
                       sources [])
           (lambda (a b)
             (string< (assoc-default 'created_at b)
                      (assoc-default 'created_at a))))))
    (with-current-buffer (get-buffer-create "*Combined*")
      (let ((inhibit-read-only t))
        (erase-buffer)
        (mastodon-tl--timeline combined)
        (mastodon-mode))
      (setq mastodon-tl--buffer-spec `(account ,(cons mastodon-active-user mastodon-instance-url) buffer-name ,(buffer-name)))
      (display-buffer (current-buffer)))))

The tricky thing is that boosting and replying in mastodon.el both use the toot IDs instead of the toot URLs, so they only work for toots that came in via my current mastodon.el account. Toots from other timelines might not have been fetched by my server yet. Adding an external property lets me find that in the item_json text property in the timeline buffer. For those toots, I can use (mastodon-url-lookup (mastodon-toot--toot-url)) to open the toot in a new buffer that does allow boosting or replying, which is probably enough for my purposes.

(defun my-mastodon-lookup-toot ()
  (interactive)
  (mastodon-url-lookup (mastodon-toot--toot-url)))

When I go through Emacs News, I have a shortcut ethat boosts a post and saves it to as an Org Mode capture with a link to the toot. I sometimes want to reply, too. So I just need to intervene before boosting and replying. Boosting and favoriting both use mastodon-toot--action, which looks up the base-item-id text property. Replying looks up the item-json property and gets the id from it.

(defun my-text-property-update-at-point (pos prop value)
  (let ((start (previous-single-property-change (or pos (point)) prop))
        (end (next-single-property-change (or pos (point)) prop)))
    (put-text-property (or start (point-min))
                       (or end (point-max))
                       prop value)))

(defun my-mastodon-update-external-item-id (&rest _)
  (when (mastodon-tl--field 'external (mastodon-tl--property 'item-json))
    ;; ask the server to resolve it
    (let* ((response (mastodon-http--get-json (format "%s/api/v2/search" mastodon-instance-url)
                                              `(("q" . ,(mastodon-toot--toot-url))
                                                ("resolve" . "t"))))
           (id (alist-get 'id (seq-first (assoc-default 'statuses response))))
           (inhibit-read-only t)
           (json (get-text-property (point) 'item-json)))
      (when (and id json)
        (my-text-property-update-at-point (point) 'base-item-id id)
        (my-text-property-update-at-point (point) 'item-json
                                          (progn
                                            (setf (alist-get 'id json) id)
                                            (setf (alist-get 'external json) nil)
                                            json))))))

So now all I need to do is make sure that this is called before the relevant mastodon.el functions if I'm looking at an external toot.

(with-eval-after-load 'mastodon-tl
  (advice-add #'mastodon-toot--action :before #'my-mastodon-update-external-item-id)
  (advice-add #'mastodon-toot--reply :before #'my-mastodon-update-external-item-id)
  (advice-add #'mastodon-tl--thread :before #'my-mastodon-update-external-item-id))

The only thing is that I need to press RET after loading a thread with T (mastodon-tl--thread) for some reason, but that's okay. Now I can boost and save posts with my usual Emacs News shortcut, and I can reply easily too.

I'm curious: how many toots would I be missing if I looked at only one instance's hashtag? Let's look at the #emacs hashtag toots on 2024-09-12:

Mastodon comparison
(defun my-three-way-comparison (seq1 seq2 seq3 &optional test-fn)
  `(("1" ,@(seq-difference seq1 (seq-union seq2 seq3 test-fn) test-fn))
    ("2" ,@(seq-difference seq2 (seq-union seq1 seq3 test-fn) test-fn))
    ("3" ,@(seq-difference seq3 (seq-union seq1 seq2 test-fn) test-fn))
    ("1&2" ,@(seq-difference (seq-intersection seq1 seq2 test-fn) seq3 test-fn))
    ("1&3" ,@(seq-difference (seq-intersection seq1 seq3 test-fn) seq2 test-fn))
    ("2&3" ,@(seq-difference (seq-intersection seq2 seq3 test-fn) seq1 test-fn))
    ("1&2&3" ,@(seq-intersection (seq-intersection seq2 seq3 test-fn) seq1 test-fn))))
(defun my-three-way-comparison-report (label1 seq1 label2 seq2 label3 seq3 &optional test-fn)
  (let ((list (my-three-way-comparison seq1 seq2 seq3)))
    `((,(format "%s only" label1) ,@(assoc-default "1" list #'string=))
      (,(format "%s only" label2) ,@(assoc-default "2" list #'string=))
      (,(format "%s only" label3) ,@(assoc-default "3" list #'string=))
      (,(format "%s & %s" label1 label2) ,@(assoc-default "1&2" list #'string=))
      (,(format "%s & %s" label1 label3) ,@(assoc-default "1&3" list #'string=))
      (,(format "%s & %s" label2 label3) ,@(assoc-default "2&3" list #'string=))
      ("all" ,@(assoc-default "1&2&3" list #'string=)))))

(assert (equal (my-three-way-comparison '("A" "A&B" "A&C" "A&B&C" "A1")
                                        '("B" "A&B" "A&B&C" "B&C")
                                        '("C" "A&C" "A&B&C" "B&C"))
               '(("1" "A" "A1")
                 ("2" "B")
                 ("3" "C")
                 ("1&2" "A&B")
                 ("1&3" "A&C")
                 ("2&3" "B&C")
                 ("1&2&3" "A&B&C"))))

(let* ((later-than "2024-09-12")
       (earlier-than "2024-09-13")
       (results
        (mapcar (lambda (o)
                  (cons (car o)
                        (seq-map (lambda (o) (assoc-default 'uri o))
                                 (seq-filter (lambda (toot)
                                               (string< (assoc-default 'created_at toot)
                                                        earlier-than))
                                             (my-mastodon-fetch-posts-after
                                              (format "%stimelines/tag/emacs?count=40" (cdr o))
                                              later-than)))))
                `((mastodon-social . "https://mastodon.social/api/v1/")
                  (emacs-ch . "https://emacs.ch/api/v1/")
                  (my-instance . ""))))
       (intersections
        (let-alist results
          (my-three-way-comparison-report
           "mastodon.social"
           .mastodon-social
           "emacs.ch"
           .emacs-ch
           "my instance"
           .my-instance
           #'string=))))
  (mapcar
   (lambda (row)
     (list (elt row 0) (length (cdr row))
           (string-join
            (seq-map-indexed (lambda (o i)
                               (org-link-make-string o (number-to-string (1+ i))))
                             (cdr row))
            " ")))
   intersections))
mastodon.social only 3 1 2 3
emacs.ch only 1 1
my instance only 0  
mastodon.social & emacs.ch 9 1 2 3 4 5 6 7 8 9
mastodon.social & my instance 0  
emacs.ch & my instance 1 1
all 11 1 2 3 4 5 6 7 8 9 10 11

Here's an Euler diagram visualizing it.

2024-09-13-12-45-10.png
Figure 1: #emacs posts on 2024-09-12 - an Euler diagram showing the table above

I love that I can tinker with mastodon.el to get it to combine the timelines. (I'm crossing the streams!) Yay Emacs!

Moving from @sachac@emacs.ch to @sacha@social.sachachua.com

| geek

tldr: I'm now at @sacha@social.sachachua.com .

Since the emacs.ch Mastodon instance is winding down, I decided to follow @technomancy's recommendation of GoToSocial as a Mastodon-compatibly lightweight ActivityPub surver. I set it up as a Docker container because I still don't trust myself when it comes to security, and I put it behind an Nginx reverse proxy because I already have my web server on port 443.

The issues I ran across were mostly user error. Instead of copying my Nginx config from one of my other subsites, I should probably have just started with the sample Nginx configuration from the documentation. Eventually I figured out that I needed the Host line and I didn't need the location /.well-known/ thing from my other config. I did add add_header 'Access-Control-Allow-Origin' '*'; to allow cross-origin resource sharing (CORS), though, so that I could use semaphore.social as needed.

I also used:

docker network list
docker network inspect gotosocial_gotosocial

to find the IP addresses I needed for my config.yaml:

log-level: "error"
host: "social.sachachua.com"
landing-page-user: "sacha"
letsencrypt-enabled: false
port: 8080
bind-address: "172.22.0.2"
trusted-proxies:
  - "127.0.0.1/32"
  - "172.22.0.1"
  - "172.22.0.2"

I don't have a lot of space left on my Linode virtual private server, so I set up GoToSocial in a 10 GB block storage volume (of which it is now using 2.5GB). I have backups enabled, but the automatic backups don't cover block storage. I think I can work around that with a crontab entry that copies the sqlite* files from gotosocial/data into a backup directory in a regular partition.

I like using mastodon.el to go through the #emacs hashtag to find interesting posts and conversations for Emacs News. Since I have a single-user instance, I might not find as many #emacs posts on my own server compared to emacs.ch or mastodon.social #emacs. GoToSocial doesn't support relays yet. Until I feel reasonably confident that I'm catching a pretty good selection of #emacs posts from the people I follow and the instances I federate with, I might also check against those other timelines.

I think I've gotten @sacha@sachachua.com pointing to @sacha@social.sachachua.com, so if you'd like to follow me on Mastodon or another ActivityPub-compatible service, you can add either ID. I usually toot links to my Emacs News posts. I'd like to do a little more microblogging to share what I'm learning and figuring out, since I notice my blog posts tend to get sidetracked by yak-shaving. =)

2024-09-09 Emacs news

| emacs, emacs-news

Links from reddit.com/r/emacs, r/orgmode, r/spacemacs, r/planetemacs, Mastodon #emacs, Hacker News, lobste.rs, programming.dev, lemmy.world, lemmy.ml, communick.news, planet.emacslife.com, YouTube, the Emacs NEWS file, Emacs Calendar, and emacs-devel. Thanks to Andrés Ramírez for emacs-devel links. Do you have an Emacs-related link or announcement? Please e-mail me at sacha@sachachua.com. Thank you!

2024-09-02 Emacs news

| emacs, emacs-news

Links from reddit.com/r/emacs, r/orgmode, r/spacemacs, r/planetemacs, Mastodon #emacs, Hacker News, lobste.rs, programming.dev, lemmy.world, lemmy.ml, communick.news, planet.emacslife.com, YouTube, the Emacs NEWS file, Emacs Calendar, and emacs-devel. Thanks to Andrés Ramírez for emacs-devel links. Do you have an Emacs-related link or announcement? Please e-mail me at sacha@sachachua.com. Thank you!

2024-08-26 Emacs news

| emacs, emacs-news

Links from reddit.com/r/emacs, r/orgmode, r/spacemacs, r/planetemacs, Mastodon #emacs, Hacker News, lobste.rs, programming.dev, lemmy.world, lemmy.ml, communick.news, planet.emacslife.com, YouTube, the Emacs NEWS file, Emacs Calendar, and emacs-devel. Thanks to Andrés Ramírez for emacs-devel links. Do you have an Emacs-related link or announcement? Please e-mail me at sacha@sachachua.com. Thank you!

2024-08-19 Emacs news

| emacs, emacs-news

Links from reddit.com/r/emacs, r/orgmode, r/spacemacs, r/planetemacs, Mastodon #emacs, Hacker News, lobste.rs, programming.dev, lemmy.world, lemmy.ml, communick.news, planet.emacslife.com, YouTube, the Emacs NEWS file, Emacs Calendar, and emacs-devel. Thanks to Andrés Ramírez for emacs-devel links. Do you have an Emacs-related link or announcement? Please e-mail me at sacha@sachachua.com. Thank you!

Turning 41; life as a 40-year-old

| yearly, review

Text from sketch
  • Cargo bike: I love the way the Load 75 makes it easy to go to playdates and on family adventures. A+ rides in it for quick trips or when she's tired of pedaling. I can tow her bike with the Bakkie bag. Glad to bike more.

    Since playdates are now 15 minutes away instead of 60, it's much easier to bring hot chocolate or popsicles to share. I used to bring store-bought popsicles, but now that the fruits are in, we've been making our own. (We've even gone fruit-picking!)

  • Crafts: I sewed a lot this year, from outdoor projects with Sunbrella fabric (covers for the awning, pizza oven, and grill) to handsewn drawstring bags and zippered pouches. I crocheted a few gifts, too.
  • Code: I automated more of EmacsConf and made a presentation about it–my first in years. In consulting, I wrote some code to support events, and I've even been able to write tests.
  • Curiosity: I've been slowly learning more about myself and us. I still worry, I still feel unsure, but this feels like a possible path.
  • Looking ahead: In some ways, it feels like we're taking two steps forward, one step back. Some things that were hard are now easier: cooking, sewing, reading together… Some things that were easy are now harder: playing with friends… I think we might be getting better at figuring out what works for us, though, so that's good. Looking forward to more experiments!
  • Topics: declarative language, co-regulation, nonviolent communication, competent/intuitive eating; testing and automation; Minecraft scripting; biking and exercise; working with what we've got

Text from sketch

40 by month

  • Aug 2023: ramp, etc.; swimming; pizza party; quilt, ukulele bag, grill cover; harvest
  • Sept: pizza parties; grade 2; consulting events
  • Oct: EmacsConf presentation prep; automation; tried cargo bikes; Mathalon medals; synchronous exemption
  • Nov: cargo bike!; skating
  • Dec: EmacsConf; one cat died; skating
  • Jan 2024: Tried livestreaming; lots of tweaks; Minecraft with cousins; consulting event
  • Feb: P52; Star Wars party; skating; other cat died; COVID booster
  • March: skirt, cloak, dress; rock-climbing; crepes; A+'s anxiety; maple syrup festival
  • April: first 3x3 comp; drawstring backpack, rose, journal cover, machine cover, 5-stone, basket; more decluttering
  • May: A+'s surgery; PS3 controllers & PC, PS Vita; cat carrier pad, skirt, poop emoji, vest pockets
  • June: strawberry picking; crochet gifts, swim skirt, flowers; big kid bed; homemade popsicles; garden
  • July: pouches; Pixel 8; swimming; blueberry & raspberry picking; bitter melon
  • Overall: cargo bike, crafts, automation, figuring things out
  • Next year: exploring more of our interests

Last year, I hoped to spend a lot of time playing outside with A+. Investing in a cargo bike has definitely helped with that, as we can more easily go on family bike adventures or head off to playdates. I've been learning more about playing inside with her as well, exploring Minecraft and figuring out how to work with the Minecraft Bedrock scripting API. This year A+ got into Star Wars and Harry Potter, so there's been lots and lots of reading, conversations, LEGO, Minecraft worlds, and pretend. I enjoy spending time with her. I think we're slowly starting to figure each other out.

We simplified the garden this year, and I feel like we've been able to water it more consistently. The bitter melon plants have been very productive, and the cherry tomatoes have been more than enough for our regular consumption.

If next year is much like this year (plus, of course, the things we'll experiment with and learn along the way), I think that'll be all right.

Blog posts

63 posts aside from Emacs News.

Sketches

59 sketches this year, down from 93 sketches the year before. I haven't made as much time to think through things.

Time

Category % 39 years % 40 years Diff % h/wk Diff h/wk
Business 1.8 3.4 1.6 5.7 2.7
Sleep 33.4 34.2 0.9 57.7 1.5
A- 39.6 40.0 0.5 67.5 0.8
Unpaid work 4.0 4.0 -0.1 6.7 -0.1
Personal 10.1 9.3 -0.8 15.7 -1.4
Discretionary 11.2 9.1 -2.1 15.3 -3.6

Even though I put a fair bit of work into documenting and automating more of EmacsConf (I put together a presentation this year, yay), I actually ended up spending a little less time on Emacs (-1.5 h/wk, averaging 5.3 h/wk) and other personal coding projects (-0.9h/wk, averaging 0.6 h/wk). That was probably because the previous year involved building a lot of infrastructure so that we could run multiple tracks, and this year was just about making it smoother so that we could run it with minimal prep. Things paying off!

I shifted much of that time to consulting (+ 2.8h/wk, averaging 5.6 h/wk), which let me help with some of my clients' time-sensitive project ideas. It's nice being able to take the time to write tests for my prototypes, which makes it easier to use little snippets of time without worrying too much about breaking things.

I think I'll be spending more time with A+ instead of less. I'll keep my expectations for focused time low so that I can enjoy this time with her. This is the time to do it!