Categories: geek » emacs » org

View topic page - RSS - Atom - Subscribe via email

Using Org Mode, Emacs Lisp, and TRAMP to parse meetup calendar entries and generate a crontab

| org, emacs

Times and time zones trip me up. Even with calendar notifications, I still fumble scheduled events. Automation helps me avoid embarrassing hiccups.

We run BigBlueButton as a self-hosted web conferencing server for EmacsConf. It needs at least 8 GB of RAM when active. When it's dormant, it fits on a 1 GB RAM virtual private server. It's easy enough to scale the server up and down as needed. Using the server for Emacs meetups in between EmacsConfs gives people a way to get together, and it also means I can regularly test the infrastructure. That makes scaling it up for EmacsConf less nerve-wracking.

I have some code that processes various Emacs meetup iCalendar files (often with repeating entries) and combines them into one iCal file that people can subscribe to calendar, as well as Org files in different timezones that they can include in their org-agenda-files. The code I use to parse the iCal seems to handle time zones and daylight savings time just fine. I set it up so that the Org files have simple non-repeating entries, which makes them easy to parse. I can use the Org file to determine the scheduled jobs to run with cron on a home server (named xu4) that's up all the time.

This code parses the Org file for schedule information, then generates pairs of crontab entries. The first entry scales the BigBlueButton server up 1 hour before the event using my bbb-testing script, and the second entry scales the server down 6 hours after the event using my bbb-dormant script (more info). That gives organizers time to test it before the event starts, and it gives people plenty of time to chat. A shared CPU 8 GB RAM Linode costs USD 0.072 per hour, so that's USD 0.50 per meetup hosted.

Using #+begin_src emacs-lisp :file "/ssh:xu4:~/bbb.crontab" :results file as the header for my code block and using an SSH agent for authentication lets me use TRAMP to write the file directly to the server. (See Results of Evaluation (The Org Manual))

(let* ((file "/home/sacha/sync/emacs-calendar/emacs-calendar-toronto.org")
       (time-format "%M %H %d %m")
       (bbb-meetups "OrgMeetup\\|Emacs Berlin\\|Emacs APAC")
       (scale-up "/home/sacha/bin/bbb-testing")
       (scale-down "/home/sacha/bin/bbb-dormant"))
  (mapconcat
   (lambda (o)
     (let ((start-time (format-time-string time-format (- (car o) 3600 )))
           (end-time (format-time-string time-format (+ (car o) (* 6 3600)))))
       (format "# %s\n%s * %s\n%s * %s\n"
               (cdr o)
               start-time
               scale-up
               end-time
               scale-down)))
   (delq nil
         (with-temp-buffer
           (insert-file-contents file)
           (org-mode)
           (goto-char (point-min))
           (org-map-entries
            (lambda ()
              (when (and
                     (string-match bbb-meetups (org-entry-get (point) "ITEM"))
                     (re-search-forward org-tr-regexp (save-excursion (org-end-of-subtree)) t))
                (let ((time (match-string 0)))
                  (cons (org-time-string-to-seconds time)
                        (format "%s - %s" (org-entry-get (point) "ITEM") time)))))
            "LEVEL=1")))
   "\n"))

The code makes entries that look like this:

# OrgMeetup (virtual) - <2025-06-11 Wed 12:00>--<2025-06-11 Wed 14:00>
00 11 11 06 * /home/sacha/bin/bbb-testing
00 18 11 06 * /home/sacha/bin/bbb-dormant

# Emacs Berlin (hybrid, in English) - <2025-06-25 Wed 12:30>--<2025-06-25 Wed 14:30>
30 11 25 06 * /home/sacha/bin/bbb-testing
30 18 25 06 * /home/sacha/bin/bbb-dormant

# Emacs APAC: Emacs APAC meetup (virtual) - <2025-06-28 Sat 04:30>--<2025-06-28 Sat 06:00>
30 03 28 06 * /home/sacha/bin/bbb-testing
30 10 28 06 * /home/sacha/bin/bbb-dormant

This works because meetups don't currently overlap. If there were, I'll need to tweak the code so that the server isn't downscaled in the middle of a meetup. It'll be a good problem to have.

I need to load the crontab entries by using crontab bbb.crontab. Again, I can tell Org Mode to run this on the xu4 home server. This time I use the :dir argument to specify the default directory, like this:

#+begin_src sh :dir "/ssh:xu4:~" :results silent
crontab bbb.crontab
#+end_src

Then cron can take care of things automatically, and I'll just get the e-mail notifications from Linode telling me that the server has been resized. This has already come in handy, like when I thought of Emacs APAC as being on Saturday, but it was actually on Friday my time.

I have another Emacs Lisp block that I use to retrieve all the info and update the list of meetups. I can add (goto-char (org-find-property "CUSTOM_ID" "crontab")) to find this section and use org-babel-execute-subtree to execute all the code blocks. That makes it an automatic part of my process for updating the Emacs Calendar and Emacs News. Here's the code that does the calendar part (Org source):

(defun my-prepare-calendar-for-export ()
  (interactive)
  (with-current-buffer (find-file-noselect "~/sync/emacs-calendar/README.org")
    (save-restriction
      (widen)
      (goto-char (point-min))
      (re-search-forward "#\\+NAME: event-summary")
      (org-ctrl-c-ctrl-c)
      (org-export-to-file 'html "README.html")
      ;; (unless my-laptop-p (my-schedule-announcements-for-upcoming-emacs-meetups))
      ;; update the crontab
      (goto-char (org-find-property "CUSTOM_ID" "crontab"))
      (org-babel-execute-subtree)
      (when my-laptop-p
        (org-babel-goto-named-result "event-summary")
        (re-search-forward "^- ")
        (goto-char (match-beginning 0))
        (let ((events (org-babel-read-result)))
          (oddmuse-edit "EmacsWiki" "Usergroups")
          (goto-char (point-min))
          (delete-region (progn (re-search-forward "== Upcoming events ==\n\n") (match-end 0))
                         (progn (re-search-forward "^$") (match-beginning 0)))
          (save-excursion (insert (mapconcat (lambda (s) (concat "* " s "\n")) events ""))))))))
(my-prepare-calendar-for-export)

I used a similar technique to generate the EmacsConf crontabs for automatically switching to the next talk. For that one, I used Emacs Lisp to write the files directly instead of using the :file header argument for Org Mode source blocks. That made it easier to loop over multiple files.

Hmm. Come to think of it, the technique of "go to a specific subtree and then execute it" is pretty powerful. In the past, I've found it handy to execute source blocks by name. Executing a subtree by custom ID is even more useful because I can easily mix source blocks in different languages or include other information. I think that's worth adding a my-org-execute-subtree-by-custom-id function to my Emacs configuration. Combined with an elisp: link, I can make links that execute functional blocks that might even be in different files. That could be a good starting point for a dashboard.

I love the way Emacs can easily work with files and scripts in different languages on different computers, and how it can help me with times and time zones too. This code should help me avoid brain hiccups and calendar mixups so that people can just enjoy getting together. Now I don't have to worry about whether I remembered to set up cron entries and if I did the math right for the times. We'll see how it holds up!

View org source for this post

Org Mode: Cutting the current list item (including nested lists) with a speed command

| emacs, org

Defining shortcuts in org-speed-commands is handy because you can use these single-key shortcuts at the beginning of a subtree. With a little modification, they'll also work at the beginning of list items.

(defun my-org-use-speed-commands-for-headings-and-lists ()
  "Activate speed commands on list items too."
  (or (and (looking-at org-outline-regexp) (looking-back "^\**" nil))
      (save-excursion (and (looking-at (org-item-re)) (looking-back "^[ \t]*" nil)))))
(setq org-use-speed-commands 'my-org-use-speed-commands-for-headings-and-lists)

I want k to be an org-speed-commands that cuts the current subtree or list item. This is handy when I'm cleaning up the Mastodon toots in my weekly review or getting rid of outline items that I no longer need. By default, k is mapped to org-cut-subtree, but it's easy to override.

(defun my-org-cut-subtree-or-list-item (&optional n)
  "Cut current subtree or list item."
  (cond
   ((and (looking-at org-outline-regexp) (looking-back "^\**" nil))
    (org-cut-subtree n))
   ((looking-at (org-item-re))
    (kill-region (org-beginning-of-item) (org-end-of-item)))))
(with-eval-after-load 'org
  (setf (alist-get "k" org-speed-commands nil nil #'string=)
        #'my-org-cut-subtree-or-list-item))

So now, if I put my cursor before "1." below and press k:

- this
  1. is a
    - nested
  2. list
- with levels

it will turn into:

  • this
    1. list
  • with levels

You can find out a little more about Org Mode speed commands in the Org manual: (info "(org) Speed Keys").

This is part of my Emacs configuration.
View org source for this post

Tweaking my 11ty blog to link to the Mastodon post defined in an Org Mode property

| 11ty, mastodon, org

One of the things I like about blogging from Org Mode in Emacs is that it's easy to add properties to the section that I'm working on and then use those property values elsewhere. For example, I've modified Emacs to simplify tooting a link to my blog post and saving the Mastodon status URL in the EXPORT_MASTODON property. Then I can use that in my 11ty static site generation process to include a link to the Mastodon thread as a comment option.

First, I need to export the property and include it in the front matter. I use .11tydata.json files to store the details for each blog post. I modified ox-11ty.el so that I could specify functions to change the front matter (org-11ty-front-matter-functions, org-11ty--front-matter):

(defvar org-11ty-front-matter-functions nil
  "Functions to call with the current front matter plist and info.")
(defun org-11ty--front-matter (info)
  "Return front matter for INFO."
  (let* ((date (plist-get info :date))
         (title (plist-get info :title))
         (modified (plist-get info :modified))
         (permalink (plist-get info :permalink))
         (categories (plist-get info :categories))
         (collections (plist-get info :collections))
         (extra (if (plist-get info :extra) (json-parse-string
                                             (plist-get info :extra)
                                             :object-type 'plist))))
    (seq-reduce
     (lambda (prev val)
       (funcall val prev info))
     org-11ty-front-matter-functions
     (append
      extra
      (list :permalink permalink
            :date (if (listp date) (car date) date)
            :modified (if (listp modified) (car modified) modified)
            :title (if (listp title) (car title) title)
            :categories (if (stringp categories) (split-string categories) categories)
            :tags (if (stringp collections) (split-string collections) collections))))))

Then I added the EXPORT_MASTODON Org property as part of the front matter. This took a little figuring out because I needed to pass it as one of org-export-backend-options, where the parameter is defined as MASTODON but the actual property needs to be called EXPORT_MASTODON.

(defun my-org-11ty-add-mastodon-to-front-matter (front-matter info)
  (plist-put front-matter :mastodon (plist-get info :mastodon)))
(with-eval-after-load 'ox-11ty
  (cl-pushnew
   '(:mastodon "MASTODON" nil nil)
   (org-export-backend-options (org-export-get-backend '11ty)))
  (add-hook 'org-11ty-front-matter-functions #'my-org-11ty-add-mastodon-to-front-matter))

Then I added the Mastodon field as an option to my comments.cjs shortcode. This was a little tricky because I'm not sure I'm passing the data correctly to the shortcode (sometimes it ends up as item.data, sometimes it's item.data.data, …?), but with ?., I can just throw all the possibilities in there and it'll eventually find the right one.

const pluginRss = require('@11ty/eleventy-plugin-rss');
module.exports = function(eleventyConfig) {
  function getCommentChoices(data, ref) {
    const mastodonUrl = data.mastodon || data.page?.mastodon || data.data?.mastodon;
    const mastodon = mastodonUrl && `<a href="${mastodonUrl}" target="_blank" rel="noopener noreferrer">comment on Mastodon</a>`;
    const url = ref.absoluteUrl(data.url || data.permalink || data.data?.url || data.data?.permalink, data.metadata?.url || data.data?.metadata?.url);
    const subject = encodeURIComponent('Comment on ' + url);
    const body = encodeURIComponent("Name you want to be credited by (if any): \nMessage: \nCan I share your comment so other people can learn from it? Yes/No\n");
    const email = `<a href="mailto:sacha@sachachua.com?subject=${subject}&body=${body}">e-mail me at sacha@sachachua.com</a>`;
    const disqusLink = url + '#comment';
    const disqusForm = data.metadata?.disqusShortname && `<div id="disqus_thread"></div>
<script>
 var disqus_config = function () {
   this.page.url = "${url}";
   this.page.identifier = "${data.id || ''} ${data.metadata?.url || ''}?p=${ data.id || data.permalink || this.page?.url}";
   this.page.disqusTitle = "${ data.title }"
   this.page.postId = "${ data.id || data.permalink || this.page?.url }"
 };
 (function() { // DON'T EDIT BELOW THIS LINE
   var d = document, s = d.createElement('script');
   s.src = 'https://${ data.metadata?.disqusShortname }.disqus.com/embed.js';
   s.setAttribute('data-timestamp', +new Date());
   (d.head || d.body).appendChild(s);
 })();
</script>
<noscript>Disqus requires Javascript, but you can still e-mail me if you want!</noscript>`;
    return { mastodon, disqusLink, disqusForm, email };
  }
  eleventyConfig.addShortcode('comments', function(data, linksOnly=false) {
    const { mastodon, disqusForm, disqusLink, email } = getCommentChoices(data, this);
    if (linksOnly) {
      return `You can ${mastodon ? mastodon + ', ' : ''}<a href="${disqusLink}">comment with Disqus (JS required)</a>${mastodon ? ',' : ''} or ${email}.`;
    } else {
      return `<div id="comment"></div>
You can ${mastodon ? mastodon + ', ' : ''}comment with Disqus (JS required)${mastodon ? ', ' : ''} or you can ${email}.
${disqusForm || ''}`;}
  });
}

I included it in my post.cjs shortcode:

module.exports = eleventyConfig =>
eleventyConfig.addShortcode('post', async function(item, index, includeComments) {
  let comments = '<div class="comments">' + (includeComments ? this.comments(item) : this.comments(item, true)) + '</div>';
  let categoryList = item.categories || item.data && item.data.categories;
  let categoriesFooter = '', categories = '';
  if (categoryList && categoryList.length > 0) {
    categoriesFooter = `<div class="footer-categories">More posts about ${this.categoryList(categoryList)}</div>`;
    categories = `| <span class="categories">${this.categoryList(categoryList)}</span>`;
  }

  return  `<article class="post" id="index${index}" data-url="${item.url || item.permalink || ''}">
<header><h2 data-pagefind-meta="title"><a href="${item.url || item.permalink || ''}">${item.title || item.data && item.data.title}</a></h2>
${this.timeInfo(item)}${categories}
</header>
<div class="entry">
${await (item.templateContent || item.layoutContent || item.data?.content || item.content || item.inputContent)}
</div>
${comments}
${categoriesFooter}
</article>`;
});

I also included it in my RSS item template to make it easier for people to send me comments without having to dig through my website for contact info.

const posthtml = require("posthtml");
const urls = require("posthtml-urls");

module.exports = (eleventyConfig) => {
  eleventyConfig.addAsyncShortcode('rssItem', async function(item) {
    let content = item.templateContent.replace(/--/g, '&#45;&#45;');
    if (this.transformWithHtmlBase) {
      content = await this.transformWithHtmlBase(content);
    }
    return `<item>
    <title>${item.data.title}</title>
    <link>${this.absoluteUrl(item.url, item.data.metadata.url)}</link>
    <dc:creator><![CDATA[${item.data.metadata.author.name}]]></dc:creator>
    <pubDate>${item.date.toUTCString()}</pubDate>
    ${item.data.categories?.map((cat) => `<category>${cat}</category>`).join("\n") || ''}
    <guid isPermaLink="false">${this.guid(item)}</guid>
    <description><![CDATA[${content}
<p>${this.comments(item, true)}</p>]]></description>
    </item>`;
  });
};

The new workflow I'm trying out seems to be working:

  1. Keep npx eleventy --serve running in the background, using .eleventyignore to make rebuilds reasonably fast.
  2. Export the subtree with C-c e s 1 1, which uses org-export-dispatch to call my-org-11ty-export with the subtree.
  3. After about 10 seconds, use my-org-11ty-copy-just-this-post and verify.
  4. Use my-mastodon-11ty-toot-post to compose a toot. Edit the toot and post it.
  5. Check that the EXPORT_MASTODON property has been set.
  6. Export the subtree again, this time with the front matter.
  7. Publish my whole blog.

Next, I'm thinking of modifying my-mastodon-11ty-toot-post so that it includes a list of links to blog posts I might be building on or responding to, and possibly the handles of people related to those blog posts or topics. Hmm…

View org source for this post

mastodon.el: Copy toot content as Org Mode

| mastodon, org, emacs

Sometimes I want to copy a toot and include it in my Org Mode notes, like when I post a thought and then want to flesh it out into a blog post. This code defines my-mastodon-org-copy-toot-content, which converts the toot text to Org Mode format using Pandoc and puts it in the kill ring so I can yank it somewhere else.

(defun my-mastodon-toot-at-url (&optional url)
  "Return JSON toot object at URL.
If URL is nil, return JSON toot object at point."
  (if url
      (let* ((search (format "%s/api/v2/search" mastodon-instance-url))
             (params `(("q" . ,url)
                       ("resolve" . "t"))) ; webfinger
             (response (mastodon-http--get-json search params :silent)))
        (car (alist-get 'statuses response)))
    (mastodon-toot--base-toot-or-item-json)))

(defun my-mastodon-org-copy-toot-content (&optional url)
  "Copy the current toot's content as Org Mode.
Use pandoc to convert.

When called with \\[universal-argument], prompt for a URL."
  (interactive (list
                (when current-prefix-arg
                  (read-string "URL: "))))

  (let ((toot (my-mastodon-toot-at-url url)))
    (with-temp-buffer
      (insert (alist-get 'content toot))
      (call-process-region nil nil "pandoc" t t nil "-f" "html" "-t" "org")
      (kill-new
       (concat
        (org-link-make-string
         (alist-get 'url toot)
         (concat "@" (alist-get 'acct (alist-get 'account toot))))
        ":\n\n#+begin_quote\n"
        (string-trim (buffer-string)) "\n#+end_quote\n"))
      (message "Copied."))))
This is part of my Emacs configuration.
View org source for this post

Org Mode: Merge top-level items in an item list

| org

I usually summarize Mastodon links, move them to my Emacs News Org file, and then categorize them. Today I accidentically categorized the links while they were still in my Mastodon buffer, so I had two lists with categories. I wanted to write some Emacs Lisp to merge sublists based on the top-level items. I could sort the list alphabetically with C-c ^ (org-sort) and then delete the redundant top-level item lines, but it's fun to tinker with Emacs Lisp.

Example input:

  • Topic A:
    • Item 1
    • Item 2
      • Item 2.1
  • Topic B:
    • Item 3
  • Topic A:
    • Item 4
      • Item 4.1

Example output:

  • Topic B:
    • Item 3
  • Topic A:
    • Item 1
    • Item 2
      • Item 2.1
    • Item 4
      • Item 4.1

The sorting doesn't particularly matter to me, but I want the things under Topic A to be combined. Someday it might be nice to recursively merge other entries (ex: if there's another "Topic A: - Item 2" subitem like "Item 2.2"), but I don't need that yet.

Anyway, we can parse the list with org-list-to-lisp (which can even delete the original list) and recreate it with org-list-to-org, so then it's a matter of transforming the data structure.

(defun my-org-merge-list-entries-at-point ()
  "Merge entries in a nested Org Mode list at point that have the same top-level item text."
  (interactive)
  (save-excursion
    (let* ((list-indentation (save-excursion
                               (goto-char (caar (org-list-struct)))
                               (current-indentation)))
           (list-struct (org-list-to-lisp t))
           (merged-list (my-org-merge-list-entries list-struct)))
      (insert (org-ascii--indent-string (org-list-to-org merged-list) list-indentation)
              "\n"))))

(defun my-org-merge-list-entries (list-struct)
  "Merge an Org list based on its top-level headings"
  (cons (car list-struct)
        (mapcar
         (lambda (g)
           (list
            (car g)
            (let ((list-type (car (car (cdr (car (cdr g))))))
                  (entries (seq-mapcat #'cdar (mapcar #'cdr (cdr g)))))
              (apply #'append (list list-type) entries nil))))
         (seq-group-by #'car (cdr list-struct)))))

A couple of test cases:

(ert-deftest my-org-merge-list-entries ()
  (should
   (equal
    (my-org-merge-list-entries
     '(unordered ("Topic B:" (unordered ("Item 3")))))
    '(unordered ("Topic B:" (unordered ("Item 3"))))))
  (should
   (equal
    (my-org-merge-list-entries
     '(unordered ("Topic B:" (unordered ("Item 3")))
                 ("Topic A:"
                  (unordered ("Item 1")
                             ("Item 2"
                              (unordered ("Item 2.1")))))
                 ("Topic A:"
                  (unordered
                   ("Item 4" (unordered ("Item 4.1")))))))
    '(unordered
      ("Topic B:" (unordered ("Item 3")))
      ("Topic A:"
       (unordered ("Item 1")
                  ("Item 2" (unordered ("Item 2.1")))
                  ("Item 4" (unordered ("Item 4.1")))))))))
View org source for this post

Remove open Org Mode clock entries

| emacs, org

Pedro pointed out that I had some incomplete clock entries in my Emacs configuration. org-resolve-clocks prompts you for what to do with each open clock entry in your Org agenda files and whatever Org Mode files you have open.

If you don't feel like cancelling each clock with C, I also wrote this function to delete all open clocks in the current file.

(defun my-org-delete-open-clocks ()
  (interactive)
  (flush-lines
   (rx
    line-start
    (zero-or-more space)
    "CLOCK:"
    (one-or-more space)
    (regexp org-ts-regexp-inactive)
    (zero-or-more space)
    line-end)))
This is part of my Emacs configuration.
View org source for this post

Getting an Org link URL from a string; debugging regex groups

| elisp, org

Sometimes I want to get the URL from a string whether the string contains a bare URL (https://example.com) or an Org bracketed link ([[https://example.com]] or [[https://example.com][Example]], ignoring any extra non-link text (blah https://example.com blah blah). org-link-any-re seemed like the right regular expression to use, but I started to get a little dizzy looking at all the parenthesis and I couldn't figure out which matching group to use. I tried using re-builder. That highlighted the groups in different colours, but I didn't know what the colours meant. All the matching information is in (match-data), but integer pairs can be a little hard to translate back to substrings. So I wrote an Emacs Lisp function to gave me the matching groups:

(defun my-match-groups (&optional object)
  "Return the matching groups, good for debugging regexps."
  (seq-map-indexed (lambda (entry i)
                     (list i entry
                           (and (car entry)
                                (if object
                                    (substring object (car entry) (cadr entry))
                                  (buffer-substring (car entry) (cadr entry))))))
                   (seq-partition
                    (match-data t)
                    2)))

There's probably a standard way to do this, but I couldn't figure out how to find it.

Anyway, if I give it a string with a bracketed link, I can tell that the URL ends up in group 2:

(let ((text "blah [[https://example.com][example]] blah blah"))
  (when (string-match org-link-any-re text)
    (pp-to-string (my-match-groups text))))
((0 (5 37) "[[https://example.com][example]]")
 (1 (5 37) "[[https://example.com][example]]")
 (2 (7 26) "https://example.com")
 (3 (28 35) "example"))

When I use a string with a bare link, I can see that the URL ends up in group 7:

(let ((text "blah https://example.com blah blah"))
  (when (string-match org-link-any-re text)
    (pp-to-string (my-match-groups text))))
((0 (5 24) "https://example.com")
 (1 (nil nil) nil) (2 (nil nil) nil)
 (3 (nil nil) nil) (4 (nil nil) nil)
 (5 (nil nil) nil) (6 (nil nil) nil)
 (7 (5 24) "https://example.com")
 (8 (5 10) "https") (9 (11 24) "//example.com"))

This makes it so much easier to refer to the right capture group. So now I can use those groups to extract the URL from a string:

(defun my-org-link-url-from-string (s)
  "Return the link URL from S."
  (when (string-match org-link-any-re s)
    (or
     (match-string 7 s)
       (match-string 2 s))))

This is handy when I summarize Emacs News links from Mastodon or from my inbox. Sometimes I add extra text after a link that I've captured from my phone, and I don't want that included in the URL. Sometimes I have a bracketed link that I've copied from org-capture note. Now I don't have to worry about the format. I can just grab the link I want.

View org source for this post