Category Archives: org

Wow, literate devops with Emacs and Org does actually work on Windows

Since I persist in using Microsoft Windows as my base system, I’m used to not being able to do the kind of nifty tricks that other people do with Emacs and shell stuff.

So I was delighted to find that the literate devops that Howard Abrams described – running shell scripts embedded in an Org Mode file on a remote server – actually worked with Plink.

Here’s my context: The Toronto Public Library publishes a list of new books on the 15th of every month. I’ve written a small Perl script that parses the list for a specified category and displays the Dewey decimal code, title, and item ID. I also have another script (Ruby on Rails, part of that lets me request multiple items by pasting in text containing the item IDs. Tying these two together, I can take the output of the library new releases script, delete the lines I’m not interested in, and feed those lines to my library request script.

Instead of starting Putty, sshing to my server, and typing in the command line myself, I can now use C-c C-c on an Org Mode block like this:

#+begin_src sh :dir [email protected]:~
perl Business

That’s in a task that’s scheduled to repeat monthly, for even more convenience, and I also have a link there to my web-based interface for bulk-requesting files. But really, now that I’ve got it in Emacs, I should add a #+NAME: above the #+RESULTS: and have Org Mode take care of requesting those books itself.

On a related note, I’d given up on being able to easily use TRAMP from Emacs on Windows before, because Cygwin SSH was complaining about a non-interactive terminal.

ssh -l sacha  -o ControlPath=c:/Users/Sacha/AppData/Local/Temp/tramp.13728lpv.%r@%h:%p -o ControlMaster=auto -o ControlPersist=no -e none && exit || exit
Pseudo-terminal will not be allocated because stdin is not a terminal.
ssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory
Permission denied, please try again.
ssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory
Permission denied, please try again.
ssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory
Permission denied (publickey,password).

As it turns out, setting the following made it work for me.

(setq tramp-default-method "plink")

Now I can do things like the following:

(find-file "[email protected]:~/")

… which is, incidentally, this file (edited to remove my credentials):

# Displays new books from the Toronto Public Library
# Author: Sacha Chua ([email protected])
# Usage:
# perl <category> - print books
# perl <file> <username> <password> - request books
use Date::Parse;

#!/usr/bin/perl -w
use strict;
use URI::URL;
use WWW::Mechanize;
use WWW::Mechanize::FormFiller;
use WWW::Mechanize::TreeBuilder;
use HTML::TableExtract;
use Data::Dumper;
sub process_account($$$);
sub trim($);
sub to_renew($$);
sub clean_string($);

# Get the arguments
if ($#ARGV < 0) {
    print "Usage:\n";
    print "perl <category/filename> [username] [password]\n";

my $agent = WWW::Mechanize->new( autocheck => 1 );
my $formfiller = WWW::Mechanize::FormFiller->new();
if ($#ARGV > 0) {
  my $filename = shift(@ARGV);
  my $username = "NOT ACTUALLY MY USERNAME";
  my $password = "NOT ACTUALLY MY PASSWORD";
  print "Requesting books\n";
  request_books($agent, $username, $password, $filename);
} else {
  my $category = shift(@ARGV);
  print_new_books($agent, $category);

## FUNCTIONS ###############################################

# Perl trim function to remove whitespace from the start and end of the string
sub trim($)
  my $string = shift;
  $string =~ s/^\s+//;
  $string =~ s/\s+$//;
  return $string;

sub request_books($$$$)
  my $agent = shift;
  my $username = shift;
  my $password = shift;
  my $filename = shift;

  # Read titles and IDs
  open(DATA, $filename) or die("Couldn't open file.");
  my @lines = <DATA>;

  my %requests = ();

  my $line;
  my $title;
  my $id;
  foreach $line (@lines) {
    ($title, $id) = split /\t/, $line;
    chomp $id;
    $requests{$id} = $title;

  # Log in
  log_in_to_library($agent, $username, $password);
  print "Retrieving checked-out and requested books...";
  # Retrieve my list of checked-out and requested books
  my $current_ids = get_current_ids($agent);

  # Retrieve a stem URL that I can use for requests
  my $base_url = '';
  my @already_out;
  my @success;
  my @failure;
  # For each line in the file
  while (($id, $title) = each(%requests)) {
    # Do I already have it checked out or on hold? Skip.
    if ($current_ids->{$id}) {
      push @already_out, $title . " (" . $current_ids->{$id} . ")";
    } else {
      # Request the hold
      my $url = $base_url . $id;
      if ($agent->content =~ /The hold was successfully placed/) {
        # print "Borrowed ", $title, "\n";
        ## Did it successfully get checked out? Save title in success list
        push @success, $title;
      } else {
        # Else, save title and ID in fail list
        push @failure, $title . "\t" . $id;
  # Print out success list
  if ($#success > 0) {
    print "BOOKS REQUESTED:\n";
    foreach my $title (@success) {
      print $title, "\n";
    print "\n";
  # Print out already-out list
  if ($#already_out > 0) {
    foreach my $s (@already_out) {
      print $s, "\n";
    print "\n";
  # Print out fail list
  if ($#failure > 0) {
    print "COULD NOT REQUEST:\n";
    foreach my $s (@failure) {
      print $s, "\n";
    print "\n";

sub get_current_ids($)
  my $agent = shift;
  my %current_ids = ();
  my $string = $agent->content;
  while ($string =~ m/TITLE\^([0-9]+)\^/g) {
    $current_ids{$1} = 'requested';
  while ($string =~ m/RENEW\^([0-9]+)\^/g) {
    $current_ids{$1} = 'checked out';
  return \%current_ids;

sub print_new_books($$)
  my $agent = shift;
  my $category = shift;
  $agent->follow_link(text_regex => qr/Our Newest Titles/);
  $agent->follow_link(text_regex => qr/$category/i);

  my $continue = 1;
  while ($continue) {
    if ($agent->form_with_fields('SCROLL^F')) {
    } else {
      $continue = 0;

# Print out all the entries on this page
sub print_titles_on_page($)
  my $agent = shift;
  my @titles = $agent->look_down(sub {
                                $_[0]->tag() eq 'strong' and
                                $_[0]->parent->attr('class') and
                                $_[0]->parent->attr('class') eq 'itemlisting'; });
  foreach my $title (@titles) {
    my $hold = $title->parent->parent->parent->parent->look_down(sub {
                                                          $_[0]->attr('alt') and
                                                          $_[0]->attr('alt') eq 'Place Hold'; });
    my $id = "";
    my $call_no = "";
    if ($hold && $hold->parent && $hold->parent->attr('href') =~ /item_id=([^&]+)&.*?callnum=([^ "&]+)/) {
      $id = $1;
      $call_no = $2;
    print $call_no . "\t" . $title->as_text . "\t" . $id . "\n";

sub clean_string($)
    my $string = shift;
    $string =~ s#^.*?(<form id="renewitems" [^>]+>)#<html><body>\1#s;
    $string =~ s#</form>.*#</form></html>#s;
    $string =~ s#<table border="0" bordercolor="red".*(<table border="0" bordercolor="blue" cellspacing="0" cellpadding="0">)#\1#s;
    $string =~ s#</table>.*</form>#</table></form>#s;
# Clean up for parsing
    $string =~ s#<!-- Print the date due -->##g;
    $string =~ s#<br> <!-- Displays Date -->##g;
    return $string;

sub log_in_to_library($$$) {
    my $agent = shift;
    my $username = shift;
    my $password = shift;
    $agent->current_form->value('userId', $username);
    $agent->current_form->value('password', $password);

Ah, Emacs!

Update on Emacs Conf 2015 videos; Org Mode tables and time calculations

I spent the day cutting up the rest of the videos from the Emacs Conference 2015 stream into individual talks. I’d already cut the set of talks before lunch, but there were quite a few more after. As it turned out, keeping the video data in .ts format instead of converting it to .mp4 is actually better for Youtube processing.

Since Camtasia Studio and Movie Maker were both having problems with the large videos, I used VLC to play the video and find the timestamps at which I needed to cut the segments. I made an Org Mode table with the start and end times, and then I used the ;T flag in a table function to get the duration. A little bit of Emacs Lisp code later, and I had my ffmpeg commands. Here’s the source from my Org file:

#+NAME: emacsconf-c.ts
| Notes                                            |      Start |        End | Duration |
| Emacs configuration                              | 4:02:25.37 | 4:27:09.30 | 00:24:44 |
| Hearing from Emacs Beginners                     |    4:27:27 |    5:01:00 | 00:33:33 |
| Lightning talk: Emacs Club                       | 5:03:19.30 | 5:19:37.83 | 00:16:18 |
| Starting an Emacs Meetup - Harry Schwartz part 1 | 5:31:52.03 |    6:01:20 | 00:29:28 |
#+TBLFM: $4=$3-$2;T

#+NAME: emacsconf-a.ts
| Notes                                                    |   Start |     End | Duration |
| Starting an Emacs Meetup - Harry Schwartz part 2         |  0:0:00 | 0:20:04 | 00:20:04 |
| Literate Devops - Howard Abrams                          | 1:28:20 | 2:08:15 | 00:39:55 |
| Lightning talk: Wanderlust and other mail clients        | 2:15:04 | 2:26:55 | 00:11:51 |
| Making Emacs a Better Tool for Scholars - Erik Hetzner   | 2:27:00 | 2:57:38 | 00:30:38 |
| Wrapping up and going forward                            | 2:58:09 | 2:59:44 | 00:01:35 |
| Lightning talk: Collaborative coding with tmux and tmate | 3:00:20 | 3:05:53 | 00:05:33 |
| Lightning talk: Cask and Pellet                          | 3:05:56 | 3:09:04 | 00:03:08 |
| Lightning talk: File sharing with Git and save hooks     | 3:09:34 | 3:17:50 | 00:08:16 |
| Lightning talk: Calc                                     | 3:18:42 | 3:33:20 | 00:14:38 |
| Lightning talk: Magit                                    | 3:35:15 | 3:49:42 | 00:14:27 |
| Lightning talk: gist.el                                  | 3:53:50 | 4:01:58 | 00:08:08 |
| Lightning talk: Go                                       | 4:02:45 | 4:16:37 | 00:13:52 |
| Question: Emacs Lisp backtraces                          | 4:16:50 | 4:20:09 | 00:03:19 |
#+TBLFM: $4=$3-$2;T

#+begin_src emacs-lisp :var data=emacsconf-a.ts :var data2=emacsconf-c.ts :colnames t :results output
(let ((format-str "ffmpeg -i %s -ss %s -t %s -c:v copy -c:a copy \"EmacsConf 2015 - %s.ts\"\n"))
  (mapc (lambda (file)
    (mapc (lambda (row) 
      (princ (format format-str (car file) (elt row 1) (elt row 3) (my/convert-sketch-title-to-filename (elt row 0))))) 
     (cdr file)))
    `(("emacsconf-c.ts" . ,data2)
      ("emacsconf-a.ts" . ,data))))

and the output:

ffmpeg -i emacsconf-c.ts -ss 4:02:25.37 -t 00:24:44 -c:v copy -c:a copy "EmacsConf 2015 - Emacs configuration.ts"
ffmpeg -i emacsconf-c.ts -ss 4:27:27 -t 00:33:33 -c:v copy -c:a copy "EmacsConf 2015 - Hearing from Emacs Beginners.ts"
ffmpeg -i emacsconf-c.ts -ss 5:03:19.30 -t 00:16:18 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - Emacs Club.ts"
ffmpeg -i emacsconf-c.ts -ss 5:31:52.03 -t 00:29:28 -c:v copy -c:a copy "EmacsConf 2015 - Starting an Emacs Meetup - Harry Schwartz part 1.ts"
ffmpeg -i emacsconf-a.ts -ss 0:0:00 -t 00:20:04 -c:v copy -c:a copy "EmacsConf 2015 - Starting an Emacs Meetup - Harry Schwartz part 2.ts"
ffmpeg -i emacsconf-a.ts -ss 1:28:20 -t 00:39:55 -c:v copy -c:a copy "EmacsConf 2015 - Literate Devops - Howard Abrams.ts"
ffmpeg -i emacsconf-a.ts -ss 2:15:04 -t 00:11:51 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - Wanderlust and other mail clients.ts"
ffmpeg -i emacsconf-a.ts -ss 2:27:00 -t 00:30:38 -c:v copy -c:a copy "EmacsConf 2015 - Making Emacs a Better Tool for Scholars - Erik Hetzner.ts"
ffmpeg -i emacsconf-a.ts -ss 2:58:09 -t 00:01:35 -c:v copy -c:a copy "EmacsConf 2015 - Wrapping up and going forward.ts"
ffmpeg -i emacsconf-a.ts -ss 3:00:20 -t 00:05:33 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - Collaborative coding with tmux and tmate.ts"
ffmpeg -i emacsconf-a.ts -ss 3:05:56 -t 00:03:08 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - Cask and Pellet.ts"
ffmpeg -i emacsconf-a.ts -ss 3:09:34 -t 00:08:16 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - File sharing with Git and save hooks.ts"
ffmpeg -i emacsconf-a.ts -ss 3:18:42 -t 00:14:38 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - Calc.ts"
ffmpeg -i emacsconf-a.ts -ss 3:35:15 -t 00:14:27 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - Magit.ts"
ffmpeg -i emacsconf-a.ts -ss 3:53:50 -t 00:08:08 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - gist.el.ts"
ffmpeg -i emacsconf-a.ts -ss 4:02:45 -t 00:13:52 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - Go.ts"
ffmpeg -i emacsconf-a.ts -ss 4:16:50 -t 00:03:19 -c:v copy -c:a copy "EmacsConf 2015 - Question - Emacs Lisp backtraces.ts"

You can watch the Emacs Conference 2015 playlist on YouTube. At some point, each talk will probably have individual wiki pages and IRC logs at . =) Enjoy!

Related tech notes: Emacs Conf video tech notes:,, livestreamer, ffmpeg

Org Mode date arithmetic

Whenever I need to get Emacs to prompt me for a date or time (even for non-Org things), I use org-read-date. I love its flexibility. It’s great to be able to say things like +3 for three days from now, fri for next Friday, +2tue for two Tuesdays from now, +1w for next week, and +1m for next month. It’s easy to use org-read-date in Emacs Lisp. Calling it with (org-read-date) (usually as an interactive argument, like so: (interactive (list (org-read-date)))) gives me a date like 2015-08-06 depending on what I type in.

I use org-read-date for non-interactive date calculations, too. For example, if I want to quickly get the Org-style date for tomorrow, I can use org-read-date‘s third parameter (from-string) like this:

(org-read-date nil nil "+1")

Here’s how to calculate relative dates based on a specified date. You can hard-code the base date or use another org-read-date to get it. In this example, I’m getting the Monday after 2015-08-31. Note the use of two + signs instead of just one.

(org-read-date nil nil "++mon" nil (org-time-string-to-time "2015-08-31"))

org-time-string-to-time converts a date or time string into the internal representation for time. You can then extract individual components (ex: month) with decode-time, or convert it to the number of seconds since the epoch with time-to-seconds. Alternatively, you can convert Org time strings directly to seconds with org-time-to-seconds.

If you’re working with days, you can convert time strings with org-time-string-to-absolute. For example, you can use this to calculate the number of days between two dates (including the first day but excluding the last day), like this:

(let ((start-date (org-read-date))
      (end-date (org-read-date)))
  (- (org-time-string-to-absolute end-date)
     (org-time-string-to-absolute start-date)))

To get the month, day, and year, you can use org-time-string-to-time and decode-time, or you can use org-time-string-to-seconds and calendar-gregorian-from-absolute.

To convert internal time representations into Org-style dates, I tend to use (format-time-string "%Y-%m-%d" ...). encode-time is useful for converting something to the internal time representation. If you’re working with absolute days, you can convert them to Gregorian, and then format the string.

So, to loop over all the days between the start and end date, you could use a pattern like this:

(let* ((start-date (org-read-date))
       (end-date (org-read-date))
       (current (org-time-string-to-absolute start-date))
       (end (org-time-string-to-absolute end-date))
  (while (< current end)
    (setq gregorian-date (calendar-gregorian-from-absolute current))
    (setq formatted-date
          (format "%04d-%02d-%02d"
                  (elt gregorian-date 2) ; month
                  (elt gregorian-date 0) ; day
                  (elt gregorian-date 1))) ; year
    ;; Do something here; ex:
    (message "%s" formatted-date)
    ;; Move to the next date
    (setq current (1+ current))))

Alternatively, you could use org-read-date with a default date, like this:

(let* ((start-date (org-read-date))
       (end-date (org-read-date))
       (current start-date))
  (while (string< current end-date)
    ;; Do something here; ex:
    (message "%s" current)
    ;; Move to the next date
    (setq current (org-read-date nil nil "++1" nil (org-time-string-to-time current)))))

There are probably more elegant ways to write this code, so if you can think of improvements, please feel free to share.

Anyway, hope that helps!

Using your own Emacs Lisp functions in Org Mode table calculations: easier dosage totals

UPDATE 2015-06-17: In the comments below, Will points out that if you use proper dates ([yyyy-mm-dd] instead of yyyy-mm-dd), Org will do the date arithmetic for you. Neato! Here’s what Will said:

Hi Sacha. Did you know you can do date arithmetic directly on org’s inactive or active timestamps? It can even give you an answer in fractional days if the time of day is different in the two timestamps:

| Start                  | End                    | Interval |
| [2015-06-16 Tue]       | [2015-06-23 Tue]       |        7 |
| <2015-06-13 Sat>       | <2015-06-15 Mon>       |        2 |
| [2015-06-10 Wed 20:00] | [2015-06-17 Wed 08:00] |      6.5 |
#+TBLFM: $3=$2 - $1 

Here’s my previous convoluted way of doing things… =)

I recently wrote about calculating how many doses you need to buy using an Org Mode table. On reflection, it’s easier and more flexible to do that calculation using an Emacs Lisp function instead of writing a function that processes and outputs entire tables.

First, we define a function that calculates the number of days between two dates, including the dates given. I put this in my Emacs config.

(defun my/org-days-between (start end)
  "Number of days between START and END.
This includes START and END."
  (1+ (- (calendar-absolute-from-gregorian (org-date-to-gregorian end))
         (calendar-absolute-from-gregorian (org-date-to-gregorian start)))))

Here’s the revised table. I moved the “Needed” column to the left of the medication type because this makes it much easier to read and confirm.

| Needed | Type         | Per day |      Start |        End | Stock |
|     30 | Medication A |       2 | 2015-06-16 | 2015-06-30 |     0 |
|      2 | Medication B |     0.1 | 2015-06-16 | 2015-06-30 |   0.2 |
#+TBLFM: @2$1..@>$1='(ceiling (- (* (my/org-days-between $4 $5) (string-to-number $3)) (string-to-number $6)))

C-c C-c on the #+TBLFM: line updates the values in column 1.

@2$1..@>$1 means the cells from the second row (@2) to the last row (@>) in the first column ($1).  '  tells Org to evaluate the following expression as Emacs Lisp, substituting the values as specified ($4 is the fourth column’s value, etc.).

The table formula calculates the value of the first column (Needed) based on how many you need per day, the dates given (inclusive), and how much you already have in stock. It rounds numbers up by using the ceiling function.

Because this equation uses the values from each row, the start and end date must be filled in for all rows. To quickly duplicate values downwards, set org-table-copy-increment to nil, then use S-return (shift-return) in the table cell you want to copy. Keep typing S-return to copy more.

This treats the calculation inputs as strings, so I used string-to-number to convert some of them to numbers for multiplication and subtraction. If you were only dealing with numbers, you can convert them automatically by using the ;N flag, like this:

| Needed | Type         | Per day | Days | Stock |
|      6 | Medication A |       2 |    3 |     0 |
|      1 | Medication B |     0.1 |    3 |   0.2 |
#+TBLFM: @2$1..@>$1='(ceiling (- (* $3 $4) $5)));N

Providing values to functions in org-capture-templates

Over at the Emacs StackExchange, Raam Dev asked how to define functions for org-capture-templates that could take arguments. For example, it would be useful to have a function that creates a Ledger entry for the specified account. Functions used in org-capture-templates can’t take any arguments, but you can use property lists instead. Here’s the answer I posted.

You can specify your own properties in the property list for the template, and then you can access those properties with plist-get and org-capture-plist. Here’s a brief example:

Here’s a brief example:

(defun my/expense-template ()
  (format "Hello world %s" (plist-get org-capture-plist :account)))
(setq org-capture-templates '(("x" "Test entry 1" plain
                               (file "~/tmp/test.txt")
                               (function my/expense-template)
                               :account "Account:Bank")
                              ("y" "Test entry 2" plain
                               (file "~/tmp/test.txt")
                               (function my/expense-template)
                               :account "Account:AnotherBank")))

I hope that helps!

Using Emacs Org Mode tables to calculate doses to buy

I got tired of manually calculating how many I needed to buy based on a daily protocol and how many I had in stock, so I wrote a little bit of Emacs Lisp to figure it out. You can specify the type, daily dose, start and end dates (inclusive; defaults to the last specified date if blank), and how many you have in stock.

First, define a table of this form, and give it a name.

#+NAME: input
| Type         | Per day |      Start |        End | Stock |
| Medication A |       2 | 2015-06-09 | 2015-06-16 |     5 |
| Medication B |       1 |            |            |     0 |
| Medication C |     0.1 | 2015-06-12 | 2015-06-16 |   0.2 |
Type Per day Start End Stock
Medication A 2 2015-06-09 2015-06-16 5
Medication B 1 0
Medication C 0.1 2015-06-12 2015-06-16 0.2

To call the code from the bottom of this post, use something like:

#+CALL: calculate-meds-needed(meds=input) :hlines yes :colnames yes
Type Total In stock Needed
Medication A 16 5 11
Medication B 8 0 8
Medication C 0.5 0.2 1

Here’s the code that processes it:

#+name: calculate-meds-needed :var meds=meds :colnames yes :hlines yes
#+begin_src emacs-lisp
(let (start end)
   (list (list "Type" "Total" "In stock" "Needed"))
   (list 'hline)
   (sort (delq nil (mapcar
                    (lambda (row)
                      (unless (or (eq row 'hline) (string= (elt row 0) "Type"))
                        (let (total)
                          (setq start (if (string< "" (elt row 3)) (elt row 3) start)
                                end (if (string< "" (elt row 2)) (elt row 2) end)
                                total (* (elt row 1)
                                         (- (calendar-absolute-from-gregorian (org-date-to-gregorian start))
                                            (calendar-absolute-from-gregorian (org-date-to-gregorian end))
                           (elt row 0)
                           (elt row 4)
                           (max 0 (ceiling (- total (elt row 4))))))))
                    meds)) (lambda (a b) (string< (car a) (car b))))))