Wow, literate devops with Emacs and Org does actually work on Windows

Since I persist in using Microsoft Windows as my base system, I’m used to not being able to do the kind of nifty tricks that other people do with Emacs and shell stuff.

So I was delighted to find that the literate devops that Howard Abrams described – running shell scripts embedded in an Org Mode file on a remote server – actually worked with Plink.

Here’s my context: The Toronto Public Library publishes a list of new books on the 15th of every month. I’ve written a small Perl script that parses the list for a specified category and displays the Dewey decimal code, title, and item ID. I also have another script (Ruby on Rails, part of quantifiedawesome.com) that lets me request multiple items by pasting in text containing the item IDs. Tying these two together, I can take the output of the library new releases script, delete the lines I’m not interested in, and feed those lines to my library request script.

Instead of starting Putty, sshing to my server, and typing in the command line myself, I can now use C-c C-c on an Org Mode block like this:

#+begin_src sh :dir /sacha@direct.sachachua.com:~
perl library-new.pl Business
#+end_src

That’s in a task that’s scheduled to repeat monthly, for even more convenience, and I also have a link there to my web-based interface for bulk-requesting files. But really, now that I’ve got it in Emacs, I should add a #+NAME: above the #+RESULTS: and have Org Mode take care of requesting those books itself.

On a related note, I’d given up on being able to easily use TRAMP from Emacs on Windows before, because Cygwin SSH was complaining about a non-interactive terminal.

ssh -l sacha  -o ControlPath=c:/Users/Sacha/AppData/Local/Temp/tramp.13728lpv.%r@%h:%p -o ControlMaster=auto -o ControlPersist=no -e none direct.sachachua.com && exit || exit
Pseudo-terminal will not be allocated because stdin is not a terminal.
ssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory
Permission denied, please try again.
ssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory
Permission denied, please try again.
ssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory
Permission denied (publickey,password).

As it turns out, setting the following made it work for me.

(setq tramp-default-method "plink")

Now I can do things like the following:

(find-file "/sacha@direct.sachachua.com:~/library-new.pl")

… which is, incidentally, this file (edited to remove my credentials):

#!/usr/bin/perl
# Displays new books from the Toronto Public Library
#
# Author: Sacha Chua (sacha@sachachua.com)
#
# Usage:
# perl library-new.pl <category> - print books
# perl library-new.pl <file> <username> <password> - request books
#
use Date::Parse;

#!/usr/bin/perl -w
use strict;
use URI::URL;
use WWW::Mechanize;
use WWW::Mechanize::FormFiller;
use WWW::Mechanize::TreeBuilder;
use HTML::TableExtract;
use Data::Dumper;
sub process_account($$$);
sub trim($);
sub to_renew($$);
sub clean_string($);

# Get the arguments
if ($#ARGV < 0) {
    print "Usage:\n";
    print "perl library-new.pl <category/filename> [username] [password]\n";
    exit;
}

my $agent = WWW::Mechanize->new( autocheck => 1 );
my $formfiller = WWW::Mechanize::FormFiller->new();
if ($#ARGV > 0) {
  my $filename = shift(@ARGV);
  my $username = "NOT ACTUALLY MY USERNAME";
  my $password = "NOT ACTUALLY MY PASSWORD";
  print "Requesting books\n";
  request_books($agent, $username, $password, $filename);
} else {
  my $category = shift(@ARGV);
  WWW::Mechanize::TreeBuilder->meta->apply($agent);
  print_new_books($agent, $category);
}


## FUNCTIONS ###############################################

# Perl trim function to remove whitespace from the start and end of the string
sub trim($)
{
  my $string = shift;
  $string =~ s/^\s+//;
  $string =~ s/\s+$//;
  return $string;
}

sub request_books($$$$)
{
  my $agent = shift;
  my $username = shift;
  my $password = shift;
  my $filename = shift;

  # Read titles and IDs
  open(DATA, $filename) or die("Couldn't open file.");
  my @lines = <DATA>;
  close(DATA);

  my %requests = ();

  my $line;
  my $title;
  my $id;
  foreach $line (@lines) {
    ($title, $id) = split /\t/, $line;
    chomp $id;
    $requests{$id} = $title;
  }

  # Log in
  log_in_to_library($agent, $username, $password);
  print "Retrieving checked-out and requested books...";
  # Retrieve my list of checked-out and requested books
  my $current_ids = get_current_ids($agent);

  # Retrieve a stem URL that I can use for requests
  my $base_url = 'https://www.torontopubliclibrary.ca/placehold?itemId=';
  my @already_out;
  my @success;
  my @failure;
  # For each line in the file
  while (($id, $title) = each(%requests)) {
    # Do I already have it checked out or on hold? Skip.
    if ($current_ids->{$id}) {
      push @already_out, $title . " (" . $current_ids->{$id} . ")";
    } else {
      # Request the hold
      my $url = $base_url . $id;
      $agent->get($url);
      $agent->form_name('form_place-hold');
      $agent->submit();
      if ($agent->content =~ /The hold was successfully placed/) {
        # print "Borrowed ", $title, "\n";
        ## Did it successfully get checked out? Save title in success list
        push @success, $title;
      } else {
        # Else, save title and ID in fail list
        push @failure, $title . "\t" . $id;
      }
    }
  }
  # Print out success list
  if ($#success > 0) {
    print "BOOKS REQUESTED:\n";
    foreach my $title (@success) {
      print $title, "\n";
    }
    print "\n";
  }
  # Print out already-out list
  if ($#already_out > 0) {
    print "ALREADY REQUESTED/CHECKED OUT:\n";
    foreach my $s (@already_out) {
      print $s, "\n";
    }
    print "\n";
  }
  # Print out fail list
  if ($#failure > 0) {
    print "COULD NOT REQUEST:\n";
    foreach my $s (@failure) {
      print $s, "\n";
    }
    print "\n";
  }
}

sub get_current_ids($)
{
  my $agent = shift;
  my %current_ids = ();
  my $string = $agent->content;
  while ($string =~ m/TITLE\^([0-9]+)\^/g) {
    $current_ids{$1} = 'requested';
  }
  while ($string =~ m/RENEW\^([0-9]+)\^/g) {
    $current_ids{$1} = 'checked out';
  }
  return \%current_ids;
}

sub print_new_books($$)
{
  my $agent = shift;
  my $category = shift;
  $agent->env_proxy();
  $agent->get('http://oldcatalogue.torontopubliclibrary.ca');
  $agent->follow_link(text_regex => qr/Our Newest Titles/);
  $agent->follow_link(text_regex => qr/$category/i);

  my $continue = 1;
  while ($continue) {
    print_titles_on_page($agent);
    if ($agent->form_with_fields('SCROLL^F')) {
      $agent->click('SCROLL^F');
    } else {
      $continue = 0;
    }
  }
}

# Print out all the entries on this page
sub print_titles_on_page($)
{
  my $agent = shift;
  my @titles = $agent->look_down(sub {
                                $_[0]->tag() eq 'strong' and
                                $_[0]->parent->attr('class') and
                                $_[0]->parent->attr('class') eq 'itemlisting'; });
  foreach my $title (@titles) {
    my $hold = $title->parent->parent->parent->parent->look_down(sub {
                                                          $_[0]->attr('alt') and
                                                          $_[0]->attr('alt') eq 'Place Hold'; });
    my $id = "";
    my $call_no = "";
    if ($hold && $hold->parent && $hold->parent->attr('href') =~ /item_id=([^&]+)&.*?callnum=([^ "&]+)/) {
      $id = $1;
      $call_no = $2;
    }
    print $call_no . "\t" . $title->as_text . "\t" . $id . "\n";
  }
}

sub clean_string($)
{
    my $string = shift;
    $string =~ s#^.*?(<form id="renewitems" [^>]+>)#<html><body>\1#s;
    $string =~ s#</form>.*#</form></html>#s;
    $string =~ s#<table border="0" bordercolor="red".*(<table border="0" bordercolor="blue" cellspacing="0" cellpadding="0">)#\1#s;
    $string =~ s#</table>.*</form>#</table></form>#s;
# Clean up for parsing
    $string =~ s#<!-- Print the date due -->##g;
    $string =~ s#<br> <!-- Displays Date -->##g;
    return $string;
}

sub log_in_to_library($$$) {
    my $agent = shift;
    my $username = shift;
    my $password = shift;
    $agent->get('http://beta.torontopubliclibrary.ca/youraccount');
    $agent->form_name('form_signin');
    $agent->current_form->value('userId', $username);
    $agent->current_form->value('password', $password);
    $agent->submit();
}

Ah, Emacs!

  • sbrisard

    HI,
    thanks for sharing these tricks! I did not know about the plink method, and would really like to use tramp with my windows box. So, I set tramp-default-method to “plink”, and tried to open one of my remote files C-x C-f /plink:sbrisard@doubs/foo/bar
    It asks for a password, but the (correct) password is not accepted.
    Of course, PuTTY works like a charm outside Emacs. Have you had this problem? Any idea where this could come from? I was thinking maybe some weird encoding of the password, but my password is pure ASCII 127!
    I’ll keep on reading your blog, it’s a real goldmine.
    Sébastien

    • Hello, Sébastien! I’ve been using pageant to manage my private keys and passphrases on Windows, so I only have to enter my passphrase once and it handles authentication. Would you consider giving that a try?

      • sbrisard

        Hi Sacha,
        thanks for the suggestion. I have now set up pageant to hold login/password. It works in the console, but not with tramp (I get the following message: “Tramp: Found remote shell prompt on `doubs'”). I will dig into this, because I like your solution based on pageant.

        Meanwhile, do you have any specific configuration in your init.el regarding tramp? You do use plink as a tramp method, right?
        Best regards,
        Sébastien

        • Hmm, just (setq tramp-default-method “plink”), I think. With pageant and your private key loaded, does putty authenticate to your server without a password prompt? If not, that’s probably what you’ll want to sort out first.

          • sbrisard

            Yes, that works fine. I haven’t gone any further on this issue: I spent some time setting up ob-ipython instead. It’s just great!