Anyway, I'm making do. Thanks to Node and the abundance of libraries available through NPM, Javascript is shaping up to be a surprisingly useful scripting language.

After I used the Flickr API library for Javascript to cross-reference my Flickr archive with my blog posts, I looked around for other things I could do with it. photoSync occasionally didn't upload new pictures I added to its folders (or at least, not as quickly as I wanted). I wanted to replace photoSync with my own script that would:

• upload the picture only if it doesn't already exist,
• add tags based on the filename,
• add the photo to my Sketchbook photoset,
• move the photo to the "To blog" folder, and
• make it easy for me to refer to the Flickr image in my blog post or index.

The flickr-with-uploads library made it easy to upload images and retrieve information, although the format was slightly different from the Flickr API library I used previously. (In retrospect, I should've checked the Flickr API documentation first – there's an example upload request right on the main page. Oh well! Maybe I'll change it if I feel like rewriting it.)

I searched my existing photos to see if a photo with that title already existed. If it did, I displayed an Org-style list item with a link. If it didn't exist, I uploaded it, set the tags, added the item to the photo set, and moved it to the folder. Then I displayed an Org-style link, but using a plus character instead of a minus character, taking advantage of the fact that both + and – can be used for lists in Org.

While using console.log(...) to display these links in the terminal allowed me to mark and copy the link, I wanted to go one step further. Could I send the links directly to Emacs? I looked into getting org-protocol to work, but I was having problems figuring this out. (I solved those problems; details later in this post.)

What were some other ways I could get the information into Emacs aside from copying and pasting from the terminal window? Maybe I could put text directly into the clipboard. The node-clipboard package didn't build for me and I couldn't get node-copy-paste to work either,about the node-copy-paste README told me about the existence of the clip command-line utility, which worked for me.

On Windows, clip allows you to pipe the output of commands into your clipboard. (There are similar programs for Linux or Mac OS X.) In Node, you can start a child process and communicate with it through pipes.

I got a little lost trying to figure out how to turn a string into a streamable object that I could set as the new standard input for the clip process I was going to spawn, but the solution turned out to be much simpler than that. Just write(...) to the appropriate stream, and call end() when you're done.

Here's the relevant bit of code that takes my result array and puts it into my clipboard:

var child = cp.spawn('clip'); child.stdin.write(result.join("\n")); child.stdin.end();

Of course, to get to that point, I had to revise my script. Instead of letting all the callbacks finish whenever they wanted, I needed to be able to run some code after everything was done. I was a little familiar with the async library, so I used that. I copied the output to the clipboard instead of displaying it so that I could call it easily using ! (dired-do-shell-command) and get the output in my clipboard for easy yanking elsewhere, although I could probably change my batch file to pipe the result to clip and just separate the stderr stuff. Hmm. Anyway, here it is!

See this on Github

/**
 * Upload the file to my Flickr sketchbook and then move it to
 * Dropbox/Inbox/To blog. Save the Org Mode links in the clipboard. -
 * means the photo already existed, + means it was uploaded.
 */

var async = require('async');
var cp = require('child_process');
var fs = require('fs');
var glob = require('glob');
var path = require('path');
var flickr = require('flickr-with-uploads');
var secret = require("./secret");
var SKETCHBOOK_PHOTOSET_ID = '72157641017632565';
var BLOG_INBOX_DIRECTORY = 'c:\\sacha\\dropbox\\inbox\\to blog\\';
var api = flickr(secret.flickrOptions.api_key,
                 secret.flickrOptions.secret,
                 secret.flickrOptions.access_token,
                 secret.flickrOptions.access_token_secret);
var result = [];

function getTags(filename) {
  var tags = [];
  var match;
  var re = new RegExp('#([^ ]+)', 'g');
  while ((match = re.exec(filename)) !== null) {
    tags.push(match[1]);
  }
  return tags.join(' ');
}
// assert(getTags("foo #bar #baz qux") == "bar baz");

function checkIfPhotoExists(filename, doesNotExist, existsFunction, done) {
  var base = path.basename(filename).replace(/.png$/, '');
  api({method: 'flickr.photos.search',
       user_id: secret.flickrOptions.user_id,
       text: base},
      function(err, response) {
        var found = undefined;
        if (response && response.photos[0].photo) {
          for (var i = 0; i < response.photos[0].photo.length; i++) {
            if (response.photos[0].photo && response.photos[0].photo[i]['$'].title == base) {
              found = i; break;
            }            
          }
        }
        if (found !== undefined) {
          existsFunction(response.photos[0].photo[found], done);
        } else {
          doesNotExist(filename, done);
        }
      });
}

function formatExistingPhotoAsOrg(photo, done) {
  var title = photo['$'].title;
  var url = 'https://www.flickr.com/photos/'
        + photo['$'].owner
        + '/' + photo['$'].id;
  result.push('- [[' + url + '][' + title + ']]');
  done();
}

function formatAsOrg(response) {
  var title = response.photo[0].title[0];
  var url = response.photo[0].urls[0].url[0]['_'];
  result.push('+ [[' + url + '][' + title + ']]');
}

function uploadImage(filename, done) {
  api({
    method: 'upload',
    title: path.basename(filename.replace(/.png$/, '')),
    is_public: 1,
    hidden: 1,
    safety_level: 1,
    photo: fs.createReadStream(filename),
    tags: getTags(filename.replace(/.png$/, ''))
  }, function(err, response) {
    if (err) {
      console.log('Could not upload photo: ', err);
      done();
    } else {
      var newPhoto = response.photoid[0];
      async.parallel(
        [
          function(done) {
            api({method: 'flickr.photos.getInfo',
                 photo_id: newPhoto}, function(err, response) {
                   if (response) { formatAsOrg(response); }
                   done();
                 });
          },
          function(done) {
            api({method: 'flickr.photosets.addPhoto',
                 photoset_id: SKETCHBOOK_PHOTOSET_ID,
                 photo_id: newPhoto}, function(err, response) {
                   if (!err) {
                     moveFileToBlogInbox(filename, done);
                   } else {
                     console.log('Could not add ' + filename + ' to Sketchbook');
                     done();
                   }
                 });
          }],
        function() {
          done();
        });
    }
  });
}

function moveFileToBlogInbox(filename, done) {
  fs.rename(filename, BLOG_INBOX_DIRECTORY + path.basename(filename),
            function(err) {
              if (err) { console.log(err); }
              done();
            });
}

var arguments = process.argv.slice(2);
async.each(arguments, function(item, done) {
  if (item.match('\\*')) {
    glob.glob(item, function(err, files) {
      if (!files) return;
      async.each(files, function(file, done) {
        checkIfPhotoExists(file, uploadImage, formatExistingPhotoAsOrg, done);
      }, function() {
        done();
      });
    });
  } else {
    checkIfPhotoExists(item, uploadImage, formatExistingPhotoAsOrg, done);
  }
}, function(err) {
  console.log(result.join("\n"));
  var child = cp.spawn('clip');
  child.stdin.write(result.join("\n"));
  child.stdin.end();
});

Wheeee! Hooray for automation. I made a Windows batch script like so:

up.bat

node g:\code\node\flickr-upload.js %*

and away I went. Not only did I have a handy way to process images from the command line, I could also mark the files in Emacs Dired with m, then type ! to execute my up command on the selected images. Mwahaha!

Anyway, I thought I'd write it up in case other people were curious about using Node to code little utilities, filling the clipboard in Windows, or getting data back into Emacs (sometimes the clipboard is enough).

Back to org-protocol, since I was curious about it. With (require 'org-protocol) (server-start), emacsclient org-protocol://store-link:/foo/bar worked when I entered it at the command prompt. I was having a hard time getting it to work under Node, but eventually I figured out that:

• I needed to pass -n as one of the arguments to emacsclient so that it would return right away.
• The : after store-link is important! I was passing org-protocol://store-link/foo/bar and wondering why it opened up a file called bar. org-protocol://store-link:/foo/bar was what I needed.

I only just figured out that last bit while writing this post. Here's a small demonstration program:

var cp = require('child_process');
var child = cp.execFile('emacsclient', ['-n', 'org-protocol://store-link:/foo/bar']);

Yay!

You can view 5 comments or e-mail me at sacha@sachachua.com.

I scan my sketches or draw them on the computer, and then I upload these sketches to Flickr using photoSync, which synchronizes folders with albums. I include these sketches in my outlines and blog posts, and I update my index of blog posts every month. I recently added a tweak to make it possible for people to go from a blog post to its index entry, so it should be easier to see a post in context. I've been thinking about keeping an additional info index to manage blog posts and sketches, including unpublished ones. We'll see how well that works. Lastly, I want to link my Flickr posts to my blog posts so that people can see the context of the sketch.

My higher goal is to be able to easily see the open ideas that I haven't summarized or linked to yet. There's no shortage of new ideas, but it might be interesting to revisit old ones that had a chance to simmer a bit. I wrote a little about this in Learning from artists: Making studies of ideas. Let me flesh out what I want this archive to be like.

2015.01.05 Thinking about my archive

When I pull on an idea, I'd like to be able to see other open topics attached to it. I also want to be able to see open topics that might jog my memory.

How about the technical details? How can I organize my data so that I can get what I want from it?

2015.01.05 Figuring out the technical details of this idea or visual archive I want – index card

Because blog posts link to sketches and other blog posts, I can model this as a directed graph. When I initially drew this, I thought I might be able to get away with an acyclic graph (no loops). However, since I habitually link to future posts (the time traveller's problem!), I can't make that simplifying assumption. In addition, a single item might be linked from multiple things, so it's not a simple tree (and therefore I can't use an outline). I'll probably start by extracting all the link information from my blog posts and then figuring out some kind of Org Mode-based way to update the graph.

To get one step closer to being able to see open thoughts and relationships, I decided that my sketches on Flickr:

• should not have duplicates despite my past mess-ups, so that:
  • I can have an accurate count
  • it's easier for me to categorize
  • people get less confused
• should have hi-res versions if possible, despite the IFTTT recipe I tried that imported blog posts but unfortunately picked up the low-res thumbnails instead of the hi-res links
• should link to the blog posts they're mentioned in, so that:
  • people can read more details if they come across a sketch in a search
  • I can keep track of which sketches haven't been blogged yet

I couldn't escape doing a bit of manual cleaning up, but I knew I could automate most of the fiddly bits. I installed node-flickrapi and cheerio (for HTML parsing), and started playing.

exports.trimTitle = function(str) {
    return str.replace(/ --.*$/g, '').replace(/#[^ ]+/g, '').replace(/[- _]/g, '');
};

exports.keepNewPhoto = function(oldPhoto, newPhoto) {
    if (newPhoto.title.length > oldPhoto.title.length)
        return true;
    if (newPhoto.title.length < oldPhoto.title.length)
        return false;
    if (newPhoto.id < oldPhoto.id) 
        return true;
    return false;
};

exports.processPhoto = function(p, flickr) {
    var trimmed = exports.trimTitle(p.title);
    if (trimmed && hash[trimmed] && p.id != hash[trimmed].id) {
        // We keep the one with the longer title or the newer date
        if (exports.keepNewPhoto(hash[trimmed], p)) {
            exports.possiblyDeletePhoto(hash[trimmed], flickr);
            hash[trimmed] = p;
        }
        else if (p.id != hash[trimmed].id) {
            exports.possiblyDeletePhoto(p, flickr);
        }
    } else {
        hash[trimmed] = p;
    }
};

Linking to blog posts

This was the trickiest part, but also the most fun. I took advantage of the fact that WordPress transforms uploaded filenames in a mostly consistent way. I'd previously added a bulk view that displayed any number of blog posts with very little additional markup, and I modified the relevant code in my theme to make parsing easier.

See this on Gist:

/**
 * Adds "Blogged" links to Flickr for images that don't yet have "Blogged" in their description.
 * Command-line argument: URL to retrieve and parse
 */

var secret = require('./secret');
var flickrOptions = secret.flickrOptions;
var Flickr = require("flickrapi");
var fs = require('fs');
var request = require('request');
var cheerio = require('cheerio');
var imageData = {};
var $;

function setDescriptionsFromURL(url) {
  request(url, function(error, response, body) {
    // Parse the images
    $ = cheerio.load(body);
    $('article').each(function() {
      var prettyLink = $(this).find("h2 a").attr("href");
      if (!prettyLink.match(/weekly/i) && !prettyLink.match(/monthly/i)) {
        collectLinks($(this), prettyLink, imageData);
      }
    });
    updateFlickrPhotos();
  });
}

function updateFlickrPhotos() {
    Flickr.authenticate(flickrOptions, function(error, flickr) {
      flickr.photos.search(
        {user_id: flickrOptions.user_id,
         per_page: 500,
         extras: 'description',
         text: ' -blogged'}, function(err, result) {
           processPage(result, flickr);
           for (var i = 2 ; i < result.photos.pages; i++) {
             flickr.photos.search(
               {user_id: flickrOptions.user_id, per_page: 500, page: i,
                extras: 'description', text: ' -blogged'},
               function(err, result) {
                 processPage(err, result, flickr);
               });
           }
         });
    });
}

function collectLinks(article, prettyLink, imageData) {
  var results = [];
  article.find(".body a").each(function() {
    var link = $(this);
    if (link.attr('href')) {
      if (link.attr('href').match(/sachachua/)
          || !link.attr('href').match(/^http/)) {
        imageData[exports.trimTitle(link.attr('href'))] = prettyLink;
      } else if (link.attr('href').match(/flickr.com/)) {
        imageData[exports.trimTitle(link.text())] = prettyLink;
      }
    }
  });
  return results;
}

exports.trimTitle = function(str) {
  return str.replace(/^.*\//, '').replace(/^wpid-/g, '').replace(/[^A-Za-z0-9]/g, '').replace(/png$/, '').replace(/[0-9]$/, '');
};

function processPage(result, flickr) {
  if (!result) return;
  for (var i = 0; i < result.photos.photo.length; i++) {
    var p = result.photos.photo[i];
    var trimmed = exports.trimTitle(p.title);
    var noTags = trimmed.replace(/#.*/g, '');
    var withTags = trimmed.replace(/#/g, '');
    var found = imageData[noTags] || imageData[withTags];
    if (found) {
      var description = p.description._content;
      if (description.match(found)) continue;
      if (description) {
        description += " - ";
      }
      description += '<a href="' + found + '">Blogged</a>';
      console.log("Updating " + p.title + " with " + description);
      flickr.photos.setMeta(
        {photo_id: p.id,
         description: description},
        function(err, res) {
          if (err) { console.log(err, res); }
        } );
    }
  }
}

setDescriptionsFromURL(process.argv[2]);

And now sketches like

are now properly linked to their blog posts. Yay! Again, this script won't get everything, but it gets a decent number automatically sorted out.

Next steps:

• Run the image extraction and set description scripts monthly as part of my indexing process
• Check my list of blogged images to see if they're matched up with Flickr sketches, so that I can identify images mysteriously missing from my sketchbook archive or not correctly linked

Yay code!

You can view 1 comment or e-mail me at sacha@sachachua.com.