Category Archives: geek

Getting started with Emacs? Empty your cup

Frustrated with Emacs because you’re just not as productive as you are with your old editor? Copying configuration snippets from the Web in order to force it to work like what you’re used to, but running into problems with conflicting code or obscure error messages?

Here’s something I’ve realized. To learn Emacs well, try emptying your cup. This is the story as told on the C2 wiki:

A master was trying to explain something to a student. Now this student was not a brand new student, but a senior student who had learned many things. He had knowledge and experience aplenty to draw upon. But each time the master tried to explain something new to the student, the student kept trying to hold it up against his own notions of the way the world is and how it ought be, and he was unable to see the lessons in what the master was trying to teach him.

Finally, the master poured a full serving of tea into his own cup, and into the cup of the student. Then he told the student he wanted to give to him some of the tea from his own cup. He began pouring tea from his cup into the student’s cup, but the student’s cup was already full, and all the tea from the master’s cup spilled out over the cup onto the surface below.

The student said, “Master, you can’t pour anything into my cup until I empty it to make room for what you are trying to give me.”, and the master replied “Yes I know.” “And I can’t give you any new thoughts or ideas or perspectives on life’s lessons until you clear out some thoughts that are already teeming in your mind to make room for what I have to teach you.” Then the master paused for a brief moment, meeting the student’s eyes with his own knowing look and calmly but sternly said: ” If you truly seek understanding, then first, empty your cup!”

The student pondered for a moment with a look of absolute bewilderment. Then a look of enlightenment came over him, followed by a smile, and a look of receptiveness. The master started to explain again, and this time the student saw what the master was trying to say.

2015-01-13 Emacs and the beginner's mind -- index card #emacs #beginner

2015-01-13 Emacs and the beginner’s mind – index card #emacs #beginner

It’s natural to get frustrated when you expect something should work a particular way and it doesn’t, or you’re used to working quickly and you have to slow down. “Why can’t I use Ctrl-X to cut? Why is it called ‘killing text’? Why doesn’t it work like __?” I know what that’s like; even after years of using Emacs, I sometimes still struggle to configure things that people who use other editors take for granted.

Some differences seem easy to address with code you can get on the Net. But if you do that – if you give in to your initial discomfort – you may find yourself fighting more and more of Emacs as you go along, without the skills to properly understand or integrate your changes.

It’s better, I think, to approach Emacs as a beginner. Empty your cup and let go of your expectations. Pretend this is your first editor. Go through the tutorial. Start with the basics. Try doing things the Emacs way.

In the beginning, you might feel agonizingly slow. You may need to do this after hours instead of when everyone is counting on you to deliver a time-sensitive project. It’s okay to open up Vim or your previous editor so that you can get something done, while you keep learning Emacs in the background. It’s okay to use the menu and the toolbar while you get the hang of the most common keyboard shortcuts.

As you become familiar with the system, you learn how to work within it. Slowly configure things. Get used to the changes before you introduce more. Eventually, you’ll understand the configuration snippets that other people post. Reading Emacs Lisp is the first step to being able to modify Emacs Lisp, and from there you can learn how to write Emacs Lisp. But you don’t have to start knowing that right away. Many people use Emacs for years before writing Emacs Lisp, and many people use it without customizing it.

But if you start learning Emacs by getting frustrated with it and trying to force it to be what you want, you might never get to the point where you can enjoy it and work with it. Be a beginner. Give yourself time and space to learn, even if you feel you’re learning slowly. Then, as you improve your understanding and skills, you’ll learn how to coax Emacs to fit you even better.

Let’s have a virtual Emacs conference in August – help me make it happen!

Why August? It’s an arbitrary target, although it tickles my brain to think about celebrating my 32nd birthday with awesome people sharing awesome ideas. (Incidentally, I’ll also reach the point of having been using Emacs for about half my life – doubly neat!)

Anyway. I think it would be great to have some kind of knowledge-swapping thing. Since I’m not particularly keen on travelling, not everyone can make it out to Canada, and it’s hard to make awesome in-person conference recordings anyway, maybe a virtual conference would be a great bet. I’m willing to spend what I would have paid for airfare on things like organization, speaker honoraria, and other good things.

I enjoyed the Emacs Conference in 2013, and I think we should figure out how to have these kinds of get-togethers more often. Emacs Chats and Emacs Hangouts are tiny steps in that direction, and I’d appreciate help in making this and many other community-ish things even better. =)

2015-02-02 Imagining an Emacs conference -- index card #emacs #conference #plans #organizing-people

2015-02-02 Imagining an Emacs conference – index card #emacs #conference #plans #organizing-people

So here’s what I imagine a virtual Emacs conference might be like. People volunteer, and somehow we organize a schedule of fascinating talks. This could be a full day, or maybe we’d spread it out over a couple of half-days (maybe even scheduled for different timezones so that everyone has something they can interact with life). We use Google Hangout on Air or a similar platform that can stream and automatically record. There’s the speaker with slides and screensharing, and there’s a moderator who can pick up questions from IRC and Google Hangout in order to ask them out loud. We might even be able to pull off panel discussions. Afterwards, there’s a playlist and a webpage with all the videos/MP3s/OGGs, and people can share their notes/discussions/follow-ups.

All this is immensely doable with the technology we have today. For free, even. Anyway, the technology should be okay.

What about topics? Here’s what I’m particularly curious about:

  • New features in Emacs 25 (and beyond)
  • Demos, workflows, and setup tips for popular toolsets/needs (ex: awesome setups for Clojure/CL, Rails, Javascript, C++, Java, writing, research)
  • Fascinating uses of Emacs
  • Good practices for Emacs Lisp: automated testing, performance, reliability, coding style/idioms (maybe even workshops along these lines)
  • Demystifying cool stuff: how core modules work, how to contribute to Emacs
  • A hackathon: get package.el headers on everything! fix bugs! make improvements! document!
  • Emacs microhabits, learning
  • Workshops: intermediate/advanced use of Org Mode, Calc, ESS, and other powerful packages
  • Emacs community-building and sharing

And people can suggest other topics, too. =) Maybe we can even figure out some kind of unconference setup: people suggesting topics they can share, quickly voting on what they’re interested in, and breaking up into separate “rooms” to share/discuss.

2015-02-02 Making a virtual Emacs conference happen -- index card #emacs #organizing-people #conference #planning #questions

2015-02-02 Making a virtual Emacs conference happen – index card #emacs #organizing-people #conference #planning #questions

An Emacs conference would be awesome. Here are my (pitiful) excuses for why I haven’t figured out how to organize one yet, and things I want to figure out (especially with people’s help):

  • Who might be interested in speaking? How does one go about organizing speakers, schedules, topics, tech, etc? I’m still slowly getting the hang of reaching out to people and inviting them to Emacs Chats.
  • Will people show up and ask questions? Part of me is worried that I’ll pick entirely the wrong date/time/topics and there’ll be awkward silence.
  • How can we handle questions? IRC, probably, so that people can chat about stuff too. I think I’m pretty comfortable at keeping an eye on stuff and repeating people’s questions. Or maybe people can join the Emacs Hangout if we can get the flow to be smooth?
  • Will the experience be pleasant and worthwhile? Maybe not as goosebump-inducingly awesome as being in a room with 80+ other Emacs geeks, but I think it will be worthwhile.
  • How can we harvest and share resources? Hangouts on Air will put videos on Youtube automatically, so that’ll be taken care of.
  • What would we need to do leading up to it? Something about a mailing list, and a webpage, and lots and lots of coordination.
  • Do I need to gain experience/confidence with smaller steps? Or maybe find some accomplices?

Of course, if someone wants to organize an in-person one, that’s cool too. Especially in Toronto. That would be awesome. =) (Although I might be able to get to New York or similar places too…)

My evil plans for a conference like this include:

  • Getting cool stuff out of people’s heads/fingers/configs and into a form that other people can look at, learn from, and link to
  • Ditto for good practices that can help us develop better code (performance)
  • Discovering resources and tips we might not have found out about otherwise
  • Sparking more conversations and follow-ups
  • Spurring people to create and share more resources

What could help the Emacs community learn even faster?

2015-02-01 Accelerating the Emacs community -- index card #accelerating #emacs

2015-02-01 Accelerating the Emacs community – index card #accelerating #emacs

How can we get more people sharing their configs, or learning from other people’s configs? How can we make it easier for people to share through blog posts, videos, animated GIFs, and presentations? How can we create spaces for people to connect, either with virtual meetups or in person? How can we swap interesting ideas, workflows, and mental habits? How can we improve our skills? How can we keep the conversation going?

Mm. Figuring out how to do virtual conferences might be a good start. Also, I’ve got this idea noodling around in my head on having some kind of an intermediate/advanced Org Mode workshop: something that covers clocking workflows, table calculations, literate programming, data analysis, publishing. Figuring out how to do virtual workshops would be awesome too.

Okay. First things first. Some kind of date and some kind of time, and some kind of help sorting out a schedule. August 8 and/or August 15, maybe? If librarians can hold an online conference through Google Hangouts, we should be able to figure this out too. (Librarians are super-cool!) If you have lots of experience in organizing virtual conferences or you have ideas for how to make this less intimidating for a non-organizer-y introvert, I’d love to hear from you in the comments or at [email protected]. Let’s make this happen!

Digital index piles with Emacs: Rapid categorization of Org Mode items

Somewhat daunted by the prospect of categorizing more than a hundred sketches and blog posts for my monthly review, I spent some time figuring out how to create the digital equivalent of sorting index cards into various piles.

2015-02-01 Digital piles of index card -- index card #indexing #organization #pkm

2015-02-01 Digital piles of index cards – index card #indexing #organization #pkm

In fact, wouldn’t it be super-cool if the items could automatically guess which category they should probably go in, prompting me only if it wasn’t clear?

I wanted to write a function that could take a list structured like this:

  • Keyword A
    • Previous links
  • Keyword B
    • Previous links
  • Link 1 with Keyword A
  • Link 2 with Keyword B
  • Link 3 with Keyword A
  • Link 4

It should file Link 1 and 3 under Keyword A, Link 2 under Keyword B, and prompt me for the category for Link 4. At that prompt, I should be able to select Keyword A or Keyword B, or specify a new category.

Inspired by John Kitchin’s recent post on defining a Helm source, I wanted to get it to work with Helm.

First step: I needed to figure out the structure of the list, maybe including a sample from the category to make it clearer what’s included. org-list.el seemed to have useful functions for this. org-list-struct gave me the structure of the current list. Let’s say that a category is anything whose text does not match org-bracket-link-regexp.

(defun sacha/org-get-list-categories ()
  "Return a list of (category indent matching-regexp sample).
List categories are items that don't contain links."
  (let ((list (org-list-struct)) last-category results)
    (save-excursion
      (mapc
       (lambda (x)
         (goto-char (car x))
         (let ((current-item
                (buffer-substring-no-properties
                 (+ (point)
                    (elt x 1)
                    (length (elt x 2)))
                 (line-end-position))))
           (if (string-match
                org-bracket-link-regexp
                (buffer-substring-no-properties
                 (point)
                 (line-end-position)))
               ;; Link - update the last category
               (when last-category
                 (if (< (elt x 1) (elt last-category 1))
                     (setq results
                           (cons (append last-category
                                         (list
                                          (match-string-no-properties
                                           3
                                           (buffer-substring-no-properties
                                            (point)
                                            (line-end-position)))))
                                 (cdr results))))
                 (setq last-category nil))
             ;; Category
             (setq results
                     (cons
                      (setq last-category
                            (list
                             current-item
                             (elt x 1)
                             (concat "^"
                                     (make-string (elt x 1) ?\ )
                                     (regexp-quote
                                      (concat (elt x 2)
                                              current-item))
                                     "$")))
                      results)))))
       list))
    results))

The next step was to write a function that guessed the list category based on the item text, and moved the item there.

(defvar sacha/helm-org-list-candidates nil)
(defun sacha/helm-org-list-categories-init-candidates ()
  "Return a list of categories from this list in a form ready for Helm."
  (setq sacha/helm-org-list-candidates
        (mapcar (lambda (x)
                  (cons (if (elt x 3)
                            (format "%s - %s" (car x) (elt x 3))
                          (car x))
                        x))
                (sacha/org-get-list-categories))))

(defun sacha/org-move-current-item-to-category (category)
  (when category
    (let* ((beg (line-beginning-position))
           (end (line-end-position))
           (string (buffer-substring-no-properties beg end)))
      (save-excursion
        (when (re-search-backward (elt category 2) nil t)
          (delete-region beg (min (1+ end) (point-max)))
          (forward-line 1)
          (insert (make-string (+ 2 (elt category 1)) ?\ )
                  string "\n")))) t))

(defun sacha/org-guess-list-category (&optional categories)
  (interactive)
  (require 'cl-lib)
  (unless categories
    (setq categories
          (sacha/helm-org-list-categories-init-candidates)))
  (let* ((beg (line-beginning-position))
         (end (line-end-position))
         (string (buffer-substring-no-properties beg end))
         (found
          (cl-member string
                     categories
                     :test
                     (lambda (string cat-entry)
                       (string-match (regexp-quote (downcase (car cat-entry)))
                                     string)))))
    (when (car found)
      (sacha/org-move-current-item-to-category
       (cdr (car found)))
      t)))

After that, I wrote a function that used Helm to prompt me for a category in case it couldn’t guess the category. It took me a while to figure out that I needed to use :init instead of :candidates because I wanted to read information from the buffer before Helm kicked in.

(setq sacha/helm-org-list-category-source
      (helm-build-sync-source
          "Non-link categories in the current list"
        :init 'sacha/helm-org-list-categories-init-candidates
        :candidates 'sacha/helm-org-list-candidates
        :action 'sacha/org-move-current-item-to-category
        :fuzzy-match t))

(defun sacha/org-guess-uncategorized ()
  (interactive)
  (sacha/helm-org-list-categories-init-candidates)
  (let (done)
    (while (not done)
      (save-excursion
        (unless (sacha/org-guess-list-category sacha/helm-org-list-candidates)
          (unless
              (helm :sources
                    '(sacha/helm-org-list-category-source
                      sacha/helm-org-list-category-create-source))
            (setq done t))))
      (unless done
        (setq done (not (looking-at "^[-+] \\[")))))))

The :action above refers to this function, which creates a category if it doesn’t exist yet.

(setq sacha/helm-org-list-category-create-source
      (helm-build-dummy-source
          "Create category"
        :action (helm-make-actions
                 "Create category"
                 (lambda (candidate)
                   (save-excursion
                     (let* ((beg (line-beginning-position))
                            (end (line-end-position))
                            (string (buffer-substring beg end)))
                       (delete-region beg (min (1+ end) (point-max)))
                       (org-beginning-of-item-list)
                       (insert "- " candidate "\n  " string "\n")))
                   (sacha/helm-org-list-categories-init-candidates)))))

I’m new to fiddling with Helm, so this implementation is not the best it could be. But it’s nifty and it works the way I want it to, hooray! Now I can generate a list of blog posts and unblogged sketches, categorize them quickly, and then tweak the categorizations afterwards.

2015-02-01 Index card sketches and monthly reviews -- index card #organization #pkm #indexing

2015-02-01 Index card sketches and monthly reviews – index card #organization #pkm #indexing

You can see the results in my January 2015 review, or check my config to see if the code has changed.

My next step for learning more about Helm sources is probably to write a Helm command that creates a montage of selected images. John Kitchin has a post about handling multiple selection in Helm, so I just need to combine that with my code for using Imagemagick to create a montage of images. Whee!

De-dupe and link: Using the Flickr API to neaten up my archive and link sketches to blog posts

I’ve been thinking about how to manage the relationships between my blog posts and my Flickr sketches. Here’s the flow of information:

2015-01-06 Figuring out information flow -- index card

2015.01.06 Figuring out information flow – index card

I scan my sketches or draw them on the computer, and then I upload these sketches to Flickr using photoSync, which synchronizes folders with albums. I include these sketches in my outlines and blog posts, and I update my index of blog posts every month. I recently added a tweak to make it possible for people to go from a blog post to its index entry, so it should be easier to see a post in context. I’ve been thinking about keeping an additional info index to manage blog posts and sketches, including unpublished ones. We’ll see how well that works. Lastly, I want to link my Flickr posts to my blog posts so that people can see the context of the sketch.

My higher goal is to be able to easily see the open ideas that I haven’t summarized or linked to yet. There’s no shortage of new ideas, but it might be interesting to revisit old ones that had a chance to simmer a bit. I wrote a little about this in Learning from artists: Making studies of ideas. Let me flesh out what I want this archive to be like.

2015-01-05 Thinking about my archive -- index card
2015.01.05 Thinking about my archive

When I pull on an idea, I’d like to be able to see other open topics attached to it. I also want to be able to see open topics that might jog my memory.

How about the technical details? How can I organize my data so that I can get what I want from it?

2015-01-05 Figuring out the technical details of this idea or visual archive I want -- index card
2015.01.05 Figuring out the technical details of this idea or visual archive I want – index card

Because blog posts link to sketches and other blog posts, I can model this as a directed graph. When I initially drew this, I thought I might be able to get away with an acyclic graph (no loops). However, since I habitually link to future posts (the time traveller’s problem!), I can’t make that simplifying assumption. In addition, a single item might be linked from multiple things, so it’s not a simple tree (and therefore I can’t use an outline). I’ll probably start by extracting all the link information from my blog posts and then figuring out some kind of Org Mode-based way to update the graph.

2015-01-07 Mapping the connections in my blog -- index card
2015.01.07 Mapping the connections in my blog – index card

To get one step closer to being able to see open thoughts and relationships, I decided that my sketches on Flickr:

  • should not have duplicates despite my past mess-ups, so that:
    • I can have an accurate count
    • it’s easier for me to categorize
    • people get less confused
  • should have hi-res versions if possible, despite the IFTTT recipe I tried that imported blog posts but unfortunately picked up the low-res thumbnails instead of the hi-res links
  • should link to the blog posts they’re mentioned in, so that:
    • people can read more details if they come across a sketch in a search
    • I can keep track of which sketches haven’t been blogged yet

I couldn’t escape doing a bit of manual cleaning up, but I knew I could automate most of the fiddly bits. I installed node-flickrapi and cheerio (for HTML parsing), and started playing.

Removing duplicates

Most of the duplicates had resulted from the Great Renaming, when I added tags in the form of #tag1 #tag2 etc. to selected filenames. It turns out that adding these tags en-masse using Emacs’ writable Dired mode broke photoSync’s ability to recognize the renamed files. As a result, I had files like this:

  • 2013-05-17 How I set up Autodesk Sketchbook Pro for sketchnoting.png
  • 2013-05-17 How I set up Autodesk Sketchbook Pro for sketchnoting #tech #autodesk-sketchbook-pro #drawing.png

This is neatly resolved by the following Javascript:

exports.trimTitle = function(str) {
    return str.replace(/ --.*$/g, '').replace(/#[^ ]+/g, '').replace(/[- _]/g, '');
};

and a comparison function that compared the titles and IDs of two photos:

exports.keepNewPhoto = function(oldPhoto, newPhoto) {
    if (newPhoto.title.length > oldPhoto.title.length)
        return true;
    if (newPhoto.title.length < oldPhoto.title.length)
        return false;
    if (newPhoto.id < oldPhoto.id) 
        return true;
    return false;
};

So then this code can process the photos:

exports.processPhoto = function(p, flickr) {
    var trimmed = exports.trimTitle(p.title);
    if (trimmed && hash[trimmed] && p.id != hash[trimmed].id) {
        // We keep the one with the longer title or the newer date
        if (exports.keepNewPhoto(hash[trimmed], p)) {
            exports.possiblyDeletePhoto(hash[trimmed], flickr);
            hash[trimmed] = p;
        }
        else if (p.id != hash[trimmed].id) {
            exports.possiblyDeletePhoto(p, flickr);
        }
    } else {
        hash[trimmed] = p;
    }
};

You can see the code on Gist: duplicate_checker.js.

High-resolution versions

I couldn’t easily automate this, but fortunately, the IFTTT script had only imported twenty images or so, clearly marked by a description that said: “via sacha chua :: living an awesome life…”. I searched for each image, deleting the low-res entry if a high-resolution image was already in the system and replacing the low-res entry if that was the only one there.

Linking to blog posts

This was the trickiest part, but also the most fun. I took advantage of the fact that WordPress transforms uploaded filenames in a mostly consistent way. I’d previously added a bulk view that displayed any number of blog posts with very little additional markup, and I modified the relevant code in my theme to make parsing easier.

See this on Gist:

/**
 * Adds "Blogged" links to Flickr for images that don't yet have "Blogged" in their description.
 * Command-line argument: URL to retrieve and parse
 */

var secret = require('./secret');
var flickrOptions = secret.flickrOptions;
var Flickr = require("flickrapi");
var fs = require('fs');
var request = require('request');
var cheerio = require('cheerio');
var imageData = {};
var $;

function setDescriptionsFromURL(url) {
  request(url, function(error, response, body) {
    // Parse the images
    $ = cheerio.load(body);
    $('article').each(function() {
      var prettyLink = $(this).find("h2 a").attr("href");
      if (!prettyLink.match(/weekly/i) && !prettyLink.match(/monthly/i)) {
        collectLinks($(this), prettyLink, imageData);
      }
    });
    updateFlickrPhotos();
  });
}

function updateFlickrPhotos() {
    Flickr.authenticate(flickrOptions, function(error, flickr) {
      flickr.photos.search(
        {user_id: flickrOptions.user_id,
         per_page: 500,
         extras: 'description',
         text: ' -blogged'}, function(err, result) {
           processPage(result, flickr);
           for (var i = 2 ; i < result.photos.pages; i++) {
             flickr.photos.search(
               {user_id: flickrOptions.user_id, per_page: 500, page: i,
                extras: 'description', text: ' -blogged'},
               function(err, result) {
                 processPage(err, result, flickr);
               });
           }
         });
    });
}

function collectLinks(article, prettyLink, imageData) {
  var results = [];
  article.find(".body a").each(function() {
    var link = $(this);
    if (link.attr('href')) {
      if (link.attr('href').match(/sachachua/)
          || !link.attr('href').match(/^http/)) {
        imageData[exports.trimTitle(link.attr('href'))] = prettyLink;
      } else if (link.attr('href').match(/flickr.com/)) {
        imageData[exports.trimTitle(link.text())] = prettyLink;
      }
    }
  });
  return results;
}

exports.trimTitle = function(str) {
  return str.replace(/^.*\//, '').replace(/^wpid-/g, '').replace(/[^A-Za-z0-9]/g, '').replace(/png$/, '').replace(/[0-9]$/, '');
};

function processPage(result, flickr) {
  if (!result) return;
  for (var i = 0; i < result.photos.photo.length; i++) {
    var p = result.photos.photo[i];
    var trimmed = exports.trimTitle(p.title);
    var noTags = trimmed.replace(/#.*/g, '');
    var withTags = trimmed.replace(/#/g, '');
    var found = imageData[noTags] || imageData[withTags];
    if (found) {
      var description = p.description._content;
      if (description.match(found)) continue;
      if (description) {
        description += " - ";
      }
      description += '<a href="' + found + '">Blogged</a>';
      console.log("Updating " + p.title + " with " + description);
      flickr.photos.setMeta(
        {photo_id: p.id,
         description: description},
        function(err, res) {
          if (err) { console.log(err, res); }
        } );
    }
  }
}

setDescriptionsFromURL(process.argv[2]);

And now sketches like 2013-11-11 How to think about a book while reading it are now properly linked to their blog posts. Yay! Again, this script won’t get everything, but it gets a decent number automatically sorted out.

Next steps:

  • Run the image extraction and set description scripts monthly as part of my indexing process
  • Check my list of blogged images to see if they’re matched up with Flickr sketches, so that I can identify images mysteriously missing from my sketchbook archive or not correctly linked

Yay code!

Getting data from Org Mode tables

Org Mode is an amazingly powerful package for Emacs. I’ve been learning a lot about how to use its support for plain-text tables and spreadsheet calculations.

Using table data in Emacs Lisp with the :var argument

For example, I wanted to be able to define my abbreviations in an Org Mode table in my config. I remembered coming across this technique a few weeks ago, but I couldn’t find the webpage with the code. It turned out to be simple to write from scratch. Here’s the plain text I added to my config.

#+NAME: abbrev
| Base  | Expansion                             |
|-------+---------------------------------------|
| bc    | because                               |
| wo    | without                               |
| wi    | with                                  |
| ex    | For example,                          |
| qm    | [email protected]                   |
| qe    | http://sachachua.com/dotemacs         |
| qw    | http://sachachua.com/                 |
| qb    | http://sachachua.com/blog/            |
| qc    | http://sachachua.com/blog/emacs-chat/ |

#+begin_src emacs-lisp :exports code :var data=abbrev
(mapc (lambda (x) (define-global-abbrev (car x) (cadr x))) data)
#+end_src

The :var data=abbrev argument to the Emacs Lisp source block is where all the magic happens. Here, it takes the data from the table named “abbrev” (which I set using #+NAME: before the table) and makes it available to the code. Emacs evaluates that data when the code is tangled (or exported) to my configuration. The code that’s in my Sacha.el looks like this:

(let ((data (quote (("bc" "because")
                    ("wo" "without")
                    ("wi" "with")
                    ("ex" "For example,")
                    ("email" "[email protected]")
                    ("dote" "http://sachachua.com/dotemacs")
                    ("web" "http://sachachua.com/")
                    ("blog" "http://sachachua.com/blog/")
                    ("ec" "http://sachachua.com/blog/emacs-chat/")))))
  (mapc (lambda (x) (define-global-abbrev (car x) (cadr x))) data) )

Looking up data with org-lookup-first, org-lookup-last, and org-lookup-all

You can do more complex things with Org tables, too. Inspired by Eric Boyd’s talk on his Epic Quest of Awesome (which he based on Steve Kamb‘s), I started putting together my own. I made a list of little achievements, guessed at the years, and assigned arbitrary experience points.

The achievements table had rows like this:

Approximate date Category XP Description ID
2014 Life 50 Became a Canadian citizen – link L_CAN
2014 Programming 20 Used NodeJS and AngularJS for a client project – link P_NOD
2014 Programming 5 Pulled information out of Evernote

I wanted to summarize the points by year: points gained, total points, level (according to a lookup table based on D&D experience points), and description. The lookup table was structured like this:

#+TBLNAME: levels
| Total XP | Level | Adjective             |
|----------+-------+-----------------------|
|        0 |     1 | trained-initiate      |
|     1000 |     2 | experienced           |
|     2250 |     3 | savvy                 |
|     3750 |     4 | veteran               |
|     5500 |     5 | unusually experienced |

Now for the summary table. I created rows for different years, and then I used Org Mode to fill in the rest. (Org Mode! Wow.)

| Year | Points gained | Cumulative points | Level | Adjective        |
|------+---------------+-------------------+-------+------------------|
| 1997 |             0 |                 0 |     1 | trained-initiate |
| 1998 |            10 |                10 |     1 | trained-initiate |
| 1999 |            50 |                60 |     1 | trained-initiate |
| 2000 |            50 |               110 |     1 | trained-initiate |
| 2001 |           100 |               210 |     1 | trained-initiate |
| 2002 |            60 |               270 |     1 | trained-initiate |
| 2003 |           245 |               515 |     1 | trained-initiate |
| 2004 |           115 |               630 |     1 | trained-initiate |
| 2005 |           140 |               770 |     1 | trained-initiate |
| 2006 |            60 |               830 |     1 | trained-initiate |
| 2007 |           270 |              1100 |     2 | experienced      |
| 2008 |           290 |              1390 |     2 | experienced      |
| 2009 |           205 |              1595 |     2 | experienced      |
| 2010 |           215 |              1810 |     2 | experienced      |
| 2011 |           115 |              1925 |     2 | experienced      |
| 2012 |           355 |              2280 |     3 | savvy            |
| 2013 |           290 |              2570 |     3 | savvy            |
| 2014 |           350 |              2920 |     3 | savvy            |
| 2015 |            45 |              2965 |     3 | savvy            |
#+TBLFM: $2='(calc-eval (format "vsum(%s)" (vconcat (org-lookup-all $1 '(remote(accomplishments,@2$1..@>$1)) '(remote(accomplishments,@2$3..@>$3))))))::$3=vsum(@2$2..@+0$2)::$4='(org-lookup-last $3 '(remote(levels,@2$1..@>$1)) '(remote(levels,@2$2..@>$2)) '>=);N::$5='(org-lookup-last $3 '(remote(levels,@2$1..@>$1)) '(remote(levels,@2$3..@>$3)) '>=);L

The TBLFM (table formula) line is very long, so let me break it down.

Points gained:

(calc-eval
 (format "vsum(%s)"
         (vconcat
          (org-lookup-all
           $1
           '(remote(accomplishments,@2$1..@>$1))
           '(remote(accomplishments,@2$3..@>$3))))))

This uses org-lookup-all to look up the value of the first column ($1) in the accomplishments table, from the second row to the last row @2..@>, looking in the first column ($1). It returns the values from the third column of the matching rows ($3). This is then passed through calc’s vsum function to calculate the sum.

Cumulative points: vsum(@2$2..@+0$2) is the sum of the second column $2 from the second row @2 to the current row @+0.

Level: This uses org-lookup-last to find the last value where the operator function returns true. In this case, testing the level from column $3 against each of the values in the levels table’s column $1 while the given level is greater than or equal to the value from levels. When it finds the last matching row, it returns the $2 second column from it. ;N means treat everything as a number.

org-lookup-first is like org-lookup-last, but it returns the first matching row.

Adjective: This one works like Level does, but it returns the value from column $3 instead. I found that it converted the return values to 0 if I used ;N, so I used ;L instead.

Passing data to R or other external programs

Of course, you’re not limited to things that Emacs can do. I wanted to summarize the data in graphs, so here’s what I did.

#+RESULTS: category_analysis

#+name: category_analysis
#+begin_src R :var data=accomplishments :exports both :results graphics :file quest_category.png :height 300
library(plyr)
library(ggplot2)
categories <- ddply(data, c("Category"), summarize, Points=sum(XP))
cat_sorted <- transform(categories, Category=reorder(Category, Points))
plot <- ggplot(data = cat_sorted, aes(x = Category, y = Points))
plot <- plot + geom_bar(stat="identity")
plot <- plot + geom_text(aes(label = Points, x = Category, y = Points + 10, hjust=0))
plot <- plot + scale_y_continuous(expand=c(0,70))
plot <- plot + coord_flip()
print(plot)
#+end_src

I like including source code in published pages for fellow geeks, but having the results come first gives people more context for the source block. So I named the source block using the #+name: directive and defined a #+RESULTS: directive before it. The source block used the :var argument to bring the data in from the accomplishments table. With R blocks, the data becomes available as a data frame that you can then do interesting things with. I used the :file argument to save the output to quest_category.png.

Those are a few ways that you can get data out of Org Mode tables and into Emacs Lisp, other Org Mode tables, or external programs. As I learn more about Org Mode, I find myself using it for more of the things that I used to use Microsoft Excel for – tracking, analyzing, and even graphing. I found it a little difficult to piece together what I needed to do from the manuals and examples on the Web, so I hope this explanation will help you (and that it’ll help me when I forget the syntax, as I’m sure I will). If you come up with something really neat that uses Org Mode tables, tell me what you’ve figured out!

Emacs microhabit: Switching windows with windmove, ace-window, and ace-jump

When I work with a large monitor, I often divide my Emacs frame (what most people call a window) into two or more windows (divisions within a frame). I like this more than dealing with multiple Emacs frames, even if I could spread those frames across multiple monitors. I find it easier to manage the windows using keyboard shortcuts than to manage the tiling and display of frames.

One of the Emacs micro-habits I’m working on is getting better at switching between windows. When there are only two windows, C-x o (other-window) works just fine. However, when there are three or more, it can take a few repetitions of C-x o to get to where I want. I could get around that by binding other-window to M-o instead, replacing the default keymap for that. Or I could try to get the hang of other ways to move around.

Here’s an 8-minute video showing windmove, ace-window, and ace-jump:

https://www.youtube.com/watch?v=nKCKuRuvAOw&list=UUlT2UAbC6j7TqOWurVhkuHQ

Windmove lets you move around with cursor keys, if you set up the appropriate keyboard shortcuts. Ace-window works like ace-jump. In addition, you can use C-u to swap windows and C-u C-u to delete windows. Ace-jump works across windows, so that’s handy too.

Here’s my relevant code snippet for Windmove. I changed this to use define-key instead of bind-key.

(defvar sacha/windmove-map (make-sparse-keymap))
(define-key sacha/windmove-map "h" 'windmove-left)
(define-key sacha/windmove-map "t" 'windmove-up)
(define-key sacha/windmove-map "n" 'windmove-down)
(define-key sacha/windmove-map "s" 'windmove-right)
(define-key sacha/windmove-map "[left]" 'windmove-left)
(define-key sacha/windmove-map "[up]" 'windmove-up)
(define-key sacha/windmove-map "[down]" 'windmove-down)
(define-key sacha/windmove-map "[right]" 'windmove-right)
(key-chord-define-global "yy"     sacha/windmove-map)

Here’s the cheat sheet I made for myself:

2015-01-12 Emacs microhabit - window management -- index card #emacs

2015-01-12 Emacs microhabit – window management – index card #emacs

And here’s a simpler reference that you can personalize with your own shortcuts:

2015-01-18 Emacs microhabit - Switching windows -- index card #emacs #microhabit

2015-01-18 Emacs microhabit – Switching windows – index card #emacs #microhabit

Naturally, after recording the video, I thought of a better way to manage my windows. I took advantage of the def-repeat-command that abo-abo posted on (or emacs so that I could repeat keybindings easily. I modified the function to accept nil as the first value if you don’t want the keymap to run a command by default, and to use kbd for the keybinding definitions.

  (defun sacha/def-rep-command (alist)
    "Return a lambda that calls the first function of ALIST.
It sets the transient map to all functions of ALIST,
allowing you to repeat those functions as needed."
    (lexical-let ((keymap (make-sparse-keymap))
                  (func (cdar alist)))
      (mapc (lambda (x)
              (when x
                (define-key keymap (kbd (car x)) (cdr x))))
            alist)
      (lambda (arg)
        (interactive "p")
        (when func
          (funcall func arg))
        (set-transient-map keymap t))))

Here’s my new binding for yy. It lets me bounce on y to use other-window as normal, use the arrow keys to move between windows thanks to windmove, and use ace-window as well: h is the regular ace-window, s swaps, and d deletes.

(key-chord-define-global "yy"   
      (sacha/def-rep-command
       '(nil
         ("<left>" . windmove-left)
         ("<right>" . windmove-right)
         ("<down>" . windmove-down)
         ("<up>" . windmove-up)
         ("y" . other-window)
         ("h" . ace-window)
         ("s" . (lambda () (interactive) (ace-window 4)))
         ("d" . (lambda () (interactive) (ace-window 16)))
         )))

Neat, eh?