Category Archives: automation

Developing Emacs micro-habits: Abbreviations and templates

When it comes to improving how you use Emacs, picking one small change and paying close attention seems to work well. Little things make a lot of difference, especially when frequently repeated over a long period of time. It reminded me of this quote I came across on Irreal:

I’ve gotten the hang of basic multiple-cursors-mode and I’m making gradual progress towards internalizing smart-parens by the simple approach of focusing on one tiny habit at a time. For example, I spent a week reminding myself to use mc/mark-all-like-this-dwim or mc/mark-lines instead of using keyboard macros.

Inspired by the Emacs Advent Calendar, I wanted to start a 52-week series on micro-habits for more effective Emacs use. I brain-dumped an outline of four sets (basic Emacs, Org, programming, meta-habits) of thirteen small tips each. Looking at my list, I realized there were many ideas there that I hadn’t quite gotten the hang of myself. I figured that this might be more of a project for 2016; in the meantime, I could learn by doing.

The first micro-habit I wanted to dig into was that of automating text: abbreviations, templates, and other ways to expand or transform text. I’d used Yasnippet before. I sometimes accidentally expanded keywords instead of indenting lines if my cursor happened to be at the wrong spot. But I hadn’t yet drilled the instinct of automation or the familiarity with templates into my fingers.

This blog post isn’t the easy-to-understand guide to automating text. I’ll write that later, when I’ve figured more things out. In the meantime, I’ll share what I’ve been learning and thinking so far, and maybe you can help me make sense of it.

Emacs has a separate manual for autotyping, which I had never read before. The short manual covers abbrev, skeleton, auto-insert, copyright messages, timestamps, and tempo. Did you know that define-skeleton lets you create a template that accepts multiple interregions if you call your skeleton function with a negative argument? (Interregions? What are those?) It took me an embarrassing amount of time to figure out how to mark interregions and use them. It turns out they have to be contiguous. It might be easier to think of marking the beginning of the region, marking some points in the middle, and then calling the command when your point is at the end – which is probably how most people would interpret the diagrams, but I was trying to mark discontinuous regions because that would be super-cool, and that totally didn’t work. And then I forgot that using helm-M-x means you need to specify numeric arguments after typing M-x instead of before. (I wrote about that very point in one of my blog posts, but it slipped my mind.) Once I got past that, I was delighted to find that it worked as advertised. I still haven’t imagined a situation where I would use it, but it seems like a good sort of thing to know.

What are the practical situations where text automation can help people work more effectively? I looked around to see how other people were using it. Coding, of course – especially if you use Emacs Lisp to transform the text. Debugging, too. Marking up text. Remembering parameters. Wrapping regions. Writing e-mails. Adding blog post metadata. Citing references. Lifehacker has a long list, too.

I came up with several categories I’m going to focus on so that I can more easily recognize opportunities to work better:

2015-01-05 Seeing opportunities for abbreviations and text automation -- index card

2015.01.05 Seeing opportunities for abbreviations and text automation – index card

  • Abbreviations are about typing long words with fewer keystrokes. For example, you might shorten “description” to desc.
  • Phrases are like word abbrevations, but longer. You might want to be able to expand btw to “by the way.”
  • Code benefits from expansion in multiple ways:
    • Automatically inserting characters that are harder to reach on a keyboard, like { and }
    • Being consistent about coding style, like the way many people like adding a comment after the closing brace of an if
    • Transforming text that shows up in multiple places, such as variable names that need getters and setters
    • Filling in the blanks: parameters, comments, etc.
    • Reducing the cognitive load of switching between languages by establishingq a common vocabulary. For example, I sometimes need to look up the syntax of for or the proper way to display a debugging statement when I switch to a language I haven’t used in a while
  • Templates are also useful for consistency in writing, planning, and other areas
  • Text transformation can save time and minimize error.

2015-01-04 Automating text - index card

2015.01.04 Automating text – index card

Translating the examples I’d seen to my personal interests, I could probably find plenty of opportunities to automate text while coding, debugging, writing, planning, or publishing. To dig deeper, I looked at each of the categories in detail.


2015-01-06 Abbreviations -- index card

2015.01.06 Abbreviations – index card

When I was curious about typing faster, I read forum posts from people who had increased their speed by developing their own form of digital shorthand. The trick works on paper, too. When I need to write quickly or in limited space, I use abbreviations like bc for “because” and w/o for “without.” Why not on the computer as well?

I often take advantage of dynamic abbreviations when I know I’ve recently typed the word I want. To trigger those, I just have to type the beginning of the word and then use dabbrev-expand. I haven’t set up my own static abbreviations, though. Main obstacles:

  • I want to write shorter words instead of longer ones
  • In the beginning, it’s faster to type the word instead of thinking of the abbreviation and expanding it
  • If I have to undo or backspace, that makes me slower
  • If I burn this into my muscle memory, I might be more frustrated on other computers or in other apps (then again, I already customize Emacs extensively, so I guess I’m okay with the tradeoff)

Anyway, here’s a short list I’m trying out with define-global-abbrev and hippie-expand:

hw however
bc because
wo without
prob probably
st sometimes

Hmm. Let’s say that it takes me two keystrokes to trigger the expansion, whether it’s the xx keychord I’ve just set up or the M-/ I’ve replaced with hippie-expand. (Hmm, maybe a double-space keychord is a good candidate for expansion too.) Is the retraining worth a ~50% possible reduction in keystrokes? Probably not.

How about text with punctuation, so I can minimize reaching for symbols?

mail [email protected]

Maybe it’s better to look at the words I frequently misspell, or that I tend to slow down then typing. I’ll keep an eye out for those.


2015-01-06 Phrases -- index card

2015.01.06 Phrases – index card

Phrases are an easier sell. Still, I’m trying not to settle into the rut of set phrases. I should cut those mercilessly or avoid writing them from the beginning.

co check out
iti I think I
otoh on the other hand,
mean in the meantime,
fe for example
fi for instance,
oc of course
ip in particular

Code insertion

This is, fortunately, well-trodden ground. The yasnippet package comes with a large collection of snippets for many programming languages. You can start by familiarizing yourself with the pre-defined snippets for the modes that you use. For example, in my installation, they’re under ~/.emacs.d/elpa/yasnippet-20141117.327/snippets. You can use the filename (or keywords defined with key, if specified) as the abbreviation, and you can expand them with yas-expand (which should be bound to TAB if you have yas-global-mode on).

I mostly work with HTML, CSS, Javascript, Ruby on Rails, and Emacs Lisp, so this is the cheat sheet I’ve made for myself:

2015-01-07 Code insertion -- index card

2015.01.07 Code insertion – index card

For HTML, I need to remember that the tags are generally expandable, and that there are a few Lorem Ipsum abbreviations triggered by lorem.1 through .5. CSS has a v abbreviation that sets up a bunch of rules with vendor prefixes. For Javascript, I’ll probably start with f to define a function and log to output something to console.log. Rails has a bunch of iterators like eai that look interesting. As for Emacs Lisp, the pre-defined templates generally add parentheses around common functions so you don’t have to type them, and there are a few shortcuts like bs for buffer-string and cc for condition-case. I think I’ll modify the default snippets to make better use of Yasnippet’s field support, though, so that I don’t have to delete and replace text.


In addition to using text expansion for code, you can use it for planning and writing other text. I saw Karl Voit use it to great effect in my Emacs Chat with him (around the 44:00 mark), and I’ve been gradually refining some templates of my own.

2015-01-07 Templates -- index card

2015.01.07 Templates – index card

For example, here’s the template I’ve been using for sketched books. Note: If you use Yasnippet for Org Mode properties, you may want to set yas-indent-line to fixed or the fields will get confused.

Gist: sbook

# key: sbook
# name: Sketched Book
# --

**** TOSKETCH ${1:short title}
      :TITLE: ${2:long title}
      :SHORT_TITLE: $1
      :AUTHOR: ${3:authors}
      :YEAR: ${4:year}
      :BUY_LINK: ${5:Amazon link}
      :BASENAME: ${6:`(org-read-date nil nil ".")`} Sketched Book - ${2:$(sacha/convert-sketch-title-to-filename yas-text)} - ${3:$(sacha/convert-sketch-title-to-filename yas-text)}
      :ISBN: ${7:ISBN}


***** TODO Sketchnote $1
:Effort: 2:00

[[elisp:sacha/prepare-sketchnote-file][Prepare the file]]

***** TODO Write personal reflection for $1
:Effort: 1:00

[[][View in calendar]]

****** Sketched Book - $2 - $3

$3's /$2/ ($4) ...

I’ve sketched the key points of the book below to make it easier to remember and share. Click on the image for a larger version that you can print if you want.

Haven't read the book yet? You can [[$5][buy it from Amazon]] (affiliate link) or get it from your favourite book sources.

Like this sketch? Check out [[][]] for more. Feel free to share – it’s under the Creative Commons Attribution License, like the rest of my blog.

***** TODO Post $1 to blog
:Effort: 1:00
:QUANTIFIED: Packaging

***** TODO Update sketched books collection
:Effort: 1:00
:QUANTIFIED: Packaging

1. [[elisp:sacha/index-sketched-book][Index sketched book]]
   - [[file:~/Dropbox/Packaging/sketched-books/][Edit index]]
   - [[file:~/Dropbox/Packaging/sketched-books/][Edit ebook]]
2. [[elisp:sacha/package-sketched-book][Compile]]
3. Update [[]]

***** TODO Tweet sneak peek of $1 with attached picture

[[elisp:(progn (kill-new (format "Sneak peek: Sketched Book: %s - %s %s" (org-entry-get-with-inheritance "SHORT_TITLE") (org-entry-get-with-inheritance "AUTHOR") (org-entry-get-with-inheritance "BLOG_POST"))) (browse-url ""))][Copy text and launch Twitter]]

It’s a lot of code and I keep tweaking it as I come across rough corners, but it’s handy to have that all captured in a template that I can easily expand. =)

Text transformation

One of the advantages of tweaking text expansion inside Emacs instead of using a general-purpose text expansion program is that you can mix in some Emacs Lisp to transform the text along the way. I’m still thinking about how to make the most of this, as you can see from this half-filled note-card:

2015-01-07 Text transformation as part of expansion -- index card

2015.01.07 Text transformation as part of expansion – index card

For example, this snippet makes it easier to share source code on my blog while also linking to a Gist copy of the code, in case I revise it or people want to comment on the code snippet itself. It doesn’t use any of the built-in text expansion capabilities, but I think of it as a text expander and transformer because it replaces work I used to do manually. You’ll need the gist package for this one.

Gist: Sacha (updated to clean up region code)

(defun sacha/copy-code-as-org-block-and-gist (beg end)
  (interactive "r")
  (let ((filename (file-name-base))
        (mode (symbol-name major-mode))
         (if (use-region-p) (buffer-substring beg end) (buffer-string)))
        (gist (if (use-region-p) (gist-region beg end) (gist-buffer))))
     (format "\n[[%s][Gist: %s]]\n#+begin_src %s\n%s\n#+end_src\n"
             (oref (oref gist :data) :html-url) filename
             (replace-regexp-in-string "-mode$" "" mode)

Both Yasnippet and Skeleton allow you to use Lisp expressions in your template. If you don’t have all the data yet, you might consider writing another Lisp function that you can call later when you do. For example, in the sketched books code above, I have an Emacs Lisp link that composes a tweet with a link, puts it in the clipboard, and then opens up a web browser. (I do this instead of posting directly because I also want to attach an image to that tweet, and I haven’t figured out how to modify any of the Emacs Twitter clients to do that.)

So that’s what I’ve learned so far about automating text in Emacs. It’ll take me more than a week to get the hang of the abbreviations I’ve just set up, and I’ll probably need to add even more before adding and using abbreviations become true habits. But hey, maybe this will help you pay closer attention to repetitive text and editing actions in Emacs so that you can automate them too, and we can swap notes on useful abbreviations. What kind of text do you expand?

For more information, see:

Automating bulk web stuff with iMacros

I found myself needing to download a whole bunch of JSON data from a server that had a weird authentication thing that Chrome could deal with but wget/curl/Ruby couldn’t. Since my Firefox was on the fritz, I couldn’t use Selenium IDE or the Selenium Webdriver. iMacros to the rescue! (Chrome, Firefox)

2013-11-21 Geek notes - automating bulk web stuff with iMacros

That plus lots of keyboard macros and text manipulation in Emacs, plus a little parsing and regexp substitution in Ruby, plus more Emacs munging got me the data I wanted. Hooray for bubblegum and string scripting!

Back to the joys of coverage testing: Vagrant, Guard, Spork, RSpec, Simplecov

Tests are important because programmers are human.  I know that I’m going to forget, make mistakes, and change things that I didn’t mean to change. Testing lets me improve my chances of at least noticing. Coverage tools show me how much of my code is covered by tests, improving my chances of seeing where my blind spots are.

In one of my last projects at IBM, I convinced my team to use coverage testing tools on a Rails development project. It was great watching the coverage numbers inch up, and we actually reached 100% (at least in terms of what rcov was looking at). I occasionally had to fix things when people broke the build, but sometimes people added and updated tests too. Although coverage testing has its weaknesses (are you testing for edge cases?), it’s better than nothing, and can catch some embarrassing bugs before they make it to the outside world.

Although I’m not currently taking on any web development work (I’m saving brainspace for other things), I have a few personal projects that I enjoy working on. For example, lets me track different measurements such as time. I had written some tests for it before, but since then, I’d been adding features and making changes without updating the tests. Like the way that unpaid credit card balances turn into financial debt (very bad habit, try to avoid this if possible), unwritten or out-of-date tests contribute to technical debt.

It was a little daunting to sit down and slowly work my way through the knots of out-of-date tests that had accumulated from haphazard coding. Here’s how I made the task more manageable:

  • I set up a virtual Linux development environment using Vagrant and Puppet. Many tools are much easier to set up on Linux than they are on Windows, and using a virtual machine for development means that I can easily isolate changes from the rest of my system. Vagrant allows you to specify the configuration of a VirtualBox machine and quickly set up a new instance of it. I copied the Puppet configuration from rails-dev-box and modified it to match my requirements. I also compiled and installed Emacs 24 from source, along with a few Ruby-related tools like haml-mode and rinari. (If you’re going to do a lot of development, you might as well do it with your favourite tools!) I copied the Vagrant private keys and imported them into Pageant (an ssh agent for Windows) for easy logins, and I forwarded the ports too.
  • I used Guard and Spork to efficiently run subsets of my tests. Guard is a tool that re-runs specific tests when it detects that files have changed, while Spork lets you run tests without reloading your entire Rails application. Instead of running rake spec (which runs the whole test suite) or rspec ./spec/path/to/file (and  having to copy and paste the failing test’s filename), I could let Guard take care of rerunning failed tests for me.
  • I worked on getting the old tests to pass. I wanted to get those cleared up before writing new tests. Good thing too – found a few bugs and some old code along the way.
  • I reviewed the recommendations for better tests. Better Specs has tips on using RSpec with Ruby.
  • Then I wrote new tests to cover the rest of my models. I used Simplecov for coverage testing. Since Vagrant uses shared folders, I could use Google Chrome to view the coverage webpages from my host computer. I still have some technical debt left – I need to write integration tests, and there’s more beyond that – but it’s satisfying to know that the models (data + logic) are covered. For some reason, my Guard+Spork combination doesn’t result in cumulative Simplecov scores, so I occasionally regenerate my coverage stats with rake spec. These tests will be useful as I write new features or move to Rails 4.


(… 21.5 hours of coding/development/infrastructure…)

New things I learned:

I use Mechanize to work with the Toronto Public Library’s webpage system so that I can retrieve the list of due books, renew items, or request holds. I wanted to test these as well. FakeWeb lets you intercept web requests and return your own responses, so I saved sample web pages to my spec/fixtures/files directory and used FakeWeb to return them. (ex: toronto_library_spec.rb)

Paperclip also needed some tweaking. I replaced my Paperclip uploads with File assignments so that I could test the image processing. (ex: clothing_spec.rb)

Always remember to wrap your RSpec tests in an it “…” instead of just context “when …” or describe “…”. I forgot to do this a few times and got confused about why stuff got left in my database and why the error messages weren’t formatted like they normally are.

Progress! Next step: Update my Cucumber tests and add some more integration tests…

I like this part of being a developer, even though writing new code is more fun. Testing is one of those time-consuming but powerful tools that can reduce frustration in the long run. I’ve experimented with applying automated testing techniques to everyday life before. I wonder what it would be like to take that even further… Any thoughts?

Sketchnotes: The Very Versatile Drip–Mathew Sweezey (Pardot)

UPDATE: Dec 13, 2012 Want to watch the webinar? Here’s the video recording.

In this marketing webinar hosted by Pardot, Mathew Sweezey shared tips on setting up a drip nurturing program for marketing and sales support. Click on the image to view a larger size, and feel free to share this with attribution!

20121206 Pardot - The Very Versatile Drip - Mathew Sweezey

Pardot has many other webinars and recordings, so check them out if you’re curious about marketing automation.

Like this? Browse through my other sketchnotes, including my visual summary of The 5 Key Elements of a Better B2B Content Marketing Strategy by Nolin LeChasseur. I sketchnote technology/business conferences and presentations – if that sounds interesting, get in touch!

[

Sketchnotes: Marketing Automation, Jeffrey Yee (#torontob2b)

UPDATE 2012-11-15: Here’s the video recap!

Marketing Automation
Jeffrey Yee, Eloqua

Like these? Check out my other sketchnotes, visual book notes/reviews, and visual metaphors.

Here’s the text from the sketchnotes to improve people’s ability to search for it:

Marketing automation

Marketing Automation
Jeffrey Yee, Eloqua

leads small
list management
-Too expensive
-Not fully used
-Not implemented correctly
-Did not address business needs

1. Focus
one thing! business need!
2. Identify
Look for what your top performers are already doing
3. Start small, then build for mass adoption
-Target the second-tier salespeople!
4. Wait patiently for the lift.
incremental improvement

Best practices from client side
Dun & Bradstreet
credit risk management sales & marketing supply risk management

1. Focus
Retention trigger-based e-mail
one need
40.1% opens
13.4% click through
10% increase in retention rates
2. Identify before you automate
Focus group?
Study top performers
How are we achieving this today?
Can we automate and scale this?


Think linear, it’s easier that way

Get personal and add value
plaint text e-mail from sales, not marketing
3. Mass adoption (but start very small)
advocates get others on board

Look for the people who are close to their quotas:
Tier 2 segmenting your salespeople!

Have reps vet leads before adding to program

3rd party data
4. Wait patiently for the lift. Set expectations.
Ex results
-6 months
pipeline value *19%
# of yes 14%
average upsize 3%
ops won 25%

Budget 12+ months

Like low-hanging fruit
Scaling up what already works
Notes by Sacha Chua, @sachac,


Drush, Simpletest, and continuous integration for Drupal using Jenkins (previously Hudson)

One of my development goals is to learn how to set up continuous integration so that I’ll always remember to run my automated tests. I picked up the inspiration to use Hudson from Stuart Robertson, with whom I had the pleasure of working on a Drupal project before he moved to BMO. He had set up continuous integration testing with Hudson and Selenium on another project he’d worked on, and they completed user acceptance testing without any defects. That’s pretty cool. =)

I’m a big fan of automated testing because I hate doing repetitive work. Automated tests also let me turn software development into a game, with clearly defined goalposts and a way to keep score. Automated tests can be a handy way of creating lots of data so that I can manually test a site set up the way I want it to be. I like doing test-driven development: write the test first, then write the code that passes it.

Testing was even better with Rails. I love the Cucumber testing framework because I could define high-level tests in English. The Drupal equivalent (Drucumber?) isn’t quite there yet. I could actually use Cucumber to test my Drupal site, but it would only be able to test the web interface, not the code, and I like to write unit tests in addition to integration tests. Still, some automated testing is better than no testing, and I’m comfortable creating Simpletest classes.

Jenkins (previously known as Hudson) is a continuous integration server that can build and test your application whenever you change the code. I set it up on my local development image by following Jenkins’ installation instructions. I enabled the Git plugin (Manage Jenkins – Manage Plugins – Available).

Then I set up a project with my local git repository. I started with a placeholder build step of Execute shell and pwd, just to see where I was. When I built the project, Hudson checked out my source code and ran the command. I then went into the Hudson workspace directory, configured my Drupal settings.php to use the database and URL I created for the integration site, and configured permissions and Apache with a name-based virtual host so that I could run web tests.

For build steps, I used Execute shell with the following settings:

mysql -u integration integration < sites/default/files/backup_migrate/scheduled/site-backup.mysql
/var/drush/drush test PopulateTestUsersTest
/var/drush/drush test PopulateTestSessionsTest
/var/drush/drush testre MyProjectName --error-on-fail

This loads the backup file created by Backup and Migrate, sets up my test content, and then uses my custom testre command.

Code below (c) 2011 Sacha Chua ([email protected]), available under GNU General Public License v2.0 (yes, I should submit this as a patch, but there’s a bit of paperwork for direct contributions, and it’s easier to just get my manager’s OK to blog about something…)

// A Drush command callback.
function drush_simpletest_test_regular_expression($test_re='') {
  global $verbose, $color;
  $verbose = is_null(drush_get_option('detail')) ? FALSE : TRUE;
  $color = is_null(drush_get_option('color')) ? FALSE : TRUE;
  $error_on_fail = is_null(drush_get_option('error-on-fail')) ? FALSE : TRUE;
  if (!preg_match("/^\/.*\//", $test_re)) {
    $test_re = "/$test_re/";
  // call this method rather than simpletest_test_get_all() in order to bypass internal cache
  $all_test_classes = simpletest_test_get_all_classes();

  // Check that the test class parameter has been set.
  if (empty($test_re)) {
    drush_print("\nAvailable test groups & classes");
    $current_group = '';
    foreach ($all_test_classes as $class => $details) {
      if (class_exists($class) && method_exists($class, 'getInfo')) {
        $info = call_user_func(array($class, 'getInfo'));
        if ($info['group'] != $current_group) {
          $current_group = $info['group'];
          drush_print('[' . $current_group . ']');
        drush_print("\t" . $class . ' - ' . $info['name']);

  // Find test classes that match
  foreach ($all_test_classes as $class => $details) {
    if (class_exists($class) && method_exists($class, 'getInfo')) {
      if (preg_match($test_re, $class)) {
        $info = call_user_func(array($class, 'getInfo'));
        $matching_classes[$class] = $info;

  // Sort matching classes by weight
  uasort($matching_classes, '_simpletest_drush_compare_weight');

  foreach ($matching_classes as $class => $info) {
    $main_verbose = $verbose;
    $results[$class] = drush_simpletest_run_single_test($class, $error_on_fail);
    $verbose = $main_verbose;

  $failures = $successes = 0;
  foreach ($results as $class => $status) {
    print $status . "\t" . $class . "\n";
    if ($status == 'fail') {
    } else {
  print "Failed: " . $failures . "/" . ($failures + $successes) . "\n";
  print "Succeeded: " . $successes . "/" . ($failures + $successes) . "\n";
  if ($failures > 0) {
    return 1;

I didn’t bother hacking Simpletest output to match the Ant/JUnit output so that Jenkins could understand it better. I just wanted a pass/fail status, as I could always look at the results to find out which test failed.

What does it gain me over running the tests from the command-line? I like having the build history and being able to remember the last successful build.

I’m going to keep this as a local build server instead of setting up a remote continuous integration server on our public machine, because it involves installing quite a number of additional packages. Maybe the other developers might be inspired to set up something similar, though!

2011-06-09 Thu 09:51