Category Archives: automation

Back to the joys of coverage testing: Vagrant, Guard, Spork, RSpec, Simplecov

Tests are important because programmers are human.  I know that I’m going to forget, make mistakes, and change things that I didn’t mean to change. Testing lets me improve my chances of at least noticing. Coverage tools show me how much of my code is covered by tests, improving my chances of seeing where my blind spots are.

In one of my last projects at IBM, I convinced my team to use coverage testing tools on a Rails development project. It was great watching the coverage numbers inch up, and we actually reached 100% (at least in terms of what rcov was looking at). I occasionally had to fix things when people broke the build, but sometimes people added and updated tests too. Although coverage testing has its weaknesses (are you testing for edge cases?), it’s better than nothing, and can catch some embarrassing bugs before they make it to the outside world.

Although I’m not currently taking on any web development work (I’m saving brainspace for other things), I have a few personal projects that I enjoy working on. For example, lets me track different measurements such as time. I had written some tests for it before, but since then, I’d been adding features and making changes without updating the tests. Like the way that unpaid credit card balances turn into financial debt (very bad habit, try to avoid this if possible), unwritten or out-of-date tests contribute to technical debt.

It was a little daunting to sit down and slowly work my way through the knots of out-of-date tests that had accumulated from haphazard coding. Here’s how I made the task more manageable:

  • I set up a virtual Linux development environment using Vagrant and Puppet. Many tools are much easier to set up on Linux than they are on Windows, and using a virtual machine for development means that I can easily isolate changes from the rest of my system. Vagrant allows you to specify the configuration of a VirtualBox machine and quickly set up a new instance of it. I copied the Puppet configuration from rails-dev-box and modified it to match my requirements. I also compiled and installed Emacs 24 from source, along with a few Ruby-related tools like haml-mode and rinari. (If you’re going to do a lot of development, you might as well do it with your favourite tools!) I copied the Vagrant private keys and imported them into Pageant (an ssh agent for Windows) for easy logins, and I forwarded the ports too.
  • I used Guard and Spork to efficiently run subsets of my tests. Guard is a tool that re-runs specific tests when it detects that files have changed, while Spork lets you run tests without reloading your entire Rails application. Instead of running rake spec (which runs the whole test suite) or rspec ./spec/path/to/file (and  having to copy and paste the failing test’s filename), I could let Guard take care of rerunning failed tests for me.
  • I worked on getting the old tests to pass. I wanted to get those cleared up before writing new tests. Good thing too – found a few bugs and some old code along the way.
  • I reviewed the recommendations for better tests. Better Specs has tips on using RSpec with Ruby.
  • Then I wrote new tests to cover the rest of my models. I used Simplecov for coverage testing. Since Vagrant uses shared folders, I could use Google Chrome to view the coverage webpages from my host computer. I still have some technical debt left – I need to write integration tests, and there’s more beyond that – but it’s satisfying to know that the models (data + logic) are covered. For some reason, my Guard+Spork combination doesn’t result in cumulative Simplecov scores, so I occasionally regenerate my coverage stats with rake spec. These tests will be useful as I write new features or move to Rails 4.


(… 21.5 hours of coding/development/infrastructure…)

New things I learned:

I use Mechanize to work with the Toronto Public Library’s webpage system so that I can retrieve the list of due books, renew items, or request holds. I wanted to test these as well. FakeWeb lets you intercept web requests and return your own responses, so I saved sample web pages to my spec/fixtures/files directory and used FakeWeb to return them. (ex: toronto_library_spec.rb)

Paperclip also needed some tweaking. I replaced my Paperclip uploads with File assignments so that I could test the image processing. (ex: clothing_spec.rb)

Always remember to wrap your RSpec tests in an it “…” instead of just context “when …” or describe “…”. I forgot to do this a few times and got confused about why stuff got left in my database and why the error messages weren’t formatted like they normally are.

Progress! Next step: Update my Cucumber tests and add some more integration tests…

I like this part of being a developer, even though writing new code is more fun. Testing is one of those time-consuming but powerful tools that can reduce frustration in the long run. I’ve experimented with applying automated testing techniques to everyday life before. I wonder what it would be like to take that even further… Any thoughts?

Automating bulk web stuff with iMacros

I found myself needing to download a whole bunch of JSON data from a server that had a weird authentication thing that Chrome could deal with but wget/curl/Ruby couldn’t. Since my Firefox was on the fritz, I couldn’t use Selenium IDE or the Selenium Webdriver. iMacros to the rescue! (Chrome, Firefox)

2013-11-21 Geek notes - automating bulk web stuff with iMacros

That plus lots of keyboard macros and text manipulation in Emacs, plus a little parsing and regexp substitution in Ruby, plus more Emacs munging got me the data I wanted. Hooray for bubblegum and string scripting!

Developing Emacs micro-habits: Abbreviations and templates

When it comes to improving how you use Emacs, picking one small change and paying close attention seems to work well. Little things make a lot of difference, especially when frequently repeated over a long period of time. It reminded me of this quote I came across on Irreal:

I’ve gotten the hang of basic multiple-cursors-mode and I’m making gradual progress towards internalizing smart-parens by the simple approach of focusing on one tiny habit at a time. For example, I spent a week reminding myself to use mc/mark-all-like-this-dwim or mc/mark-lines instead of using keyboard macros.

Inspired by the Emacs Advent Calendar, I wanted to start a 52-week series on micro-habits for more effective Emacs use. I brain-dumped an outline of four sets (basic Emacs, Org, programming, meta-habits) of thirteen small tips each. Looking at my list, I realized there were many ideas there that I hadn’t quite gotten the hang of myself. I figured that this might be more of a project for 2016; in the meantime, I could learn by doing.

The first micro-habit I wanted to dig into was that of automating text: abbreviations, templates, and other ways to expand or transform text. I’d used Yasnippet before. I sometimes accidentally expanded keywords instead of indenting lines if my cursor happened to be at the wrong spot. But I hadn’t yet drilled the instinct of automation or the familiarity with templates into my fingers.

This blog post isn’t the easy-to-understand guide to automating text. I’ll write that later, when I’ve figured more things out. In the meantime, I’ll share what I’ve been learning and thinking so far, and maybe you can help me make sense of it.

Emacs has a separate manual for autotyping, which I had never read before. The short manual covers abbrev, skeleton, auto-insert, copyright messages, timestamps, and tempo. Did you know that define-skeleton lets you create a template that accepts multiple interregions if you call your skeleton function with a negative argument? (Interregions? What are those?) It took me an embarrassing amount of time to figure out how to mark interregions and use them. It turns out they have to be contiguous. It might be easier to think of marking the beginning of the region, marking some points in the middle, and then calling the command when your point is at the end – which is probably how most people would interpret the diagrams, but I was trying to mark discontinuous regions because that would be super-cool, and that totally didn’t work. And then I forgot that using helm-M-x means you need to specify numeric arguments after typing M-x instead of before. (I wrote about that very point in one of my blog posts, but it slipped my mind.) Once I got past that, I was delighted to find that it worked as advertised. I still haven’t imagined a situation where I would use it, but it seems like a good sort of thing to know.

What are the practical situations where text automation can help people work more effectively? I looked around to see how other people were using it. Coding, of course – especially if you use Emacs Lisp to transform the text. Debugging, too. Marking up text. Remembering parameters. Wrapping regions. Writing e-mails. Adding blog post metadata. Citing references. Lifehacker has a long list, too.

I came up with several categories I’m going to focus on so that I can more easily recognize opportunities to work better:

2015-01-05 Seeing opportunities for abbreviations and text automation -- index card

2015.01.05 Seeing opportunities for abbreviations and text automation – index card

  • Abbreviations are about typing long words with fewer keystrokes. For example, you might shorten “description” to desc.
  • Phrases are like word abbrevations, but longer. You might want to be able to expand btw to “by the way.”
  • Code benefits from expansion in multiple ways:
    • Automatically inserting characters that are harder to reach on a keyboard, like { and }
    • Being consistent about coding style, like the way many people like adding a comment after the closing brace of an if
    • Transforming text that shows up in multiple places, such as variable names that need getters and setters
    • Filling in the blanks: parameters, comments, etc.
    • Reducing the cognitive load of switching between languages by establishingq a common vocabulary. For example, I sometimes need to look up the syntax of for or the proper way to display a debugging statement when I switch to a language I haven’t used in a while
  • Templates are also useful for consistency in writing, planning, and other areas
  • Text transformation can save time and minimize error.

2015-01-04 Automating text - index card

2015.01.04 Automating text – index card

Translating the examples I’d seen to my personal interests, I could probably find plenty of opportunities to automate text while coding, debugging, writing, planning, or publishing. To dig deeper, I looked at each of the categories in detail.


2015-01-06 Abbreviations -- index card

2015.01.06 Abbreviations – index card

When I was curious about typing faster, I read forum posts from people who had increased their speed by developing their own form of digital shorthand. The trick works on paper, too. When I need to write quickly or in limited space, I use abbreviations like bc for “because” and w/o for “without.” Why not on the computer as well?

I often take advantage of dynamic abbreviations when I know I’ve recently typed the word I want. To trigger those, I just have to type the beginning of the word and then use dabbrev-expand. I haven’t set up my own static abbreviations, though. Main obstacles:

  • I want to write shorter words instead of longer ones
  • In the beginning, it’s faster to type the word instead of thinking of the abbreviation and expanding it
  • If I have to undo or backspace, that makes me slower
  • If I burn this into my muscle memory, I might be more frustrated on other computers or in other apps (then again, I already customize Emacs extensively, so I guess I’m okay with the tradeoff)

Anyway, here’s a short list I’m trying out with define-global-abbrev and hippie-expand:

hw however
bc because
wo without
prob probably
st sometimes

Hmm. Let’s say that it takes me two keystrokes to trigger the expansion, whether it’s the xx keychord I’ve just set up or the M-/ I’ve replaced with hippie-expand. (Hmm, maybe a double-space keychord is a good candidate for expansion too.) Is the retraining worth a ~50% possible reduction in keystrokes? Probably not.

How about text with punctuation, so I can minimize reaching for symbols?

mail [email protected]

Maybe it’s better to look at the words I frequently misspell, or that I tend to slow down then typing. I’ll keep an eye out for those.


2015-01-06 Phrases -- index card

2015.01.06 Phrases – index card

Phrases are an easier sell. Still, I’m trying not to settle into the rut of set phrases. I should cut those mercilessly or avoid writing them from the beginning.

co check out
iti I think I
otoh on the other hand,
mean in the meantime,
fe for example
fi for instance,
oc of course
ip in particular

Code insertion

This is, fortunately, well-trodden ground. The yasnippet package comes with a large collection of snippets for many programming languages. You can start by familiarizing yourself with the pre-defined snippets for the modes that you use. For example, in my installation, they’re under ~/.emacs.d/elpa/yasnippet-20141117.327/snippets. You can use the filename (or keywords defined with key, if specified) as the abbreviation, and you can expand them with yas-expand (which should be bound to TAB if you have yas-global-mode on).

I mostly work with HTML, CSS, Javascript, Ruby on Rails, and Emacs Lisp, so this is the cheat sheet I’ve made for myself:

2015-01-07 Code insertion -- index card

2015.01.07 Code insertion – index card

For HTML, I need to remember that the tags are generally expandable, and that there are a few Lorem Ipsum abbreviations triggered by lorem.1 through .5. CSS has a v abbreviation that sets up a bunch of rules with vendor prefixes. For Javascript, I’ll probably start with f to define a function and log to output something to console.log. Rails has a bunch of iterators like eai that look interesting. As for Emacs Lisp, the pre-defined templates generally add parentheses around common functions so you don’t have to type them, and there are a few shortcuts like bs for buffer-string and cc for condition-case. I think I’ll modify the default snippets to make better use of Yasnippet’s field support, though, so that I don’t have to delete and replace text.


In addition to using text expansion for code, you can use it for planning and writing other text. I saw Karl Voit use it to great effect in my Emacs Chat with him (around the 44:00 mark), and I’ve been gradually refining some templates of my own.

2015-01-07 Templates -- index card

2015.01.07 Templates – index card

For example, here’s the template I’ve been using for sketched books. Note: If you use Yasnippet for Org Mode properties, you may want to set yas-indent-line to fixed or the fields will get confused.

Gist: sbook

# key: sbook
# name: Sketched Book
# --

**** TOSKETCH ${1:short title}
      :TITLE: ${2:long title}
      :SHORT_TITLE: $1
      :AUTHOR: ${3:authors}
      :YEAR: ${4:year}
      :BUY_LINK: ${5:Amazon link}
      :BASENAME: ${6:`(org-read-date nil nil ".")`} Sketched Book - ${2:$(sacha/convert-sketch-title-to-filename yas-text)} - ${3:$(sacha/convert-sketch-title-to-filename yas-text)}
      :ISBN: ${7:ISBN}


***** TODO Sketchnote $1
:Effort: 2:00

[[elisp:sacha/prepare-sketchnote-file][Prepare the file]]

***** TODO Write personal reflection for $1
:Effort: 1:00

[[][View in calendar]]

****** Sketched Book - $2 - $3

$3's /$2/ ($4) ...

I’ve sketched the key points of the book below to make it easier to remember and share. Click on the image for a larger version that you can print if you want.

Haven't read the book yet? You can [[$5][buy it from Amazon]] (affiliate link) or get it from your favourite book sources.

Like this sketch? Check out [[][]] for more. Feel free to share – it’s under the Creative Commons Attribution License, like the rest of my blog.

***** TODO Post $1 to blog
:Effort: 1:00
:QUANTIFIED: Packaging

***** TODO Update sketched books collection
:Effort: 1:00
:QUANTIFIED: Packaging

1. [[elisp:sacha/index-sketched-book][Index sketched book]]
   - [[file:~/Dropbox/Packaging/sketched-books/][Edit index]]
   - [[file:~/Dropbox/Packaging/sketched-books/][Edit ebook]]
2. [[elisp:sacha/package-sketched-book][Compile]]
3. Update [[]]

***** TODO Tweet sneak peek of $1 with attached picture

[[elisp:(progn (kill-new (format "Sneak peek: Sketched Book: %s - %s %s" (org-entry-get-with-inheritance "SHORT_TITLE") (org-entry-get-with-inheritance "AUTHOR") (org-entry-get-with-inheritance "BLOG_POST"))) (browse-url ""))][Copy text and launch Twitter]]

It’s a lot of code and I keep tweaking it as I come across rough corners, but it’s handy to have that all captured in a template that I can easily expand. =)

Text transformation

One of the advantages of tweaking text expansion inside Emacs instead of using a general-purpose text expansion program is that you can mix in some Emacs Lisp to transform the text along the way. I’m still thinking about how to make the most of this, as you can see from this half-filled note-card:

2015-01-07 Text transformation as part of expansion -- index card

2015.01.07 Text transformation as part of expansion – index card

For example, this snippet makes it easier to share source code on my blog while also linking to a Gist copy of the code, in case I revise it or people want to comment on the code snippet itself. It doesn’t use any of the built-in text expansion capabilities, but I think of it as a text expander and transformer because it replaces work I used to do manually. You’ll need the gist package for this one.

Gist: Sacha (updated to clean up region code)

(defun sacha/copy-code-as-org-block-and-gist (beg end)
  (interactive "r")
  (let ((filename (file-name-base))
        (mode (symbol-name major-mode))
         (if (use-region-p) (buffer-substring beg end) (buffer-string)))
        (gist (if (use-region-p) (gist-region beg end) (gist-buffer))))
     (format "\n[[%s][Gist: %s]]\n#+begin_src %s\n%s\n#+end_src\n"
             (oref (oref gist :data) :html-url) filename
             (replace-regexp-in-string "-mode$" "" mode)

Both Yasnippet and Skeleton allow you to use Lisp expressions in your template. If you don’t have all the data yet, you might consider writing another Lisp function that you can call later when you do. For example, in the sketched books code above, I have an Emacs Lisp link that composes a tweet with a link, puts it in the clipboard, and then opens up a web browser. (I do this instead of posting directly because I also want to attach an image to that tweet, and I haven’t figured out how to modify any of the Emacs Twitter clients to do that.)

So that’s what I’ve learned so far about automating text in Emacs. It’ll take me more than a week to get the hang of the abbreviations I’ve just set up, and I’ll probably need to add even more before adding and using abbreviations become true habits. But hey, maybe this will help you pay closer attention to repetitive text and editing actions in Emacs so that you can automate them too, and we can swap notes on useful abbreviations. What kind of text do you expand?

For more information, see: