Tags: automation

RSS - Atom - Subscribe via email

Scripting and the grocery store flyer

| geek

We plan the week’s meals around the grocery store flyers, taking advantage of what’s on sale (chicken, ground beef, etc.) and stocking up when the opportunity presents itself (for example, diced tomatoes or cream of mushroom soup).

The flyers are usually delivered on Thursdays. We do most of our grocery shopping at No Frills because it’s convenient and almost always a good price, but sometimes we’ll go to Freshco or Metro if there’s a particularly good sale. I might flip through the other flyers if we’re looking for something in particular, but most of the time, we just toss them out. I’d love to opt out of paper flyers, but that doesn’t seem to be an option in our neighbourhood.

It doesn’t take a lot of time to review the flyers, but I figured it would be fun to write a script that highlights specific items for us. Now I have a script that parses the output of the No Frills accessible flyer to create a table like this:

Y Clementines 2.47 2 lb bag product of Spain $2.47
Y Smithfield Bacon 3.97 500 g selected varieties $3.97
Y Thomas’ Cinnamon Raisin Bread 2.5 675 g or Weston Kaisers 12’s selected varieties $5.00 or $2.50 ea.
Y Unico Tomatoes 0.97 796 mL or Beans 540 mL selected varieties $0.97
Fresh Boneless Skinless Chicken Breast 3.33 2.78 BIG Pack!™ DECEMBER 18TH – 24TH ONLY! $3.33 lb/$7.34/kg save $2.78/lb
Purex 3.97 2.02 2.03 L $3.97 save $2.02
Frozen Steelhead Trout Fillets 5.97 2.0 filets de truite $5.97 lb/$13.16/kg save $2.00/lb
Heinz Tomato Juice 0.97 1.52 1.36 L selected varieties $0.97 save $1.52
Nestlé Multi-Pack Chocolate or Bagged Chocolate 2.88 0.61 45-246 g selected varieties $2.88 save 61¢
Source 4.97 0.5 16 x 100 g selected varieties $4.97 save 50 ¢
Franco Gravy 0.67 0.32 284 mL selected varieties $0.67 save 32¢
Ocean Spray Cranberry Sauce 1.67 0.32 348 mL whole or jellied $1.67 save 32¢
10 lb Bag Yellow Potatoes, 5 lb Bag Carrots or 10 lb Bag Yellow Cooking Onions 1.87 product of Ontario, Canada no. 1 grade or 5 lb Bag Rutabaga product of Canada, no. 1 grade $1.87
Betty Crocker Hamburger Helper 1.5 158-255 g or Mashed or Scallop Potatoes 141-215 g selected varieties $3.00 or $1.50 ea.
Blackberries 1.25 6 oz product of U.S.A. or Mexico DECEMBER 18TH – 24TH ONLY! 4/$5.00 or $1.25 ea.

(more lines omitted)

The table is sorted by whether the item name matches one of the things we usually buy (first column: Y), then how much the sale is for, and then the name of the item. Over time, I’ll add more things to the priority list, and the script will get smarter and smarter.

I can use Org commands to move the rows up or down or remove the rows I’m not interested in. Then I can take the second column of the script’s output with Emacs’ copy-rectangle-as-kill command (C-x r M-w), and paste it into OurGroceries‘ import dialog. That builds a shopping list that’s sorted by the aisles I’ve previously set up, and this list is synchronized with our phones.

I’ve added the script to https://github.com/sachac/scripts/blob/master/check-grocery-flyer.js, so you can check there for updates. The flyer URL and the list of staples are defined in a separate configuration file that I haven’t included in the repository, but you can probably come up with your own if you want to adapt the idea. =)

Here’s the source, just in case:

#!/usr/bin/env node

/*
  Creates a prioritized list based on the flyers, like this:

Y Clementines 2.47    2 lb bag product of Spain $2.47
Y Smithfield Bacon  3.97    500 g selected varieties $3.97
Y Thomas' Cinnamon Raisin Bread 2.50    675 g or Weston Kaisers 12's selected varieties $5.00 or $2.50 ea.
Y Unico Tomatoes  0.97    796 mL or Beans 540 mL selected varieties $0.97
  Fresh Boneless Skinless Chicken Breast  3.33  2.78  BIG Pack!™ DECEMBER 18TH - 24TH ONLY! $3.33 lb/$7.34/kg save $2.78/lb
  Purex 3.97  2.02  2.03 L $3.97 save $2.02
  Frozen Steelhead Trout Fillets  5.97  2.00  filets de truite $5.97 lb/$13.16/kg save $2.00/lb
  Heinz Tomato Juice  0.97  1.52  1.36 L selected varieties $0.97 save $1.52
  Nestlé Multi-Pack Chocolate or Bagged Chocolate 2.88  0.61  45-246 g selected varieties $2.88 save 61¢
  ...
  */

var rp = require('request-promise');
var cheerio = require('cheerio');
var homeDir = require('home-dir');
var config = require(homeDir() + '/.secret');
var staples = config.grocery.staples; // array of lower-case text to match against flyer items
var flyerURL = config.grocery.flyerURL; // accessible URL

function parseValue(details) {
  var matches;
  var price;
  if ((matches = details.match(/\$([\.0-9]+)( | )+(ea|lb|\/kg)/i))) {
    price = matches[1];
  }
  else if ((matches = details.match(/\$([\.0-9]+)/i))) {
    price = matches[1];
  }
  else if ((matches = details.match(/([0-9]+) *¢/))) {
    price = parseInt(matches[1]) / 100.0;
  }
  return price;
}

function getFlyer(url) {
  return rp.get(url).then(function(response) {
    var $ = cheerio.load(response);
    var results = [];
    $('table[colspan="2"]').each(function() {
      var cells = $(this).find('td');
      // $0.67  or  2/$3.00 or $1.25ea
      var item = $(cells[0]).text().replace(/^[ \t\r\n]+|[ \t\r\n]+$/g, '');
      var details = $(cells[1]).text().replace(/([ \t\r\n\u00a0\u0000]| )+/g, ' ').replace(/^[ \t\r\n]+|[ \t\r\n]+$/g, '');
      var matches;
      var save = '';
      var price = parseValue(details);
      details = details.replace(/ \/ [^A-Z$]+/, ' ');
      if (details.match(/To Our Valued Customers/)) {
        details = details.replace(/To Our Valued Customers.*/, 'DELAYED');
      }
      if ((matches = details.match(/save .*/))) {
        save = parseValue(matches[0]);
      }
      results.push({item: item,
                    details: details,
                    price: price,
                    save: save});
    });
    return results;
  });
}

function prioritizeFlyer(data) {
  for (var i = 0; i < data.length; i++) {
    var name = data[i].item.toLowerCase();
    for (var j = 0; j < staples.length; j++) {
      if (name.match(staples[j])) {
        data[i].priority = true;
      }
    }
  }
  return data.sort(function(a, b) {
    if (a.priority && !b.priority) return -1;
    if (!a.priority && b.priority) return 1;
    if (a.save > b.save) return -1;
    if (a.save < b.save) return 1;
    if (a.item < b.item) return -1;
    if (a.item > b.item) return 1;
  });
}

function displayFlyerData(data) {
  for (var i = 0; i < data.length; i++) {
    var o = data[i];
    console.log((o.priority ? 'Y' : '') + '\t' + o.item + "\t" + o.price + "\t" + o.save + "\t" + o.details);
  }
}

getFlyer(flyerURL).then(prioritizeFlyer).then(displayFlyerData);

I’ll check next week to see if the accessible flyer URL changes each time, or if I can determine the correct publication ID by going to a stable URL. Anyway, this was fun to write!

Scripting and the Toronto Public Library’s movie collection

| geek

We hardly ever watch movies in the theatre now, since we prefer watching movies with subtitles and the ability to pause. Fortunately, the Toronto Public Library has a frequently updated collection of DVDs. The best time to grab a movie is when it’s a new release, since DVDs that have been in heavy circulation can get pretty scratched up from use. However, newly released movies can’t be reserved. You need to find them at the library branches they’re assigned to, and then you can borrow them for seven days. You can check the status of each movie online to see if it’s in the library or when it’s due to be returned.

Since there are quite a few movies on our watch list, quite a few library branches we can walk to, and some time flexibility as to when to go, checking all those combinations is tedious. I wrote a script that takes a list of branches and a list of movie URLs, checks the status of each, and displays a table sorted by availability and location. My code gives me a list like this:

In Library Annette Street Mad Max Fury Road M 10-8:30 T 12:30-8:30 W 10-6 Th 12:30-8:30 F 10-6 Sat 9-5
In Library Bloor/Gladstone Inside Out M 9-8:30 T 9-8:30 W 9-8:30 Th 9-8:30 F 9-5 Sat 9-5 Sun 1:30-5
In Library Bloor/Gladstone Match M 9-8:30 T 9-8:30 W 9-8:30 Th 9-8:30 F 9-5 Sat 9-5 Sun 1:30-5
In Library Jane/Dundas Avengers: Age of Ultron M 9-8:30 T 9-8:30 W 9-8:30 Th 9-8:30 F 9-5 Sat 9-5
In Library Jane/Dundas Ant-Man M 9-8:30 T 9-8:30 W 9-8:30 Th 9-8:30 F 9-5 Sat 9-5
In Library Jane/Dundas Mad Max Fury Road M 9-8:30 T 9-8:30 W 9-8:30 Th 9-8:30 F 9-5 Sat 9-5
In Library Jane/Dundas Minions M 9-8:30 T 9-8:30 W 9-8:30 Th 9-8:30 F 9-5 Sat 9-5
In Library Perth/Dupont Chappie T 12:30-8:30 W 10-6 Th 12:30-8:30 F 10-6 Sat 9-5
In Library Runnymede Ant-Man M 9-8:30 T 9-8:30 W 9-8:30 Th 9-8:30 F 9-5 Sat 9-5
In Library Runnymede Minions M 9-8:30 T 9-8:30 W 9-8:30 Th 9-8:30 F 9-5 Sat 9-5
In Library St. Clair/Silverthorn Kingsman: the Secret Service T 12:30-8:30 W 10-6 Th 12:30-8:30 F 10-6 Sat 9-5
In Library St. Clair/Silverthorn Mad Max Fury Road T 12:30-8:30 W 10-6 Th 12:30-8:30 F 10-6 Sat 9-5
In Library Swansea Memorial Ant-Man T 10-6 W 1-8 Th 10-6 Sat 10-5
In Library Swansea Memorial Chappie T 10-6 W 1-8 Th 10-6 Sat 10-5
In Library Swansea Memorial Kingsman: the Secret Service T 10-6 W 1-8 Th 10-6 Sat 10-5
In Library Swansea Memorial Kingsman: the Secret Service T 10-6 W 1-8 Th 10-6 Sat 10-5
In Library Swansea Memorial Mad Max Fury Road T 10-6 W 1-8 Th 10-6 Sat 10-5
In Library Swansea Memorial Minions T 10-6 W 1-8 Th 10-6 Sat 10-5
2015-12-08 Perth/Dupont Terminator Genisys T 12:30-8:30 W 10-6 Th 12:30-8:30 F 10-6 Sat 9-5
2015-12-08 Perth/Dupont Mad Max Fury Road T 12:30-8:30 W 10-6 Th 12:30-8:30 F 10-6 Sat 9-5
2015-12-09 Swansea Memorial Avengers: Age of Ultron T 10-6 W 1-8 Th 10-6 Sat 10-5

… many more rows omitted. =)

With this data, I can decide that Swansea Memorial has a bunch of things I might want to check out, and pick that as the destination for my walk. Sure, there’s a chance that someone else might check out the movies before I get there (although I can minimize that by getting to the library as soon as it opens), or that the video has been misfiled or misplaced, but overall, the system tends to work fine.

It’s easy for me to send the output to myself by email, too. I just select the part of the table I care about and use Emacs’ M-x shell-command-on-region (M-|) to mail it to myself with the command mail -s "Videos to check out" sacha@sachachua.com.

The first time I ran my script, I ended up going to Perth/Dupont to pick up seven movies in addition to the two I picked up from Annette Library. Many of the movies had been returned but not yet shelved, so the librarian retrieved them from his bin and gave them to me. When I got back, W- looked at the stack of DVDs by the television and said, “You know that’s around 18 hours of viewing, right?” It’ll be fine for background watching. =)

Little things like this make me glad that I can write scripts and other tiny tools to make my life better. Anything that involves multiple steps or combining information from multiple sources might be simpler with a script. I wrote this script as a command-line tool with NodeJS, since I’m comfortable with the HTML request and parsing libraries available there.

Anyway, here’s the code, in case you want to build on the idea. Have fun!

/* Shows you which videos are available at which libraries.

   Input: A json filename, which should be a hash of the form:
   {"branches": {"Branch name": "Additional branch details (ex: hours)", ...},
   "videos": [{"Title": "URL to library page"}, ...]}.

   Example: {
   "branches": {
   "Runnymede": "M 9-8:30 T 9-8:30 W 9-8:30 Th 9-8:30 F 9-5 Sat 9-5"
   },
   "videos": [
   {"title": "Avengers: Age of Ultron", "url": "http://www.torontopubliclibrary.ca/detail.jsp?Entt=RDM3350205&R=3350205"}
   ]}

   Output:
   Status,Branch,Title,Branch notes
*/

var rp = require('request-promise');
var moment = require('moment');
var async = require('async');
var cheerio = require('cheerio');
var q = require('q');
var csv = require('fast-csv');
var fs = require('fs');

if (process.argv.length < 3) {
  console.log('Please specify the JSON file to read the branches and videos from.');
  process.exit(1);
}

var config = JSON.parse(fs.readFileSync(process.argv[2]));
var branches = config.branches;
var videos = config.videos;

/*
  Returns a promise that will resolve with an array of [status,
  branch, movie, info], where status is either the next due date, "In
  Library", etc. */
function checkStatus(branches, movie) {
  var url = movie.url;
  var matches = url.match(/R=([0-9]+)/);
  return rp.get(
    'http://www.torontopubliclibrary.ca/components/elem_bib-branch-holdings.jspf?print=&numberCopies=1&itemId='
      + matches[1]).then(function(a) {
        var $ = cheerio.load(a);
        var results = [];
        var lastBranch = '';              
        $('tr.notranslate').each(function() {
          var row = $(this);
          var cells = row.find('td');
          var branch = $(cells[0]).text().replace(/^[ \t\r\n]+|[ \t\r\n]+$/g, '');
          var due = $(cells[2]).text().replace(/^[ \t\r\n]+|[ \t\r\n]+$/g, '');
          var status = $(cells[3]).text().replace(/^[ \t\r\n]+|[ \t\r\n]+$/g, '');
          if (branch) { lastBranch = branch; }
          else { branch = lastBranch; }
          if (branches[branch]) {
            if (status == 'On loan' && (matches = due.match(/Due: (.*)/))) {
              status = moment(matches[1], 'DD/MM/YYYY').format('YYYY-MM-DD');
            }
            if (status != 'Not Available - Search in Progress') {
              results.push([status, branch, movie.title, branches[branch]]);
            }
          }
        });
        return results;
      });
}

function checkAllVideos(branches, videos) {
  var results = [];
  var p = q.defer();
  async.eachLimit(videos, 5, function(video, callback) {
    checkStatus(branches, video).then(function(result) {
      results = results.concat(result);
      callback();
    });
  }, function(err) {
    p.resolve(results.sort(function(a, b) {
      if (a[0] == 'In Library') {
        if (b[0] == 'In Library') {
          if (a[1] < b[1]) return -1;
          if (a[1] > b[1]) return 1;
          if (a[2] < b[2]) return -1;
          if (a[2] > b[2]) return 1;
          return 0;
        } else {
          return -1;
        }
      }
      if (b[0] == 'In Library') { return 1; }
      if (a[0] < b[0]) { return -1; }
      if (a[0] > b[0]) { return 1; }
      return 0;
    }));
  });
  return p.promise;
}

checkAllVideos(branches, videos).then(function(result) {
  csv.writeToString(result, {}, function(err, data) {
    console.log(data);
  });
});

P.S. Okay, I’m really tempted to walk over to Swansea Memorial, but W- reminds me that we’ve got a lot of movies already waiting to be watched. So I’ll probably just walk to the supermarket, but I’m looking forward to running this script once we get through our backlog of videos!

Developing Emacs micro-habits: Abbreviations and templates

Posted: - Modified: | emacs

When it comes to improving how you use Emacs, picking one small change and paying close attention seems to work well. Little things make a lot of difference, especially when frequently repeated over a long period of time. It reminded me of this quote I came across on Irreal:

I've gotten the hang of basic multiple-cursors-mode and I'm making gradual progress towards internalizing smart-parens by the simple approach of focusing on one tiny habit at a time. For example, I spent a week reminding myself to use mc/mark-all-like-this-dwim or mc/mark-lines instead of using keyboard macros.

Inspired by the Emacs Advent Calendar, I wanted to start a 52-week series on micro-habits for more effective Emacs use. I brain-dumped an outline of four sets (basic Emacs, Org, programming, meta-habits) of thirteen small tips each. Looking at my list, I realized there were many ideas there that I hadn't quite gotten the hang of myself. I figured that this might be more of a project for 2016; in the meantime, I could learn by doing.

The first micro-habit I wanted to dig into was that of automating text: abbreviations, templates, and other ways to expand or transform text. I'd used Yasnippet before. I sometimes accidentally expanded keywords instead of indenting lines if my cursor happened to be at the wrong spot. But I hadn't yet drilled the instinct of automation or the familiarity with templates into my fingers.

This blog post isn't the easy-to-understand guide to automating text. I'll write that later, when I've figured more things out. In the meantime, I'll share what I've been learning and thinking so far, and maybe you can help me make sense of it.

Emacs has a separate manual for autotyping, which I had never read before. The short manual covers abbrev, skeleton, auto-insert, copyright messages, timestamps, and tempo. Did you know that define-skeleton lets you create a template that accepts multiple interregions if you call your skeleton function with a negative argument? (Interregions? What are those?) It took me an embarrassing amount of time to figure out how to mark interregions and use them. It turns out they have to be contiguous. It might be easier to think of marking the beginning of the region, marking some points in the middle, and then calling the command when your point is at the end – which is probably how most people would interpret the diagrams, but I was trying to mark discontinuous regions because that would be super-cool, and that totally didn't work. And then I forgot that using helm-M-x means you need to specify numeric arguments after typing M-x instead of before. (I wrote about that very point in one of my blog posts, but it slipped my mind.) Once I got past that, I was delighted to find that it worked as advertised. I still haven't imagined a situation where I would use it, but it seems like a good sort of thing to know.

What are the practical situations where text automation can help people work more effectively? I looked around to see how other people were using it. Coding, of course – especially if you use Emacs Lisp to transform the text. Debugging, too. Marking up text. Remembering parameters. Wrapping regions. Writing e-mails. Adding blog post metadata. Citing references. Lifehacker has a long list, too.

I came up with several categories I'm going to focus on so that I can more easily recognize opportunities to work better:

2015-01-05 Seeing opportunities for abbreviations and text automation -- index card

2015.01.05 Seeing opportunities for abbreviations and text automation – index card

  • Abbreviations are about typing long words with fewer keystrokes. For example, you might shorten "description" to desc.
  • Phrases are like word abbrevations, but longer. You might want to be able to expand btw to "by the way."
  • Code benefits from expansion in multiple ways:
    • Automatically inserting characters that are harder to reach on a keyboard, like { and }
    • Being consistent about coding style, like the way many people like adding a comment after the closing brace of an if
    • Transforming text that shows up in multiple places, such as variable names that need getters and setters
    • Filling in the blanks: parameters, comments, etc.
    • Reducing the cognitive load of switching between languages by establishingq a common vocabulary. For example, I sometimes need to look up the syntax of for or the proper way to display a debugging statement when I switch to a language I haven't used in a while
  • Templates are also useful for consistency in writing, planning, and other areas
  • Text transformation can save time and minimize error.

2015-01-04 Automating text - index card

2015.01.04 Automating text – index card

Translating the examples I'd seen to my personal interests, I could probably find plenty of opportunities to automate text while coding, debugging, writing, planning, or publishing. To dig deeper, I looked at each of the categories in detail.

Abbreviations

2015-01-06 Abbreviations -- index card

2015.01.06 Abbreviations – index card

When I was curious about typing faster, I read forum posts from people who had increased their speed by developing their own form of digital shorthand. The trick works on paper, too. When I need to write quickly or in limited space, I use abbreviations like bc for "because" and w/o for "without." Why not on the computer as well?

I often take advantage of dynamic abbreviations when I know I've recently typed the word I want. To trigger those, I just have to type the beginning of the word and then use dabbrev-expand. I haven't set up my own static abbreviations, though. Main obstacles:

  • I want to write shorter words instead of longer ones
  • In the beginning, it's faster to type the word instead of thinking of the abbreviation and expanding it
  • If I have to undo or backspace, that makes me slower
  • If I burn this into my muscle memory, I might be more frustrated on other computers or in other apps (then again, I already customize Emacs extensively, so I guess I'm okay with the tradeoff)

Anyway, here's a short list I'm trying out with define-global-abbrev and hippie-expand:

hw however
bc because
wo without
prob probably
st sometimes

Hmm. Let's say that it takes me two keystrokes to trigger the expansion, whether it's the xx keychord I've just set up or the M-/ I've replaced with hippie-expand. (Hmm, maybe a double-space keychord is a good candidate for expansion too.) Is the retraining worth a ~50% possible reduction in keystrokes? Probably not.

How about text with punctuation, so I can minimize reaching for symbols?

blog https://sachachua.com/blog/
mail sacha@sachachua.com

Maybe it's better to look at the words I frequently misspell, or that I tend to slow down then typing. I'll keep an eye out for those.

Phrases

2015-01-06 Phrases -- index card

2015.01.06 Phrases – index card

Phrases are an easier sell. Still, I'm trying not to settle into the rut of set phrases. I should cut those mercilessly or avoid writing them from the beginning.

co check out
iti I think I
otoh on the other hand,
mean in the meantime,
fe for example
fi for instance,
oc of course
ip in particular

Code insertion

This is, fortunately, well-trodden ground. The yasnippet package comes with a large collection of snippets for many programming languages. You can start by familiarizing yourself with the pre-defined snippets for the modes that you use. For example, in my installation, they're under ~/.emacs.d/elpa/yasnippet-20141117.327/snippets. You can use the filename (or keywords defined with key, if specified) as the abbreviation, and you can expand them with yas-expand (which should be bound to TAB if you have yas-global-mode on).

I mostly work with HTML, CSS, Javascript, Ruby on Rails, and Emacs Lisp, so this is the cheat sheet I've made for myself:

2015-01-07 Code insertion -- index card

2015.01.07 Code insertion – index card

For HTML, I need to remember that the tags are generally expandable, and that there are a few Lorem Ipsum abbreviations triggered by lorem.1 through .5. CSS has a v abbreviation that sets up a bunch of rules with vendor prefixes. For Javascript, I'll probably start with f to define a function and log to output something to console.log. Rails has a bunch of iterators like eai that look interesting. As for Emacs Lisp, the pre-defined templates generally add parentheses around common functions so you don't have to type them, and there are a few shortcuts like bs for buffer-string and cc for condition-case. I think I'll modify the default snippets to make better use of Yasnippet's field support, though, so that I don't have to delete and replace text.

Templates

In addition to using text expansion for code, you can use it for planning and writing other text. I saw Karl Voit use it to great effect in my Emacs Chat with him (around the 44:00 mark), and I've been gradually refining some templates of my own.

2015-01-07 Templates -- index card

2015.01.07 Templates – index card

For example, here's the template I've been using for sketched books. Note: If you use Yasnippet for Org Mode properties, you may want to set yas-indent-line to fixed or the fields will get confused.

Gist: sbook

# key: sbook
# name: Sketched Book
# --

**** TOSKETCH ${1:short title}
      :PROPERTIES:
      :TITLE: ${2:long title}
      :SHORT_TITLE: $1
      :AUTHOR: ${3:authors}
      :YEAR: ${4:year}
      :BUY_LINK: ${5:Amazon link}
      :BASENAME: ${6:`(org-read-date nil nil ".")`} Sketched Book - ${2:$(sacha/convert-sketch-title-to-filename yas-text)} - ${3:$(sacha/convert-sketch-title-to-filename yas-text)}
      :ISBN: ${7:ISBN}
      :BLOG_POST: https://sachachua.com/blog
      :END:

$0

***** TODO Sketchnote $1
:PROPERTIES:
:Effort: 2:00
:QUANTIFIED: Drawing
:END:

[[elisp:sacha/prepare-sketchnote-file][Prepare the file]]

***** TODO Write personal reflection for $1
:PROPERTIES:
:Effort: 1:00
:QUANTIFIED: Writing
:END:

[[https://sachachua.com/blog/wp-admin/edit.php?page=cal][View in calendar]]

****** Sketched Book - $2 - $3

$3's /$2/ ($4) ...

I’ve sketched the key points of the book below to make it easier to remember and share. Click on the image for a larger version that you can print if you want.

Haven't read the book yet? You can [[$5][buy it from Amazon]] (affiliate link) or get it from your favourite book sources.

Like this sketch? Check out [[http://sketchedbooks.com][sketchedbooks.com]] for more. Feel free to share – it’s under the Creative Commons Attribution License, like the rest of my blog.

***** TODO Post $1 to blog
:PROPERTIES:
:Effort: 1:00
:QUANTIFIED: Packaging
:END:


***** TODO Update sketched books collection
:PROPERTIES:
:Effort: 1:00
:QUANTIFIED: Packaging
:END:

1. [[elisp:sacha/index-sketched-book][Index sketched book]]
   - [[file:~/Dropbox/Packaging/sketched-books/index.org][Edit index]]
   - [[file:~/Dropbox/Packaging/sketched-books/ebook.org][Edit ebook]]
2. [[elisp:sacha/package-sketched-book][Compile]]
3. Update [[https://gumroad.com/products/pBtS/edit]]

***** TODO Tweet sneak peek of $1 with attached picture

[[elisp:(progn (kill-new (format "Sneak peek: Sketched Book: %s - %s %s" (org-entry-get-with-inheritance "SHORT_TITLE") (org-entry-get-with-inheritance "AUTHOR") (org-entry-get-with-inheritance "BLOG_POST"))) (browse-url "http://twitter.com"))][Copy text and launch Twitter]]

It's a lot of code and I keep tweaking it as I come across rough corners, but it's handy to have that all captured in a template that I can easily expand. =)

Text transformation

One of the advantages of tweaking text expansion inside Emacs instead of using a general-purpose text expansion program is that you can mix in some Emacs Lisp to transform the text along the way. I'm still thinking about how to make the most of this, as you can see from this half-filled note-card:

2015-01-07 Text transformation as part of expansion -- index card

2015.01.07 Text transformation as part of expansion – index card

For example, this snippet makes it easier to share source code on my blog while also linking to a Gist copy of the code, in case I revise it or people want to comment on the code snippet itself. It doesn't use any of the built-in text expansion capabilities, but I think of it as a text expander and transformer because it replaces work I used to do manually. You'll need the gist package for this one.

Gist: Sacha (updated to clean up region code)

(defun sacha/copy-code-as-org-block-and-gist (beg end)
  (interactive "r")
  (let ((filename (file-name-base))
        (mode (symbol-name major-mode))
        (contents
         (if (use-region-p) (buffer-substring beg end) (buffer-string)))
        (gist (if (use-region-p) (gist-region beg end) (gist-buffer))))
    (kill-new
     (format "\n[[%s][Gist: %s]]\n#+begin_src %s\n%s\n#+end_src\n"
             (oref (oref gist :data) :html-url) filename
             (replace-regexp-in-string "-mode$" "" mode)
             contents))))

Both Yasnippet and Skeleton allow you to use Lisp expressions in your template. If you don't have all the data yet, you might consider writing another Lisp function that you can call later when you do. For example, in the sketched books code above, I have an Emacs Lisp link that composes a tweet with a link, puts it in the clipboard, and then opens up a web browser. (I do this instead of posting directly because I also want to attach an image to that tweet, and I haven't figured out how to modify any of the Emacs Twitter clients to do that.)

So that's what I've learned so far about automating text in Emacs. It'll take me more than a week to get the hang of the abbreviations I've just set up, and I'll probably need to add even more before adding and using abbreviations become true habits. But hey, maybe this will help you pay closer attention to repetitive text and editing actions in Emacs so that you can automate them too, and we can swap notes on useful abbreviations. What kind of text do you expand?

For more information, see:

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

Automating bulk web stuff with iMacros

Posted: - Modified: | geek

I found myself needing to download a whole bunch of JSON data from a server that had a weird authentication thing that Chrome could deal with but wget/curl/Ruby couldn’t. Since my Firefox was on the fritz, I couldn’t use Selenium IDE or the Selenium Webdriver. iMacros to the rescue! (Chrome, Firefox)

2013-11-21 Geek notes - automating bulk web stuff with iMacros

That plus lots of keyboard macros and text manipulation in Emacs, plus a little parsing and regexp substitution in Ruby, plus more Emacs munging got me the data I wanted. Hooray for bubblegum and string scripting!

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

Back to the joys of coverage testing: Vagrant, Guard, Spork, RSpec, Simplecov

Posted: - Modified: | rails

Tests are important because programmers are human.  I know that I’m going to forget, make mistakes, and change things that I didn’t mean to change. Testing lets me improve my chances of at least noticing. Coverage tools show me how much of my code is covered by tests, improving my chances of seeing where my blind spots are.

In one of my last projects at IBM, I convinced my team to use coverage testing tools on a Rails development project. It was great watching the coverage numbers inch up, and we actually reached 100% (at least in terms of what rcov was looking at). I occasionally had to fix things when people broke the build, but sometimes people added and updated tests too. Although coverage testing has its weaknesses (are you testing for edge cases?), it’s better than nothing, and can catch some embarrassing bugs before they make it to the outside world.

Although I’m not currently taking on any web development work (I’m saving brainspace for other things), I have a few personal projects that I enjoy working on. For example, QuantifiedAwesome.com lets me track different measurements such as time. I had written some tests for it before, but since then, I’d been adding features and making changes without updating the tests. Like the way that unpaid credit card balances turn into financial debt (very bad habit, try to avoid this if possible), unwritten or out-of-date tests contribute to technical debt.

It was a little daunting to sit down and slowly work my way through the knots of out-of-date tests that had accumulated from haphazard coding. Here’s how I made the task more manageable:

image

(… 21.5 hours of coding/development/infrastructure…)

New things I learned:

I use Mechanize to work with the Toronto Public Library’s webpage system so that I can retrieve the list of due books, renew items, or request holds. I wanted to test these as well. FakeWeb lets you intercept web requests and return your own responses, so I saved sample web pages to my spec/fixtures/files directory and used FakeWeb to return them. (ex: toronto_library_spec.rb)

Paperclip also needed some tweaking. I replaced my Paperclip uploads with File assignments so that I could test the image processing. (ex: clothing_spec.rb)

Always remember to wrap your RSpec tests in an it “…” instead of just context “when …” or describe “…”. I forgot to do this a few times and got confused about why stuff got left in my database and why the error messages weren’t formatted like they normally are.

Progress! Next step: Update my Cucumber tests and add some more integration tests…

I like this part of being a developer, even though writing new code is more fun. Testing is one of those time-consuming but powerful tools that can reduce frustration in the long run. I’ve experimented with applying automated testing techniques to everyday life before. I wonder what it would be like to take that even further… Any thoughts?

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

Sketchnotes: The Very Versatile Drip–Mathew Sweezey (Pardot)

Posted: - Modified: | sketchnotes

UPDATE: Dec 13, 2012 Want to watch the webinar? Here’s the video recording.

In this marketing webinar hosted by Pardot, Mathew Sweezey shared tips on setting up a drip nurturing program for marketing and sales support. Click on the image to view a larger size, and feel free to share this with attribution!

20121206 Pardot - The Very Versatile Drip - Mathew Sweezey

Pardot has many other webinars and recordings, so check them out if you’re curious about marketing automation.

Like this? Browse through my other sketchnotes, including my visual summary of The 5 Key Elements of a Better B2B Content Marketing Strategy by Nolin LeChasseur. I sketchnote technology/business conferences and presentations – if that sounds interesting, get in touch!

Text:

THE VERY VERSATILE DRIP
Mathew Sweezey
Dec 6 2012
Pardot

Email marketing
Drip nurturing
one-to-one conversations!

#1 LOOK AT MARKETING LIKE SALES LOOKS AT DEALS

Unidentified need -> Identified need not yet ready -> Starting to evaluate

competitive vs greenfield

Figure out your stages and buying cycles

#2 RELEVANT

context intelligence -> communication (must be relevant!)

Not just automation!
Relevance is key.

#3 NOT E-MAIL MARKETING AS YOU KNOW IT
HTML? marketer

One-to-one relationship
inbox = battlefield
Make it feel like a one-to-one email (rich text)

#4 HAVE A GOAL, THEN STICK TO IT!

(I feel bad not working with this guy.. He’s so ATTENTIVE)
Lost a deal? Nurture the relationship!

#5 USE THAT COLD DATABASE

Value per lead x Size of database = $$$
* maximize your database
* Find leads that slipped through cracks

#6 GIVE SALES AN EXCUSE.

Don’t get overwhelmed!
Sales drip = great ROI

Marketing – Sales
Bridge the gap

expecially for prospects that are challenging

engaged -> excuse to call (Whew!)

PROBLEMS/GOALS

cold database
automate lead nurturing
event pre-/post-follow ups
cold marketing/cold sales
competitive
lost deal

TYPES

3..2..1
start with stage 3, then 2, then 1
Good for cold databases, tradeshow lists..

Why reverse?
Engage HOT prospects right away!

Event-specific
straightforward

Stage-specific
Give people “carrots” to encourage them to move stages

Straight
simplest drip goal: engage, excuse to reach out

(Give me your best emails)

(Thought you might enjoy this..)
..doesn’t have to be your stuff
Nurture relationship

Q&A:
Q: Whitepaper vs video for scoring?
A: Topic, sales readiness
Q: Rich text vs images?
A: Images often make people think “marketing!” Images okay if fake-forwarded.
FWD:…
Hey take a look at this
Q: # of emails?
A: Start small. 3 emails, ~12-18 days. Then more. Iterate.
Test List. at least 1 day before. ALWAYS.
Q: Length?
A: 1 goal, 1-2 actions
Q: Replace e-mail blasts?
A: Nope.
Q: Bcc?
A: Not needed – CRM, reply.
Q: Timing?
A: Overanalyzed
Q: Timing?
A: Ask your salespeople. 6-45 days is good. -> not twice in a week
Q: From?
A: Depends, can be dynamic.
Q: Not engaging?
A: Don’t remove the, keep on going.

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

Sketchnotes: Marketing Automation, Jeffrey Yee (#torontob2b)

Posted: - Modified: | sketchnotes

UPDATE 2012-11-15: Here’s the video recap!

Marketing Automation
Jeffrey Yee, Eloqua

Like these? Check out my other sketchnotes, visual book notes/reviews, and visual metaphors.

Here’s the text from the sketchnotes to improve people’s ability to search for it:

Marketing automation

Marketing Automation
Jeffrey Yee, Eloqua

leads small
list management
forms
scoring
analytics
events
challenge
-Too expensive
-Not fully used
-Not implemented correctly
-Did not address business needs

1. Focus
one thing! business need!
2. Identify
Look for what your top performers are already doing
3. Start small, then build for mass adoption
-Target the second-tier salespeople!
4. Wait patiently for the lift.
incremental improvement

Best practices from client side
Dun & Bradstreet
credit risk management sales & marketing supply risk management

1. Focus
Example
Retention trigger-based e-mail
one need
40.1% opens
13.4% click through
10% increase in retention rates
2. Identify before you automate
Focus group?
Study top performers
How are we achieving this today?
Can we automate and scale this?

Repurpose

Think linear, it’s easier that way

Get personal and add value
plaint text e-mail from sales, not marketing
3. Mass adoption (but start very small)
advocates get others on board

Look for the people who are close to their quotas:
Tier 2 segmenting your salespeople!

Have reps vet leads before adding to program

3rd party data
4. Wait patiently for the lift. Set expectations.
Ex results
-6 months
pipeline value *19%
# of yes 14%
average upsize 3%
ops won 25%

Budget 12+ months

Like low-hanging fruit
Scaling up what already works
Notes by Sacha Chua, @sachac, LivingAnAwesomeLife.com

 

View or add comments (Disqus), or e-mail me at sacha@sachachua.com