On this page:

Helping kids learn about automation

J- shuffled in and out of the living room, listless and bored. As part of a 9-week simulation of real life in school, she and her classmates had been assigned jobs. Her job was to be an accountant, and the tedium of checking dozens of pretend tax returns had long sunk in. W- had encouraged her to use a calculator, so at least she didn’t have to multiple all those figures by hand, but there were still so many numbers to verify.

My geek sense tingled, as it does whenever there’s an opportunity for a quick win through automation. I coaxed her back to her homework. “Come on, let’s set up a spreadsheet,” I said. “That way, you don’t have to redo each of the calculations or worry about getting things wrong.”

We brought up OpenOffice.org Calc. She was still lackluster, so I took the lead in creating the spreadsheet. I asked her which tax return we could use as a model, and she picked hers. We started filling in the formulas, checking her work along the way. (We found and fixed an error in her tax return, too!) Then we tested the spreadsheet on a few other tax returns she had manually done, and she used it to check the rest.

Result: Not only could she verify a correct tax return in less than a minute, but she perked up and started having fun with it. She made a pile of correct tax returns and a pile of incorrect ones, with sticky notes pointing out the deficiencies. She still doesn’t want to be an accountant again, but at least she knows that tedious tasks might be automated away.

The next time J- finds herself doing tedious calculations or verifications, I hope she thinks about how much faster, more reliable, and more enjoyable the spreadsheet was compared to calculating things step by step, and perhaps invest time into learning how to automate whatever she needs to do.

How do people learn how to automate? It’s such a time-saving skill, but it doesn’t seem all that common. Maybe people are intimidated by spreadsheets and programming languages, and that fear of losing more time keeps them from gradually building the knowledge they need to save lots of time. If we can show J- and other kids the benefits of automating, maybe that light at the end of the tunnel will encourage them to learn. If we expose them to the methods for automating tasks, such as putting calculations into a spreadsheet, creating keyboard macros, or writing short programs, maybe they’ll realize it’s not scary – and maybe they’ll start modifying or creating new tools.

In my experience, working with new automating frameworks is always slow and somewhat frustrating in the beginning. It helps that I don’t usually need or want to automate everything right away. I break things down into small things, small wins. I might start by figuring out the most time-consuming parts and automating that 10%, or automating the most common operations. As I become more familiar with the tools and the process, I automate a little bit more, and more, and more. Eventually I might even create a tool that other people can use, like the way my Community Toolkit for Lotus Connections is off and running.

The hardest thing, I guess, is knowing where to start. I run into that problem a lot, because I work with lots of different technologies and frameworks. It’s like looking for the end of a tangled piece of string. That can be hard to find in the confusion, but once you do, you can start unknotting the mess. I want J- to be able to think: ah, this has to do with calculations, maybe I can get a handle on it by using a spreadsheet, putting in manual steps if needed.

How do you use teachable moments to encourage people to automate?

2011-03-29 Tue 21:09

Cucumber, Capybara, and the joys of integration testing in Rails

Development is so much more fun with test cases. They give you a big target to aim for, and it feels fantastic when you write the code to make them pass. Tests also avoid or shorten those late-night “oh no! I broke something!” sessions, because you can backtrack to versions that pass the tests. (You are using version control, right?)

So naturally, as I worked on my first IBM project using Ruby on Rails, I wanted to know about how to perform automated testing – not just at the unit level, but at the web/integration level.

I like using Simpletest in Drupal. I love the testing frameworks available in Rails.

You see, Cucumber for Rails allows you to write your tests in English (or something reasonably close to it). For example:

Feature: Contributor
  In order to maintain security
  As a contributor
  I want to be able to edit existing submissions
  Scenario: Contributor should not be able to create or delete submissions
    Given I am a company contributor
    And there is a 2010 survey for "Company X"
    When I view the dashboard
    Then I should not be able to delete a submission
    And I should not be able to create a submission

Putting that in my features/contributor.feature" file and executing that with =bundle execute cucumber features/contributor.feature gets me a lovely test with green signs all around.

You’re thinking: Rails is awesome, but it’s not that awesome, is it? How can it know about the specifics of the application?

Rails knows because I’ve written my own step definitions for Cucumber. Step definitions are simple. You can define them with a regular expression like this:

When /^I view the dashboard/ do
  visit root_path
end

Then /^I should not be able to create a submission/ do
  page.should_not have_button("Create submission")
end

You can also define steps that parse arguments from the string or call other steps:

Given /^there is a ([^ ]+) survey for \"([^\"]+)\"$/ do |year,name|
  @company = Company.find_by_name(name)
  assert !@company.nil?
  Given "there is a #{year} survey"
end

You can even take multi-line input, such as tables.

Automated testing is so awesome!

Rails: Exporting data from specific tables into fixtures

Rails is pretty darn amazing. There are plenty of gems (Ruby packages) that provide additional functionality. They’re like Drupal modules, except with more customizability (not just hooks) and fewer pre-built administrative interfaces (you win some, you lose some).

For example, the client asked me, “Can we edit the static content?” Now if I had asked about this as a requirement at the beginning of the project, we might have gone with Drupal instead–although the Rails Surveyor still feels cleaner than a CCK-based survey type, so we might’ve stayed with Rails.

Anyway, we were well into Rails now, so I looked for a content management system that I could integrate into the Rails 3-based website. After some experimenting with Refinery CMS (looks slick, but couldn’t get it to do what I wanted) and Comfortable Mexican Sofa (looked pretty geeky), I settled on Rich CMS. I nearly gave up on Rich CMS, actually, because I’d gotten stuck, but the web demo helped me figure out what I needed to do in order to enable it.

We’re still emptying and reloading the database a lot, though, so I wanted to make sure that I could save the CmsContent items and reload them. I didn’t want to back up the entire database, just a table or two. There were some gems that promised the ability to back up specific models, but I couldn’t figure it out. Eventually I decided to use the table-focused Rake code I saw in order to export the data to fixtures (seems to be based on code from the Rails Recipes book).

task :extract_fixtures => :environment do
  sql  = "SELECT * FROM %s"
  skip_tables = ["schema_info"]
  ActiveRecord::Base.establish_connection
  if (not ENV['TABLES'])
    tables = ActiveRecord::Base.connection.tables - skip_tables
  else
    tables = ENV['TABLES'].split(/, */)
  end
  if (not ENV['OUTPUT_DIR'])
    output_dir="#{RAILS_ROOT}/test/fixtures"
  else
    output_dir = ENV['OUTPUT_DIR'].sub(/\/$/, '')
  end
  (tables).each do |table_name|
    i = "000"
    File.open("#{output_dir}/#{table_name}.yml", 'w') do |file|
      data = ActiveRecord::Base.connection.select_all(sql % table_name)
      file.write data.inject({}) { |hash, record|
        hash["#{table_name}_#{i.succ!}"] = record
        hash
      }.to_yaml
      puts "wrote #{table_name} to #{output_dir}/"
    end
  end
end

Being a lazy programmer who doesn’t want to remember table names, I also defined the following Rake tasks:

task :save_content => :environment do
  ENV["TABLES"] = "cms_contents"
  Rake.application.invoke_task("myproj:extract_fixtures")
end
task :load_content do
  Rake.application.invoke_task("db:fixtures:load")
end

Then I can call rake myproj:save_content and rake myproj:load_content to do the right thing. Or rather, my co-developer (a new IBMer – hello, Vijay!) can do so, and then check his work into our git repository. =)

Now we can re-create the development database as often as we’d like without losing our page content!

2011-04-24 Sun 16:29

Drush, Simpletest, and continuous integration for Drupal using Jenkins (previously Hudson)

One of my development goals is to learn how to set up continuous integration so that I’ll always remember to run my automated tests. I picked up the inspiration to use Hudson from Stuart Robertson, with whom I had the pleasure of working on a Drupal project before he moved to BMO. He had set up continuous integration testing with Hudson and Selenium on another project he’d worked on, and they completed user acceptance testing without any defects. That’s pretty cool. =)

I’m a big fan of automated testing because I hate doing repetitive work. Automated tests also let me turn software development into a game, with clearly defined goalposts and a way to keep score. Automated tests can be a handy way of creating lots of data so that I can manually test a site set up the way I want it to be. I like doing test-driven development: write the test first, then write the code that passes it.

Testing was even better with Rails. I love the Cucumber testing framework because I could define high-level tests in English. The Drupal equivalent (Drucumber?) isn’t quite there yet. I could actually use Cucumber to test my Drupal site, but it would only be able to test the web interface, not the code, and I like to write unit tests in addition to integration tests. Still, some automated testing is better than no testing, and I’m comfortable creating Simpletest classes.

Jenkins (previously known as Hudson) is a continuous integration server that can build and test your application whenever you change the code. I set it up on my local development image by following Jenkins’ installation instructions. I enabled the Git plugin (Manage Jenkins – Manage Plugins – Available).

Then I set up a project with my local git repository. I started with a placeholder build step of Execute shell and pwd, just to see where I was. When I built the project, Hudson checked out my source code and ran the command. I then went into the Hudson workspace directory, configured my Drupal settings.php to use the database and URL I created for the integration site, and configured permissions and Apache with a name-based virtual host so that I could run web tests.

For build steps, I used Execute shell with the following settings:

mysql -u integration integration < sites/default/files/backup_migrate/scheduled/site-backup.mysql
/var/drush/drush test PopulateTestUsersTest
/var/drush/drush test PopulateTestSessionsTest
/var/drush/drush testre MyProjectName --error-on-fail

This loads the backup file created by Backup and Migrate, sets up my test content, and then uses my custom testre command.

Code below (c) 2011 Sacha Chua ([email protected]), available under GNU General Public License v2.0 (yes, I should submit this as a patch, but there’s a bit of paperwork for direct contributions, and it’s easier to just get my manager’s OK to blog about something…)

// A Drush command callback.
function drush_simpletest_test_regular_expression($test_re='') {
  global $verbose, $color;
  $verbose = is_null(drush_get_option('detail')) ? FALSE : TRUE;
  $color = is_null(drush_get_option('color')) ? FALSE : TRUE;
  $error_on_fail = is_null(drush_get_option('error-on-fail')) ? FALSE : TRUE;
  if (!preg_match("/^\/.*\//", $test_re)) {
    $test_re = "/$test_re/";
  }
  // call this method rather than simpletest_test_get_all() in order to bypass internal cache
  $all_test_classes = simpletest_test_get_all_classes();

  // Check that the test class parameter has been set.
  if (empty($test_re)) {
    drush_print("\nAvailable test groups & classes");
    drush_print("-------------------------------");
    $current_group = '';
    foreach ($all_test_classes as $class => $details) {
      if (class_exists($class) && method_exists($class, 'getInfo')) {
        $info = call_user_func(array($class, 'getInfo'));
        if ($info['group'] != $current_group) {
          $current_group = $info['group'];
          drush_print('[' . $current_group . ']');
        }
        drush_print("\t" . $class . ' - ' . $info['name']);
      }
    }
    return;
  }

  // Find test classes that match
  foreach ($all_test_classes as $class => $details) {
    if (class_exists($class) && method_exists($class, 'getInfo')) {
      if (preg_match($test_re, $class)) {
        $info = call_user_func(array($class, 'getInfo'));
        $matching_classes[$class] = $info;
      }
    }
  }

  // Sort matching classes by weight
  uasort($matching_classes, '_simpletest_drush_compare_weight');

  foreach ($matching_classes as $class => $info) {
    $main_verbose = $verbose;
    $results[$class] = drush_simpletest_run_single_test($class, $error_on_fail);
    $verbose = $main_verbose;
  }

  $failures = $successes = 0;
  foreach ($results as $class => $status) {
    print $status . "\t" . $class . "\n";
    if ($status == 'fail') {
      $failures++;
    } else {
      $successes++;
    }
  }
  print "Failed: " . $failures . "/" . ($failures + $successes) . "\n";
  print "Succeeded: " . $successes . "/" . ($failures + $successes) . "\n";
  if ($failures > 0) {
    return 1;
  }
}

I didn’t bother hacking Simpletest output to match the Ant/JUnit output so that Jenkins could understand it better. I just wanted a pass/fail status, as I could always look at the results to find out which test failed.

What does it gain me over running the tests from the command-line? I like having the build history and being able to remember the last successful build.

I’m going to keep this as a local build server instead of setting up a remote continuous integration server on our public machine, because it involves installing quite a number of additional packages. Maybe the other developers might be inspired to set up something similar, though!

2011-06-09 Thu 09:51

Sketchnotes: Marketing Automation, Jeffrey Yee (#torontob2b)

UPDATE 2012-11-15: Here’s the video recap!

Marketing Automation
Jeffrey Yee, Eloqua

Like these? Check out my other sketchnotes, visual book notes/reviews, and visual metaphors.

Here’s the text from the sketchnotes to improve people’s ability to search for it:

Marketing automation

Marketing Automation
Jeffrey Yee, Eloqua

leads small
list management
forms
scoring
analytics
events
challenge
-Too expensive
-Not fully used
-Not implemented correctly
-Did not address business needs

1. Focus
one thing! business need!
2. Identify
Look for what your top performers are already doing
3. Start small, then build for mass adoption
-Target the second-tier salespeople!
4. Wait patiently for the lift.
incremental improvement

Best practices from client side
Dun & Bradstreet
credit risk management sales & marketing supply risk management

1. Focus
Example
Retention trigger-based e-mail
one need
40.1% opens
13.4% click through
10% increase in retention rates
2. Identify before you automate
Focus group?
Study top performers
How are we achieving this today?
Can we automate and scale this?

Repurpose

Think linear, it’s easier that way

Get personal and add value
plaint text e-mail from sales, not marketing
3. Mass adoption (but start very small)
advocates get others on board

Look for the people who are close to their quotas:
Tier 2 segmenting your salespeople!

Have reps vet leads before adding to program

3rd party data
4. Wait patiently for the lift. Set expectations.
Ex results
-6 months
pipeline value *19%
# of yes 14%
average upsize 3%
ops won 25%

Budget 12+ months

Like low-hanging fruit
Scaling up what already works
Notes by Sacha Chua, @sachac, LivingAnAwesomeLife.com

 

Sketchnotes: The Very Versatile Drip–Mathew Sweezey (Pardot)

UPDATE: Dec 13, 2012 Want to watch the webinar? Here’s the video recording.

In this marketing webinar hosted by Pardot, Mathew Sweezey shared tips on setting up a drip nurturing program for marketing and sales support. Click on the image to view a larger size, and feel free to share this with attribution!

20121206 Pardot - The Very Versatile Drip - Mathew Sweezey

Pardot has many other webinars and recordings, so check them out if you’re curious about marketing automation.

Like this? Browse through my other sketchnotes, including my visual summary of The 5 Key Elements of a Better B2B Content Marketing Strategy by Nolin LeChasseur. I sketchnote technology/business conferences and presentations – if that sounds interesting, get in touch!

[Read more →]