On this page:

Automating bulk web stuff with iMacros

I found myself needing to download a whole bunch of JSON data from a server that had a weird authentication thing that Chrome could deal with but wget/curl/Ruby couldn’t. Since my Firefox was on the fritz, I couldn’t use Selenium IDE or the Selenium Webdriver. iMacros to the rescue! (Chrome, Firefox)

2013-11-21 Geek notes - automating bulk web stuff with iMacros

That plus lots of keyboard macros and text manipulation in Emacs, plus a little parsing and regexp substitution in Ruby, plus more Emacs munging got me the data I wanted. Hooray for bubblegum and string scripting!

Back to the joys of coverage testing: Vagrant, Guard, Spork, RSpec, Simplecov

Tests are important because programmers are human.  I know that I’m going to forget, make mistakes, and change things that I didn’t mean to change. Testing lets me improve my chances of at least noticing. Coverage tools show me how much of my code is covered by tests, improving my chances of seeing where my blind spots are.

In one of my last projects at IBM, I convinced my team to use coverage testing tools on a Rails development project. It was great watching the coverage numbers inch up, and we actually reached 100% (at least in terms of what rcov was looking at). I occasionally had to fix things when people broke the build, but sometimes people added and updated tests too. Although coverage testing has its weaknesses (are you testing for edge cases?), it’s better than nothing, and can catch some embarrassing bugs before they make it to the outside world.

Although I’m not currently taking on any web development work (I’m saving brainspace for other things), I have a few personal projects that I enjoy working on. For example, QuantifiedAwesome.com lets me track different measurements such as time. I had written some tests for it before, but since then, I’d been adding features and making changes without updating the tests. Like the way that unpaid credit card balances turn into financial debt (very bad habit, try to avoid this if possible), unwritten or out-of-date tests contribute to technical debt.

It was a little daunting to sit down and slowly work my way through the knots of out-of-date tests that had accumulated from haphazard coding. Here’s how I made the task more manageable:

  • I set up a virtual Linux development environment using Vagrant and Puppet. Many tools are much easier to set up on Linux than they are on Windows, and using a virtual machine for development means that I can easily isolate changes from the rest of my system. Vagrant allows you to specify the configuration of a VirtualBox machine and quickly set up a new instance of it. I copied the Puppet configuration from rails-dev-box and modified it to match my requirements. I also compiled and installed Emacs 24 from source, along with a few Ruby-related tools like haml-mode and rinari. (If you’re going to do a lot of development, you might as well do it with your favourite tools!) I copied the Vagrant private keys and imported them into Pageant (an ssh agent for Windows) for easy logins, and I forwarded the ports too.
  • I used Guard and Spork to efficiently run subsets of my tests. Guard is a tool that re-runs specific tests when it detects that files have changed, while Spork lets you run tests without reloading your entire Rails application. Instead of running rake spec (which runs the whole test suite) or rspec ./spec/path/to/file (and  having to copy and paste the failing test’s filename), I could let Guard take care of rerunning failed tests for me.
  • I worked on getting the old tests to pass. I wanted to get those cleared up before writing new tests. Good thing too – found a few bugs and some old code along the way.
  • I reviewed the recommendations for better tests. Better Specs has tips on using RSpec with Ruby.
  • Then I wrote new tests to cover the rest of my models. I used Simplecov for coverage testing. Since Vagrant uses shared folders, I could use Google Chrome to view the coverage webpages from my host computer. I still have some technical debt left – I need to write integration tests, and there’s more beyond that – but it’s satisfying to know that the models (data + logic) are covered. For some reason, my Guard+Spork combination doesn’t result in cumulative Simplecov scores, so I occasionally regenerate my coverage stats with rake spec. These tests will be useful as I write new features or move to Rails 4.

image

(… 21.5 hours of coding/development/infrastructure…)

New things I learned:

I use Mechanize to work with the Toronto Public Library’s webpage system so that I can retrieve the list of due books, renew items, or request holds. I wanted to test these as well. FakeWeb lets you intercept web requests and return your own responses, so I saved sample web pages to my spec/fixtures/files directory and used FakeWeb to return them. (ex: toronto_library_spec.rb)

Paperclip also needed some tweaking. I replaced my Paperclip uploads with File assignments so that I could test the image processing. (ex: clothing_spec.rb)

Always remember to wrap your RSpec tests in an it “…” instead of just context “when …” or describe “…”. I forgot to do this a few times and got confused about why stuff got left in my database and why the error messages weren’t formatted like they normally are.

Progress! Next step: Update my Cucumber tests and add some more integration tests…

I like this part of being a developer, even though writing new code is more fun. Testing is one of those time-consuming but powerful tools that can reduce frustration in the long run. I’ve experimented with applying automated testing techniques to everyday life before. I wonder what it would be like to take that even further… Any thoughts?

Sketchnotes: The Very Versatile Drip–Mathew Sweezey (Pardot)

UPDATE: Dec 13, 2012 Want to watch the webinar? Here’s the video recording.

In this marketing webinar hosted by Pardot, Mathew Sweezey shared tips on setting up a drip nurturing program for marketing and sales support. Click on the image to view a larger size, and feel free to share this with attribution!

20121206 Pardot - The Very Versatile Drip - Mathew Sweezey

Pardot has many other webinars and recordings, so check them out if you’re curious about marketing automation.

Like this? Browse through my other sketchnotes, including my visual summary of The 5 Key Elements of a Better B2B Content Marketing Strategy by Nolin LeChasseur. I sketchnote technology/business conferences and presentations – if that sounds interesting, get in touch!

[Read more →]

Sketchnotes: Marketing Automation, Jeffrey Yee (#torontob2b)

UPDATE 2012-11-15: Here’s the video recap!

Marketing Automation
Jeffrey Yee, Eloqua

Like these? Check out my other sketchnotes, visual book notes/reviews, and visual metaphors.

Here’s the text from the sketchnotes to improve people’s ability to search for it:

Marketing automation

Marketing Automation
Jeffrey Yee, Eloqua

leads small
list management
forms
scoring
analytics
events
challenge
-Too expensive
-Not fully used
-Not implemented correctly
-Did not address business needs

1. Focus
one thing! business need!
2. Identify
Look for what your top performers are already doing
3. Start small, then build for mass adoption
-Target the second-tier salespeople!
4. Wait patiently for the lift.
incremental improvement

Best practices from client side
Dun & Bradstreet
credit risk management sales & marketing supply risk management

1. Focus
Example
Retention trigger-based e-mail
one need
40.1% opens
13.4% click through
10% increase in retention rates
2. Identify before you automate
Focus group?
Study top performers
How are we achieving this today?
Can we automate and scale this?

Repurpose

Think linear, it’s easier that way

Get personal and add value
plaint text e-mail from sales, not marketing
3. Mass adoption (but start very small)
advocates get others on board

Look for the people who are close to their quotas:
Tier 2 segmenting your salespeople!

Have reps vet leads before adding to program

3rd party data
4. Wait patiently for the lift. Set expectations.
Ex results
-6 months
pipeline value *19%
# of yes 14%
average upsize 3%
ops won 25%

Budget 12+ months

Like low-hanging fruit
Scaling up what already works
Notes by Sacha Chua, @sachac, LivingAnAwesomeLife.com

 

Drush, Simpletest, and continuous integration for Drupal using Jenkins (previously Hudson)

One of my development goals is to learn how to set up continuous integration so that I’ll always remember to run my automated tests. I picked up the inspiration to use Hudson from Stuart Robertson, with whom I had the pleasure of working on a Drupal project before he moved to BMO. He had set up continuous integration testing with Hudson and Selenium on another project he’d worked on, and they completed user acceptance testing without any defects. That’s pretty cool. =)

I’m a big fan of automated testing because I hate doing repetitive work. Automated tests also let me turn software development into a game, with clearly defined goalposts and a way to keep score. Automated tests can be a handy way of creating lots of data so that I can manually test a site set up the way I want it to be. I like doing test-driven development: write the test first, then write the code that passes it.

Testing was even better with Rails. I love the Cucumber testing framework because I could define high-level tests in English. The Drupal equivalent (Drucumber?) isn’t quite there yet. I could actually use Cucumber to test my Drupal site, but it would only be able to test the web interface, not the code, and I like to write unit tests in addition to integration tests. Still, some automated testing is better than no testing, and I’m comfortable creating Simpletest classes.

Jenkins (previously known as Hudson) is a continuous integration server that can build and test your application whenever you change the code. I set it up on my local development image by following Jenkins’ installation instructions. I enabled the Git plugin (Manage Jenkins – Manage Plugins – Available).

Then I set up a project with my local git repository. I started with a placeholder build step of Execute shell and pwd, just to see where I was. When I built the project, Hudson checked out my source code and ran the command. I then went into the Hudson workspace directory, configured my Drupal settings.php to use the database and URL I created for the integration site, and configured permissions and Apache with a name-based virtual host so that I could run web tests.

For build steps, I used Execute shell with the following settings:

mysql -u integration integration < sites/default/files/backup_migrate/scheduled/site-backup.mysql
/var/drush/drush test PopulateTestUsersTest
/var/drush/drush test PopulateTestSessionsTest
/var/drush/drush testre MyProjectName --error-on-fail

This loads the backup file created by Backup and Migrate, sets up my test content, and then uses my custom testre command.

Code below (c) 2011 Sacha Chua ([email protected]), available under GNU General Public License v2.0 (yes, I should submit this as a patch, but there’s a bit of paperwork for direct contributions, and it’s easier to just get my manager’s OK to blog about something…)

// A Drush command callback.
function drush_simpletest_test_regular_expression($test_re='') {
  global $verbose, $color;
  $verbose = is_null(drush_get_option('detail')) ? FALSE : TRUE;
  $color = is_null(drush_get_option('color')) ? FALSE : TRUE;
  $error_on_fail = is_null(drush_get_option('error-on-fail')) ? FALSE : TRUE;
  if (!preg_match("/^\/.*\//", $test_re)) {
    $test_re = "/$test_re/";
  }
  // call this method rather than simpletest_test_get_all() in order to bypass internal cache
  $all_test_classes = simpletest_test_get_all_classes();

  // Check that the test class parameter has been set.
  if (empty($test_re)) {
    drush_print("\nAvailable test groups & classes");
    drush_print("-------------------------------");
    $current_group = '';
    foreach ($all_test_classes as $class => $details) {
      if (class_exists($class) && method_exists($class, 'getInfo')) {
        $info = call_user_func(array($class, 'getInfo'));
        if ($info['group'] != $current_group) {
          $current_group = $info['group'];
          drush_print('[' . $current_group . ']');
        }
        drush_print("\t" . $class . ' - ' . $info['name']);
      }
    }
    return;
  }

  // Find test classes that match
  foreach ($all_test_classes as $class => $details) {
    if (class_exists($class) && method_exists($class, 'getInfo')) {
      if (preg_match($test_re, $class)) {
        $info = call_user_func(array($class, 'getInfo'));
        $matching_classes[$class] = $info;
      }
    }
  }

  // Sort matching classes by weight
  uasort($matching_classes, '_simpletest_drush_compare_weight');

  foreach ($matching_classes as $class => $info) {
    $main_verbose = $verbose;
    $results[$class] = drush_simpletest_run_single_test($class, $error_on_fail);
    $verbose = $main_verbose;
  }

  $failures = $successes = 0;
  foreach ($results as $class => $status) {
    print $status . "\t" . $class . "\n";
    if ($status == 'fail') {
      $failures++;
    } else {
      $successes++;
    }
  }
  print "Failed: " . $failures . "/" . ($failures + $successes) . "\n";
  print "Succeeded: " . $successes . "/" . ($failures + $successes) . "\n";
  if ($failures > 0) {
    return 1;
  }
}

I didn’t bother hacking Simpletest output to match the Ant/JUnit output so that Jenkins could understand it better. I just wanted a pass/fail status, as I could always look at the results to find out which test failed.

What does it gain me over running the tests from the command-line? I like having the build history and being able to remember the last successful build.

I’m going to keep this as a local build server instead of setting up a remote continuous integration server on our public machine, because it involves installing quite a number of additional packages. Maybe the other developers might be inspired to set up something similar, though!

2011-06-09 Thu 09:51

Rails: Exporting data from specific tables into fixtures

Rails is pretty darn amazing. There are plenty of gems (Ruby packages) that provide additional functionality. They’re like Drupal modules, except with more customizability (not just hooks) and fewer pre-built administrative interfaces (you win some, you lose some).

For example, the client asked me, “Can we edit the static content?” Now if I had asked about this as a requirement at the beginning of the project, we might have gone with Drupal instead–although the Rails Surveyor still feels cleaner than a CCK-based survey type, so we might’ve stayed with Rails.

Anyway, we were well into Rails now, so I looked for a content management system that I could integrate into the Rails 3-based website. After some experimenting with Refinery CMS (looks slick, but couldn’t get it to do what I wanted) and Comfortable Mexican Sofa (looked pretty geeky), I settled on Rich CMS. I nearly gave up on Rich CMS, actually, because I’d gotten stuck, but the web demo helped me figure out what I needed to do in order to enable it.

We’re still emptying and reloading the database a lot, though, so I wanted to make sure that I could save the CmsContent items and reload them. I didn’t want to back up the entire database, just a table or two. There were some gems that promised the ability to back up specific models, but I couldn’t figure it out. Eventually I decided to use the table-focused Rake code I saw in order to export the data to fixtures (seems to be based on code from the Rails Recipes book).

task :extract_fixtures => :environment do
  sql  = "SELECT * FROM %s"
  skip_tables = ["schema_info"]
  ActiveRecord::Base.establish_connection
  if (not ENV['TABLES'])
    tables = ActiveRecord::Base.connection.tables - skip_tables
  else
    tables = ENV['TABLES'].split(/, */)
  end
  if (not ENV['OUTPUT_DIR'])
    output_dir="#{RAILS_ROOT}/test/fixtures"
  else
    output_dir = ENV['OUTPUT_DIR'].sub(/\/$/, '')
  end
  (tables).each do |table_name|
    i = "000"
    File.open("#{output_dir}/#{table_name}.yml", 'w') do |file|
      data = ActiveRecord::Base.connection.select_all(sql % table_name)
      file.write data.inject({}) { |hash, record|
        hash["#{table_name}_#{i.succ!}"] = record
        hash
      }.to_yaml
      puts "wrote #{table_name} to #{output_dir}/"
    end
  end
end

Being a lazy programmer who doesn’t want to remember table names, I also defined the following Rake tasks:

task :save_content => :environment do
  ENV["TABLES"] = "cms_contents"
  Rake.application.invoke_task("myproj:extract_fixtures")
end
task :load_content do
  Rake.application.invoke_task("db:fixtures:load")
end

Then I can call rake myproj:save_content and rake myproj:load_content to do the right thing. Or rather, my co-developer (a new IBMer – hello, Vijay!) can do so, and then check his work into our git repository. =)

Now we can re-create the development database as often as we’d like without losing our page content!

2011-04-24 Sun 16:29