September 16, 2008

Bulk view

Drupal and Drush: Updating the database from the command-line

So now that we're doing all our configuration changes in source code, it makes sense to automate database updates as much as I can. Here's something I've added to drush_tools so that I can run all the schema changes from the command-line:
function drush_tools_update($command = '') {
  require_once 'includes/';
  require_once 'update.php';
  $ret = ob_get_contents();

  $list = module_list();
  $update_list = array();
  foreach ($list as $module) {
    $updates = drupal_get_schema_versions($module);
    if ($updates !== FALSE) {
      $latest = 0;
      $base = drupal_get_installed_schema_version($module);
      foreach ($updates as $update) {
        if ($update > $base) {
          if ($update > $latest) { $latest = $update; }
          $update_list[$module][] = $update;
      if ($latest) {
        printf("%-30s %5d -> %5d (%s)\n", $module, $base, $latest, join(', ', $update_list[$module]));
      } else {
        printf("%-30s %5d\n", $module, $base);
  if (count($update_list) == 0) return;
  if ($command != 'force' && !drush_confirm(t('Do you really want to continue?'))) {
  foreach ($update_list as $module => $versions) {
    foreach ($versions as $v) {
      print "Running " . $module . "_update_" . $v . "\n";
      update_data($module, $v);
  $updates = ob_get_contents();

  cache_clear_all('*', 'cache', TRUE);
  cache_clear_all('*', 'cache_page', TRUE);
  cache_clear_all('*', 'cache_menu', TRUE);
  cache_clear_all('*', 'cache_filter', TRUE);
  $output = '';
  if (!empty($_SESSION['update_results'])) {
    $output .= "The following queries were executed:\n";
    foreach ($_SESSION['update_results'] as $module => $updates) {
      $output .= "\n" . $module . "\n--------------------------\n";
      foreach ($updates as $number => $queries) {
        $output .= 'Update #'. $number . ":\n";
        foreach ($queries as $query) {
          if ($query['success']) {
            $output .= "SUCCESS: " . $query['query'] . "\n";
          else {
            $output .= "FAILURE: " . $query['query'] . "\n";
        if (!count($queries)) {
          $output .= "No queries\n";
    $output .= "\n";
    print $output;

Drupal: Deploying two branches to three systems

To keep track of the bugfixes we'll need to make for our next release, I've created a Subversion branch called branches/release-1. Development of new features will continue on trunk, but we'll merge in the bugfixes from release-1 every so often. There are three environments we deploy to:
Developers should be able to easily test both versions on their local machines.
QA server
We should be able to deploy both versions to a publicly-accessible QA server for acceptance testing.
Production server
We should be able to deploy release-1 (and then later, release-2 and so on) to the production server, preferably after a lot of testing
Editorial changes happen on the production server, where our users update content. We would like to be able to take a snapshot of that database and use that to test our development code on the QA server or in our local development environments. Because we use Domain Access to serve multiple subdomains with shared content, it's not just a matter of using mysqldump to back up the database and copy it over. We also need to replace URLs inside the database, and we need to override domain_root using the $conf array in settings.php. I'm the only one running Linux, so the other developers don't really benefit from the Makefiles I've defined or the tools I use. For the simpler build system we had before (all development on trunk), I wrote a deployment script that allowed users to:
  • Download a stripped copy of the production database with the URLs changed for their local testing environment
  • Deploy a stripped copy of the production database to the QA server
  • Deploy a specified revision of the source code to the QA server
  • Deploy a specified revision of the source code to the production server
The new deployment script needed to allow users to do the same, but for both branches of the code. Both branches of the code would be simultaneously available on the QA server, so the script would need to deploy the code to different directories. After some fiddling around with the page design (because I care about making interfaces make sense!), I came up with something that looks like this:
  Development Release-1
Local Database
QA Database
QA Deployment
Production Deployment None
r988 | somegeek | 2008-09-15 13:00:00 -0500 (Mon, 15 Sep 2008) | 2 lines

[more changelog entries go here]
r986 | somegeek | 2008-09-15 11:03:38 -0500 (Mon, 15 Sep 2008) | 3 lines

Starting a branch for release-1
[more changelog entries go here]
The deployment script allows the user to get a copy of the database, deploy a copy of the database, or deploy specific revisions of branches. Because I was having a hard time figuring out how to do ssh key-based operations from Apache (which runs as a no-login user), I use two shell scripts to do the dirty work. One shell script connects to the production server, creates a partial backup, copies the information over, and does any necessary replacements. Another shell script takes a domain name and optionally a revision, and deploys the revision from the appropriate branch. Here's my totally small-scale PHP way to show the revisions log:
$dev_output = shell_exec("svn log $dev_url $details --limit 20");
$dev_revisions = preg_match_all('/r([0-9]+)/', $dev_output, $dev_matches);
where $dev_url is the URL of the trunk in Subversion, and $details contains the username and password specified as options for the Subversion command-line. I'm going to see if I can get my regression tests running on the server that I've got my deployment script on. Wouldn't that be awesome?

Working on a small project

Working on a small project means that I wear multiple hats, and that's helping me grow so much as a developer. Yesterday, I planned some changes to our build process and developed tools to make deployment painless. In a large project, all of this would have been decided already, and all I would need to do would be just to fit into the scheme. Here, I get to weigh the pros and cons, then make things better. What's the next step? If I integrate regression testing into the production deployment script, then we will be one step closer to development nirvana. Another step would be to do daily regression testing, per-checkin tests, or even pre-checkin tests. Wouldn't that be awesome? I'm definitely working above my pay grade. ;) That's because I don't have to make all the mistakes myself. Working on and watching open source development means that I can see what problems other developers run into and how they've solved them, and that's taught me ways to keep my development environment sane. I'm still going to make mistakes, but at least I can avoid some of them. Kaizen--relentless improvement--means that I'm not only solving problems right now, I'm also working on making things better. As a blogger (and somewhat of a teacher), I not only grow as a developer, but I'm helping other people grow too.