Deploying Drupal updates onto a live site

Our configuration management discipline paid off last week, when we rolled out lots of bugfixes onto our production server. I used my deployment script to test the release-1 branch against our regression tests and copy the source code to the production server after it passed. A quick database update later, and everything worked without a hitch.

Why did it work? Here’s what’s behind our build system:

  • We have a branch for release-1 bugfixes and a trunk branch for new features. It’s easier than cherrypicking features to push to production.
  • Module .install files handle all behavior-related changes (variables, new modules, etc.). Drupal can automatically apply those changes during the database upgrade process. Even the process of enabling a module is done through an update function in our main module’s .install file. Note that the module install files are called in alphabetic order, so you may need to think about the sequence of functions.
  • The Simpletest framework for unit and functional testing allows us to write regression tests that minimize risks of updates.
  • Updates and tests are always run after a stripped-down copy of the production database is loaded, and are also occasionally tested against a full copy of the production database. This makes sure that the changes will cleanly apply to the production server. Domain Access complicates things a little because the domain name is encoded in the database, so my deployment script also takes care of substituting the correct domain name.
  • To automate testing and upgrades, we use the Drupal Shell (drush) module in our local and QA environments.
  • I wrote a deployment script webpage that provides a form for managing database changes and source code deployments to our QA server and to our production server.

I think it’s pretty darn good. What would make it even awesomer?

  • An updated deployment plan. I’m going to work on this document later.
  • Testing before commits, instead of after. maybe I can set up a pre-hook on my own system, or use Emacs to do this somehow.
  • Buttons to get full copy of database for local, QA. Full copies of the database take much more time, but are useful for getting a realistic feel for the functionality.
  • An article on how other people can get up and running with this kind of thing. DeveloperWorks, maybe?

Drupal: Deploying two branches to three systems

To keep track of the bugfixes we’ll need to make for our next release, I’ve created a Subversion branch called branches/release-1. Development of new features will continue on trunk, but we’ll merge in the bugfixes from release-1 every so often.

There are three environments we deploy to:

Local
Developers should be able to easily test both versions on their local machines.
QA server
We should be able to deploy both versions to a publicly-accessible QA server for acceptance testing.
Production server
We should be able to deploy release-1 (and then later, release-2 and so on) to the production server, preferably after a lot of testing

Editorial changes happen on the production server, where our users update content. We would like to be able to take a snapshot of that database and use that to test our development code on the QA server or in our local development environments. Because we use Domain Access to serve multiple subdomains with shared content, it’s not just a matter of using mysqldump to back up the database and copy it over. We also need to replace URLs inside the database, and we need to override domain_root using the $conf array in settings.php.

I’m the only one running Linux, so the other developers don’t really benefit from the Makefiles I’ve defined or the tools I use. For the simpler build system we had before (all development on trunk), I wrote a deployment script that allowed users to:

  • Download a stripped copy of the production database with the URLs changed for their local testing environment
  • Deploy a stripped copy of the production database to the QA server
  • Deploy a specified revision of the source code to the QA server
  • Deploy a specified revision of the source code to the production server

The new deployment script needed to allow users to do the same, but for both branches of the code. Both branches of the code would be simultaneously available on the QA server, so the script would need to deploy the code to different directories.

After some fiddling around with the page design (because I care about making interfaces make sense!), I came up with something that looks like this:

  Development Release-1
Local Database
QA Database
QA Deployment
Production Deployment None
Changelog
-------
r988 | somegeek | 2008-09-15 13:00:00 -0500 (Mon, 15 Sep 2008) | 2 lines

Use dev.qa.example.com
------------
[more changelog entries go here]
------------------------------------------------------------------------
r986 | somegeek | 2008-09-15 11:03:38 -0500 (Mon, 15 Sep 2008) | 3 lines

Starting a branch for release-1
------------------------------------------------------------------------
[more changelog entries go here]

The deployment script allows the user to get a copy of the database, deploy a copy of the database, or deploy specific revisions of branches.

Because I was having a hard time figuring out how to do ssh key-based operations from Apache (which runs as a no-login user), I use two shell scripts to do the dirty work. One shell script connects to the production server, creates a partial backup, copies the information over, and does any necessary replacements. Another shell script takes a domain name and optionally a revision, and deploys the revision from the appropriate branch.

Here’s my totally small-scale PHP way to show the revisions log:

$dev_output = shell_exec("svn log $dev_url $details --limit 20");
$dev_revisions = preg_match_all('/r([0-9]+)/', $dev_output, $dev_matches);

where $dev_url is the URL of the trunk in Subversion, and $details contains the username and password specified as options for the Subversion command-line.

I’m going to see if I can get my regression tests running on the server that I’ve got my deployment script on. Wouldn’t that be awesome?