Category Archives: drupal

On this page:

Drupal, Emacs, and templates: Module update functions

Drupal’s coding conventions make it easier to hook into system behavior, but they also result in a lot of repetitive typing. For example, you can run code when upgrading a module by putting the code in a function named modulename_update_N() in your module’s install file. I found myself scrolling up and copy-pasting stuff too many times, so I decided to automate it instead.

I’ve been using yasnippet for my Emacs templates. All I needed to do to automate that little update bit was to write some code that figured out what the next update number should be. Here’s the snippet file I’ve just added (~/elisp/snippets/php-mode/drupal-mode/_update):

function `(sacha/drupal-module-name)`_update_`(sacha/drupal-module-update-number)`() {
  $ret = array();
  $0
  $ret[] = array(
    'success' => true,
    'query' => '$1',
  );
  return $ret;
}

The relevant functions from my ~/.emacs:

(defun sacha/drupal-module-update-number ()
  "Return the number of the next module update function.
This is one more than the highest number used so far.
This function should be called in a module's .install file."
  (save-excursion
    (save-restriction
      (widen)
      (goto-char (point-min))
      (let ((module-name (sacha/drupal-module-name))
	    (max 0))
	(while (re-search-forward
		(concat "function[ \t\n]+" module-name "_update_\\([0-9]+\\)") nil t)
	  (setq max (max (string-to-number (match-string 1)) max)))
	(number-to-string (1+ max))))))

(defun sacha/drupal-module-name ()
  "Return the Drupal module name for .module and .install files."
  (file-name-sans-extension (file-name-nondirectory (buffer-file-name))))

I can’t think of how I’d do that in Eclipse. =) Don’t get me wrong–I still like Eclipse–but I heart being able to hack my editor on the fly.

Drupal 5: Migrating a production database to a QA server

Building on the configuration management strategy I described last time, I wrote some scripts to make it easier for other developers to migrate the production database to the QA server or to get a copy of the production database for their local system. I needed to consider the following factors:

  • Domain name changes: Because we use Domain Access to serve multiple subdomains using a single Drupal installation and shared sign-on, we needed to make sure that all instances of the domain root are replaced and the correct domain_root is set. For example, the site URLs might be:
    Production QA Local
    example.com qa.example.com local.example.com
    foo.example.com foo.qa.example.com foo.local.example.com
    bar.example.com bar.qa.example.com bar.local.example.com
  • Privacy concerns: The QA database and local copies of the database should not contain sensitive user information. All users aside from the administrator should be removed from the system, and all content on site should be limited to editorial content.
  • Efficiency: We don’t need data like access logs or watchdog logs in QA or for local testing. This saves quite a lot of time and space during the data migration process.

Here’s how I did it:

  1. I identified tables I needed to copy over and tables that I could just leave empty. I did this by going through the output of “SHOW TABLES” in a MySQL console.
  2. In my Makefile, I declared a DB_DATA variable that contained a list of tables I wanted to copy.
  3. I wrote a backup_partial target in my Makefile that issued a series of commands:
    mysqldump -u ${DB_USER} --password=${DB_PASSWORD} ${DB} --no-data > partial.dump  # export the schema
    mysqldump -u ${DB_USER} --password=${DB_PASSWORD} ${DB} --opt --complete-insert ${DB_DATA} >> partial.dump # Dump some of the data
    mysqldump -u ${DB_USER} --password=${DB_PASSWORD} ${DB} --opt --complete-insert --where='uid< =1' users users_roles >> partial.dump # Dump the admin and anonymous users
    echo "UPDATE node SET uid=1;" >> partial.dump # Set all the node authors to admin
    echo "REPLACE INTO search_index_queue (nid, module, timestamp)  select nid, type, unix_timestamp(now()) FROM node;" >> partial.dump # Prepare for reindexing
    
  4. I wrote a shell script on an internal server that accepted an argument (the domain to translate to) and performed the following steps:
    1. ssh into the production server and run make with my backup_partial target, compressing the resulting partial.dump
    2. scp the partial.dump.gz from the production server onto the internal server
    3. unpack partial.dump.gz
    4. figure out what $DOMAIN is supposed to be based on the arguments
    5. run perl -pi -e "s/example.com/$DOMAIN/" partial.dump
    6. load partial.dump into my database
    7. run cron.php if it can
  5. I added two buttons to my web-based deploy script: one button to migrate the production database to the QA server, one button to make a copy of the production database for the domain “local.example.com”. Both buttons call
  6. I created multisite settings.php in my Drupal directory (ex: sites/local.example.com and sites/qa.example.com). The production settings go in default/settings.php, and the multisite settings.php override it like this:
    $conf = array(
      'domain_root' => 'local.example.com',
    );
    $cookie_domain = '.' . $conf['domain_root'];
    

    $conf allows you to override Drupal variables returned by variable_get.

So now, I can click on a button to migrate a sanitized copy of the production database to the QA server or to my local system. Sweet!

Running groups of Drupal tests from the command line

I’ve written about using drush to evaluate PHP statements in the Drupal context using the command line before, and it turns out that Drush is also quite useful for running Simpletest scripts. Drush comes with a module that allows you to display all the available tests with “drush test list”, run all the tests with “drush test run”, or run specified tests with “drush test run test1,test2″.

‘Course, I wanted to run groups of tests and tests matching regular expressions, so I defined two new commands:

drush test run re regular-expression
Run all tests matching a regular expression that uses ereg(..) to match.
Ex: drush test run re Example.*
drush test run group group1,group2…
Run all tests matching the given groups
Ex: drush test run group Example

Here’s the patch to make it happen:

Index: drush_simpletest.module
===================================================================
--- drush_simpletest.module	(revision 884)
+++ drush_simpletest.module	(working copy)
@@ -12,9 +12,13 @@
 function drush_simpletest_help($section) {
   switch ($section) {
       case 'drush:test run':
-        return t("Usage drush [options] test run.\n\nRun the specified specified unit tests. If  is omitted, all tests are run.  should be a list of classes separated by a comma. For example: PageCreationTest,PageViewTest.");
+        return t("Usage drush [options] test run .\n\nRun the specified unit tests. If  is omitted, all tests are run.  should be a list of classes separated by a comma. For example: PageCreationTest,PageViewTest.");
       case 'drush:test list':
         return t("Usage drush [options] test list.\n\nList the available tests. Use drush test run command to run them. ");
+      case 'drush:test group':
+        return t("Usage drush [options] test group .\n\nRun all unit tests in the specified groups. For example: drush test group Group1,Group2");
+      case 'drush:test re':
+        return t("Usage drush [options] test re .\n\nRun all unit tests matching this regular expression. For example: drush test re Page.*");
   }
 }
 
@@ -30,10 +34,18 @@
     'callback' => 'drush_test_list',
     'description' => 'List the available Simpletest test classes.',
   );
+  $items['test re'] = array(
+    'callback' => 'drush_test_re',
+    'description' => 'Run one or more Simpletest tests based on regular expressions.',
+  );
+  $items['test group'] = array(
+    'callback' => 'drush_test_group',
+    'description' => 'Run one or more Simpletest test groups.',
+  );
   return $items;
 }
 
-function drush_test_list() {
+function drush_test_get_list() {
   simpletest_load();
   // TODO: Refactor simpletest.module so we don't copy code from DrupalUnitTests
   $files = array();
@@ -60,6 +72,11 @@
       $rows[] = array($class, $info['name'], truncate_utf8($info['desc'], 30, TRUE, TRUE));
     }
   }
+  return $rows;
+}
+
+function drush_test_list() {
+  $rows = drush_test_get_list();
   return drush_print_table($rows, 0, TRUE);
 }
 
@@ -75,3 +92,31 @@
   }
   return $result;
 }
+
+function drush_test_re($expression) {
+  if (!$expression) {
+    die('You must specify a regular expression.');
+  }
+  $rows = drush_test_get_list();
+  $tests = array();
+  foreach ($rows as $row) {
+    if (ereg($expression, $row[0])) {
+      $tests[] = $row[0];
+    }
+  }
+  simpletest_run_tests($tests, 'text');
+  return $result;
+}
+
+function drush_test_group($groups) {
+  $rows = drush_test_get_list();
+  $tests = array();
+  $groups = explode(',', $groups);
+  foreach ($rows as $row) {
+    if (in_array($row[1], $groups)) {
+      $tests[] = $row[0];
+    }
+  }
+  simpletest_run_tests($tests, 'text');
+  return $result;
+}

That makes running tests so much easier and more fun!

Drupal: Programmatically installing and enabling modules in the .install file

To make configuration management easier, we decided to make sure that all behavior-related changes are in the source code repository. So when I needed to add the reCAPTCHA module to the project, I needed to figure out how to programmatically install and enable the module with update code in another module’s .install file.

Here is some sample code to do so:

/**
 * Install and enable the captcha module.
 */
function yourmodule_update_1() {
  $ret = array();
  include_once('includes/install.inc');
  module_rebuild_cache();
  drupal_install_modules(array('recaptcha'));
  variable_set('recaptcha_public_key', 'PUBLIC KEY GOES HERE');
  variable_set('recaptcha_private_key', 'SECRET KEY GOES HERE');
  $ret[] = array(
    'success' => true,
    'query' => 'Installed recaptcha module and enabled it',
  );
  return $ret;
}

Drupal: Making our build system better

Hooray! We have running code, and we’re about to make another release after our code exits quality assurance. This means, of course, that we’ll need some way to differentiate the inevitable bugfixes that the next production release will require, and development of new features.

What’s the best way to do this? Making a release branch seems like a good idea. Here’s how I did it:

svn copy http://subversion.example.com/myproject/trunk http://subversion.example.com/myproject/branches/release-1 

Bugfixes for release-1 will be committed to the release-1 branch, while new development continues on trunk. Bugfixes will be periodically merged into trunk to make it easier to roll the next release, which will be the release-2 branch.

Next, I need to configure my local system so that it’s easy to switch back and forth. I could work with a single source tree for local.example.com, but that means switching back and forth. The best thing to do would be to have two separate source code directories: one for trunk, and one for production releases.

svn co http://subversion.example.com/myproject/branches/release-1 /var/www/example.com-prod

For each site, I’ll need

  • a source code directory
  • a database
  • entries in /etc/hosts
  • entries in my Apache configuration
  • a local site directory (ex: sites/dev.local.example.com) with settings.php
  • a QA site directory (ex: sites/dev.qa.example.com) with settings.php

I’ll also need to update my Makefile to make it easier to work. For example, the Makefile should connect me to the right database depending on which branch I’m on. How do I determine this? One way is to have an unversioned file that overrides some of the Makefile variables, and to include that file in my Makefile. I can do this by adding the following to my Makefile:

-include *.mk

and then creating a dev-local.mk file that changes the values of my variables.

I’ll need copies of the production database translated for the different domains, which means I need to update my deploy script and format it a little to make it easier to deal with all these options. Hmm…

This will be fun.

UPDATE: Fixed HTML tags. Thanks for pointing it out!

Drupal: Deploying two branches to three systems

To keep track of the bugfixes we’ll need to make for our next release, I’ve created a Subversion branch called branches/release-1. Development of new features will continue on trunk, but we’ll merge in the bugfixes from release-1 every so often.

There are three environments we deploy to:

Local
Developers should be able to easily test both versions on their local machines.
QA server
We should be able to deploy both versions to a publicly-accessible QA server for acceptance testing.
Production server
We should be able to deploy release-1 (and then later, release-2 and so on) to the production server, preferably after a lot of testing

Editorial changes happen on the production server, where our users update content. We would like to be able to take a snapshot of that database and use that to test our development code on the QA server or in our local development environments. Because we use Domain Access to serve multiple subdomains with shared content, it’s not just a matter of using mysqldump to back up the database and copy it over. We also need to replace URLs inside the database, and we need to override domain_root using the $conf array in settings.php.

I’m the only one running Linux, so the other developers don’t really benefit from the Makefiles I’ve defined or the tools I use. For the simpler build system we had before (all development on trunk), I wrote a deployment script that allowed users to:

  • Download a stripped copy of the production database with the URLs changed for their local testing environment
  • Deploy a stripped copy of the production database to the QA server
  • Deploy a specified revision of the source code to the QA server
  • Deploy a specified revision of the source code to the production server

The new deployment script needed to allow users to do the same, but for both branches of the code. Both branches of the code would be simultaneously available on the QA server, so the script would need to deploy the code to different directories.

After some fiddling around with the page design (because I care about making interfaces make sense!), I came up with something that looks like this:

  Development Release-1
Local Database
QA Database
QA Deployment
Production Deployment None
Changelog
-------
r988 | somegeek | 2008-09-15 13:00:00 -0500 (Mon, 15 Sep 2008) | 2 lines

Use dev.qa.example.com
------------
[more changelog entries go here]
------------------------------------------------------------------------
r986 | somegeek | 2008-09-15 11:03:38 -0500 (Mon, 15 Sep 2008) | 3 lines

Starting a branch for release-1
------------------------------------------------------------------------
[more changelog entries go here]

The deployment script allows the user to get a copy of the database, deploy a copy of the database, or deploy specific revisions of branches.

Because I was having a hard time figuring out how to do ssh key-based operations from Apache (which runs as a no-login user), I use two shell scripts to do the dirty work. One shell script connects to the production server, creates a partial backup, copies the information over, and does any necessary replacements. Another shell script takes a domain name and optionally a revision, and deploys the revision from the appropriate branch.

Here’s my totally small-scale PHP way to show the revisions log:

$dev_output = shell_exec("svn log $dev_url $details --limit 20");
$dev_revisions = preg_match_all('/r([0-9]+)/', $dev_output, $dev_matches);

where $dev_url is the URL of the trunk in Subversion, and $details contains the username and password specified as options for the Subversion command-line.

I’m going to see if I can get my regression tests running on the server that I’ve got my deployment script on. Wouldn’t that be awesome?