'Course, I wanted to run groups of tests and tests matching regular expressions, so I defined two new commands:

drush test run re regular-expression

Run all tests matching a regular expression that uses ereg(..) to match.

Ex: drush test run re Example.*

drush test run group group1,group2…

Run all tests matching the given groups

Ex: drush test run group Example

Here's the patch to make it happen:

Index: drush_simpletest.module
===================================================================
--- drush_simpletest.module	(revision 884)
+++ drush_simpletest.module	(working copy)
@@ -12,9 +12,13 @@
 function drush_simpletest_help($section) {
   switch ($section) {
       case 'drush:test run':
-        return t("Usage drush [options] test run.\n\nRun the specified specified unit tests. If  is omitted, all tests are run.  should be a list of classes separated by a comma. For example: PageCreationTest,PageViewTest.");
+        return t("Usage drush [options] test run .\n\nRun the specified unit tests. If  is omitted, all tests are run.  should be a list of classes separated by a comma. For example: PageCreationTest,PageViewTest.");
       case 'drush:test list':
         return t("Usage drush [options] test list.\n\nList the available tests. Use drush test run command to run them. ");
+      case 'drush:test group':
+        return t("Usage drush [options] test group .\n\nRun all unit tests in the specified groups. For example: drush test group Group1,Group2");
+      case 'drush:test re':
+        return t("Usage drush [options] test re .\n\nRun all unit tests matching this regular expression. For example: drush test re Page.*");
   }
 }

@@ -30,10 +34,18 @@
     'callback' => 'drush_test_list',
     'description' => 'List the available Simpletest test classes.',
   );
+  $items['test re'] = array(
+    'callback' => 'drush_test_re',
+    'description' => 'Run one or more Simpletest tests based on regular expressions.',
+  );
+  $items['test group'] = array(
+    'callback' => 'drush_test_group',
+    'description' => 'Run one or more Simpletest test groups.',
+  );
   return $items;
 }

-function drush_test_list() {
+function drush_test_get_list() {
   simpletest_load();
   // TODO: Refactor simpletest.module so we don't copy code from DrupalUnitTests
   $files = array();
@@ -60,6 +72,11 @@
       $rows[] = array($class, $info['name'], truncate_utf8($info['desc'], 30, TRUE, TRUE));
     }
   }
+  return $rows;
+}
+
+function drush_test_list() {
+  $rows = drush_test_get_list();
   return drush_print_table($rows, 0, TRUE);
 }

@@ -75,3 +92,31 @@
   }
   return $result;
 }
+
+function drush_test_re($expression) {
+  if (!$expression) {
+    die('You must specify a regular expression.');
+  }
+  $rows = drush_test_get_list();
+  $tests = array();
+  foreach ($rows as $row) {
+    if (ereg($expression, $row[0])) {
+      $tests[] = $row[0];
+    }
+  }
+  simpletest_run_tests($tests, 'text');
+  return $result;
+}
+
+function drush_test_group($groups) {
+  $rows = drush_test_get_list();
+  $tests = array();
+  $groups = explode(',', $groups);
+  foreach ($rows as $row) {
+    if (in_array($row[1], $groups)) {
+      $tests[] = $row[0];
+    }
+  }
+  simpletest_run_tests($tests, 'text');
+  return $result;
+}

That makes running tests so much easier and more fun!

Technorati Tags: drupal, drush, patch, simpletest, testing

• Domain name changes: Because we use Domain Access to serve multiple subdomains using a single Drupal installation and shared sign-on, we needed to make sure that all instances of the domain root are replaced and the correct domain_root is set. For example, the site URLs might be:
  

  Production	QA	Local
  example.com	qa.example.com	local.example.com
  foo.example.com	foo.qa.example.com	foo.local.example.com
  bar.example.com	bar.qa.example.com	bar.local.example.com

• Privacy concerns: The QA database and local copies of the database should not contain sensitive user information. All users aside from the administrator should be removed from the system, and all content on site should be limited to editorial content.
• Efficiency: We don't need data like access logs or watchdog logs in QA or for local testing. This saves quite a lot of time and space during the data migration process.

Here's how I did it:

1. I identified tables I needed to copy over and tables that I could just leave empty. I did this by going through the output of "SHOW TABLES" in a MySQL console.
2. In my Makefile, I declared a DB_DATA variable that contained a list of tables I wanted to copy.
3. I wrote a backup_partial target in my Makefile that issued a series of commands:

   mysqldump -u ${DB_USER} --password=${DB_PASSWORD} ${DB} --no-data > partial.dump  # export the schema
   mysqldump -u ${DB_USER} --password=${DB_PASSWORD} ${DB} --opt --complete-insert ${DB_DATA} >> partial.dump # Dump some of the data
   mysqldump -u ${DB_USER} --password=${DB_PASSWORD} ${DB} --opt --complete-insert --where='uid< =1' users users_roles >> partial.dump # Dump the admin and anonymous users
   echo "UPDATE node SET uid=1;" >> partial.dump # Set all the node authors to admin
   echo "REPLACE INTO search_index_queue (nid, module, timestamp)  select nid, type, unix_timestamp(now()) FROM node;" >> partial.dump # Prepare for reindexing
   

4. I wrote a shell script on an internal server that accepted an argument (the domain to translate to) and performed the following steps:
   1. ssh into the production server and run make with my backup_partial target, compressing the resulting partial.dump
   2. scp the partial.dump.gz from the production server onto the internal server
   3. unpack partial.dump.gz
   4. figure out what $DOMAIN is supposed to be based on the arguments
   5. run perl -pi -e "s/example.com/$DOMAIN/" partial.dump
   6. load partial.dump into my database
   7. run cron.php if it can
5. I added two buttons to my web-based deploy script: one button to migrate the production database to the QA server, one button to make a copy of the production database for the domain "local.example.com". Both buttons call
6. I created multisite settings.php in my Drupal directory (ex: sites/local.example.com and sites/qa.example.com). The production settings go in default/settings.php, and the multisite settings.php override it like this:

   $conf = array(
     'domain_root' => 'local.example.com',
   );
   $cookie_domain = '.' . $conf['domain_root'];

   $conf allows you to override Drupal variables returned by variable_get.

So now, I can click on a button to migrate a sanitized copy of the production database to the QA server or to my local system. Sweet!

Technorati Tags: makefile, mysql

I've been using yasnippet for my Emacs templates. All I needed to do to automate that little update bit was to write some code that figured out what the next update number should be. Here's the snippet file I've just added (~/elisp/snippets/php-mode/drupal-mode/_update):

function `(sacha/drupal-module-name)`_update_`(sacha/drupal-module-update-number)`() {
  $ret = array();
  $0
  $ret[] = array(
    'success' => true,
    'query' => '$1',
  );
  return $ret;
}

The relevant functions from my ~/.emacs:

(defun sacha/drupal-module-update-number ()
  "Return the number of the next module update function.
This is one more than the highest number used so far.
This function should be called in a module's .install file."
  (save-excursion
    (save-restriction
      (widen)
      (goto-char (point-min))
      (let ((module-name (sacha/drupal-module-name))
	    (max 0))
	(while (re-search-forward
		(concat "function[ \t\n]+" module-name "_update_\\([0-9]+\\)") nil t)
	  (setq max (max (string-to-number (match-string 1)) max)))
	(number-to-string (1+ max))))))

(defun sacha/drupal-module-name ()
  "Return the Drupal module name for .module and .install files."
  (file-name-sans-extension (file-name-nondirectory (buffer-file-name))))

I can't think of how I'd do that in Eclipse. =) Don't get me wrong–I still like Eclipse–but I heart being able to hack my editor on the fly.

Technorati Tags: template, yasnippet

--- sites/local.example.com/modules/drush/drush_tools.module.orig    2008-08-05 17:18:48.000000000 -0400
+++ sites/local.example.com/modules/drush/drush_tools.module    2008-08-05 17:18:55.000000000 -0400
@@ -46,6 +46,10 @@
     'callback' => 'drush_tools_sync',
     'description' => 'Rsync the Drupal tree to/from another server using ssh'
   );
+  $items['eval'] = array(
+    'callback' => 'drush_tools_eval',
+    'description' => 'Evaluate a command',
+  );
   return $items;
 }
 
@@ -156,3 +160,6 @@
   }
 }
 
+function drush_tools_eval($command) {
+  eval($command);
+}

I also added an alias to my ~/.bashrc along the lines of:

alias drush='php ~/drupal/sites/local.example.com/modules/drush/drush.php -r ~/drupal -l http://local.example.com'

where ~/drupal is my multisite Drupal directory root.

After I loaded the alias with "source ~/.bashrc", I can now execute PHP statements in my Drupal context with commands like:

drush eval "variable_set('hello', 'world');"

Good stuff!

Technorati Tags: drush

Thinking about what could make the most difference in the other developers' productivity, I decided to invest time into making it easier for them to deploy code to the testing server. I had written a Makefile target that efficiently transferred only the updated files, but the other developers worked on Microsoft Windows and did not have all the necessary tools. I spent the morning writing a web-based interface for them: a password-protected PHP script that displayed a list of recent revisions and allowed people to deploy a selected revision to a separate server. Behind the scenes, it was a mess of bubblegum and string. To work around various limitations, I strung together sudo and suid and rsync and ssh key-based authentication. It wasn't pretty, but it worked. I e-mailed instructions to my team members, and they started using it right away.

After solving that problem, I focused on improving our tests. Fixing one bug often led to creating or recreating another, and these regression errors resulted in back-and-forth communication and wasted time. I explored the Simpletest automated testing framework for Drupal, and found out that I could write both unit tests and Web-based tests using the framework. However, I had a hard time figuring out why several of my Web-based tests consistently failed. I found out that the latest version of Simpletest for Drupal 5 did not understand the Location: header, which we use extensively in order to direct people to different subdomains and external sites. I fixed it and wrote a number of tests for one of our key modules. By the time I was ready to pack up and go home, I'd gotten into the swing of writing test cases.

Easier deployment and automated testing go a long way towards making a project almost a joy to work on. I'm glad I spent some time paving the way for my team and for other projects to come. =) Kaizen: relentless improvement.

I really like being able to diff a file with C-x v = (vc-diff) and check in a file with C-x v v (vc-next-action). I also like the way that svn-status (from psvn.el) lets me examine a directory tree, mark a set of files, and commit them–all without using the mouse. I probably should've set up keyboard shortcuts for these in Eclipse, but Eclipse made it too easy to just use the mouse. Emacs encourages you to use the keyboard, and it's easy to customize any keyboard shortcut.

php-mode's C-c . (php-show-arglist) works beautifully with the TAGS file that I'd set up using Exuberant Ctags, so I don't need to do anything special in order to get function definitions. Definition functions for PHP functions would be nice. In the meantime, there's C-c C-f (php-search-documentation).

The yasnippet template engine came in handy when I was writing test cases. I updated my module template to include the simpletest hook, added a test case template, and added a template for the simpletest hook as well. Yay dynamic templates!

And I just got Xdebug working with PhpMode and Geben… Sweet!

Index: browser.php
===================================================================
--- browser.php	(revision 753)
+++ browser.php	(working copy)
@@ -321,6 +321,22 @@
 
         $this->_history->recordEntry(
                 $this->_page->getUrl(),
                 $this->_page->getRequestData());
+        $counter = 0;
+        $url = $this->_page->_headers->_location;
+        if ($url) {
+          $url = &new SimpleURL($url);
+        }
+        while ($url && $counter < 10) {
+          $this->_page = &$this->_fetch($url, $parameters);
+          $this->_history->recordEntry(
+            $this->_page->getUrl(),
+            $this->_page->getRequestData());
+          $url = $this->_page->_headers->_location;
+          if ($url) {
+            $url = &new SimpleURL($url);
+          }
+          $counter++;
+        }
         return $this->_page->getRaw();
     }

(update: okay, let's try diff syntax highlighting with wp-syntax…)

We're getting ready for the next phase of our Drupal project, and it's a good time to think about how we can make our process better.

• Automated builds: I have a few Makefile commands related to pushing the latest development source code to the testing server, but the other developers can't use the commands on Microsoft Windows. However, they have access to another Linux-based server. If I can simplify the deployment process (maybe a password-protected webpage that allows you to choose which revision to deploy?), then they won't be held up when I go on vacation.
• Automated testing: We used simpletest for a few pieces of functionality, but we don't have anything close to coverage. I'd like to learn how to write proper tests for Drupal so that I can avoid regression errors, which I often made during development.
• Switching between hosts: Because we use Domain Access, I can't just use a local domain name and a copy of the server's database. My current approach is to use the same domain name as on the testing server, and then keep editing /etc/hosts to switch back and forth. An alternative might be to create a Makefile target that grabs the server's database, runs it through sed to translate all the domain names to my local domain, and restores the database from this translated file. That way, I don't need to edit /etc/hosts all the time.
• Coding environment: I'm thinking of moving my development from Eclipse to Emacs in order to be able to customize my environment more effectively. I'll post more notes about it as I figure out what works for me and what doesn't. It's a good excuse to learn even more about Emacs…

What worked well:

• Source code control, I love you so much, even if you're Subversion.
• Adding a CSS person to our team meant that the other developer and I were much less stressed out about cross-browser issues. Hooray!
• Using a defect-tracking system was infinitely better than sending e-mail around, even if that defect-tracking system was ClearQuest.

It's only one minute and 25 seconds, and it'll probably make you smile. Plus, it's a (mostly) working sitemap, and how cool is that? The prize for this category is an iPod Touch, which (if I win it) I will promptly put to good use. After all, if I used my Nintendo DS to make a presentation, what might I do with something like the iPod Touch?

Honorable mention would get me a copy of Presentation Zen book, which I liked so much I bought it already, so I'd be happy to raffle it off and keep just the warm and fuzzy feelings that my fledgling sketching and writing skills got noticed. Want it? Leave a comment on this entry (just one comment is fine), and I'll pick a random commenter if I win the book.

In addition to either blog posts about doing interesting things with the iPod Touch or a chance at winning a free book, you'll also get the satisfaction of supporting a geek in an area dominated by graphic designers. =) (Not that graphic designers can't be geeks, of course.)

Anyway, please vote for my presentation! =) Voting ends on July 31, 2008.

World's Best Presentation Contest

There's a hook_mail_alter function that allows you to modify any message your system sends. Example:

function mymodule_mail_alter($mailkey, &$to, &$subject, &$body, &$from, &$headers) {
   $body .= "\n\n" . t('This is a system-generated email, please do not reply to this message');
}

You gotta love developers who plan for extensibility and put all sorts of hooks into the code…

127.0.0.1 example.com sub1.example.com sub2.example.com

in my /etc/hosts file.

However, Firefox caches the domain name information, and it can be confusing to figure out which server I'm on. The following setup makes this much easier:

1. Install Domain Details to show the IP address of the server in your status bar. Good for a quick check - am I on 127.0.0.1 or elsewhere? (NOTE: I started with ShowIP, but it looks like ShowIP caches the information.)
2. Install the Clear Cache Button extension.
3. Restart your browser and customize your toolbar. Add the Clear Cache button. Click on this before switching sites.

In an ideal world, I'll have a plugin that automatically twiddles my DNS entries, too. =)

• developers have a consistent environment for development and testing, so all changes need to be checked into the Subversion source code control system
• all significant changes are captured so that we can deploy the tested system on the production server

Here's our current plan:

• Editorial changes (creating new nodes, editing existing nodes) happen on the testing server. Developers can test their local systems with real data by downloading a data dump from the testing server and loading it locally.
• Code changes that don't require database updates can be pushed to the testing server after local testing.
• Code changes that require database changes or one-time administrative actions can be handled in modules' .install files as modulename_update_somenumber() and _install() functions. This can be tested locally by loading a data dump from the testing server and then running update.php. It should also be tested from a fresh install. If the change works locally, then the testing server can be updated. (Make sure to run update.php on the testing server as well).

When we deploy the system to production, we'll review all the content, remove the test data, and push the data and the code to the live site.

See http://heyrocker.com/drupal/content/deployment-and-change-management-problem for more analysis.

I had spent last Thursday and Friday searching the Internet for people's blog posts about the topic. I came across exact descriptions of the problems I encountered, but no solutions–only notes about critical bugs in Drupal 5. Asking the developers in #drupal didn't yield any tips, either. They told me to get ready for a world of pain.

Then I remembered that the password-reset module used a multi-step form, and validation actually worked. I had already incorporated part of the source code into one of my modules, and it was easy to extend that example. I defined a new multi-step form in my PHP code as a replacement for the system form in user/register. There were some complications around the fact that other modules modify the behavior of user_register, but I got the code to validate and submit the data. And it worked!

No code snippet, sorry. Too many interactions for me to explain neatly. Some hints:

• You can use logic to set the value of $step.
• Pass information to the next step using hidden fields.
• Make sure the step field is included in your form template. Watch out for this if you've got a form template that specifies all the fields explicitly.
• Captcha uses hook_form_alter to add a captcha based on the form_id. It doesn't know about steps. You'll need your own hook_form_alter to unset the captcha if you don't need it on the other steps. Make sure your module runs after captcha by changing the weights in the system table.

So now I have a multi-step form that can skip step 1 (employee verification) if a valid invite code is provided, asks for CAPTCHA once, and does all the validation and user registration magic. Hooray!

Following the instructions for programmatically creating, inserting, and updating CCK nodes will get you most of the way there. Here are some of the gotchas you'll want to watch out for.

I was using Path to define aliases for my nodes and the path aliases were just not getting created. After lots and lots of tracing with XDebug, I found out that it was checking for user_access, but no user was active during the installation process. Solution: create the admin user and load it into a global $user variable.

I also spent a fair bit of time being very annoyed with CCK and creating nodes programmatically in my install profile. Turns out that you first need to use install_add_content_type to create the base node type, _then_ follow the instructions to drupal_execute the form that adds custom fields.

Here's a quick sketch of the code:

function profilename_profile_final() {
  // lots of other code here
  install_add_user('admin', 'admin', 'admin@example.com', array(), 1);
  global $user;
  $user = user_load(array('uid' => 1));
  install_add_content_type(array(
    // take this from Install Profile Wizard
  ));
  node_get_types('types', NULL, TRUE);  // flush cache
  $content = array();
  $content['type'] = array(
   // take this from content copy export
  );
  $content['fields'] = array(
   // take this from content copy export
  );
    _install_create_content($content);
}
function _install_create_content($content) {
  global $_install_macro;
  $type_name = $content['type']['type'];
  $_install_macro[$type_name] = $content;
  include_once drupal_get_path('module', 'node') .'/content_types.inc';
  include_once drupal_get_path('module', 'content') .'/content_admin.inc';
  $macro = 'global $_install_macro; $content = $_install_macro['. $type_name .'];';
  drupal_execute('content_copy_import_form', array('type_name' => $type_name, 'macro' => $macro));
  content_clear_type_cache();
}

Now I feel more confident about CCK…

Technorati Tags: drupal cck

• Use source code control. No, seriously. Use it. Religiously. It's going to save you someday. CVS is popular. Subversion is also easy to set up. It's a must when you're working with other people, but even if you're on your own, it's really handy to be able to go back to previous versions or to check your changes.
• Get to know your integrated development environment (IDE). For example, I use Eclipse + PHP Development Toolkit (PDT) + Xdebug, and the keyboard shortcuts save me so much time and make it easier for me to think about my code instead of thinking about editing my code. F3 to jump to a function definition, Ctrl-Shift-R to open a file by typing part of the filename, Ctrl-space to complete text… I've also gotten XDebug integration to work, so I can step through code and examine variables without putting var_dump everywhere. Know your tools.
• Add the XDebug extension to your PHP installation. Even if you don't get XDebug integrated with Eclipse, you'll like the new and improved var_dump and other debugging functions. I've heard good things about Zend Debugger, too.
• Put your entire Drupal directory into your project in the IDE, not just your site-specific code. You can even check your entire Drupal directory into your source code control repository. This makes it easier to look up functions in Drupal core, not just your sites/all/modules directory. You _could_ version-control just your site-specific code (profiles/, sites/, etc.), but set-up becomes a little more complicated.
• Use an installation profile to manage your configuration. Experiment with things using the administration interface, then create/update an installation profile that sets up a brand-new Drupal installation. This will save you lots of time and head-scratching when you need to reproduce your setup.
• Make starting from scratch easier, and do it often. I have a Makefile that drops my database and recreates it, resets my settings.php, and even opens up a Firefox browser that uses my installation profile to set everything up from scratch. Drupal 5: Copy your settings.php to settings-default.php or something like that before you install your system, then copy settings-default.php over settings.php when you want to refresh your setup. Drupal 6: Delete settings.php when you want to refresh your setup.

• Use multiple profiles. You can start completely independent Firefox processes by using profiles. For example, you could have your default profile with all the bells and whistles, and a test profile with just FireBug. You can log in as different users, too. Use this command to start a new Firefox process: firefox -ProfileManager -no-remote
• Install and use FireBug. Best web development extension ever. I use this to examine the document object model, debug my CSS rules, experiment with my CSS and HTML on the fly, and monitor the network requests.
• Set up quick searches for drupal.org and php.net. You can define quick searches by creating bookmarks that use keywords and %s placeholders in Location:. Example: Do a quick Google-search of Drupal.org by adding a bookmark with Location: http://google.com/search?q=inurl:drupal.org+%s and Keyword: gd (short for Google Drupal). To use your new quick search, type Ctrl-L or Alt-D to get to the address bar, then type gd keywords. Set up ways to search php.net and drupal.org/project, too. =)

• Simplify switching between your computer and your remote server. If you've set up your local system for virtual hosts and you have a remote server for testing, you'll want to be able to switch between testing locally and testing remotely. Set up your local system to respond to all the domain names for the site (ex: example.com, foo.example.com, bar.example.com) and a local subdomain (ex: local.example.com). Set up your remote server to respond to all the domain names for the site (example.com, foo.example.com, bar.example.com) and a remote subdomain (ex: test.example.com). Then set up your /etc/hosts (c:\windows\system32\drivers\etc\hosts on Microsoft Windows) to look like this:

  127.0.0.1 localhost local.example.com
  remote.server.ip.address test.example.com
  
  127.0.0.1 example.com foo.example.com bar.example.com
  #remote.server.ip.address example.com foo.example.com bar.example.com
  

  local.example.com will always resolve to your computer, test.example.com will always resolve to the other computer, and everything else depends on which line is active. # comments out a line. When you want to switch to testing on the remote server, add # to the beginning of the line for 127.0.0.1 example.com… and remove # from the line for your remote server. If you want to automate this even further, write a script that swaps different /etc/hosts files around.

Technorati Tags: development, drupal, eclipse, ide, xdebug

Should be fun. =)

Technorati Tags: drupal toronto

Backporting from Drupal 6 to Drupal 5: Change the menu items

• Move the menu path from the array key to the path attribute.
• Rename

  page callback to callback and page arguments to callback arguments.

• Remove the file and file path attributes. Move the included files to include_once lines for now, and look for instances of include that need to be changed to include_once. Read the included files to check for unintended execution.
• Change tokenized menu items (%/%/%) to ones that dynamically build the menu path (arg(1) . '/' . arg(2) . '/' . arg(3)). Don't forget - this shouldn't be cached.
• Change 'access arguments' => array( to 'access' => user_access( or something similar.

Those were most of the changes I needed to make in order to backport bawstats to Drupal 5. Seems to work so far… =) More notes when I run into other things!

Technorati Tags: drupal sketches

global $user;
if ($user) {
  $array = array('fieldname' => 'value_to_save');
  user_save($user, $array);
}

Then access it with $user->fieldname. For more configurability, check out the Profile module.

Technorati Tags: drupal newbie

Here's the code I added to the end of the profilename_profile_final() function:

    // Add the following to the end of settings.php
    $file = fopen("sites/default/settings.php", "a");
    if ($file) {
      fputs($file, "\$cookie_domain = '.transitions2.org';\n");
      fputs($file, "require_once './sites/all/modules/domain/domain_conf/settings_domain_conf.inc';\n");
      fputs($file, "require_once './sites/all/modules/domain/domain_prefix/settings_domain_prefix.inc';\n");
      fclose($file);
    } else {
      drupal_set_message("Can't add domain-related lines to sites/default/settings.php");
    }