Category Archives: drupal

Managing configuration changes in Drupal

One of our clients asked if we had any tips for documenting and managing Drupal configuration, modules, versions, settings, and so on. She wrote, “It’s getting difficult to keep track of what we’ve changed, when, for that reason, and what settings are in that need to be moved to production versus what settings are there for testing purposes.” Here’s what works for us.

Version control: A good distributed version control system is key. This allows you to save and log versions of your source code, merge changes from multiple developers, review differences, and roll back to a specified version. I use Git whenever I can because it allows much more flexibility in managing changes. I like the way it makes it easy to branch code, too, so I can start working on something experimental without interfering with the rest of the code.

Issue tracking: Use a structured issue-tracking or trouble-ticketing system to manage your to-dos. That way, you can see the status of different items, refer to specific issues in your version control log entries, and make sure that nothing gets forgotten. Better yet, set up an issue tracker that’s integrated with your version control system, so you can see the changes that are associated with an issue. I’ve started using Redmine, but there are plenty of options. Find one that works well with the way your team works.

Local development environments and an integration server: Developers should be able to experiment and test locally before they share their changes, and they shouldn’t have to deal with interference from other people’s changes. They should also be able to refer to a common integration server that will be used as the basis for production code.

I typically set up a local development environment using a Linux-based virtual machine so that I can isolate all the items for a specific project. When I’m happy with the changes I’ve made to my local environment, I convert them to code (see Features below) and commit the changes to the source code repository. Then I update the integration server with the new code and confrm that my changes work there. I periodically load other developers’ changes and a backup of the integration server database into my local environment, so that I’m sure I’m working with the latest copy.

Database backups: I use Backup and Migrate for automatic trimmed-down backups of the integration server database. These are regularly committed to the version control repository so that we can load the changes in our local development environment or go back to a specific point in time.

Turning configuration into code: You can use the Features module to convert most Drupal configuration changes into code that you can commit to your version control repository.

There are some quirks to watch out for:

  • Features aren’t automatically enabled, so you may want to have one overall feature that depends on any sub-features you create. If you are using Features to manage the configuration of a site and you don’t care about breaking Features into smaller reusable components, you might consider putting all of your changes into one big Feature.
  • Variables are under the somewhat unintuitively named category of Strongarm.
  • Features doesn’t handle deletion of fields well, so delete fields directly on the integration server.
  • Some changes are not exportable, such as nodequeue. Make those changes directly on the integration server.

You want your integration server to be at the default state for all features. On your local system, make the changes you want, then create or update features to encapsulate those changes. Commit the features to your version control repository. You can check if you’ve captured all the changes by reverting your database to the server copy and verifying your functionality (make a manual backup of your local database first!). When you’re happy with the changes, push the changes to the integration server.

Using Features with your local development environment should minimize the number of changes you need to directly make on the server.

Documenting specific versions or module sources: You can use Drush Make to document the specific versions or sources you use for your Drupal modules.

Testing: In development, there are few things as frustrating as finding you’ve broken something that was working before. Save yourself lots of time and hassle by investing in automated tests. You can use Simpletest to test Drupal sites, and you can also use external testing tools such as Selenium. Tests can help you quickly find and compare working and non-working versions of your code so that you can figure out what went wrong.

What are your practices and tips?

2011-06-09 Thu 12:25

Drush, Simpletest, and continuous integration for Drupal using Jenkins (previously Hudson)

One of my development goals is to learn how to set up continuous integration so that I’ll always remember to run my automated tests. I picked up the inspiration to use Hudson from Stuart Robertson, with whom I had the pleasure of working on a Drupal project before he moved to BMO. He had set up continuous integration testing with Hudson and Selenium on another project he’d worked on, and they completed user acceptance testing without any defects. That’s pretty cool. =)

I’m a big fan of automated testing because I hate doing repetitive work. Automated tests also let me turn software development into a game, with clearly defined goalposts and a way to keep score. Automated tests can be a handy way of creating lots of data so that I can manually test a site set up the way I want it to be. I like doing test-driven development: write the test first, then write the code that passes it.

Testing was even better with Rails. I love the Cucumber testing framework because I could define high-level tests in English. The Drupal equivalent (Drucumber?) isn’t quite there yet. I could actually use Cucumber to test my Drupal site, but it would only be able to test the web interface, not the code, and I like to write unit tests in addition to integration tests. Still, some automated testing is better than no testing, and I’m comfortable creating Simpletest classes.

Jenkins (previously known as Hudson) is a continuous integration server that can build and test your application whenever you change the code. I set it up on my local development image by following Jenkins’ installation instructions. I enabled the Git plugin (Manage Jenkins – Manage Plugins – Available).

Then I set up a project with my local git repository. I started with a placeholder build step of Execute shell and pwd, just to see where I was. When I built the project, Hudson checked out my source code and ran the command. I then went into the Hudson workspace directory, configured my Drupal settings.php to use the database and URL I created for the integration site, and configured permissions and Apache with a name-based virtual host so that I could run web tests.

For build steps, I used Execute shell with the following settings:

mysql -u integration integration < sites/default/files/backup_migrate/scheduled/site-backup.mysql
/var/drush/drush test PopulateTestUsersTest
/var/drush/drush test PopulateTestSessionsTest
/var/drush/drush testre MyProjectName --error-on-fail

This loads the backup file created by Backup and Migrate, sets up my test content, and then uses my custom testre command.

Code below (c) 2011 Sacha Chua ([email protected]), available under GNU General Public License v2.0 (yes, I should submit this as a patch, but there’s a bit of paperwork for direct contributions, and it’s easier to just get my manager’s OK to blog about something…)

// A Drush command callback.
function drush_simpletest_test_regular_expression($test_re='') {
  global $verbose, $color;
  $verbose = is_null(drush_get_option('detail')) ? FALSE : TRUE;
  $color = is_null(drush_get_option('color')) ? FALSE : TRUE;
  $error_on_fail = is_null(drush_get_option('error-on-fail')) ? FALSE : TRUE;
  if (!preg_match("/^\/.*\//", $test_re)) {
    $test_re = "/$test_re/";
  }
  // call this method rather than simpletest_test_get_all() in order to bypass internal cache
  $all_test_classes = simpletest_test_get_all_classes();

  // Check that the test class parameter has been set.
  if (empty($test_re)) {
    drush_print("\nAvailable test groups & classes");
    drush_print("-------------------------------");
    $current_group = '';
    foreach ($all_test_classes as $class => $details) {
      if (class_exists($class) && method_exists($class, 'getInfo')) {
        $info = call_user_func(array($class, 'getInfo'));
        if ($info['group'] != $current_group) {
          $current_group = $info['group'];
          drush_print('[' . $current_group . ']');
        }
        drush_print("\t" . $class . ' - ' . $info['name']);
      }
    }
    return;
  }

  // Find test classes that match
  foreach ($all_test_classes as $class => $details) {
    if (class_exists($class) && method_exists($class, 'getInfo')) {
      if (preg_match($test_re, $class)) {
        $info = call_user_func(array($class, 'getInfo'));
        $matching_classes[$class] = $info;
      }
    }
  }

  // Sort matching classes by weight
  uasort($matching_classes, '_simpletest_drush_compare_weight');

  foreach ($matching_classes as $class => $info) {
    $main_verbose = $verbose;
    $results[$class] = drush_simpletest_run_single_test($class, $error_on_fail);
    $verbose = $main_verbose;
  }

  $failures = $successes = 0;
  foreach ($results as $class => $status) {
    print $status . "\t" . $class . "\n";
    if ($status == 'fail') {
      $failures++;
    } else {
      $successes++;
    }
  }
  print "Failed: " . $failures . "/" . ($failures + $successes) . "\n";
  print "Succeeded: " . $successes . "/" . ($failures + $successes) . "\n";
  if ($failures > 0) {
    return 1;
  }
}

I didn’t bother hacking Simpletest output to match the Ant/JUnit output so that Jenkins could understand it better. I just wanted a pass/fail status, as I could always look at the results to find out which test failed.

What does it gain me over running the tests from the command-line? I like having the build history and being able to remember the last successful build.

I’m going to keep this as a local build server instead of setting up a remote continuous integration server on our public machine, because it involves installing quite a number of additional packages. Maybe the other developers might be inspired to set up something similar, though!

2011-06-09 Thu 09:51

VMWare, Samba, Eclipse, and XDebug: Mixing a virtual Linux environment with a Microsoft Windows development environment

I’m starting the second phase of a Drupal development project, which means I get to write about all sorts of geeky things again. Hooray! So I’m investing some time into improving my environment set-up, and taking notes along the way.

This time, I’m going to try developing code in Eclipse instead of Emacs, although I’ll dip into Emacs occasionally if I need to do anything involving keyboard macros or custom automation. Setting up a good Eclipse environment will help me use XDebug for line-by-line debugging. var_dump> can only take me so far, and I still haven’t figured out how to properly use XDebug under Emacs. Configuring Eclipse will also help me help my coworkers, who tend to not be big Emacs fans. (Sigh.)

So here’s my current setup:

  • A Linux server environment in VMWare, so that I can use all the Unix tools I like and so that I don’t have to fuss about with a WAMP stack
  • Samba for sharing the source code between the Linux VM image and my Microsoft Windows laptop
  • XDebug for debugging
  • Eclipse and PDT for development

I like this because it allows me to edit files in Microsoft Windows or in Linux, and I can use step-by-step debugging instead of relying on var_dump.

Setting up Samba

Samba allows you to share folders on the network. Edit your smb.conf (mine’s in /etc/samba/) and uncomment/edit the following lines:

security = user

[homes]
   comment = Home Directories
   browseable = no
   read only = no
   valid users = %S

You may also need to use smbpasswd to set the user’s password.

Xdebug

Install php5-xdebug or whatever the Xdebug package is for PHP on your system. Edit xdebug.ini (mine’s in /etc/php5/conf.d) and add the following lines to the end:

[Xdebug]
xdebug.remote_enable=on
xdebug.remote_port=9000
xdebug.remote_handler=dbgp
xdebug.remote_autostart=1
xdebug.remote_connect_back=1

Warning: this allows debugging access from any computer that connects to it. Use this only on your development image. If you want to limit debugging access to a specific computer, remove the line that refers to remote_connect_back and replace it with this:

xdebug.remote_host=YOUR.IP.ADDRESS.HERE

Eclipse and PDT

I downloaded the all-in-one PHP Development Toolkit (PDT) from http://www.eclipse.org/pdt/, unpacked it, and imported my project. After struggling with Javascript and HTML validation, I ended up disabling most of those warnings. Then I set up a debug configuration that used Xdebug and the server in the VM image, and voila! Line by line debugging with the ability to look in variables. Hooray!

2011-05-31 Tue 17:37

Drupal fixes: Modifying the entries in Calendar

The Drupal Calendar module is great. You can show view results in a decent-looking calendar easily. However, if you want to show more than what’s provided out of the box, you need to do a bit of undocumented and confusing hacking.

Let’s say that instead of displaying the items, you want to display the number of items. You need to implement hook_calendar_add_items, which, despite its name, actually overrides the items displayed. Your hook_calendar_add_items($view) should return an array of an array of items, which is also confusing, because the receiving function (template_preprocess_calendar in calendar/theme/theme.inc) doesn’t actually do anything with the outermost array, or indeed with multiple entries – it simply takes the last array and sets the items to it.

Each item should be something like this:

$item = new stdClass();
$item->node_title = t('Hello world');
$item->calendar_start_date = $date;
$item->calendar_end_date = $date;
$items[] = $item;

and at the end, you do this:

return array($items);

Your hook_calendar_add_items gets the $view, but not the existing items, so you’ll have to recalculate things using $view->result. Remember: if you return anything, your results will replace the items instead of being added to them.

Because hook_calendar_add_items doesn’t actually do what it advertises, there are probably bugs in this area of code, and this behaviour might change in future releases of Calendar. Be careful when using it. However, it seems to be the easiest way to change Calendar behavior. Caveat developer.

Also useful: http://drupal.org/node/412418

2010-12-24 Fri 10:19

Reflections on mentoring new developers in Drupal

UPDATE: Fixed formatting. Thanks, Brock!

Two developers recently joined our team. Johnny has worked with Drupal before, and needs a little help getting used to Drupal 6 and Views 2. Elena is an IT architect who is new to both IBM and Drupal. She needs a lot more help getting started, because she doesn’t know what things are called yet and she isn’t yet accustomed to the Drupal way of doing things. For my part, I work on Workflow, node access, and other requirements that require deep Drupal hacking.

I’m learning to check on Elena more frequently and to help her break down tasks. Otherwise, she might get lost or stuck, because she might not yet know where things are or whether she’s getting closer to an answer. I’ve made good progress on the things we’ve planned for this iteration, and I can invest the time into helping our new team members be more productive and learn more effectively.

Both Elena and Johnny have set up their debuggers in Eclipse, so they don’t have to figure out the right places to insert var_dumps. Instead, they can trace through the relevant pieces of code, learning more about the structures and the flow of Drupal websites along the way.

Although I occasionally struggle to explain things I take for granted, I enjoy helping someone who’s new to an area. It helps me remember the things people need to learn. For example, Elena’s work on surveys requires her to learn about nodes, getting values from the $_REQUEST, loading nodes, working with CCK, altering forms, adding new form fields using the Form API, and using Drupal functions for links and text. We broke down the task into the following steps:

  1. Create a CCK node type.
  2. Use hook_form_alter to add some text to the form.
  3. Load a node and fill the information in the form.
  4. Get the extra node ID from the URL.
  5. Adapt form_alter for the case where you’re editing the node.

We’ve managed our planning well, so I don’t feel overcommitted or stretched with the additional mentoring I’ve taken on. The time is an investment that will pay off both in the short-term as well as the long-term. If I can slow down and write more, then the investment can benefit to other people too.

I like this. It’s certainly much better than leaving developers to flounder and work things out on their own, and I learn a lot in the process of helping. Maybe that will be one of my specialties: projects where other people are learning a lot on the fly.

Using Simpletest and spreadsheets to populate Drupal with data

One of the challenges of testing views or custom Drupal code is generating the right kind of data. Devel can generate hundreds of random nodes, but you might need more custom data than that. For example, on our project, we need to have test users, their content profiles, and nodes that follow a certain node reference structure. By creating a class that extends DrupalWebTestCase and provides convenience functions on top of drupalCreateNode, we can easily create test data as part of our test cases. Copying the code from drupalCreateUser and making our own version that uses roles and content profiles helps us set up the right users, too.

We wanted our tests and changes to use the same database tables used by the web interface, so we overrode the setUp methods to use the actual database. This not only makes the tests faster, it also makes them more useful for the web testing and demos.

Many of our test cases create the data they need. However, some test cases need even more complex structures that are similar from one test to another. Instead of creating and recreating them on each test, I’ve written another test case for populating the data. For example, PopulateTestUsers sets up about 30 users with different characteristics. I can then write other tests that assume PopulateTestUsers has been run and the sample users and nodes are available.

How do we generate the users and nodes without getting tangled in lots of PHP? Here’s a technique I picked up from Stuart Robertson, an IT architect with lots of good ideas. He fills in a spreadsheet with the values he wants test data to have. He then uses other columns to generate PHP that set the individual fields, and another column that generates PHP based on the generated PHP. For example, a formula to set a CCK value might look like this:

=IF(B3<>"",CONCATENATE("'field_widget_doohickey' => array(array('value' => '",B3,"')),"),"")

which turns a value of “foo” in B3 to

'field_widget_doohickey' => array(array('value' => 'foo'))

which is then something you can pass to the node creation function. To figure out the syntax for other node attributes, use var_dump or your favourite debugging tool to look at a node that has the information you want.

You might have the final generation like this:

=CONCATENATE("$this->createWidget('",A3,"',array(",E3,"));")

where createWidget is a function you’ve defined to make things more readable. It would be a wrapper around drupalCreateNode that sets the type and does other things.

This spreadsheet makes it so much easier for us to work with our test data because we can refer to it to find test data matching criteria when designing our tests or trying things out using the web interface. Adding new test items is easy: just fill in the rows, copy the equations, and then copy the generated code and paste it into the test case.

Naming tip: Using a different naming convention makes it easy for me to use our custom-coded Drush testre command (run tests matching a regular expression) to run just the tests that populate data, or just the tests that assume data is there. Likewise, test cases that use SimpleTest’s web client ($this->get, etc.) have a different naming convention so that I can avoid running these slower tests when I just want to do a quick check.

Simpletest is a powerful tool. I’ve used it on every Drupal project I’ve worked on, even when I was the only one writing tests. Combined with a spreadsheet for generating structured test data, it’s a great help for development and demonstration, because we can set up or refresh complicated sets of users and nodes in little time. Well worth investing time to learn and use.

SCHEDULED: 2010-12-23 Thu 08:00