Drupal staging and deployment tips: It’s all code

As I talk to more and more developers about practices for working with Drupal, I get the idea that the staging and deployment process adopted by my team isn’t widespread.

Many developers make their changes directly through the web-based interface of a testing server, or even on the production site itself. I think that’s both tedious and scary. =)

Putting all of our behavior-related changes in update function in the install file makes it easier to merge changes and repeatedly test upgrades. Editorial changes (fixing typos, etc.) can happen on the site, but if it’s behavior-related code, it should be in the code repository. In moments of weakness, we’ve made web-based changes to our site, and we almost always regret those–either right away because things broke, or when we try to reconstruct our changes.

The Module Developer’s Guide documents how to write .install files for Drupal 5 and Drupal 6, but doesn’t go into much detail about what else you can put in the update functions. Maybe that’s why many developers use install files only for database-related changes. But you can do so much more: creating nodes, adding permissions, enabling other modules, and so on. 

Watch out for these potential pitfalls:

  • $user is set to the superuser (uid = 1) if you run update.php after logging in as the superuser, but if $access_check is set to FALSE in update.php, then $user will be null. That means that if you’re creating nodes or doing other things that check user_access or node_access, you should temporarily switch to the superuser.
  • If your update works if you apply them one at a time, but not if you apply all of them in one go, look for functions that cache information in static variables. You may need to modify the source code to add an argument that allows you to reset the cache or find another way to deal with the static variables. Static variables are a pain.
  • Another reason why batch updates may fail while incremental updates work has to do with the order that the update functions are called in. Modules are processed according to weight and alphabetical order, and all applicable update functions within one module are run before moving onto the next. If you have a set of related modules, put update functions that affect the related modules into the module with the heaviest weight.
  • .install files may get really long if you create _update_N functions for every change and you frequently deploy to a test server. You can refactor your update functions if other developers know that they should test the updates from a fresh copy of the production database instead of from incremental updates to their current system. Make sure you don’t add code below the update level on the production server.
  • Don’t forget to return an array containing results, or even just array().
  • Here are some general tips on how to find out the programmatic equivalent of a web-based action:

    • Find the form or form_submit function that processes the action and see if there are any API functions you can call to produce the same effect. If you can’t find an API function, consider writing one.
    • If you can’t write an API function, use the Macro module (part of the Devel module for Drupal 5) to record the form submission, and then use drupal_execute to run the recorded macro, OR
    • Directly manipulate the database, making sure to call any hooks necessary.

    And here are some examples of programmatically doing things:

    Setting a variable
    variable_set(‘yourvariable’, ‘yourvalue’);

    Enabling modules
    include_once(‘includes/install.inc’);
    module_rebuild_cache();
    drupal_install_modules(array(‘module1′, ‘module2′));

    Disabling modules
    db_query(“UPDATE {system} SET status=0 WHERE type=’module’ AND name=’%s'”, ‘modulename’);

    Creating nodes 
    global $user;
    $old_user = $user;
    $user = user_load(array(‘uid’ => 1));
    $session = session_save_session();
    $session_save_session(FALSE);
    // Do the work
    $node = new stdClass();
    $node->type = ‘page';
    $node->title = ‘Title';
    $node->body = ‘Body';
    node_save($node);
    // Restore the user
    $user = $old_user;
    session_save_session($session);

    Deleting nodes
    global $user;
    $old_user = $user;
    $user = user_load(array(‘uid’ => 1));
    $session = session_save_session();
    session_save_session(FALSE);
    // Do the work
    node_delete($nid);
    // Restore the user
    $user = $old_user;
    session_save_session($session);

    Updating the list of blocks
    global $theme_key;
    $theme_key = ‘yourtheme';
    _block_rehash();

    Convenience functions for working with permissions

    function _add_permissions($roles, $permissions) {
      $ret = array();
      foreach ($roles as $rid) {
        if (is_numeric($rid)) {
          $role = db_fetch_array(db_query("SELECT rid, name FROM {role} WHERE rid=%d",
    				      $rid));
        }
        else {
          $role = db_fetch_array(db_query("SELECT rid, name FROM {role} WHERE name='%s'",
    				      $rid));
        }
        $role_permissions =
          explode(', ',
    	      db_result(db_query('SELECT perm FROM {permission} WHERE rid=%d',
    				 $role['rid'])));
        $role_permissions = array_unique(array_merge($role_permissions, $permissions));
        db_query('DELETE FROM {permission} WHERE rid = %d', $role['rid']);
        db_query("INSERT INTO {permission} (rid, perm) VALUES (%d, '%s')",
    	     $role['rid'],
    	     implode(', ', $role_permissions));
        $ret[] = array('success' => true,
    		   'query' => "Added " . implode(', ', $permissions)
    		   . ' permissions for ' . $role['name']);
      }
      return $ret;
    }
    
    function _remove_permissions($roles, $permissions) {
      $ret = array();
      foreach ($roles as $rid) {
        if (is_numeric($rid)) {
          $role = db_fetch_array(db_query("SELECT rid, name FROM {role} WHERE rid=%d",
    				      $rid));
        }
        else {
          $role = db_fetch_array(db_query("SELECT rid, name FROM {role} WHERE name='%s'",
    				      $rid));
        }
        $role_permissions =
          explode(', ',
    	      db_result(db_query('SELECT perm FROM {permission} WHERE rid=%d',
    				 $role['rid'])));
        $role_permissions = array_diff($role_permissions, $permissions);
        db_query('DELETE FROM {permission} WHERE rid = %d', $role['rid']);
        db_query("INSERT INTO {permission} (rid, perm) VALUES (%d, '%s')",
    	     $role['rid'],
    	     implode(', ', $role_permissions));
        $ret[] = array('success' => true,
    		   'query' => "Removed " . implode(', ', $permissions)
    		   . ' permissions for ' . $role['name']);
      }
      return $ret;
    }
    

    Use update functions for all of your behavioral changes. Use source code control. Write regression tests. These practices won’t take all the challenges out of Drupal development, but they certainly make it less stressful–and more fun. =)

    2 Pingbacks/Trackbacks

    • http://geoffhankerson.com Geoff Hankerson

      Definitely going to give this a try so much better than clicking until you get carpal tunnel. Do you do the same for permission’s? If so would love to see a code snippet.

    • dereine

      I understand your arguments, but

      how can non php developer can use it.
      there are quite some people working with me, which cannot code.

    • Just Passing Through

      I think it’s great you have this down to a coding science. But I have to say, this kind of overhead just to push dev changes to production is unacceptable for a small to medium organization. I’m working very hard to grow drupal’s presence in my organization (mostly ms office and sharepoint based) and meeting with a lot of resistance of the type “open source is penny wise & pound foolish”. They already think that there is much more labor involved in an OS solution– if management ever found out this is the kind of effort required to push changes from dev to live they’d kill my drupal project in a second. For drupal to gain a foothold in the space I work in, something like http://drupal.org/project/deploy is a must I’m afraid.

    • http://sachachua.com Sacha Chua

      dereine: Heh, sorry. We can get away with using this approach because we’re all PHP-skilled developers who need to be able to do good quality tests. I suppose that’s part of the value we add as IBM! =)

      Just Passing Through: I’m also looking forward to other user-friendly modules for recording and pushing web-based administrative changes. Drupal’s web-based administrative interface is good, and I’m glad lots of people use it to develop sites that would’ve previously taken programming skill.

      That said, there’s a whole lot you can do when you can get into the source code. If you’re building a site that could work out of the box with Sharepoint, go ahead and do that. If you’re building a site that needs a bit more custom functionality, or you want to customize the way things behave beyond the limited parameters that closed source software often gives you, then you can save time and money with open source.

      Works for us.

      Geoff: Permissions? Sure. The key idea is to retrieve the perm value from {permissions}, and either add the new permission at the end or remove the permission you want to delete. I’ll update the post with a convenience function I use for permissions. =)

    • http://www.developmentseed.org adrian

      check out install_profile_api, which has a lot of convenience functions like these.

    • http://sachachua.com Sacha Chua

      Right, and I borrowed quite a number of ideas from it. (I don’t like its block function, though. =) ). If you use that, don’t forget to include_once or require_once the appropriate .inc file.

    • Just Passing Through

      @Sacha Chua

      You’re preaching to the choir here, lol. Personally I detest sharepoint and i’m on a mission to completely replace it with drupal but atm I’m a one person show and I can’t do my ‘real’ job, develop the drupal sites, do upgrades and maintenance (d5->d6 was a much larger time sink then I ever imagined), AND handle dev->prod migration with code. And i’ll never be able to make the case that having to code dev->prod migration is a value-add and not an avoidable waste of time.

      I’ve learned a lot since finding drupal, so I’ve solved the upgrade headaches with drush and cvs, but i can’t really push for drupal until the dev->prod migration issue has a viable solution that doesn’t require a highly skilled team of IBM programmers. ;-)

      Disclaimer: i used to wok for ibm. ;-)

    • Prajwala

      I fully agree with your point that, “Many developers make their changes directly through the web-based interface of a testing server, or even on the production site itself. I think that’s both tedious and scary. =)” .

      I used to work with django framework before working on drupal. In django there are more shell based commands to do the database changes and there are many ways to keep track of these behavior changes. So that while updating code on production server, update code and just run those commands to get the required behavior changes.

      I did not see that kind of things in drupal. When I say this to my PM they did not consider it as a problem. But I know that how it makes life easy to have that kind of environment.

      I started working on drupal from just 2 months and feeling happy to see this post and giving value to my point.

      I think install_profile module helps only at the first time while installing drupal on production instance.

      I have one question. How much this supports the updation of code between developers. Suppose 2 people working on project. If the 2 developers change the behavior then, they both have to update the behavior of the other person right.

      With the way you suggested, each individual person write update hook with some number. Then there will be more update hooks right?

      Some times if the person need to remove the previous behavior then also he need to write another update_hook. Do you think this good way.

      This is very good approach to update the production.

    • http://sachachua.com Sacha Chua

      Yes, writing update functions allows multiple developers to merge their changes–and even to find out whose change broke the system. ;) (Handy!) If you find that you need to undo a change, then you have a number of options. If the change has already been applied to the production system, you need to write a new update function. If the change hasn’t been applied to the production system, you can fix the old update function IF all the developers know that they need to build from a fresh copy of the production database.

    • Pingback: Artículos destacados de Abril 2009 | cambrico.net()

    • http://mc-kenna.com/ Damien McKenna

      This is a great article, Sacha. The first large Drupal site I built was done the old way – make changes in the UI, then to push changes to production hope you remembered all of your steps. With the newest site I’ve done I’ve changed to using a) install profiles, b) update scripts for everything, my core idea that everything included in the initial launch should be in an install profile, and changes post-launch should be in an update script. I’ve used install_profile_api to simplify some things, but am not completely satisfied with how it works – I’m starting to think building classes with save() functions might be better, especially when you are dealing with lots of inter-related data structures (taxonomies, nodes, module settings, etc).

    • Prajwala

      @Damien McKenna

      I am also thinking the same, tracking all these changes some how. Then we can save the time of writing update function separately and there is no chance of forgetting some change.

    • http://sachachua.com Sacha Chua

      Damien: Good for you! You might be interested in the Deploy module mentioned above: http://drupal.org/project/deploy

      Prajwala: You can record macros, but I find that I tend to make a lot of experimental changes that I don’t need to keep anyway, so for me, it’s been easier to just write the code as I go along. =)

    • Prajwala

      Sacha Chua: I like the deploy module. It helps a lot. I don’t know how to record macros, can you please provide some more info about macros.

    • http://sachachua.com Sacha Chua

      Sure! Download and install the Devel module (Macro if you’re using Drupal 6), then enable the Macro module that’s included in the package. See this videocast for a tutorial on how to use it. =)

    • Prajwala

      Thanks Sacha :)

    • ize

      Thanks for this very helpful post!!!

      Could you describe how to programatically manipulate the display settings for cck fields in a content type (in Drupal 6)? In other words, how can I use code in .install file to control the form at /admin/content/node-type/[foo]/display ?

      I’m thinking that as you suggest, I could use the macro module to record the form submission, and then use drupal_execute to run the recorded macro. But I can’t find documentation on the proper syntax, and I’m getting the following error:

      warning: call_user_func_array() [function.call-user-func-array]: First argument is expected to be a valid callback, ” was given in /…/includes/form.inc on line 366.

    • http://hci.org Dwight Aspinwall

      Thanks Sacha.

      There’s a type in your code above:

      $session_save_session(FALSE);

      Function should not be preceeded with ‘$’.

    • http://hci.org Dwight Aspinwall

      Hey Sacha, maybe you could explain the session saving code that wraps the node creation code above? This for me causes HTTP 500 error to be thrown. Pulling that code worked fine. Thx. -Dwight

    • http://twitter.com/dtkinzer David

      Great article. I found it quite informative!

    • Pingback: Drupal Features and Drush: updating our development workflow | sacha chua :: living an awesome life()