Integrating Drupal with Enterprise Back-Office Systems to Deliver a Best of Breed e-Commerce Sites

Join Drupal experts from IBM Global Services as they detail the process of defining business requirements, selecting the appropriate technologies, overcoming the technical enterprise integration challenges, and ultimately launching best of breed e-commerce sites with Drupal.

Key takeaways will include:
A review of the key business and technology decision criteria for selecting Drupal and Acquia
An example of a 3 tiered architecture designed to deliver end-end automation and a flexible platform for future innovation
Tips on how to integrate front-end and back-end systems including Drupal, Websphere, DB2, and ERP system for launching a best of breed e- commerce solution.
A discussion of which Drupal modules and Acquia services are recommended.
Best practices and technologies for building a development and test environment to optimize parallel development and ensure security

This Webinar is designed for those who want to learn more about best practices for designing an enterprise architecture with Drupal at the core and integrating Drupal with other enterprise systems.

http://acquia.com/resources/webinars/integrating-drupal-enterprise-back-office-systems-deliver-best-breed-e-commerce-s

Enjoy!

(Update: Thanks for catching that!)

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Free Drupal-IBM-Acquia webinar: Integrating Drupal with Enterprise Back-office Systems – Thu Jan 21 2010 1PM EST

As a Drupal developer who really hates working with cross-browser fiddliness, I think it’s almost always a good idea to have someone else do the HTML and CSS theming. In an ideal world, that person would also create the Drupal theme and whatever Javascript behaviors are desired, but it seems hard to find people who are comfortable with both front-end and back-end development. A reasonable compromise is to have someone else translate the visual design into cross-browser HTML and CSS for the splash page, an inside page (anything representative of the main layout), and other layouts. What takes me four hours of teeth-gnashing may take an experienced designer just one, and he or she might even be happier doing it. So go ahead and optimize for developer productivity and happiness.

Another benefit of splitting up the work is that you can work in parallel. While you work on the technical bones of the application, the designer can get the theme just right. The earlier in the project that you know about complex parts that could be problematic, the better, and this is true for both theme and function. You’ll still need to integrate the theme at the end, so budget at least a few days for that.

So here’s what the workflow might look like:

1. Developer installs modules and configures them to work together.
   Designer prepares basic HTML and CSS.
2. Developer fleshes out site and adds custom functionality.
   Designer fleshes out individual forms/pages requiring special treatment.
3. Developer ports HTML and CSS theme to Drupal.
4. Developer and designer collaborate on fixing any cross-browser problems that show up.

Build the functional parts of the application before worrying about the theme. Problems in the theme layer can mask problems in the functional layer. If you build the theme first, then build the features, tracking down bugs might take more time.

It’s usually a good idea to build the hard parts first, so you can get a better sense of how much more effort is needed and whether you need to scale things down. If you’re new to Drupal, you may want to build a few easy parts first so that you can familiarize yourself with the system.

Hope that helps!

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Drupal workflow for small projects

This is definitely worth the subway ride. I create more than CAD $4.50 of value and save people more than 60 minutes of time total. Besides, I usually get to sit down and write a blog post during the commute.

Some things from today:

• Want to protect files from being downloaded? Set your Drupal file system handling to Private, manage your view uploaded files permission, and don’t forget to add an .htaccess in your files directory with the following:
  SetHandler Drupal_Security_Do_Not_Remove_See_SA_2006_006

  Options None

  Options +FollowSymLinks

• Use hook_form_alter to print out the form_id if you want to change where a form redirects. After you figure out the form_id, set the appropriate form’s #redirect property.
• var_dump may look different if you don’t have xdebug installed. Check the HTML source for your expected output.

Lots of other HTML- and Drupal-related things, too.

Good leverage on time and experience. =D I wonder how to get even better at this…

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

IBM Drupal Drop-in Clinic

• I learned how to write low-level design documentation. I used to hate doing this because I was much happier prototyping things in order to figure things out, and writing in a word processor instead of a code editor seemed slow. Also, laying out images was a pain in both OpenOffice.org and Microsoft Word. The key breakthrough was that after a lot of iterations, I finally got to a document that had the level of detail my project manager wanted. Once I knew what he wanted, it was easy to write documents like that. In fact, I did one better–I filled in lots of details in the validation section of the design document, and I used that for test-driven development.
• I smoothed out more of the kinks in our deployment process. Always testing our update scripts with update.php’s access check set to false helped flush out assumptions about user ID. Our regression tests came in handy, too.
• I practiced more test-driven development. And I loved it! =) Yay yay yay yay.
• I learned JQuery and the ins and outs of Views+Calendar. We customized our calendar implementation extensively, and it doesn’t look half-bad. Working with timezones had given me a lot of pain, though, but I managed to address all of those issues (and I have the regression tests to prove it! ).
• I learned that I should’ve gotten a second computer a long time ago. Having a dedicated development computer and another computer I can leave on Microsoft Windows means less hassle and less stress when it comes to working with other Windows-centric people and tools.
• I learned that I have a lot of things I can teach people about Drupal. =) Mentoring is fun!

I’m moving to strategy-focused projects next quarter, but if I were to continue on this project, I’d probably:

• Learn more about optimization (started doing a little bit of this, speeding up a page by an order of magnitude)
• Write more validation tests in my design documents
• Figure out how to check that coding standards are consistently applied
• Find some way to convince folks to go for Drupal 6, or find another project that uses Drupal 6.

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Lessons learned from this phase of our Drupal project

I needed to look into this because my update function was just taking so much time and memory. We have a little over 3,000 nodes, most of which are usernodes automatically created by the usernode module. I needed to rebuild the node access entries for a few hundred content nodes to take into account the new partner access module I wrote, but node_access_rebuild took quite a while.

The first approach I thought of taking was to delete usernode because we didn’t seem to need it any more. In order to do that, though, I would still have to delete all the user nodes, though, and invoking all the hooks for each of the 2000+ user nodes took a while as well.

The second approach was to delete just the node access records related to the nodes I needed to update. I found a way to do it even faster – instead of querying the node IDs and then deleting individual rows, I could do something like this instead:

db_query("DELETE FROM na USING {node} n
          INNER JOIN {node_access} na ON (n.nid=na.nid) WHERE n.type='%s'", $type);
$rs = db_query("SELECT nid FROM {node} WHERE type='%s'", $type);
while ($row = db_fetch_array($rs)) {
  $node = node_load($row['nid'], NULL, TRUE);
  if (!empty($node)) {
    node_access_acquire_grants($node);
  }
}

(UPDATE: Fixed MySQL statement)

That was interesting. =)

I was also running into memory problems trying to run all the unit tests in one go. I broke it into individual tests with:

drush test list | grep KEYWORD | xargs -n 1 drush test run | tee out

where KEYWORD is the keyword I use to filter the list for just the tests I wanted to run, and out is the file that stores a copy of the results. It’s not neatly summarized, but oh well. =) It works.

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Drupal in the trenches: MySQL and DELETE joins; multiple tests

I offered to get things started. The senior architect asked me how much time I thought it would take. “Two hours,” I said, which was the first number that came to mind.

After lunch, I headed to the senior architect’s desk with my laptop and wireless mouse. I thought about asking him to change his password to something I could easily type, just in case I needed to start multiple sessions. Then I realized a much better way to do it would be to use my Emacs environment, which is already set up for doing really cool things with Drupal. So I switched my keyboard layout to QWERTY, used ssh-copy-id to copy my authentication ID to the server, and then opened the directory in Emacs using the location /ssh:user@host:/usr/share/drupal6.

Emacs worked like a charm. I edited files on the server as easily as those on my own computer, with all the syntax highlighting and keyboard shortcuts I’d gotten used to. I split windows, moved windows around, copied and pasted regions, and even did a little autocompleting.

I think I made the senior architect’s jaw drop.

I finished almost all the basic theming (minus a few quirky CSS things) in one hour and fifty minutes, ten minutes less than my thumb-in-the-air estimate. The senior architect said it would’ve probably taken him 16 hours over the weekend.

While we were chatting about the changes he’d need to make and the other things he could learn, the senior architect asked me if I played any games. I told him that I play one computer game–Nethack (an old text-based roleplaying game)–and I only play it in airports. I pointed to my laptop and said, “This is my game.” Programming has its own major challenges and minor opponents, it has progress, it has points, it has that adrenaline rush of trial and triumph. Programming is my game. Life is my game.

And it’s tons of fun. =)

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Drupal from the trenches: This is my game

Then my project manager said, “If I go to the event details page from the month view and hit back, it takes me to the day view instead.”

I said, “Welcome to the wonderful world of AJAX. This might be tough to fix.”

Making the back button work in AJAX applications requires a lot of effort. It doesn’t look like people have a nice and clean solution for it yet, although there are a number of libraries that try to address the situation.

Following the principle of progressive enhancement, I had built all the plain HTML functionality first, then layered the Javascript on top of it using jQuery callbacks. In addition to ensuring that the site still works even if Javascript is disabled, this approach also helps make sure that I have proper URLs for almost all the webpages involved. (I didn’t bother with explicitly transient pages like the year navigator or the day pop-up.)

I started with this Hijax-based approach, because it had the most documentation. I had problems getting it to behave, though, because my AJAX pages have other AJAX links that fail with the history-remote plugin. The history_remote plugin works by replacing all the links with the current page and a code (#remote-1, for example). When the back button is pressed, the library looks for the appropriate link and triggers the click event. This breaks down when the link isn’t actually on the first page. For example, when a user switches from a week to a month view, then goes to the next month, the plugin can’t find the link to the next month on the week view’s page, which is where the user started.

What I really needed to do is encode more information in the URL. If I encode information in the anchor portion of the URL (#…), I can use that to request the page and put that into the appropriate div. For example, if I pass in #transitions_content?new_path=connect/calendar/2009/05 , I might be able to parse that and put the content of new_path into the transitions_content div.

I started going down that rabbit-hole, and then I got myself thoroughly confused, so I decided that the best way would be to just rip out the major AJAX navigation and go back to the simple stuff (which fortunately still works).

Gah. Brain hurts!

Does anyone have a clean way to do this?

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Drupal in the trenches: AJAX history makes my brain hurt

A long list of cities can be hard to work with, though. This list is equally long, but it’s organized by GMT offset, which people are also likely to know.

Index: date_api.module
===================================================================
--- date_api.module	(revision 2404)
+++ date_api.module	(working copy)
@@ -490,6 +490,14 @@
           }
         }
       }
+
+      // Now reformat the zonenames so that they're of the form (GMT+0800) Asia/Manila
+      foreach ($zonenames as $name => $zone) {
+        $x = date_make_date('now', $name);
+        $list[$name] = '(GMT' . date_format($x, 'P') . ') ' . str_replace(' ', '_', $zone);
+      }
+      asort($list);
+      $zonenames = $list;
       if (!empty($zonenames)) {
         cache_set('date_timezone_identifiers_list', 'cache', serialize($zonenames));
       }

There’s probably a much more efficient way to do this, but hey, it works.

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Drupal: Timezones and places

I started breaking down the SQL query to find out which joined table didn’t have the data that was supposed to be there. At first, I thought it might be that the publication dates or expiration dates were incorrect, but that wasn’t it. All of the nodes were set to ‘published’ (status = 1), and all remained correctly associated with the domain through the Domain module.

I checked the search_fuzzy_index table. Hmm. The table was smaller than it should be. When I queried it for a node that should’ve been there, I got zero rows.

I tried forcing the system to reindex the node. Still nothing.

I printed out the SQL statements and ran them myself. ERROR 1062 (23000): Duplicate entry '2147483647' for key 1.

Wha?

That looked like a suspiciously large number. I popped it into Google and found a lot of people with the same problem. Turns out that the autoincrement sequence fuzzysearch was using had jumped all the way to the end, for some reason or another (we couldn’t have been reindexing _that_ much!). I changed the column to a BIGINT, reindexed all the nodes using Drush commands like:

drush eval "\$rs=db_query(\"select nid from node where type='transitions_job'\");while(\$nid=db_fetch_array(\$rs)){fuzzysearch_reindex(\$nid['nid'], 'fuzzysearch');} fuzzysearch_cron();"

(using my custom eval command),

That worked. I reported the results to my team members in our group chat.

The IT architect said we were still having disk space issues. We’d removed all the unnecessary files and stale backups already, and I didn’t think there was much more that we could trim. I used du --max-depth=2 /var | sort -n to look at the disk usage in the /var tree, which was where we were having those problems. The three biggest directories were /var/www, /var/lib/mysql, and /var/spool/mail, and we’d already scrubbed as much as we could.

I used ls -lSr to look at the contents of /var/lib/mysql, and noticed that our watchdog table was 2.3 GBs. Gah! Turned out that Drupal had logged each of the PHP errors raised when we were trying to fix the previous problem. We didn’t want to delete all the logs, so we just deleted the logs of type ‘php’. After the IT architect ran DELETE FROM watchdog WHERE type='php'" in the MySQL client, though, we still didn't have free space.

I guessed that the lack of free space could be solved by compacting the MySQL database, which I did with OPTIMIZE TABLE watchdog. That solved it!

That was an interesting day. =)

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Drupal from the trenches: Database glitches

• hiding the year view
• creating a context-sensitive year navigator that displays the entire year, and linking that to the title of the image
• adding AJAX effects
• adding a pop-up callout with three of the day’s events
• displaying times in the user’s timezone generally, and in the user’s timezone and the event’s timezone on the actual event page
• including dates for the previous month in the month view
• allowing people to subscribe for notifications for new events in their interest group
• allowing people to get iCal feeds for their events, all the events, or a group’s events, and these feeds should work without login
• allowing people to sign up for e-mail reminders

We went with Date+Calendar instead of Event because Date+Calendar seemed more up to date, and its integration with Views meant that it was easy to add in domain access and other constraints. I learned quite a lot of new things in the process of implementing these features, such as:

• writing a lot of Javascript using jQuery in order to bind events and do AJAX calls
• writing test cases to check event subscription, event notification, timezone handling, and so on
• programmatically creating a CCK node type with a date field
• overriding calendar.inc to modify the way Drupal prepares the calendar
• overriding calendar.theme to modify the way Drupal presents the calendar
• making my own set of functions to generate the year navigator, based on the year view in calendar.inc

It took me a bit of time to figure out how this Date+Calendar AJAX patch worked, and I ended up modifying it extensively. I had been getting confused by mini= and view=ajax and all the other parameters floating around. I tried different approaches, including creating a callback function that generated just the HTML for the block, but then I found myself passing in too many parameters to control the URLs for the links.

My aha! moment was when I realized that the way the patch was handling the AJAX was to generate the entire page. When it got to processing the calendar block in the sidebar, the code checked for the $_GET['view'] parameter, and if an AJAX view was requested, it would print out the block and exit without printing the rest of the page. While that worked for the general case, we needed to modify our code so that the calendar blocks don’t appear on the calendar detail pages, so I wrote some Javascript code that requested a page within the right context.

This approach of generating the whole page didn’t quite work when it came to the subscription form that we embedded in event node page templates, though, because it printed out the node content before it generated the form. I used jQuery to retrieve the entire page, and then I extracted just the DIV I wanted to keep.

I still don’t like fussing with CSS (particularly when it comes to collapsing borders or dealing with browser issues), so I’ll leave that in the capable hands of our information architect. But now I’m the jQuery ninja on our team, too, and I know I can rock CCK+Views and calendars for future projects. =D

(p.s. Left out details, but if you’re curious about any of the bullet points, comment and I might flesh it out into its own blog post!)

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Drupal: I’m learning how to be a JQuery/Date+Calendar ninja! =)

Many developers make their changes directly through the web-based interface of a testing server, or even on the production site itself. I think that’s both tedious and scary. =)

Putting all of our behavior-related changes in update function in the install file makes it easier to merge changes and repeatedly test upgrades. Editorial changes (fixing typos, etc.) can happen on the site, but if it’s behavior-related code, it should be in the code repository. In moments of weakness, we’ve made web-based changes to our site, and we almost always regret those–either right away because things broke, or when we try to reconstruct our changes.

The Module Developer’s Guide documents how to write .install files for Drupal 5 and Drupal 6, but doesn’t go into much detail about what else you can put in the update functions. Maybe that’s why many developers use install files only for database-related changes. But you can do so much more: creating nodes, adding permissions, enabling other modules, and so on. 

Watch out for these potential pitfalls:

• $user is set to the superuser (uid = 1) if you run update.php after logging in as the superuser, but if $access_check is set to FALSE in update.php, then $user will be null. That means that if you’re creating nodes or doing other things that check user_access or node_access, you should temporarily switch to the superuser.

Here are some general tips on how to find out the programmatic equivalent of a web-based action:

• Find the form or form_submit function that processes the action and see if there are any API functions you can call to produce the same effect. If you can’t find an API function, consider writing one.
• If you can’t write an API function, use the Macro module (part of the Devel module for Drupal 5) to record the form submission, and then use drupal_execute to run the recorded macro, OR
• Directly manipulate the database, making sure to call any hooks necessary.

And here are some examples of programmatically doing things:

Setting a variable
variable_set(‘yourvariable’, ‘yourvalue’);

Enabling modules
include_once(‘includes/install.inc’);
module_rebuild_cache();
drupal_install_modules(array(‘module1′, ‘module2′));

Disabling modules
db_query(“UPDATE {system} SET status=0 WHERE type=’module’ AND name=’%s’”, ‘modulename’);

Creating nodes 
global $user;
$old_user = $user;
$user = user_load(array(‘uid’ => 1));
$session = session_save_session();
$session_save_session(FALSE);
// Do the work
$node = new stdClass();
$node->type = ‘page’;
$node->title = ‘Title’;
$node->body = ‘Body’;
node_save($node);
// Restore the user
$user = $old_user;
session_save_session($session);

Deleting nodes
global $user;
$old_user = $user;
$user = user_load(array(‘uid’ => 1));
$session = session_save_session();
session_save_session(FALSE);
// Do the work
node_delete($nid);
// Restore the user
$user = $old_user;
session_save_session($session);

Updating the list of blocks
global $theme_key;
$theme_key = ‘yourtheme’;
_block_rehash();

Convenience functions for working with permissions

function _add_permissions($roles, $permissions) {
  $ret = array();
  foreach ($roles as $rid) {
    if (is_numeric($rid)) {
      $role = db_fetch_array(db_query("SELECT rid, name FROM {role} WHERE rid=%d",
				      $rid));
    }
    else {
      $role = db_fetch_array(db_query("SELECT rid, name FROM {role} WHERE name='%s'",
				      $rid));
    }
    $role_permissions =
      explode(', ',
	      db_result(db_query('SELECT perm FROM {permission} WHERE rid=%d',
				 $role['rid'])));
    $role_permissions = array_unique(array_merge($role_permissions, $permissions));
    db_query('DELETE FROM {permission} WHERE rid = %d', $role['rid']);
    db_query("INSERT INTO {permission} (rid, perm) VALUES (%d, '%s')",
	     $role['rid'],
	     implode(', ', $role_permissions));
    $ret[] = array('success' => true,
		   'query' => "Added " . implode(', ', $permissions)
		   . ' permissions for ' . $role['name']);
  }
  return $ret;
}

function _remove_permissions($roles, $permissions) {
  $ret = array();
  foreach ($roles as $rid) {
    if (is_numeric($rid)) {
      $role = db_fetch_array(db_query("SELECT rid, name FROM {role} WHERE rid=%d",
				      $rid));
    }
    else {
      $role = db_fetch_array(db_query("SELECT rid, name FROM {role} WHERE name='%s'",
				      $rid));
    }
    $role_permissions =
      explode(', ',
	      db_result(db_query('SELECT perm FROM {permission} WHERE rid=%d',
				 $role['rid'])));
    $role_permissions = array_diff($role_permissions, $permissions);
    db_query('DELETE FROM {permission} WHERE rid = %d', $role['rid']);
    db_query("INSERT INTO {permission} (rid, perm) VALUES (%d, '%s')",
	     $role['rid'],
	     implode(', ', $role_permissions));
    $ret[] = array('success' => true,
		   'query' => "Removed " . implode(', ', $permissions)
		   . ' permissions for ' . $role['name']);
  }
  return $ret;
}

Use update functions for all of your behavioral changes. Use source code control. Write regression tests. These practices won’t take all the challenges out of Drupal development, but they certainly make it less stressful–and more fun. =)

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Drupal staging and deployment tips: It’s all code

This would be a nightmare to do if we had made all of our changes on the Web. Fortunately, most of our changes are in .install files. The changes had gotten out of sync with the installation profile we created near the beginning of the project, but it was relatively easy to work through the updates and incorporate them in the profile. The regression tests I wrote are proving to be really useful, too.

There are still a number of variables that we set through the web-based interface (blame moments of weakness!), so after we clean up the profile, we’ll compare some of production tables against the tables produced by our from-scratch installation.

I applied the lots-of-little-inc-files-that-automatically-get-included-an-run pattern from our last project, so now everything is neatly compartmentalized. I improved that pattern to add weights, so now I can make sure that some inc files are run after others.

Which brings me to thinking about the proper way to backport changes from update functions back into the install functions or to the installation profile. _install functions should reflect the latest versions of the database schema, but some code (node creation, etc.) logically feels more like a part of the installation profile’s final steps. My current approach is to use the _install function for code that doesn’t require other modules to be correctly configured, and to use my custom _install_final method for the finishing touches. I’ve also broken site-specific code out into separate .inc files so that I can include any of them as needed, and I’m using regression tests to check that things are okay. Command-line drush and my Makefile make it easy to switch between the two virtual hosts.

Most of my tests are still fail, but some of them pass with from-scratch installs. Hooray! =D

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Software reconstruction

I customize it for different testing or production environments with .mk files (ex: qa.mk) that contain the variables from the top part of the Makefile. These files are automatically included in this Makefile. Benefits: I don’t have to commit my database password to the source code tree, and I don’t have to think about which environment I’m in.

I use “make cycle” and “make mysql” a lot. “make cycle” depends on Drush being set up properly, and patched to allow you to update all the modules from the command-line.

I think this works for the Drush update command:

function drush_tools_update($command = '') {
  global $user;
  ob_start();
  require_once 'includes/install.inc';
  include_once('update.php');
  $ret = ob_get_contents();
  drupal_load_updates();
  ob_end_clean();
  $user = user_load(array('uid' => 1));
  $list = module_list();
  $update_list = array();
  foreach ($list as $module) {
    $updates = drupal_get_schema_versions($module);
    if ($updates !== FALSE) {
      $latest = 0;
      $base = drupal_get_installed_schema_version($module);
      foreach ($updates as $update) {
        if ($update > $base) {
          if ($update > $latest) { $latest = $update; }
          $update_list[$module][] = $update;
        }
      }
      if ($latest) {
        sort($update_list[$module]);
        printf("%-30s %5d -> %5d (%s)\n", $module, $base, $latest, join(', ', $update_list[$module]));
      } else {
        printf("%-30s %5d\n", $module, $base);
      }
    }
  }
  if (count($update_list) == 0) return;
  if ($command != 'force' && !drush_confirm(t('Do you really want to continue?'))) {
    drush_die('Aborting.');
  }
  ob_start();
  foreach ($update_list as $module => $versions) {
    foreach ($versions as $v) {
      print "Running " . $module . "_update_" . $v . "\n";
      update_data($module, $v);
    }
  }
  $updates = ob_get_contents();
  cache_clear_all('*', 'cache', TRUE);
  cache_clear_all('*', 'cache_page', TRUE);
  cache_clear_all('*', 'cache_menu', TRUE);
  cache_clear_all('*', 'cache_filter', TRUE);
  drupal_clear_css_cache();
  ob_end_clean();
  print $updates;
  $output = '';

  print $updates;
  if (!empty($_SESSION['update_results'])) {
    $output .= "The following queries were executed:\n";
    foreach ($_SESSION['update_results'] as $module => $updates) {
      $output .= "\n" . $module . "\n--------------------------\n";
      foreach ($updates as $number => $queries) {
        $output .= 'Update #'. $number . ":\n";
        foreach ($queries as $query) {
          if ($query['success']) {
            $output .= "SUCCESS: " . $query['query'] . "\n";
          }
          else {
            $output .= "FAILURE: " . $query['query'] . "\n";
          }
        }
        if (!count($queries)) {
          $output .= "No queries\n";
        }
      }
    }
    $output .= "\n";
    print $output;
    unset($_SESSION['update_results']);
  }
}

(and you’ll need to define it in drush_tools_drush_command also).

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

My Drupal Makefile

Clearly, we can get better at managing the crunch time around deployment.

The last time we deployed, there were a few tense moments, but our rigorous test-everything-from-a-production-install process helped us do it smoothly. This time, not so much. Here are a few reasons why, and here’s what I can do to make things better.

• I had set $access_check to FALSE because I wasn’t sure if we could get in to update the system. The IT architect logged in as a super administrator and ran upgrade.php. However, since $access_check was FALSE, it apparently didn’t check at all if the user was logged in as a super administrator, and so we ran into bugs that assumed account 1 was running the update (related to node saving). Symptom: The updates ran, but some updates didn’t get fully applied. We only detected this the day after (the perils of doing an evening deployment when you’re tired). I thought that just reloading the database backup and reapplying the changes (properly, this time!) would’ve been a cleaner way to do it, but my other team members voted for manually fixing things. So that was stressful.

  The problem occurred a couple of times during QA testing, which is how I realized that update.php was misbehaving. I wrote about it, but I didn’t review the other developers’ code for potential issues, and I didn’t emphasize the potential pitfalls during our meeting.

  To do this better next time, we can come up with a more formal and regular code review process, and I can communicate more explicitly. We could try to always run update.php with $access_check = TRUE, but it may need to be false in some case in the future, and it’s better to be aware of potential problems.

• After we deployed, we found out that a subdomain we were using hadn’t been set up in DNS. We were no longer in control of the domain record because we had turned that over to the nonprofit partner who was supposed to be managing the site.

  To do this better next time, we should make sure our QA and production setups are as close as possible (we had been using wildcards for QA), and we should test new domains.

• I had been in crunch mode for 10 days (since the weekend before). It’s difficult to maintain sprint-like energy and focus for that long, and I was feeling physically fatigued after I stayed up relatively late to finish the deployment.

  To do this better next time, I need to insist on taking breaks, even if it doesn’t seem to be being much like a team player. Also, I need to reset my sleep cycle as quickly as possible.

During deployment, I also learned to:

• Give people feedback and send them patches instead of just fixing the code for them. I don’t get fazed when code changes underneath me. I’ve worked with too much open source, I guess. I just try to figure out what changed, why, and how to work with the new structure. Other people can feel alienated from their code, though, and they lose that feeling of ownership. Better to hand things over to other people, perhaps with a few tips, even if it means it won’t be finished as quickly.
• Communicate changes more often and more explicitly. I liked having a Sametime group chat running. I don’t like sending lots of e-mail, and having the chat made it easier for me to keep others in the loop.
• Make sure tests are up to date, and run them regularly. There were a few bugs I missed because I hadn’t run the test suite, and I hadn’t run it because it takes a lot of time on my system. I should make the time to do that (using it as break time if necessary), and I can also set up a testing environment so that other people can run the tests easily. Speaking of that – I spent nearly a day tracking down failures due to other people’s changes because they didn’t verify their work against our test suite. I need to figure out how to build more common ownership of our test suite, and how to get them to run the tests themselves. The SimpleTest web interface is okay, but it’s still not as convenient as Drush. Maybe a line item in our administration interface… Hmm… Next time, I could also set up regular tests that e-mail us the results.
• Build little tools to help. Instead of analyzing the source code by hand in order to come up with the number of lines we changed (needed this for IBM Legal), I wrote a tool that analyzed our source code based on the Subversion history. It was pretty cool. It took me about 30 minutes to write, and we ended up running it twice. I expect it would’ve taken us three hours to do that all by hand. Yay! =)

So my key things for next time are:

• Make sure developers know about the gotchas we encountered.
• Set up an automated test environment and make sure other developers take ownership of the results.
• Keep a group chat running. I participate in that quite a lot. E-mail, not so much.
• Take more breaks.

Becoming a better developer, one step at a time…

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Lessons learned from this phase of our Drupal project

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Haciendo que tu entorno de desarrollo de Drupal rocks

Problem: Blocks based on views with dependencies on custom tables sometimes don’t show up when we refreshed from the database dump, although things work if we update the database one revision at a time.

After far too much pain and suffering, I figured out that _views_get_default_views was caching the results. When it checked our newly-enabled modules for default views, it found some views whose dependencies hadn’t gotten enabled yet, so it didn’t cache those. The next time _views_get_default_views was called, it used the values it had stored in a static variable available only in that function.

_views_get_default_views did not provide a way to reset that static variable, so once it was called, the data was practically written in stone.

_views_get_tables is similarly evil. (ARGH!)

This might’ve been fixed in a Views update (and we’re still using Drupal 5, in any case), but I’m not going to suggest updating the module this close to external acceptance testing.

So our options were:

• Make sure _block_rehash and similar functions never get called before the update function that enables the module. Problems: Not only do we need to untangle the order of update functions that get called and be hyper-aware of what other modules do, but we’ll also need to change old update functions if we ever need to do this for another view in the future.
• Hack _views_get_default_views and _views_get_tables to accept an optional parameter that resets the static variable.

When there are no clean alternatives, you just gotta get your fingers dirty and hack code.

Was that really only one hour of my life? It felt so much longer.

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Drupal in the Trenches: Fighting with Views

I learned this the hard way when my blocks suddenly started failing. I jumped back to the last known-good revision, and then stepped forward. All the update functions ran okay, one after the other. When I ran all the update functions after a database refresh, though, things failed. It turned out that another developer had changed the title of a block to a blank string, but left a db_query(“DELETE FROM {blocks} WHERE title=’%s’, $title”); in the function. And since it was called fairly early in our update cycle, it didn’t turn up during the incremental updates from my known-good database.

That took me about 45 minutes to find and fix. Thank goodness for tests and version control!

LESSON: Never ever ever use anything less than module AND delta to specify blocks.

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Drupal gotchas: Never ever ever use anything less than module AND delta to specify blocks

To enable this only on your testing server, add the following to that domain’s settings.php:

$conf['smtp_library'] = drupal_get_filename('module', 'devel');

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Drupal tip: Test mail sending with Devel

To fix this, temporarily assume the identity of the superuser in the update functions that need it:

global $user;
$old_user = $user;
$user = user_load(array('uid' => 1));
$session = session_save_session();
session_save_session(FALSE);

and then restore the old user afterwards:

$user = $old_user;
session_save_session($session);

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Drupal Gotcha: Watch out for $user during update.php

Building APIs that Rock
By: Jeff Eaton
Links: Slides, Video
Talked about the importance of making it easier for other modules to use your code (not just users through the Web interface); showed an example of using hooks for ultimate flexibility. Key takeaway: Try building your module as an API, then layer a user interface on top of it (ex: Views). Also, puppets!

Keynote: The State of Drupal
By: Dries Buytaert
Links:  Video , Text Outline
Drupal community, code growing exponentially. Next steps: Connecting data

Totally Rocking Your Development Environment
By:  Sacha Chua
Links:  Video, Installation Profile
I had a lot of fun learning from people’s tips, too!

Handling Asynchronous Data with Drupal
By: Josh Koenig
Links:  Video, PDF Presentation
Key takeaway: Use Drupal.behaviors to attach contextualized Javascript code. Also, you can write data to files in order to make polling more efficient.

Advanced Theming Techniques
By: Trevor Twining
Links: Video, Slideshow
Most of the presentation was about how to define subthemes and extend something like Zen.

Business Analytics with Views
By: Irakli Nadareishvili
Links: Video, Slideshow
Ooh, pretty charts. Key takeaway: look into charts and views_charts modules for integration with Google Charts and other charting engines

Boosting Our Raw Capacity to Provide Drupal Training
By: Sean Effel
Links: Training Models, Notes from Audience
See my notes at http://sachachua.com/wp/2009/03/05/drupalcon-day-1-notes-and-links-from-march-4-2009/ . Key takeaway: Core training has lots of common ground, but then need to customize training for skill level / needs; group clinics/workshops handy

Building Infrastructure You Can Scale, Monitor and Maintain
By: David Strauss
Links: Video
Great slide breaking down flow of traffic to rough percentages. Key point: adding more components is easy (content delivery network, reverse proxy, etc.), but adding more than one component (ex: multiple database servers) is harder. Coherency issues, replication issues, etc. Try to minimize dynamic pages.

When Efficiency and Manageability Matter, Drupal at Scale,
By: Scott Mattoon
Links: Slideshow, Video
Left this session after a short while, as it seemed to be mostly about Sun tools.

Powering Collaboration in a Distributed Enterprise
By: Dan Karran
Links: Video

Interesting demo of a featureful Drupal site used on the intranet. Check out their auto-saved drafts.

Drupal Patterns: Managing and Automating Site Configurations
By: Chris Bryant
Links: Slideshow, Video
Patterns allow you to bring in groups of functionality. Interesting: can publish and share patterns. No support for change management yet.

Staging and Deployment – A Panel Discussion
By: Greg Dunlap
Links: Video
Deploy and db_scripts look interesting. Also, I promised to upload my .install file snippets…

Why I Hate Drupal
By: James Walker
Links: Video
Good stuff. Interesting contrast to kumbaya keynote: contrast of Drupal’s growth with Sharepoint, Wordpress, and Joomla.

Advanced Drupal Security
By: Neil Drumm
Links: Video
Went through key parts of Drupal security handbook. My takeaway: use session_save_session(FALSE); when changing global $user;

Selling Drupal Services
By: Neil Giarratana
Links: Video, Slideshow
Lots of tips about the business side of it. Key takeaway: RFP process is inefficient; try partnering up with agencies instead, so you can build on relationships

Project Management For Fun and Profit
By: Crystal Williams
Links: Video
Not much new here if you’ve managed or worked on projects before

How do Drupal, Joomla! and WordPress Stack Up?
By: Amy Stephen
Links: Video
Didn’t go into an in-depth technical comparison / benchmarking

Token: The Little API That Could
By: Greg Knaddison
Links: Video
Walked through how Token module works, how to implement your own tokens

Sessions I wish I also attended

JQuery, Dmitri Gaskin Links: Video
Drupal and the Geospatial Web, Jeff Miccolis Links: Video, Notes from Audience
Optimizing your LAMP stack for Drupal, Eric Mandel Links: None
Learning jQuery UI, Richard Worth Links: Video
SEO & Drupal: Search Engine Optimization Tips, Tricks and Best Practices, Gregory Heller Links: PDF Slideshow, Video
Building advanced social networks at a large US University, Kyle Mathews Links: Slideshow, Video
Building a Frankenstein monster and how to maintain it, mortendk Links: Video
Front End Performance – Make Your Website Lightning Fast, Konstantin Käfer Links: Video
Communicating Data Online: Data Visualizations and Open Data, Eric Gundersen Links: Video
Project Flow and Tracker: From business objects and user stories to test-driven Drupal based website application, Victor Kane Links: PDF within page
Scaling Drupal using Amazon Web Services (AWS), Frank Febrarro Links: Video , Slideshow
Drupal Process Management, Drew Gorton Links: Video
The Business of Open Source, Liza Kindred Links: Video
Quality Assurance and the Drupal Development Process, Fen Labalme Links: Video, Slideshow
Inside Drupal Caching: From Static Variables to Memcache, John VanDyk Links: Video

Note: Thanks to Ana Macatiag for all these handy links! =)

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

DrupalCon 2009 Recap