Finally got my new landing page off the ground! =)

I’m starting to get the hang of using Web analytics to look at how people are moving through the site. I’ll go through the numbers in detail, but if you’re in a rush and you just want to check the potential landing page out, I’d love your feedback.

The numbers: Both sachachua.com and livinganawesomelife.com redirect to the front page of this blog, which contains links to all the other pieces. Last month, 1015 people visited this page. About 50% of those people were new to my site. This represents about 7% of the total new visits to the site (I get lots of people coming in through search engines), but that’s still 500 or so people who might be confused by the range of things I write about. Of all the visitors to the front page (including people who’ve been here before), about 166 people followed the “About” link from the home page. Others clicked through the categories I feature along the top.

I’d like to get your feedback on the new site landing page first, before I make it the default. I’m also curious about A/B testing, so if I can figure out how to do that consistently (so that once you’re either A or B you don’t switch unless you want to), that’ll be something cool to learn. =)

Anyway, here’s the potential landing page. It’s Drupal-based, mostly done with unmodified  third-party modules. ;) I’d love your feedback. You’ll also notice that it’s starting to accumulate my reading history. I wrote a Perl script to grab the data from the Toronto Public Library, and it still needs some tweaking to make it more useful. I’ve already started using it to keep track of my notes, but those are private for now. =)

Whee!

Also: a whole slew of talks and events!

My session on Totally Rocking Your Development Environment has been accepted for DrupalCon 2009, hooray! Thanks!

This is great! And handy, because I’ve already promised to give an IBM-flavored version of the talk at the first community call for the newly-formed (or -revived, not sure) IBM Drupal community, which means I will have to have it all ready to go by two weeks from now instead of two months.

Two weeks from now is also when I’ll be giving a lecture on Enterprise 2.0 and knowledge management to Dorit Nevo’s MBA class at Schulich.

And I’ve volunteered to help organize or otherwise make these upcoming conferences awesomer: IBM Web 2.0 Summit, DrupalCampTO, Mesh.

And there’s LifeCampTO.

Busy, busy, busy.

Fortunately, talks are so much more fun to prepare when you think of them as learning opportunities. And I’ve volunteered to help conferences out with either things I know how to do well (say hi to people at registration desks, etc.) or that I’m interested in transforming/scaling (abstract submission, voting, schedules) or that I’m interested in learning (selling sponsorship, buying merchandise). And the conferences are a bit further out.

But “slew” is such a good word, because if I’m not careful and if I don’t intentionally slow down as I get into the busy-busy-busy times, then another sense (slew: past tense of slay) may figuratively kick in. That wouldn’t be fun at all.

It’ll all be great fun, though, and I’m sure I’ll learn tons! You’ll hear about all of it here, of course.

So if I’m slow at e-mail, you know why. =)

Notes from today’s Drupal hacking

  • Avatar selection/upload during registration: Oddly enough, avatar_selection doesn’t handle this. You need the reg_with_pic module, too.
  • Shared and per-site file directories: We’re segregating uploaded files by microsite, but we want to share some files as well. Hacking the shared path into file.inc is a kludge, but it works. Like so:
    --- file.inc    (revision 1046)
    +++ file.inc    (revision 1589)
    @@ -66,6 +66,11 @@
    else if (file_check_location($file_path . '/' . $dest, $file_path)) {
    return $file_path . '/' . $dest;
    }
    +  // Not found, check by going up one more
    +  else if (file_check_location($file_path . '/' . $dest, $file_path . '/../')) {
    +    return $file_path  . '/' . $dest;
    +  }
    +
    // File not found.
    return FALSE;
    }
    
  • Group names with signed-up events: signup.module doesn’t make it easy to theme the list of events, so I created my own block based on the code in signup_list_user_signups. To determine the possible group a node came from, I check for the intersection between $node->og_groups and the array keys of $user->og_groups.
  • Fixed panes: I couldn’t figure out a way to set a block as fixed in Panels 2, but a workaround is to define a custom layout that includes the block in the right place. Here’s an example:
    /**
     * Implements hook_panels_layouts
     */
    function mymodule_panels_layouts() {
      $items['my_panel'] = array(
        'title'  => t('My Panel Layout'),
        'icon'   => 'my_panel.png',
        'theme'  => 'my_panel',
        'css'    => 'my_panel.css',
        'panels' => array(
          'top' => t('Top'),
          'left' => t('Left'),
          'right' => t('Right'),
        ),
      )
      return $items
    }
    
    /**
     * This function uses heredoc notation to make it easier to convert
     * to a template.
     */
    function theme_my_panel($id, $content) {
      if ($id) {
        $idstr = " id='$id'"
      }
      // Pull in the Organic Groups - 0 block.
      $group_details = og_block('view', 0)
      $group_block = < <<EOT
      <div class="block block-og" id="block-og-0">               
        <h2 class="title">$group_details[subject]</h2>
        <div class="content">
        $group_details[content]
        </div>
        
    EOT
      $output = < <<EOT
    <div class="panel-2col-stacked clear-block panel-display" $idstr>
      <div class="panel-col-top panel-panel">
        <div class="inside">$content[top]</div>
      </div>
        <div class="center-wrapper">
        <div class="panel-col-first panel-panel">
        <div class="inside">
        $content[left]
        </div>
        </div>
        <div class="panel-col-last panel-panel">
        <div class="inside">
        $group_block
        $content[right]
        </div>
        </div>
      </div>
      <div class="panel-col-bottom panel-panel">
        <div class="inside">$content[bottom]</div>
      </div>
    EOT
      return $output;
    }
    

    To limit the choices to just that layout, put this in your installation profile:

      panels_load_include('common');
      $allowed_layouts = new panels_allowed_layouts();
      $allowed_layouts->allow_new = TRUE;
      $allowed_layouts->module_name = 'panels_common';
      $allowed_layouts->allowed_layout_settings = array(
        'threecol_33_34_33_stacked' => 0,
        'twocol' => 0,
        'flexible' => 0,
        'threecol_25_50_25' => 0,
        'threecol_25_50_25_stacked' => 0,
        'onecol' => 0,
        'threecol_33_34_33' => 0,
        'twocol_stacked' => 0,
        'twocol_bricks' => 0,
        'my_panel' => 1
      );
      $allowed_layouts->api_save();
    
  • Organic Groups RSS feed: There’s an og_aggregator module, although it doesn’t quite fit in neatly with everything else.
  • Organic Groups tag cloud: I was surprised to find that this wasn’t built in. I ended up defining my own block using a SQL query like this:
            if (($node = og_get_group_context()) && node_access('view', $node)) {
              $result = db_query(
                'SELECT COUNT(*) AS count, d.tid, d.name, d.vid FROM {term_data} d
                 INNER JOIN {term_node} n ON (d.tid=n.tid)
                 INNER JOIN {og_ancestry} og ON (og.nid=n.nid)
                 WHERE og.group_nid=%d AND d.vid=0 GROUP BY d.tid, d.name, d.vid ORDER BY count DESC', $node->nid);
              $steps = 6;
              $tags = tagadelic_build_weighted_tags($result, $steps);
              $tags = tagadelic_sort_tags($tags);
              $output = theme('og_tagadelic_weighted', $node, $tags);
              $block['content'] = $output;
            }
    

    og_tagadelic_weighted was a custom theme function that constructed a link to a faceted_search using the og: and taxonomy: filters. I had to hack faceted_search, too. First I needed to enable the og filter, and then I needed to make taxonomy_facets.module recognize free tags (vid=0). Patch for taxonomy_facets.module:

    ===================================================================
    --- taxonomy_facets.module	(revision 1046)
    +++ taxonomy_facets.module	(revision 1582)
    @@ -147,6 +147,7 @@
           // and create facets from it.
           if ($found_text = search_query_extract($arg, 'taxonomy')) {
             $vocabularies = taxonomy_get_vocabularies();
    +        $vocabularies[0] = 1;
             // Extract separate facets
             $paths = explode(',', $found_text);
             foreach ($paths as $path_index => $tids) {
    

It’s a good thing I can find my way around sparsely-documented code with few or no examples… <laugh>

Drupal and Drush: Updating the database from the command-line

So now that we’re doing all our configuration changes in source code, it makes sense to automate database updates as much as I can. Here’s something I’ve added to drush_tools so that I can run all the schema changes from the command-line:

function drush_tools_update($command = '') {
  ob_start();
  require_once 'includes/install.inc';
  require_once 'update.php';
  $ret = ob_get_contents();
  drupal_load_updates();
  ob_end_clean();

  $list = module_list();
  $update_list = array();
  foreach ($list as $module) {
    $updates = drupal_get_schema_versions($module);
    if ($updates !== FALSE) {
      $latest = 0;
      $base = drupal_get_installed_schema_version($module);
      foreach ($updates as $update) {
        if ($update > $base) {
          if ($update > $latest) { $latest = $update; }
          $update_list[$module][] = $update;
        }
      }
      if ($latest) {
        sort($update_list[$module]);
        printf("%-30s %5d -> %5d (%s)\n", $module, $base, $latest, join(', ', $update_list[$module]));
      } else {
        printf("%-30s %5d\n", $module, $base);
      } 
    }      
  }
  if (count($update_list) == 0) return;
  if ($command != 'force' && !drush_confirm(t('Do you really want to continue?'))) {
    drush_die('Aborting.');
  }
  ob_start();
  foreach ($update_list as $module => $versions) {
    foreach ($versions as $v) {
      print "Running " . $module . "_update_" . $v . "\n";
      update_data($module, $v);
    }
  }
  $updates = ob_get_contents();

  cache_clear_all('*', 'cache', TRUE);
  cache_clear_all('*', 'cache_page', TRUE);
  cache_clear_all('*', 'cache_menu', TRUE);
  cache_clear_all('*', 'cache_filter', TRUE);
  drupal_clear_css_cache();
  ob_end_clean();
  $output = '';
  if (!empty($_SESSION['update_results'])) {
    $output .= "The following queries were executed:\n";
    foreach ($_SESSION['update_results'] as $module => $updates) {
      $output .= "\n" . $module . "\n--------------------------\n";
      foreach ($updates as $number => $queries) {
        $output .= 'Update #'. $number . ":\n";
        foreach ($queries as $query) {
          if ($query['success']) {
            $output .= "SUCCESS: " . $query['query'] . "\n";
          }
          else {
            $output .= "FAILURE: " . $query['query'] . "\n";
          }
        }
        if (!count($queries)) {
          $output .= "No queries\n";
        }
      }
    }
    $output .= "\n";
    print $output;
    unset($_SESSION['update_results']);
  }
}

Drupal: Deploying two branches to three systems

To keep track of the bugfixes we’ll need to make for our next release, I’ve created a Subversion branch called branches/release-1. Development of new features will continue on trunk, but we’ll merge in the bugfixes from release-1 every so often.

There are three environments we deploy to:

Local
Developers should be able to easily test both versions on their local machines.
QA server
We should be able to deploy both versions to a publicly-accessible QA server for acceptance testing.
Production server
We should be able to deploy release-1 (and then later, release-2 and so on) to the production server, preferably after a lot of testing

Editorial changes happen on the production server, where our users update content. We would like to be able to take a snapshot of that database and use that to test our development code on the QA server or in our local development environments. Because we use Domain Access to serve multiple subdomains with shared content, it’s not just a matter of using mysqldump to back up the database and copy it over. We also need to replace URLs inside the database, and we need to override domain_root using the $conf array in settings.php.

I’m the only one running Linux, so the other developers don’t really benefit from the Makefiles I’ve defined or the tools I use. For the simpler build system we had before (all development on trunk), I wrote a deployment script that allowed users to:

  • Download a stripped copy of the production database with the URLs changed for their local testing environment
  • Deploy a stripped copy of the production database to the QA server
  • Deploy a specified revision of the source code to the QA server
  • Deploy a specified revision of the source code to the production server

The new deployment script needed to allow users to do the same, but for both branches of the code. Both branches of the code would be simultaneously available on the QA server, so the script would need to deploy the code to different directories.

After some fiddling around with the page design (because I care about making interfaces make sense!), I came up with something that looks like this:

  Development Release-1
Local Database
QA Database
QA Deployment
Production Deployment None
Changelog
-------
r988 | somegeek | 2008-09-15 13:00:00 -0500 (Mon, 15 Sep 2008) | 2 lines

Use dev.qa.example.com
------------
[more changelog entries go here]
------------------------------------------------------------------------
r986 | somegeek | 2008-09-15 11:03:38 -0500 (Mon, 15 Sep 2008) | 3 lines

Starting a branch for release-1
------------------------------------------------------------------------
[more changelog entries go here]

The deployment script allows the user to get a copy of the database, deploy a copy of the database, or deploy specific revisions of branches.

Because I was having a hard time figuring out how to do ssh key-based operations from Apache (which runs as a no-login user), I use two shell scripts to do the dirty work. One shell script connects to the production server, creates a partial backup, copies the information over, and does any necessary replacements. Another shell script takes a domain name and optionally a revision, and deploys the revision from the appropriate branch.

Here’s my totally small-scale PHP way to show the revisions log:

$dev_output = shell_exec("svn log $dev_url $details --limit 20");
$dev_revisions = preg_match_all('/r([0-9]+)/', $dev_output, $dev_matches);

where $dev_url is the URL of the trunk in Subversion, and $details contains the username and password specified as options for the Subversion command-line.

I’m going to see if I can get my regression tests running on the server that I’ve got my deployment script on. Wouldn’t that be awesome?

Running groups of Drupal tests from the command line

I’ve written about using drush to evaluate PHP statements in the Drupal context using the command line before, and it turns out that Drush is also quite useful for running Simpletest scripts. Drush comes with a module that allows you to display all the available tests with “drush test list”, run all the tests with “drush test run”, or run specified tests with “drush test run test1,test2″.

‘Course, I wanted to run groups of tests and tests matching regular expressions, so I defined two new commands:

drush test run re regular-expression
Run all tests matching a regular expression that uses ereg(..) to match.
Ex: drush test run re Example.*
drush test run group group1,group2…
Run all tests matching the given groups
Ex: drush test run group Example

Here’s the patch to make it happen:

Index: drush_simpletest.module
===================================================================
--- drush_simpletest.module	(revision 884)
+++ drush_simpletest.module	(working copy)
@@ -12,9 +12,13 @@
 function drush_simpletest_help($section) {
   switch ($section) {
       case 'drush:test run':
-        return t("Usage drush [options] test run.\n\nRun the specified specified unit tests. If  is omitted, all tests are run.  should be a list of classes separated by a comma. For example: PageCreationTest,PageViewTest.");
+        return t("Usage drush [options] test run .\n\nRun the specified unit tests. If  is omitted, all tests are run.  should be a list of classes separated by a comma. For example: PageCreationTest,PageViewTest.");
       case 'drush:test list':
         return t("Usage drush [options] test list.\n\nList the available tests. Use drush test run command to run them. ");
+      case 'drush:test group':
+        return t("Usage drush [options] test group .\n\nRun all unit tests in the specified groups. For example: drush test group Group1,Group2");
+      case 'drush:test re':
+        return t("Usage drush [options] test re .\n\nRun all unit tests matching this regular expression. For example: drush test re Page.*");
   }
 }
 
@@ -30,10 +34,18 @@
     'callback' => 'drush_test_list',
     'description' => 'List the available Simpletest test classes.',
   );
+  $items['test re'] = array(
+    'callback' => 'drush_test_re',
+    'description' => 'Run one or more Simpletest tests based on regular expressions.',
+  );
+  $items['test group'] = array(
+    'callback' => 'drush_test_group',
+    'description' => 'Run one or more Simpletest test groups.',
+  );
   return $items;
 }
 
-function drush_test_list() {
+function drush_test_get_list() {
   simpletest_load();
   // TODO: Refactor simpletest.module so we don't copy code from DrupalUnitTests
   $files = array();
@@ -60,6 +72,11 @@
       $rows[] = array($class, $info['name'], truncate_utf8($info['desc'], 30, TRUE, TRUE));
     }
   }
+  return $rows;
+}
+
+function drush_test_list() {
+  $rows = drush_test_get_list();
   return drush_print_table($rows, 0, TRUE);
 }
 
@@ -75,3 +92,31 @@
   }
   return $result;
 }
+
+function drush_test_re($expression) {
+  if (!$expression) {
+    die('You must specify a regular expression.');
+  }
+  $rows = drush_test_get_list();
+  $tests = array();
+  foreach ($rows as $row) {
+    if (ereg($expression, $row[0])) {
+      $tests[] = $row[0];
+    }
+  }
+  simpletest_run_tests($tests, 'text');
+  return $result;
+}
+
+function drush_test_group($groups) {
+  $rows = drush_test_get_list();
+  $tests = array();
+  $groups = explode(',', $groups);
+  foreach ($rows as $row) {
+    if (in_array($row[1], $groups)) {
+      $tests[] = $row[0];
+    }
+  }
+  simpletest_run_tests($tests, 'text');
+  return $result;
+}

That makes running tests so much easier and more fun!