Category Archives: development

Drupal 5: Migrating a production database to a QA server

Building on the configuration management strategy I described last time, I wrote some scripts to make it easier for other developers to migrate the production database to the QA server or to get a copy of the production database for their local system. I needed to consider the following factors:

  • Domain name changes: Because we use Domain Access to serve multiple subdomains using a single Drupal installation and shared sign-on, we needed to make sure that all instances of the domain root are replaced and the correct domain_root is set. For example, the site URLs might be:
    Production QA Local
    example.com qa.example.com local.example.com
    foo.example.com foo.qa.example.com foo.local.example.com
    bar.example.com bar.qa.example.com bar.local.example.com
  • Privacy concerns: The QA database and local copies of the database should not contain sensitive user information. All users aside from the administrator should be removed from the system, and all content on site should be limited to editorial content.
  • Efficiency: We don’t need data like access logs or watchdog logs in QA or for local testing. This saves quite a lot of time and space during the data migration process.

Here’s how I did it:

  1. I identified tables I needed to copy over and tables that I could just leave empty. I did this by going through the output of “SHOW TABLES” in a MySQL console.
  2. In my Makefile, I declared a DB_DATA variable that contained a list of tables I wanted to copy.
  3. I wrote a backup_partial target in my Makefile that issued a series of commands:
    mysqldump -u ${DB_USER} --password=${DB_PASSWORD} ${DB} --no-data > partial.dump  # export the schema
    mysqldump -u ${DB_USER} --password=${DB_PASSWORD} ${DB} --opt --complete-insert ${DB_DATA} >> partial.dump # Dump some of the data
    mysqldump -u ${DB_USER} --password=${DB_PASSWORD} ${DB} --opt --complete-insert --where='uid< =1' users users_roles >> partial.dump # Dump the admin and anonymous users
    echo "UPDATE node SET uid=1;" >> partial.dump # Set all the node authors to admin
    echo "REPLACE INTO search_index_queue (nid, module, timestamp)  select nid, type, unix_timestamp(now()) FROM node;" >> partial.dump # Prepare for reindexing
    
  4. I wrote a shell script on an internal server that accepted an argument (the domain to translate to) and performed the following steps:
    1. ssh into the production server and run make with my backup_partial target, compressing the resulting partial.dump
    2. scp the partial.dump.gz from the production server onto the internal server
    3. unpack partial.dump.gz
    4. figure out what $DOMAIN is supposed to be based on the arguments
    5. run perl -pi -e "s/example.com/$DOMAIN/" partial.dump
    6. load partial.dump into my database
    7. run cron.php if it can
  5. I added two buttons to my web-based deploy script: one button to migrate the production database to the QA server, one button to make a copy of the production database for the domain “local.example.com”. Both buttons call
  6. I created multisite settings.php in my Drupal directory (ex: sites/local.example.com and sites/qa.example.com). The production settings go in default/settings.php, and the multisite settings.php override it like this:
    $conf = array(
      'domain_root' => 'local.example.com',
    );
    $cookie_domain = '.' . $conf['domain_root'];
    

    $conf allows you to override Drupal variables returned by variable_get.

So now, I can click on a button to migrate a sanitized copy of the production database to the QA server or to my local system. Sweet!

Working on a small project

Working on a small project means that I wear multiple hats, and that’s helping me grow so much as a developer. Yesterday, I planned some changes to our build process and developed tools to make deployment painless. In a large project, all of this would have been decided already, and all I would need to do would be just to fit into the scheme. Here, I get to weigh the pros and cons, then make things better.

What’s the next step?

If I integrate regression testing into the production deployment script, then we will be one step closer to development nirvana. Another step would be to do daily regression testing, per-checkin tests, or even pre-checkin tests. Wouldn’t that be awesome?

I’m definitely working above my pay grade. ;) That’s because I don’t have to make all the mistakes myself. Working on and watching open source development means that I can see what problems other developers run into and how they’ve solved them, and that’s taught me ways to keep my development environment sane. I’m still going to make mistakes, but at least I can avoid some of them. Kaizen–relentless improvement–means that I’m not only solving problems right now, I’m also working on making things better. As a blogger (and somewhat of a teacher), I not only grow as a developer, but I’m helping other people grow too.

Subversion: Restore deleted files

If you accidentally delete a file and then commit the change, you can retrieve the file along with the rest of the history by using svn copy to copy it into your working directory. svn copy -r revision-number your-full-repository-path-to-the-file file

UPDATE: pgier suggested svn copy http://svn/url/to/file@revision-number file

Common Use-Cases for Merging

Drupal rockitude

I joined my current Drupal project two weeks ago after documenting and wrapping up my other Drupal-based project (which I’m happy to report is chugging along quite nicely without me). Since then I’d been quickly going through my task list. I’ve also been poaching other people’s tasks, such as the configurable group home pages and the deployment scripts. As I told my cat, I’m having fun rocking Drupal. ;)

Today, one of my project team members demonstrated the site to members of the other consulting team. He wasn’t sure if the reception would be neutral or even hostile. He walked through the various features we’d built in two weeks of work, a short period of time that included the numerous issues other people encountered and my ramp-up time as a new member of the team.

One consultant from the other team couldn’t help but say, “Wow.” And the rest of the team really liked it, too!

So I guess I can count that as my first “melting a tough client” story. =)

What worked well?

  • Even though I was new to the team, my team members gave me plenty of latitude in building tools and reorganizing things. There were a couple of changes that we needed to discuss, but usually, the practices I suggested made things a bit easier and the changes weren’t difficult to get used to.
  • I set up my local development environment (my uber-tricked-out Emacs) to make it easy for me to quickly shift between files in different branches of code, look up definitions of functions, and re-build the site.
  • As a lazy programmer, it’s in my best interest to automate as much as I can. I kept tweaking the build process until I could get it to work without manual intervention, and I plugged it into the web-based deployment script I’d written for my previous project.
  • I reused as much as I could, relying on a combination of grep, var_dump, printf, tags, and asking coworkers what’s responsible for what.
  • Also, starting work at 5:15 AM turns out to be pretty good for me. I get almost a full day of work into the morning, I can spend a little time in the afternoon attending to things that require less creative energy, and I can even take a short catnap in the afternoon sun (complete with purring cat!).

I’m looking forward to tidying this site up, and I’m sure I’ll have lots of fun getting the next site off the ground, too!

Drupal, Makefiles: save time, make awesome

One day I’ll post a generalized version of the Makefile that makes my Drupal life so much easier, but in the meantime, here’s the general structure I use:

  1. Lots of configuration settings:
    DB
    Database name for Drupal site
    DB_USER
    Database user for Drupal site
    DB_PASSWORD
    Database user password for Drupal site
    ROOT_DB_PASSWORD
    Root MySQL password, used to drop and recreate the database, and grant permissions
    SVN
    Subversion branch for source code control
    SVN_USER
    Subversion user name
    SVN_PASSWORD
    Subversion password
    SITE_DOMAIN
    Particularly useful when working with sites that use Domain Access, but handy even with other sites because of BASE_PATH
    SRC_DIR
    Location of source code, needed for Drush
    DRUSH
    Drush command line, just in case I want to fiddle with it
    DB_PREFIX
    Database prefix, for make clearcache

    BASE_PATH
    Base URL

    DRUSH_GROUPRE
    Regular expression to match against simpletests using my modified Drush_simpletest that lets me run all groups of tests matching a regular expression
  2. An include statement
    -include *.mk
    This is really useful. This lets you override the variables using a file like local.mk, dropped into your current directory and kept out of your source code control system. I use this to make my Makefile just Do The Right Thing when it’s in different directories or on different servers. Do not forget to tweak your included file (or make sure there is one) if you need different configuration settings.
  3. Lots of targets:
    backup
    Uses mysqldump to back up the database.
    restore
    Drops the database and restores it from the backup.
    clean
    Drops the database
    install
    Calls the site’s install.php with the appropriate profile.
    cron
    Calls the site’s cron.php
    mysql
    Starts a MySQL connection to the Drupal database. I don’t have to remember what the database name is or what the connection details are, hooray!
    clearcache
    Clears the {cache}, {cache_menu}, {cache_views}, {cache_content}, {cache_filter}, and {cache_page} tables
    revert
    Hits the panic button and reverts my tree. (because svn revert . -R is too long)

    tags
    Regenerate the TAGS file that lets me find function definitions really quickly
    test
    Use Drush to run all of my simpletest functional tests for this project

Sometimes I’ll have an “rsync” target to deploy files to a server and a “doc” target to regenerate my low-level documentation using doxygen, too.

This Makefile is seriously cool and seriously time-saving because it means that no matter which project, branch, directory, or server I’m working on, I have a consistent set of commands for the most common things I need to do. The less time I spend thinking about fiddly administrative tasks, the more time I can enjoy in the coding zone.

This is also one of the reasons why Drupal development is _so_ much easier on Linux or other Unix-ish systems, or even Microsoft Windows with the Cygwin environment. If you see yourself doing a fair bit of Drupal work, it’s well worth investing the time in setting up a virtual machine or a second partition, learning a new operating system, and re-setting-up your development environment.

make awesome. =)

What’s in your Makefile, or what other tricks do you use to increase your developer happiness?

Scaffolds and structures

I often talk about leverage and scale: creating as much value as I can for as many people as I can. Now that I’ve been with IBM for a while, I see the personal benefits of that practice in the assignments I get and the help I offer my coworkers. In a way, I build scaffolds to help people to do more.

Scaffolding Photo (c) 2008 Kevin Dooley (Creative Commons Attribution License)

One of the ways I’ve contributed to my team is by integrating a regression-testing framework into our Drupal project.On the team call last Tuesday, the project manager asked if everyone had seen the e-mail he had sent about testing, and if anyone had any comments. I reported that I had built the features that were assigned to me, but I was waiting for someone else to confirm testing. He asked if we could look into automated regression tests, too. I laughed and said we’d had them for months. I had even integrated them into the build and deployment script I’d written, but the other developers said that running the whole regression suite of project-related tests took too much time, so I turned that off. Still, I regularly ran full regression tests on my system, using customized versions of Drush and Simpletest.

I enjoy doing things like that: completing my tasks ahead of schedule and using the rest of the time to think of ways to optimize how people work. =) Kaizen – relentless improvement!