Categories: geek » development

RSS - Atom - Subscribe via email

Geek tidbits: Postfix configuration for development and testing

| development, geek, ibm, rails, work

From November:

We got the mail information for our Rails site, so I reconfigured the mail server. We’re doing a lot of development, and testing mail is much easier when you can send example mail addresses to one bucket of mail. Here’s how I set up the server to redirect everything addressed to @example.org to a test mail address.

First, I set the mail server up for local delivery only, and I confirmed that I could send mail from a local user to another local user account. Then I experimented with rewriting, using virtual_alias_maps to rewrite addresses to a specific account. I confirmed that worked. Then I checked the database to make sure we didn’t have live e-mail addresses, reconfigured the mail server to deliver Internet mail, and crossed my fingers. A few quick tests showed that mail was being delivered as planned – example.org mail routed to our temporary address, everything else delivered normally. We’ll see how this works!

Here’s our /etc/postfix/main.cf

smtpd_banner = $myhostname ESMTP $mail_name
biff = no
append_dot_mydomain = no
readme_directory = no
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
myhostname = domain.goes.here
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = domains.go.here, example.org
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
default_transport = smtp
relay_transport = smtp
virtual_alias_maps = hash:/etc/postfix/virtual
inet_protocols = ipv4

And /etc/postfix/virtual:

@example.org change_to_real_email@example.com

Don’t forget to run postmap /etc/postfix/virtual; /etc/init.d/postfix reload after changing /etc/postfix/virtual and your configuration.

Rails experiences: Building an interactive tutorial

| development, geek, rails, work

One of the risks for this Rails project that I’m working on is that new users won’t have enough ramp-up time before we finish the project. We’re planning to wrap up in December, which is the end-users’ busiest time of year. The project also highly depends on external factors we can’t control, so it might be weeks or even a few months before people get a chance to try the most important parts of the application.

To make it easier for people to get started, we decided to build an interactive tutorial into the system. When people log in, the tutorial should create an offer that they can respond to, walk them through the process of working with it, and then tell them about the next steps they can take. People should be able to stop the tutorial at any time, and they should be able to start the tutorial from the beginning whenever they want a refresher.

People will be on a system that other people use and that generates reports, so all the tutorial information needs to be hidden from reporting and from people who are not in tutorial mode.

I started off by writing a Cucumber test that described how things should work: what people should see, what they could do, and so on.

To keep all the tutorial-related methods in one place, I put them in a file called tutorial_methods.rb and I included these methods in my controller. I added a conditional div to my application.html.erb that displayed the tutorial in a consistent spot if a tutorial was specified for the current page. Then I defined a function that took the current page and figured out what needed to be done for a tutorial. This function created a sample offer at the beginning of a tutorial, performed the behind-the-scenes work to approve the offer once people finished the first step, and loaded the tutorial text from the localization file into an instance variable.

I decided to use Rails’ built-in internationalization support instead of putting the tutorial in the database so that it could easily support multiple languages, although I might use a gem to support internationalization of database values if we need to.

To make things easier on the reporting side, I extended ActiveRecord::Base with my own association methods that filtered the queries depending on whether or not the user was in tutorial mode. These custom association methods made it much easier to make sure all the relevant queries were filtered.

I really liked adding an interactive tutorial to this project, and I think I’ll use that technique for Quantified Awesome as well. Online help is good, but it’s even better if people can practice on something and know it won’t mess up anything else.

Ruby on Rails: Extending ActiveRecord::Base to define your own ActiveRecord association methods

Posted: - Modified: | development, geek, rails, work

One of the things I really like about Rails is the ability to add to existing classes so that your code can be cleaner. For example, in the app we’re working on, I need to be able to display a list of offers associated with an organization. I also need to filter that list of offers by different criteria. If the user is not in tutorial mode, I need to filter out any tutorial-related offers. I want to show offers with different workflow states, too.

If I had to do this in straight SQL, I would need to write many queries to cover the different cases, or write my own query-building engine that takes conditions into account. In the Drupal world, I might try to build a View with lots of arguments, and then use a views_pre_execute hook to monkey around with the generated SQL.

In the Rails world, things are much simpler. I started off by chaining queries, because you can add conditions to the end of an ActiveRelation and go from there. That gave me code that looked like this:

base = Offer.includes(:donation).where("organization_id = ? AND (donations.deadline IS null OR donations.deadline >= ?) AND (NOT (offers.workflow_state IN (?, ?, ?)))", @organization.id, Time.now, Offer::DRAFT, Offer::ALLOCATED, Offer::CONFIRMED).order('offers.deadline')
@direct_offers = base.where("offers.workflow = ?", Donation::DIRECT)
@open_offers = base.where("offers.workflow = ?", Donation::OPEN)

Then I asked myself: How can I make this code even cleaner? I thought about adding instance methods. For example, in my Organization class, I could define the following:

class Organization
  # Other stuff goes here
  def current_offers
    self.offers.includes(:donation).where("(donations.deadline IS null OR donations.deadline >= ?) AND (NOT (offers.workflow_state IN (?, ?, ?)))", Time.now, Offer::DRAFT, Offer::ALLOCATED, Offer::CONFIRMED).order('offers.deadline')
  end
  def current_offers_by_workflow(workflow)
    self.current_offers.where("offers.workflow = ?", Donation::OPEN)
  end
end

That would allow me to replace the code above with something like this:

@direct_offers = @organization.current_offers_by_workflow(Donation::DIRECT)
@open_offers = @organization.current_offers_by_workflow(Donation::OPEN)

… so if I wanted to filter out tutorial entries, I could do that in def current_offers by adding a where clause for the tutorial column.

But it seemed clunky to have to specify all these instance methods in order to filter by different ways. What I really wanted was to be able to chain my custom filters together, so that I could write code like this:

@direct_offers = @organization.offers.filter(current_user).direct
@open_offers = @organization.offers.filter(current_user).open

and then eventually be able to do things like:

list = @organization.offers.filter(current_user).current.direct.allocated

(If I really wanted to.)

I couldn’t figure out where to add the methods so that they’d be defined in the right place. If I added the methods to the Organization class, they couldn’t be called on the ActiveRecord relations. A little bit of searching, and I figured out how to do it in Rails. It turns out that you can extend ActiveRecord relations with your own methods! Here’s how.

You’ll need to extend ActiveRecord::Base with your own methods. I put this in config/initializers/activerecord_extensions.rb.

module ProjectNameActiveRecordExtensions
  def filter(control)
    exclude_tutorial = true
    # Include the tutorial offers for users in tutorial mode
    if control.is_a? User and control.tutorial
      exclude_tutorial = false
    # You can also pass filter(false) to turn off these filters for testing
    elsif !control 
      exclude_tutorial = false
    end    
    if exclude_tutorial
      scoped.joins(:donation).where('donations.tutorial=?', false)
    else
      scoped
    end
  end
  # other methods go here...
end
ActiveRecord::Base.extend ProjectNameActiveRecordExtensions

The trickiest part was figuring out how to do a conditional filter, and that’s what scoped is for. I wanted to include the tutorials if the user was in tutorial mode, so my function should be a pass-through in that case. I couldn’t return self or nil, because that broke the associations. scoped turned out to be the magic keyword that refers to the current scope of the query.

What if you want to use the same words in different contexts? For example, “pending” might need to result in two different queries depending on whether you’re asking for pending offers or pending requests. ActiveRecord::Base is used for all classes, but you can use self to find out what class is being used for scoping. For example:

def pending
  if self == StandingRequest
    scoped.where("standing_requests.workflow_state=?", StandingRequest::PENDING)
  else
    # Replace with other cases as I find the need for them
    raise "Undefined behaviour"
  end
end

I love the fact that Rails lets you modify so much in order to make building sites easier. It’s like Emacs for the Web, and it makes my brain happy.

Quantified Awesome: Development-driven behaviour and integrated tests for life

Posted: - Modified: | development, geek, quantified

2011-12-14: Updated step code

In terms of testing code, behaviour-driven development is fantastic. You can write your tests in pretty much plain English using a testing tool like Cucumber for Rails, which makes it easier to communicate with other people (including clients!). There’s a certain satisfaction in getting your tests to pass, and when they break, you know something needs fixing.

I’ve been thinking about what automated tests might look like in life. It turned out to be easy to prototype, thanks to the data I’m already collecting. It’s almost like development-driven behavior: can I apply the tools I use in software development to help me change my behaviours in life?

Here are some results from my very first integration test of real life:

Feature: Development-driven behaviour

  Scenario: Check for overdue books                # features/life.feature:2
    When I check our library items                 # features/step_definitions/life.rb:3
    Then there should be no items that are overdue # features/step_definitions/life.rb:7

  Scenario: Check my work load                       # features/life.feature:5
    When I look at my time use for the past 7 days   # features/step_definitions/life.rb:11
    Then I should have time data                     # features/step_definitions/life.rb:19
    And I should have worked between 40 and 44 hours # features/step_definitions/life.rb:24
      <46.5166666666667> expected to be
      <=
      <44.0>. (Test::Unit::AssertionFailedError)
      ./features/step_definitions/life.rb:26:in `/^I should have worked between (\d+) and (\d+) hours$/'
      features/life.feature:8:in `And I should have worked between 40 and 44 hours'

  Scenario: Check if I'm sleeping                        # features/life.feature:9
    When I look at my time use for the past 7 days       # features/step_definitions/life.rb:11
    Then I should have slept between 8 and 9 hours a day # features/step_definitions/life.rb:29

Failing Scenarios:
cucumber features/life.feature:5 # Scenario: Check my work load

3 scenarios (1 failed, 2 passed)
7 steps (1 failed, 6 passed)
0m0.833s

Cucumber highlights failing tests in red and it lists the failures as well.

Here’s the steps.rb that I’ve started fleshing out:


When /^I look at my time use for the past (\d+) days?$/ do |arg1|
  @start_time = (Date.today - arg1.to_i.days).midnight.in_time_zone
  @end_time = Date.today.midnight.in_time_zone
  @log = TimeTrackerLog.new(User.first)
  @entries = @log.entries(@start_time, @end_time)
  @summary = @log.summarize(@start_time, @end_time)
end

Then /^I should have time data$/ do
  assert @entries != nil
  assert @entries.size > 0
end

Then /^I should have worked between (\d+) and (\d+) hours$/ do |min, max|
  assert_operator @summary['A - Work'] / 1.hour, :>=, min.to_f
  assert_operator @summary['A - Work'] / 1.hour, :< =, max.to_f
end

Then /^I should have slept between (\d+) and (\d+) hours a day$/ do |min, max|
  average = @summary['A - Sleep'] * 1.0 / (1.hour * ((@end_time - @start_time) / 1.day))
  assert_operator average, :>=, min.to_f
  assert_operator average, :< =, max.to_f
end

# LIBRARY

When /^I check our library items$/ do
  nil # Actually stored in database, so we don't need anything here. This is more for semantics
end

Then /^there should be no items that are overdue$/ do
  assert_equal 0, LibraryItem.where('status = ? AND due < ?', 'due', Date.today).size
end

I am pleasantly boggled that this is possible, and will probably write all sorts of odd tests now. Because Cucumber can fill in web forms, click on stuff, and so on, I might even be able to use it to check information on other sites. (When I check my mail, then all the messages in my inbox should be less than a week old?)

Oh, the possibilities…

The joys of development with Selenium web-testing

Posted: - Modified: | development, geek

I’ve started using the Selenium web-testing framework as part of regular development, and I like it. Selenium makes it easy to automate testing web applications, but it’s also useful for developing web applications.

I was working on improving the administrative interface of a Drupal site. To test the new features, I needed to switch back and forth between different users, and I needed to move nodes through a workflow. Masquerade made it a little bit easier to switch between user roles without logging out and logging in, but there was still a lot of clicking and waiting involved. Selenium made it easy to record and tweak the steps I took, and I could even add assertions so that I didn’t have to check things myself.

Timestamps helped me doublecheck that new data was successfully submitted, and that my e-mail messages weren’t using the old data. Here’s how to store a timestamp:

storeEval { var stamp = new Date(); var year = stamp.getYear() + 1900; var month = stamp.getMonth() + 1; var day = stamp.getDate(); var hours = stamp.getHours(); var mins = stamp.getMinutes(); var time; var secs = stamp.getSeconds(); if (month < 10) { month = "0" + month; } if (day < 10) { day = "0" + day; } if (mins < 10) { mins = "0" + mins; } if (secs < 10) { secs = "0" + secs; } timestamp = "" + year + "-" + month + "-" + day + " " + hours + ":" + mins + ":" + secs; } timestamp

Then I can refer to that timestamp when filling out forms, like this:

type Node title ${timestamp}

I can check if the timestamp is present on a page:

assertTextPresent Node title ${timestamp}

I thought I would get impatient with the slow pace of web testing compared to directly testing the underlying models using code (Simpletest rocks for this). The visual feedback from watching my browser step through the tests helped me appreciate the time I saved compared to clicking through things myself. (And I had extra time to write notes in Emacs or comment my code while the tests are running!)

The Stored Variables Viewer (http://seleniumhq.org/download/) plugin was great for viewing timestamps, saved HTML snippets, and other things stored during test cases. In a pinch, it’s a decent way to explore something in the Selenium IDE context – store a variable, then view it.

Selenium IDE is a free Firefox plugin that you can get at http://seleniumhq.org/download/ Good stuff!

Learning browser-based testing with Selenium

Posted: - Modified: | development, geek, work

I want to get better at testing my applications so that clients and end-users run into fewer bugs. I’m learning how to use Selenium to write browser-based tests. So far, I’ve written eight tests and fixed three bugs. This is good.

I’m using the Selenium IDE, and I’m looking forward to trying other options. I like the way that the Selenium IDE lets me record and step through tests easily. The Selenium Stored Variables Viewer plugin was really helpful, because it made it easy for me to store values and view them. I’m slowly getting the hang of different commands and asserts. Next week, I’m going to read the command reference so that I can index the possibilities.

People tell me I’m a fast developer. I want to try swapping some of that speed for better accuracy – slowing down and doing things right, with tests to back that up. It feels like it takes a lot of time to click around and wait for the pages to respond, or even to run these web-based tests and iterate until I’ve gotten them right, but it’s better for me to do it than for other people to run into these errors. Besides, with tools and metrics, I can make testing more like a game.

Onward and upward!

Git bisect and reversing a mistaken patch

Posted: - Modified: | development, geek

2011/12/09: Updated links

Using version control software such as git is like slicing the bread of programming. It lets you deal with changes in small chunks instead of having to troubleshoot everything at the same time. This comes in really handy when you’re trying to isolate a problem. If you can tell which change broke your system, then you can review just those changes and look for ways to fix them.

For example, some views I’d created in Drupal 6 had mysteriously vanished from the web interface. Fortunately, I’d exported them to code using Features, so I knew I could get them back. I needed to find out which change removed them so that I could make sure I didn’t accidentally undo the other relevant changes.

git bisect to the rescue! The idea behind git bisect is the same one behind the marvelous efficiencies of binary search: test something in the middle of what you’re looking at. If it’s good, take the later half and test the middle. If it’s bad, take the earlier half and test the middle. It’s like what people do when guessing a number between 1 and 100. It makes sense to start at 50 and ask: is the number greater than 50? If it is, ask: is the number greater than 75? And so on. Handy trick, except sometimes it can be difficult to add or subtract in your head and figure out the next number you should ask.

git bisect does that adding-up for you. You start with git bisect start in the root of your source tree. You tell it if the current version is considered broken, using git bisect bad. You tell it the last known working version, with git bisect good changeset-identifier. Then it takes the middle of that range. Test it to see whether it works, and type in git bisect good or git bisect bad depending on what you get. It’ll present you with another changeset, and another, until it can identify the first changeset that fails. If you can automate the test, you can even use the git bisect run command to quickly identify the problem.

Now that you’ve identified the relevant changeset, you can use git show changeset-identifier to look at the changes. If you save it to a file, you can edit the diff and then use the patch command to reverse and apply the diff. Alternatively, you can undo or tweak your changes by hand.

The git bisect section in the free Git SCM book has more information, as does the manual page. Hope this helps!