I spent some time working on upgrading Quantified Awesome from Rails 3 to Rails 4. I was so glad that I had invested the time into writing enough RSpec and Cucumber tests to cover all the code, since that flushed out a lot of the differences between versions: deprecated methods, strong parameters, and so on.

rspec-mode was really helpful while testing upgrade-related changes. I quickly got into the habit of using C-c , m (rspec-verify-matching) to run the spec file related to the current file. If I needed to test specific things, I headed over to the spec file and used C-c , s (rspec-verify-single). Because RSpec had also changed a little bit, I needed to change the way rspec-verify-single worked.

(defun sacha/rspec-verify-single ()
  "Runs the specified example at the point of the current buffer."
  (interactive)
  (rspec-run-single-file
   (concat
     (rspec-spec-file-for (buffer-file-name))
     ":"
     (save-restriction
               (widen)
               (number-to-string (line-number-at-pos))))
   (rspec-core-options)))

(sacha/package-install 'rspec-mode)
(use-package rspec-mode
  :config
  (progn
    (setq rspec-command-options "--fail-fast --color")
    (fset 'rspec-verify-single 'sacha/rspec-verify-single)))

C-c , c (rspec-verify-continue) was also a handy function, especially with an .rspec containing the --color --fail-fast options. rspec-verify-continue starts from the current test and runs other tests following it, so you don't have to re-run the tests that worked until you're ready for everything.

I should probably get back to setting up Guard so that the tests are automatically re-run whenever I save files, but this is a good start. Also, yay, I'm back to all the tests working!

Test coverage didn't mean I could avoid all the problems, though. I hadn't properly frozen the versions in my Gemfile or checked the asset pipeline. When I deployed to my webserver, I ran into problems with incompatible changes between Rails 4.0 and 4.1, and Bootstrap 2 and Bootstrap 3. Whoops! Now I'm trying to figure out how to get formtastic-bootstrap to play nicely with Bootstrap 3 and Rails 4 and the latest Formtastic… There are some git repositories that claim to do this correctly, but they don't seem to work.

Grr.

Fine, I'll switch to simple_form, since that seems to be sort of okay with Bootstrap 3. I ended up using this simple_form_bootstrap3 initializer along with

# You can wrap a collection of radio/check boxes in a pre-defined tag, defaulting to none.
config.collection_wrapper_tag = :div

# You can define the class to use on all collection wrappers. Defaulting to none.
config.collection_wrapper_class = 'collection'

and this in my style.css.sass:

.form-horizontal
  .control-group
    @extend .form-group
  .control-label
    @extend .col-sm-2
  .control-group > .form-control, .form-group > .form-control, .form-group > .collection
    @extend .col-sm-10
  .help-block
    @extend .col-sm-offset-2
  .control-label.boolean
    text-align: left
    font-weight: normal
    @extend .col-sm-offset-2
  label.radio
    font-weight: normal

which is totally a hack, but it sort-of-semi-works for now.

More changes to come, since I've got to get it sorted out for Rails 4.1 too! Mrph.

Tests are important because programmers are human.  I know that I’m going to forget, make mistakes, and change things that I didn’t mean to change. Testing lets me improve my chances of at least noticing. Coverage tools show me how much of my code is covered by tests, improving my chances of seeing where my blind spots are.

In one of my last projects at IBM, I convinced my team to use coverage testing tools on a Rails development project. It was great watching the coverage numbers inch up, and we actually reached 100% (at least in terms of what rcov was looking at). I occasionally had to fix things when people broke the build, but sometimes people added and updated tests too. Although coverage testing has its weaknesses (are you testing for edge cases?), it’s better than nothing, and can catch some embarrassing bugs before they make it to the outside world.

Although I’m not currently taking on any web development work (I’m saving brainspace for other things), I have a few personal projects that I enjoy working on. For example, QuantifiedAwesome.com lets me track different measurements such as time. I had written some tests for it before, but since then, I’d been adding features and making changes without updating the tests. Like the way that unpaid credit card balances turn into financial debt (very bad habit, try to avoid this if possible), unwritten or out-of-date tests contribute to technical debt.

It was a little daunting to sit down and slowly work my way through the knots of out-of-date tests that had accumulated from haphazard coding. Here’s how I made the task more manageable:

• I set up a virtual Linux development environment using Vagrant and Puppet. Many tools are much easier to set up on Linux than they are on Windows, and using a virtual machine for development means that I can easily isolate changes from the rest of my system. Vagrant allows you to specify the configuration of a VirtualBox machine and quickly set up a new instance of it. I copied the Puppet configuration from rails-dev-box and modified it to match my requirements. I also compiled and installed Emacs 24 from source, along with a few Ruby-related tools like haml-mode and rinari. (If you’re going to do a lot of development, you might as well do it with your favourite tools!) I copied the Vagrant private keys and imported them into Pageant (an ssh agent for Windows) for easy logins, and I forwarded the ports too.
• I used Guard and Spork to efficiently run subsets of my tests. Guard is a tool that re-runs specific tests when it detects that files have changed, while Spork lets you run tests without reloading your entire Rails application. Instead of running rake spec (which runs the whole test suite) or rspec ./spec/path/to/file (and  having to copy and paste the failing test’s filename), I could let Guard take care of rerunning failed tests for me.
• I worked on getting the old tests to pass. I wanted to get those cleared up before writing new tests. Good thing too – found a few bugs and some old code along the way.
• I reviewed the recommendations for better tests. Better Specs has tips on using RSpec with Ruby.
• Then I wrote new tests to cover the rest of my models. I used Simplecov for coverage testing. Since Vagrant uses shared folders, I could use Google Chrome to view the coverage webpages from my host computer. I still have some technical debt left – I need to write integration tests, and there’s more beyond that – but it’s satisfying to know that the models (data + logic) are covered. For some reason, my Guard+Spork combination doesn’t result in cumulative Simplecov scores, so I occasionally regenerate my coverage stats with rake spec. These tests will be useful as I write new features or move to Rails 4.

(… 21.5 hours of coding/development/infrastructure…)

New things I learned:

I use Mechanize to work with the Toronto Public Library’s webpage system so that I can retrieve the list of due books, renew items, or request holds. I wanted to test these as well. FakeWeb lets you intercept web requests and return your own responses, so I saved sample web pages to my spec/fixtures/files directory and used FakeWeb to return them. (ex: toronto_library_spec.rb)

Paperclip also needed some tweaking. I replaced my Paperclip uploads with File assignments so that I could test the image processing. (ex: clothing_spec.rb)

Always remember to wrap your RSpec tests in an it “…” instead of just context “when …” or describe “…”. I forgot to do this a few times and got confused about why stuff got left in my database and why the error messages weren’t formatted like they normally are.

Progress! Next step: Update my Cucumber tests and add some more integration tests…

I like this part of being a developer, even though writing new code is more fun. Testing is one of those time-consuming but powerful tools that can reduce frustration in the long run. I’ve experimented with applying automated testing techniques to everyday life before. I wonder what it would be like to take that even further… Any thoughts?

Rails is awesome. We built a workflow/reporting system for ~120 users using Rails 3. My part of the project came to about 468 hours, or roughly 60 workdays (~ 3 months), and I worked with another developer who also put in around the same number of hours. We worked with a graphic designer, a CSS/HTML developer, a tester, a project manager, and the client, and we put together a surprisingly large set of functionality.

It’s amazing how quickly the site came together. I built a simple prototype to help the other developer get started with Rails, and we fleshed it out with the client’s input while waiting for the creative design. I started with web_app_theme so that we could have a decent-looking interface for starters. When the client approved the graphic design, another developer sliced it up into HTML, CSS, and images for us. I took those, converted them into HAML and ERB, and we were off and running. Every weekly sprint meant a chance to show off useful functionality and get feedback. It was awesome.

We were initially worried that building all the UIs from scratch in Ruby on Rails would mean taking up more time because we couldn’t use CCK or Views to quickly throw everything together. It turned out that HAML, partials, and semantic_form_for made the forms and reports easy to do. Filtering reports was straightforward with ActiveRecord and scopes. Because we built the screens ourselves, we didn’t have to fight with Views or CCK for the last 20% of tweaking, and I didn’t have to kludge any SQL queries (yay, no views_pre_execute!).

I was working on a Drupal project shortly before this, and I spent some time supporting the Drupal project during this one. Rails made my brain much happier. I felt that I could organize my code more cleanly, and I could test it more thoroughly, too. I didn’t have to fight so much with other people’s modules or themes. I like Drupal, and I’m still looking forward to doing more projects with it. But I wouldn’t mind working on more Rails projects, and I’m glad I’ve got Quantified Awesome as a personal project.

Drupal does many things better than Rails. Drupal modules tend to be more mature and better-documented, and it seems like there’s been more work on scaling Drupal. Internationalization is also more straightforward in Drupal, although Rails I18n is easy to use once you’ve gotten the hang of it. Drupal module dependencies seem a little easier to sort out, too. But Rails is fun!

Tests will keep you sane. This was the first project where we invested in developing a large suite of automated tests. We used Cucumber for high-level tests and RSpec for everything else. The tests caught many regression errors we might have otherwise missed. Test-driven development was fun, too, because the tests gave us tangible progress and simplified checking.

There were times when I gave in to the temptation to commit without running the tests, and I almost always regretted them. (Particularly after friendly finger-wagging from the other developer!)

Issue-tracking rocks. We’ve liked using Rational Team Concert in the past. Getting an externally-accessible instance was complicated, so I set up a Redmine issue tracker as soon as we started the project. We used Redmine to plan work, track bugs, and even collect feedback from the client. As of the time of writing, we had created 766 items and closed 683 of them (89%).

We started with story points, but didn’t end up continuing with them for the rest of the tasks. When we needed to prioritize, we estimated the hours required for each task in order to help the client decide. That worked out quite well. I haven’t tracked item-level time spent, but that seemed to be roughly around my estimates.

I now estimate more time than I used to, because I’ve started factoring in both writing and running tests. It’s a little strange being the pessimistic estimator instead of the optimistic one, but it’s good for the project.

Selenium is great for screenshots, too. Not only is Selenium good for automated browser-based testing of web pages, but it’s also a handy way to capture screenshots for documentation or demos.

Lotus Symphony and Microsoft Word don’t get along. We wasted a few hours trying to update the user guide with the new screenshots, only to find out that the PDF still got screwed up because I didn’t have the fonts the client used. Those fonts were part of Microsoft Office and weren’t on my system. The client took care of updating the user guide so that she could format it the way she wanted, and we focused on code.

Plan with the end in mind. Short projects mean that milestones can sneak up on you before you notice. Half-way through the project, we realized that the project end date was coming Really Soon Now, and we scrambled to put together a launch plan. We wanted to launch a few weeks before the end of the project, to give people time for feedback and updates. That meant that we needed to send pre-launch e-mails one week and two weeks before the launch, which meant… that we needed to start sending those e-mails within a week. Fortunately, the client, IBM PR, and everyone involved managed to get it all sorted out, and we launched.

I like launching. I would like to do more of it. We don’t do it nearly enough on short projects, but these projects are much more likely to launch if we’re around to help with the transition than if only the client is there.

On a related note, I get antsy about adding new functionality before the end of a project. This makes sense, of course. I don’t mind adding new reports and other reading-related functionality, but workflow tweaks are scary. When we’re planning future projects, we can consider similar risks for late-project tasks.

Things I want to improve for future projects:

• More Rails! More!
• More automated tests
• Track time estimates and actual time in order to improve estimation accuracy
• More launch planning
• More blog posts about what I’m learning
• Optimization

Things I want to reduce on future projects:

• Avoid documentation formatting problems – I guess that means Microsoft Office when working with clients who use it
• Must not give in to temptation to skip tests

From November:

We got the mail information for our Rails site, so I reconfigured the mail server. We’re doing a lot of development, and testing mail is much easier when you can send example mail addresses to one bucket of mail. Here’s how I set up the server to redirect everything addressed to @example.org to a test mail address.

First, I set the mail server up for local delivery only, and I confirmed that I could send mail from a local user to another local user account. Then I experimented with rewriting, using virtual_alias_maps to rewrite addresses to a specific account. I confirmed that worked. Then I checked the database to make sure we didn’t have live e-mail addresses, reconfigured the mail server to deliver Internet mail, and crossed my fingers. A few quick tests showed that mail was being delivered as planned – example.org mail routed to our temporary address, everything else delivered normally. We’ll see how this works!

Here’s our /etc/postfix/main.cf

smtpd_banner = $myhostname ESMTP $mail_name
biff = no
append_dot_mydomain = no
readme_directory = no
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
myhostname = domain.goes.here
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = domains.go.here, example.org
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
default_transport = smtp
relay_transport = smtp
virtual_alias_maps = hash:/etc/postfix/virtual
inet_protocols = ipv4

And /etc/postfix/virtual:

@example.org change_to_real_email@example.com

Don’t forget to run postmap /etc/postfix/virtual; /etc/init.d/postfix reload after changing /etc/postfix/virtual and your configuration.

One of the risks for this Rails project that I’m working on is that new users won’t have enough ramp-up time before we finish the project. We’re planning to wrap up in December, which is the end-users’ busiest time of year. The project also highly depends on external factors we can’t control, so it might be weeks or even a few months before people get a chance to try the most important parts of the application.

To make it easier for people to get started, we decided to build an interactive tutorial into the system. When people log in, the tutorial should create an offer that they can respond to, walk them through the process of working with it, and then tell them about the next steps they can take. People should be able to stop the tutorial at any time, and they should be able to start the tutorial from the beginning whenever they want a refresher.

People will be on a system that other people use and that generates reports, so all the tutorial information needs to be hidden from reporting and from people who are not in tutorial mode.

I started off by writing a Cucumber test that described how things should work: what people should see, what they could do, and so on.

To keep all the tutorial-related methods in one place, I put them in a file called tutorial_methods.rb and I included these methods in my controller. I added a conditional div to my application.html.erb that displayed the tutorial in a consistent spot if a tutorial was specified for the current page. Then I defined a function that took the current page and figured out what needed to be done for a tutorial. This function created a sample offer at the beginning of a tutorial, performed the behind-the-scenes work to approve the offer once people finished the first step, and loaded the tutorial text from the localization file into an instance variable.

I decided to use Rails’ built-in internationalization support instead of putting the tutorial in the database so that it could easily support multiple languages, although I might use a gem to support internationalization of database values if we need to.

To make things easier on the reporting side, I extended ActiveRecord::Base with my own association methods that filtered the queries depending on whether or not the user was in tutorial mode. These custom association methods made it much easier to make sure all the relevant queries were filtered.

I really liked adding an interactive tutorial to this project, and I think I’ll use that technique for Quantified Awesome as well. Online help is good, but it’s even better if people can practice on something and know it won’t mess up anything else.

One of the things I really like about Rails is the ability to add to existing classes so that your code can be cleaner. For example, in the app we’re working on, I need to be able to display a list of offers associated with an organization. I also need to filter that list of offers by different criteria. If the user is not in tutorial mode, I need to filter out any tutorial-related offers. I want to show offers with different workflow states, too.

If I had to do this in straight SQL, I would need to write many queries to cover the different cases, or write my own query-building engine that takes conditions into account. In the Drupal world, I might try to build a View with lots of arguments, and then use a views_pre_execute hook to monkey around with the generated SQL.

In the Rails world, things are much simpler. I started off by chaining queries, because you can add conditions to the end of an ActiveRelation and go from there. That gave me code that looked like this:

base = Offer.includes(:donation).where("organization_id = ? AND (donations.deadline IS null OR donations.deadline >= ?) AND (NOT (offers.workflow_state IN (?, ?, ?)))", @organization.id, Time.now, Offer::DRAFT, Offer::ALLOCATED, Offer::CONFIRMED).order('offers.deadline')
@direct_offers = base.where("offers.workflow = ?", Donation::DIRECT)
@open_offers = base.where("offers.workflow = ?", Donation::OPEN)

Then I asked myself: How can I make this code even cleaner? I thought about adding instance methods. For example, in my Organization class, I could define the following:

class Organization
  # Other stuff goes here
  def current_offers
    self.offers.includes(:donation).where("(donations.deadline IS null OR donations.deadline >= ?) AND (NOT (offers.workflow_state IN (?, ?, ?)))", Time.now, Offer::DRAFT, Offer::ALLOCATED, Offer::CONFIRMED).order('offers.deadline')
  end
  def current_offers_by_workflow(workflow)
    self.current_offers.where("offers.workflow = ?", Donation::OPEN)
  end
end

That would allow me to replace the code above with something like this:

@direct_offers = @organization.current_offers_by_workflow(Donation::DIRECT)
@open_offers = @organization.current_offers_by_workflow(Donation::OPEN)

… so if I wanted to filter out tutorial entries, I could do that in def current_offers by adding a where clause for the tutorial column.

But it seemed clunky to have to specify all these instance methods in order to filter by different ways. What I really wanted was to be able to chain my custom filters together, so that I could write code like this:

@direct_offers = @organization.offers.filter(current_user).direct
@open_offers = @organization.offers.filter(current_user).open

and then eventually be able to do things like:

list = @organization.offers.filter(current_user).current.direct.allocated

(If I really wanted to.)

I couldn’t figure out where to add the methods so that they’d be defined in the right place. If I added the methods to the Organization class, they couldn’t be called on the ActiveRecord relations. A little bit of searching, and I figured out how to do it in Rails. It turns out that you can extend ActiveRecord relations with your own methods! Here’s how.

You’ll need to extend ActiveRecord::Base with your own methods. I put this in config/initializers/activerecord_extensions.rb.

module ProjectNameActiveRecordExtensions
  def filter(control)
    exclude_tutorial = true
    # Include the tutorial offers for users in tutorial mode
    if control.is_a? User and control.tutorial
      exclude_tutorial = false
    # You can also pass filter(false) to turn off these filters for testing
    elsif !control 
      exclude_tutorial = false
    end    
    if exclude_tutorial
      scoped.joins(:donation).where('donations.tutorial=?', false)
    else
      scoped
    end
  end
  # other methods go here...
end
ActiveRecord::Base.extend ProjectNameActiveRecordExtensions

The trickiest part was figuring out how to do a conditional filter, and that’s what scoped is for. I wanted to include the tutorials if the user was in tutorial mode, so my function should be a pass-through in that case. I couldn’t return self or nil, because that broke the associations. scoped turned out to be the magic keyword that refers to the current scope of the query.

What if you want to use the same words in different contexts? For example, “pending” might need to result in two different queries depending on whether you’re asking for pending offers or pending requests. ActiveRecord::Base is used for all classes, but you can use self to find out what class is being used for scoping. For example:

def pending
  if self == StandingRequest
    scoped.where("standing_requests.workflow_state=?", StandingRequest::PENDING)
  else
    # Replace with other cases as I find the need for them
    raise "Undefined behaviour"
  end
end

I love the fact that Rails lets you modify so much in order to make building sites easier. It’s like Emacs for the Web, and it makes my brain happy.

Inspired by my sister’s photo-assisted organization of her shoes, I decided to tackle my wardrobe. Taking an inventory would make it easier to simplify, replace, or supplement my clothes. Analyzing colour would help me substitute mathematics for a sense of style. Combining the images with the clothes log I’ve been keeping would make it easier to see patterns and maybe do some interesting visualizations. Geek time!

I took pictures of all my clothes against a convenient white wall. I corrected the images using Bibble 5 Pro and renamed the files to match my clothes-tracking database, creating new records as needed. AutoHotkey and Colorette made the task of choosing representative colours much less tedious than it would’ve been otherwise. After I created a spreadsheet of IDs, representative colours, and tags, I imported the data into my Rails-based personal dashboard, programming in new functionality along the way. (Emacs keyboard macros + Rails console = quick and easy data munging.) I used Acts as Taggable On for additional structure.

It turns out that the math for complementary and triadic colour schemes is easy when you convert RGB to HSL (hue, saturation, lightness). I used the Color gem for my RGB-HSL conversions, then calculated the complementary and triadic colours by adding or subtracting degrees as needed (180 for complementary, +/- 120 for triadic).

Here’s what the detailed view looks like now:

And the clothing log:

Clothing summary, sorted by frequency (30 days of data as of writing)

Thoughts:

• White balance and exposure are a little off in some shots. I tweaked some representative colours to account for that. It would be neat to get that all sorted out, and maybe drop out the background too. It’s fine the way it is. =)
• Matches are suggested based on tags, and are not yet sorted by colour. Sorting by colour or some kind of relevance factor would be extra cool.
• Sorting by hue can be tricky. Maybe there’s a better way to do this…
• My colour combinations don’t quite agree with other color scheme calculators I’ve tried. They’re in the right neighbourhood, at least. Rounding errors?
• I’ll keep an eye out for accessories that match triadic colours for the clothes I most frequently wear.
• Quick stats: 28 casual tops, 15 skirts, 12 office-type tops, 8 pairs of pants, 5 pairs of slacks – yes, there’s definitely room to trim. It would be interesting to visualize this further. Graph theory can help me figure out if there are clothing combinations that will help me simplify my wardrobe, and it might be fun to plot colours and perhaps usage. Hmm…

Other resources:

• Academichic: Fashion 101: How to combine colours: complementary colors and the two-thirds rule
• Work with Color