Growing as a developer: Automated tests

For this project, I put a lot of time into writing tests. Now they’re paying off. User acceptance testing and beta testing is going to be limited thanks to some real-world project constraints, so we’ll have to catch more bugs ourselves.

I want to get to 100% test coverage. That will go a long way to increasing our confidence in the results.

We had neglected our tests in the beginning, so the autogenerated tests fell out of sync. A few weeks ago, I put in around six hours to get everything back into shape.

Once that was sorted out, I looked for ways to encourage us to keep the tests working. After briefly experimenting with autotest, I settled for using git precommit hooks to force us to run tests before checking code in. Fortunately, the other developer agreed with this idea, so I set up the same script on her system.

Rspec was great for detailed tests, but I missed the expressiveness of Cucumber. After a few more hours of work, I got our Cucumber tests to work again. I wrote a number of tests from different users’ perspectives, then added Cucumber to the pre-commit hook.

The next step in our testing journey was to set up code coverage analysis. It took a little hacking to get rcov to play nicely with cucumber and rspec, but I got that to work too. Now we can get a sense of what still needs to be tested, and we have clear goals: 100% of code covered by tests, and 100% of user stories matched up with tests too.

On another project, I’ve been working on automated tests with Selenium. I hate letting the clients run into bugs, and I hate breaking things I thought I’d already fixed. I also hate clicking around and doing repetitive actions. As we run into things I’ve broken and as I change things, I write tests.

For me, test-driven development is incredibly motivating: the carrot of seeing tests pass and coverage stats go up, the stick of embarrassment when things break. =) I can take advantage of that energy and change my development habits.

My next development goals related to testing:

  • behaviour-driven development and business-user-compatible test cases
  • 100% code coverage from day one
  • integrating Selenium tests into our build scripts
  • Selenium web driver screenshots to assist in keeping user documentation up to date
  • writing automated tests for life – making progress!
  • TDD is an awesome tool, and I’m happy to read that you’re considering TDD as helpful. At some places I worked TDD is no requirement, and in fact a lot them dedicated to software development don’t even know TDD or don’t consider this approach as important. That’s a shame.

    It’s really nice to hear the company you work actually allow you to use those tools, and they seem to even embrace them. That’s übercool. Really.

    I guess you are using testing tools in a Rails environment? There are so many of them around for Rails, that when it comes to choose the best tools, you could actually spend weeks of research. Personally, I also like Rspec, and not Cucumber so much, as rspec allows you write specifications in code, whereas in Cucumber writing specifications in human-language seems redundant to me.

    “Writing automated tests for life”… I bet I will enjoy your blog post about that topic in the future! Thanks for sharing your insights. Really.

  • I love test-driven development, and hope all of my future projects involve automated tests one way or another. Not having tests is like not having version control – setting yourself up for a world of pain later! And if you’re going to write tests, you might as well have some tests up front so that you can understand the requirements and motivate yourself to check things off, right? =)

    I like both Rspec and Cucumber, even though Test::Unit and other tools are much more terse. I like Cucumber because my tests look like English, which makes them easier to write in the beginning. I like Rspec’s should-type matchers, which make it easier for me to talk through my tests. I typically start by writing high-level Cucumber tests with the @wip tag, then fleshing out additional functionality or deeper specs using Rspec.

    Automated tests for life – check out my first draft. I’m currently failing those tests: 2 overdue books (manga borrowed by J-); 36.8 hours of work in the last 7 days (last Friday was a short day for me, but I’m actually ahead of schedule this week; I should tweak the test so that it checks if I’m on track this week or over); an average of 9.04 hours of sleep over the last 7 days. Still, it’s good to have a standard, and maybe I’ll find a more visible way to hook the data into my life. My Selenium tests for checking credit card balances all pass, though, so that’s something! =)

    What kinds of tests would you like to have for your life?

  • Good job Sacha, keep it up.

    You have to know that TDD also has a dark side if done improperly and that is your application becomes less resilient to change because there is now an additional cost involved in maintaining the test scripts when there is a project change. Not really a big deal, but it needs to be costed as part of the delivery.

    Also note that once it leaves your team and gets transitioned to another group, unlike the application code, test code is rarely an important thing. Don’t get discouraged that the group flowing after you won’t maintain it. It’s their problem to deal with any issues anyway not yours.

    I have found that transitioning test maintenance skills isn’t easy, but in general transition projects are difficult, they can ask all the questions they want and you can prep everything you can think of but in the end scenario is likely (especially since you did TDD) is the people who got transitioned by the people you transitioned to will have to face the problem. (on the bright side at least it is one degree of separation from you).

  • > Not having tests is like not having version control – setting yourself up for a world of pain later!
    *shudder* A project without VC… hard for me to imagine to work on such a project nowadays.

    > Automated tests for life – check out my first draft.
    Cucumber! That’s an awesome share! :)

    > What kinds of tests would you like to have for your life?
    – Quit smoking
    – Get 8 hours of sleep
    – Don’t spend money on “things nice to have”

    I will start with those, and see how I can make that a habit. :)

  • Archie: Our high-level behavioral tests with Cucumber have been fairly forgiving of changes. I use I18n when defining steps, so we can change the wording of the interface without having to rejig our tests. If we change the way something is handled, I can change a few step definitions without rewriting lots of tests. If there are major spec changes (let’s totally change the workflow!), I think rejigging tests would be the least of our problems. ;)

    Kenny: It happens, even today! I was working on this PHP+JQuery real-time dashboard for a call center. Their server was Windows-based, so I ended up just copying files over, backing things up, and using a test directory for trying out changes. =( On another project of mine – Drupal this time – I only had FTP access to the server, so I had neither git push nor rsync even though I used git to manage my versions locally. I sorely missed my usual development workflow.