Categories: geek » development

RSS - Atom - Subscribe via email

Quantified Awesome: Squishing my excuses

| analysis, decision, development, planning, quantified

I’ve been fiddling with Quantified Awesome, this personal dashboard that I’m building so that I can keep track of what’s going on in my life and use that data to make it even more awesome. For example:

  • Tracking my time helps me make sure work doesn’t tempt me too much, and that I make time for both personal projects as well as connecting with other people. It also helps me improve my time estimates: How much time does it really take to walk to the subway station? How instant are instant noodles?
  • Tracking library books reminds me before they’re overdue, helps me collect my reading history, and gives me a greater appreciation for where my tax dollars go.
  • Tracking my clothes helps me remember to wear different types of clothes more often, makes it easier to donate items I don’t typically wear, and encourages me to try new combinations.
  • Tracking the produce we get from community-supported agriculture helps us avoid waste.
  • Tracking stuff helps me remember where infrequently-accessed items are.

It turns out that other people are interested in this too. 21 people have signed up through my “I’ll e-mail you when I figure out how to get this ready for other people” page, and my mom wants to use it too. That’s awesome!

Now I have to go ahead and actually build it so that other people can use it. That’s scary.

And like the way I deal with other scary, intimidating, procrastination-inducing things, I’m going to list my excuses here, so that I can shine a light on those assumptions and watch them scurry away like the cockroaches they are and, if necessary, squishing them with a well-applied flipflop.

  • Excuse #1: Idiosyncrasy. The way I work might be really weird, and other people may not be able to figure out what to do.
    • What’s the worst-case scenario? “I have no idea how this works!” I end up with lots of crufty special cases because I can’t figure out how to reconcile different ways of working.
    • What’s the best case? I adapt the system to the way other people work, and I get inspired by what they do. I build a lovely, flexible web app and API.
  • Excuse #2: Risk. I’m fine with loading my own data into an experimental system, but if I mess up and delete other people’s data, I’ll feel terrible. Also, they might trigger bugs.
    • What’s the worst-case scenario? Catastrophic data failure, nothing saved.
    • What’s the best case? Regular backups help me recover from any major mishaps, and careful coding avoids more common mistakes.
  • Excuse #3: Support. I’m going to spend more time handling bug reports and feature requests, and less time building little things that might be useful only for me.
    • What’s the worst-case scenario? People get annoyed and frustrated because I’m currently focused on other things, like my work.
    • What’s the best case? I get the system to become mostly usable for people, and I use my discretionary time to build more features. People’s requests inspire me to build more stuff and create more value.
  • Excuse #4: Documentation. I’ll need to write documentation, or at the very least online help. This means confronting the less-than-intuitive parts of the system. ;)
    • What’s the worst-case scenario? I describe what currently exists, get frustrated because I want to improve it, and end up cycling between updating documentation and improving the system.
    • What’s the best case? I describe what currently exists, and end up improving it along the way. I build online help into the system so that it’s easy to change. There’s a blog that helps people learn about updates, too.
  • Excuse #5: Offline access. A web-based time tracker might be of limited use if you don’t have web access often. I’ve been working on an offline HTML5 interface, but it’s still buggy.
    • What’s the worst-case scenario? Early testers try it out, but get frustrated because of the lack of offline access.
    • What’s the best case? I figure out the HTML5 offline thing. Someone else might be interested in building a native app, and we work together on fleshing out an API.
  • Excuse #6: Impatience. If I bring people on too early, they might get annoyed with a buggy system, and lose interest.
    • What’s the worst-case scenario? People give it a cursory try, and give up in annoyance.
    • What’s the best case? Early users are extraordinarily patient. We figure out a minimal viable product for each of them – the simplest thing that could possibly support what they want to do. Over time, things keep getting better and better. Also, I build a decent export interface, so even if people move on to a different system, they’ll still have their data.
  • Excuse #7: Privacy and control. A bug might accidentally expose people’s information, which is not fun. I also don’t want to have to police the system for objectionable content, considering the thumbnail uploads.
    • What’s the worst-case scenario? Someone’s private notes get accidentally published.
    • What’s the best case? People sign on knowing that I might have bugs, and don’t save any super-secret or inappropriate information on the system.

Okay. I think I can deal with that. So, what are the smallest, least-intimidating steps I need to take in order to get closer to opening up?

  • Write a quick test to make sure that people’s data will stay private. We’ll make people’s accounts private by default, although mine will stay mostly-public.
  • Make a list of things that people should be able to do right now. (Not including new functionality!) Gradually write tests to nail down that behaviour.
  • Make a list of things that people may want to do some day. Eventually set up an issue tracker.
  • Enable Devise’s invitable feature so that I can set up accounts for people easily.
  • Doublecheck backups.
  • Bring one person on. Then the next, then the next…

It will still be better than nothing, it will be a good learning experience, and participation is purely voluntary anyway.

One step at a time.

CSS theming, magnification, and foot pedals

Posted: - Modified: | development, geek, kaizen

I’m working on the stylesheets for a site, which means lots of fiddly little changes.

I decided to make all of my styling changes to my Sass source files instead of editing the attributes in Google Chrome because I found myself forgetting to copy attribute values back from Chrome. Editing the source files directly meant that the changes would be persistent – a slightly slower workflow, but a more reliable one.

I used Chrome to set selected divs to show up with display: block. This meant that I didn’t have to keep triggering hover behaviours myself. Then I used CSS Reloader to reload the stylesheet. Chrome kept my manual attribute changes, like display: block, while applying the new styles. Awesome!

I wanted a quick way to update my browser windows after I saved the file. Saving would automatically trigger Compass’ conversion of the Sass files into CSS, but the browser still used the old stylesheet until I trigged CSS Reloader with F9 or the context menu. I didn’t want to refresh the entire page because that would lose the display: block I’d manually set.

AutoHotkey to the rescue! I wrote a function that saved the current file, waited for Compass to convert the Sass file into CSS, and then used the CSS Reloader shortcut key on all open Chrome windows. This meant that I could have a translucent browser window superimposed on the design PDF for easy comparison (thanks, WinWarden), and an opaque browser window for inspection and navigation.

RefreshStylesheets() {
  Send, ^x^s
  Sleep, 2000
  SetTitleMatchMode, 2
  WinGet, id, list, Chrome
  current := WinExist("A")
  Loop, %id%
  {
    StringTrimRight, this_id, id%a_index%, 0
    WinActivate, ahk_id %this_id%
    WinWaitActive, ahk_id %this_id%,,2
    Send {F9}
  }
  WinActivate, ahk_id %current%
}

Then I mapped it to my foot pedal, just because I could. I didn’t need to take advantage of the 3-way switch, so I mapped all 6 possible functions to RefreshStylesheets.

+F1::RefreshStylesheets()
+F2::RefreshStylesheets()
+F3::RefreshStylesheets()
+F4::Send, +F1
+F5::Send, +F2
+F6::Send, +F3

To make it even easier to fine-tune tiny differences, I installed Magnifixer. I used the “Fixed” mode to magnify the translucent portion I was working on, and I moved the display next to my code. That meant that I could avoid turning my head all the time. I could simply tweak my code, nudge the pedal with my toes, glance at the display (or use peripheral vision!), get it right, and then check the overall view.

Foot pedals are fun. More fun than mapping the shortcut to something like F9 or F12, which would involve taking my fingers off home row and finding a small key. You can literally stamp out bugs.

All this put me in such a good humour that I ended up getting the homepage almost pixel-matched to the specs, except for the limitations on letter-spacing and the adjustments I made for the inconsistencies in the spec layout.

Whee! I can’t wait to use this idea when developing backend code. I’ve played around with Autotest on a Rails project, and it should be a simple matter to write a shell script running selected tests on any project. Fun!

Foot pedal + second monitor = awesome squared.

Figuring out my CSS workflow

| development, geek, kaizen, work

Yesterday’s coding session with CSS was fantastic. I used WinWarden to make my browser translucent, and I overlaid it on my reference documents. This made it a breeze to check alignment, because I didn’t have to use any measuring tools. I used Chrome’s developer tools to manually adjust the stylesheets until things looked right, adding display: block to the parts I was working with. Then I copied the numbers into my SASS file so that it could generate the CSS.

I also found a GIMP script for exporting all layers as separate images. I had to rename a few layers, but the results made it much easier to flip through images instead of toggling visibility trying to find the logos I needed. (It turned out that the logos were not included, so I’ve asked the design firm to send them to me.)

I converted the complex front page into a Drupal panel layout, getting rid of thirteen regions that were cluttering up the main block management screen. This also makes it much easier to update the content, yay! I’m looking forward to converting other pages. The previous developer used multiple regions instead of controlling visibility through configuration, so there are a lot of templates and regions.

Dual-screen worked out great, too, although I still need to fiddle a little with my ergonomics to make sure everything works out.

I’m looking forward to making this even better. I’ve only got a few more weeks on this project, but I might take on more styling in the future if it turns out I can deal with the headaches associated with cross-browser styling.

After I get the rest of the basic requirements in place, I want to automate testing and screenshots, particularly for regression-checking and for cross-browser compatibility. Selenium and WebDriver look like the way to go if I want to simulate hover events. If I can’t figure out how to use WebDriver within the time I’ll set aside for learning this, I can use JQuery to fake toggling the classes. Automated screenshots + PDF Split and Merge + ImageMagick for compositing (maybe 50% opacity?) will make it easy to spot glaring errors.

That will have to wait for next week. In the meantime, there’s a three-day weekend ahead, so I’m going to make lots of progress on Quantified Awesome. Yay!

Learning from CSS tweaking

| development, kaizen

My current project is so different from the others I’ve worked on. Instead of building logic, I’m doing front-end HTML/CSS/Javascript, working from Photoshop layers and design PDFs. I installed SASS so that I could gradually untangle the long strings of selectors my predecessor left me. Reading the code, I have a lot of sympathy for him. I imagine he felt like a fish out of water with both CSS and Drupal. I’m doing reasonably well. I’m not as fast as I am when working on Rails or Drupal back-ends, but I get stuff done. I don’t feel like I’m floundering in the land of “I don’t even know what I don’t know,” like it was when I was working with Microsoft SQL Server 2000 and IIS 5. This one I can handle. Who knows, we might even turn the project around.

I’m learning more about refactoring code, adding CSS3 styles, using Cufon for typography, and dealing with a large number of small changes. AutoHotkey scripts and Emacs macros have been amazing time-savers. On the AutoHotkey side, I’ve been taking advantage of tools like SmallMeasure and WhatColor.

Emacs makes editing lots of PHP files easy. The previous developer used dozens and dozens of node templates instead of using panels or block visibility, and there was a lot of copy-and-paste code. I started moving common parts to files that I could include, but I wanted to make sure that I didn’t accidentally overwrite something he had customized. I grepped the directory for the strings I was interested in. Then I used a keyboard macro to interactively go through each of the files and replace the common text. Win!

I’m looking forward to making my workflow even better. I’ve got a couple of weeks more in this project, and I might work on other CSS theming things in the future. (Good to encompass more of the development pipeline!) Here’s what I’m thinking of trying:

  • Use the monitor downstairs. Keep code on my laptop, and the browser/reference documents on the big screen.
  • Make my browser window translucent with WinWarden, and position it so that the browser window overlaps the reference. If I undock the Chrome developer console, that will make it even easier to work.
  • Deal with IE sooner rather than later, although IE 8 should be reasonably okay.

Looking forward to it. =)

Rails experiences: Things I learned from project O

| development, geek, kaizen, rails, work

Rails is awesome. We built a workflow/reporting system for ~120 users using Rails 3. My part of the project came to about 468 hours, or roughly 60 workdays (~ 3 months), and I worked with another developer who also put in around the same number of hours. We worked with a graphic designer, a CSS/HTML developer, a tester, a project manager, and the client, and we put together a surprisingly large set of functionality.

It’s amazing how quickly the site came together. I built a simple prototype to help the other developer get started with Rails, and we fleshed it out with the client’s input while waiting for the creative design. I started with web_app_theme so that we could have a decent-looking interface for starters. When the client approved the graphic design, another developer sliced it up into HTML, CSS, and images for us. I took those, converted them into HAML and ERB, and we were off and running. Every weekly sprint meant a chance to show off useful functionality and get feedback. It was awesome.

We were initially worried that building all the UIs from scratch in Ruby on Rails would mean taking up more time because we couldn’t use CCK or Views to quickly throw everything together. It turned out that HAML, partials, and semantic_form_for made the forms and reports easy to do. Filtering reports was straightforward with ActiveRecord and scopes. Because we built the screens ourselves, we didn’t have to fight with Views or CCK for the last 20% of tweaking, and I didn’t have to kludge any SQL queries (yay, no views_pre_execute!).

I was working on a Drupal project shortly before this, and I spent some time supporting the Drupal project during this one. Rails made my brain much happier. I felt that I could organize my code more cleanly, and I could test it more thoroughly, too. I didn’t have to fight so much with other people’s modules or themes. I like Drupal, and I’m still looking forward to doing more projects with it. But I wouldn’t mind working on more Rails projects, and I’m glad I’ve got Quantified Awesome as a personal project.

Drupal does many things better than Rails. Drupal modules tend to be more mature and better-documented, and it seems like there’s been more work on scaling Drupal. Internationalization is also more straightforward in Drupal, although Rails I18n is easy to use once you’ve gotten the hang of it. Drupal module dependencies seem a little easier to sort out, too. But Rails is fun!

Tests will keep you sane. This was the first project where we invested in developing a large suite of automated tests. We used Cucumber for high-level tests and RSpec for everything else. The tests caught many regression errors we might have otherwise missed. Test-driven development was fun, too, because the tests gave us tangible progress and simplified checking.

There were times when I gave in to the temptation to commit without running the tests, and I almost always regretted them. (Particularly after friendly finger-wagging from the other developer!)

Issue-tracking rocks. We’ve liked using Rational Team Concert in the past. Getting an externally-accessible instance was complicated, so I set up a Redmine issue tracker as soon as we started the project. We used Redmine to plan work, track bugs, and even collect feedback from the client. As of the time of writing, we had created 766 items and closed 683 of them (89%).

We started with story points, but didn’t end up continuing with them for the rest of the tasks. When we needed to prioritize, we estimated the hours required for each task in order to help the client decide. That worked out quite well. I haven’t tracked item-level time spent, but that seemed to be roughly around my estimates.

I now estimate more time than I used to, because I’ve started factoring in both writing and running tests. It’s a little strange being the pessimistic estimator instead of the optimistic one, but it’s good for the project.

Selenium is great for screenshots, too. Not only is Selenium good for automated browser-based testing of web pages, but it’s also a handy way to capture screenshots for documentation or demos.

Lotus Symphony and Microsoft Word don’t get along. We wasted a few hours trying to update the user guide with the new screenshots, only to find out that the PDF still got screwed up because I didn’t have the fonts the client used. Those fonts were part of Microsoft Office and weren’t on my system. The client took care of updating the user guide so that she could format it the way she wanted, and we focused on code.

Plan with the end in mind. Short projects mean that milestones can sneak up on you before you notice. Half-way through the project, we realized that the project end date was coming Really Soon Now, and we scrambled to put together a launch plan. We wanted to launch a few weeks before the end of the project, to give people time for feedback and updates. That meant that we needed to send pre-launch e-mails one week and two weeks before the launch, which meant… that we needed to start sending those e-mails within a week. Fortunately, the client, IBM PR, and everyone involved managed to get it all sorted out, and we launched.

I like launching. I would like to do more of it. We don’t do it nearly enough on short projects, but these projects are much more likely to launch if we’re around to help with the transition than if only the client is there.

On a related note, I get antsy about adding new functionality before the end of a project. This makes sense, of course. I don’t mind adding new reports and other reading-related functionality, but workflow tweaks are scary. When we’re planning future projects, we can consider similar risks for late-project tasks.

Things I want to improve for future projects:

  • More Rails! More!
  • More automated tests
  • Track time estimates and actual time in order to improve estimation accuracy
  • More launch planning
  • More blog posts about what I’m learning
  • Optimization

Things I want to reduce on future projects:

  • Avoid documentation formatting problems – I guess that means Microsoft Office when working with clients who use it
  • Must not give in to temptation to skip tests

Growing as a developer: Automated tests

Posted: - Modified: | development, kaizen, work

For this project, I put a lot of time into writing tests. Now they’re paying off. User acceptance testing and beta testing is going to be limited thanks to some real-world project constraints, so we’ll have to catch more bugs ourselves.

I want to get to 100% test coverage. That will go a long way to increasing our confidence in the results.

We had neglected our tests in the beginning, so the autogenerated tests fell out of sync. A few weeks ago, I put in around six hours to get everything back into shape.

Once that was sorted out, I looked for ways to encourage us to keep the tests working. After briefly experimenting with autotest, I settled for using git precommit hooks to force us to run tests before checking code in. Fortunately, the other developer agreed with this idea, so I set up the same script on her system.

Rspec was great for detailed tests, but I missed the expressiveness of Cucumber. After a few more hours of work, I got our Cucumber tests to work again. I wrote a number of tests from different users’ perspectives, then added Cucumber to the pre-commit hook.

The next step in our testing journey was to set up code coverage analysis. It took a little hacking to get rcov to play nicely with cucumber and rspec, but I got that to work too. Now we can get a sense of what still needs to be tested, and we have clear goals: 100% of code covered by tests, and 100% of user stories matched up with tests too.

On another project, I’ve been working on automated tests with Selenium. I hate letting the clients run into bugs, and I hate breaking things I thought I’d already fixed. I also hate clicking around and doing repetitive actions. As we run into things I’ve broken and as I change things, I write tests.

For me, test-driven development is incredibly motivating: the carrot of seeing tests pass and coverage stats go up, the stick of embarrassment when things break. =) I can take advantage of that energy and change my development habits.

My next development goals related to testing:

  • behaviour-driven development and business-user-compatible test cases
  • 100% code coverage from day one
  • integrating Selenium tests into our build scripts
  • Selenium web driver screenshots to assist in keeping user documentation up to date
  • writing automated tests for life – making progress!

Quantified Awesome: Thinking about what I want to build next

| development, geek, quantified

One of the best ways to make the most of those little pockets of time is to keep a someday list. What does my someday list for Quantified Awesome look like? What do I want to build next?

Emacs interface, so that I can add data and synchronize my Org-mode files: I’m split between Emacs and the Web right now. I love the customizability of Emacs, but I like the way my web app lets me easily share information. Solution: Build an API for Quantified Awesome, then integrate Emacs with it!

Android app: I’ll probably start with simple reports using the data exported from Tap Log Records, and maybe a synchronization tool that reuses the REST API I’ll be developing for Emacs integration anyway. After that, I can build my own tracking tools, which would make updating things like stuff locations much easier.

Shared memories: One of my projects is to collect and share family memories. It would be wonderful to organize these into a rough timeline and allow other people like my mom to add corrections or stories of their own. Maybe it will grow into a collaborative memory system someday!

100% test coverage: It’s my own project, so I might as well.

More graphs and statistics: Because slicing and visualizing data is fun!

Integrate my financial data, perhaps? Good to have my envelope-tracking system reflected in my personal dashboard, and maybe I can cross-reference this with time and feelings.

*Integrate automated tests for life.* Write more tests.

Build a web interface for tracking time, etc. That way, I can quickly jump over and enter a note.

Integrate my data from ThinkUp: I’ve been using ThinkUp to archive and analyze my tweets. Maybe I can overlay tweets onto my timeline and convert my current TapLogRecordsController into something more generic.

Learn how to use Mongo: Mongo’s freeform storage might make it much easier for me to stash all sorts of data in a rough timeline. Exploration needed!

So many good things to build. =)

Related: