sacha chua living an awesome life

One of the best ways to make the most of those little pockets of time is to keep a someday list. What does my someday list for Quantified Awesome look like? What do I want to build next?

Emacs interface, so that I can add data and synchronize my Org-mode files: I’m split between Emacs and the Web right now. I love the customizability of Emacs, but I like the way my web app lets me easily share information. Solution: Build an API for Quantified Awesome, then integrate Emacs with it!

Android app: I’ll probably start with simple reports using the data exported from Tap Log Records, and maybe a synchronization tool that reuses the REST API I’ll be developing for Emacs integration anyway. After that, I can build my own tracking tools, which would make updating things like stuff locations much easier.

Shared memories: One of my projects is to collect and share family memories. It would be wonderful to organize these into a rough timeline and allow other people like my mom to add corrections or stories of their own. Maybe it will grow into a collaborative memory system someday!

100% test coverage: It’s my own project, so I might as well.

More graphs and statistics: Because slicing and visualizing data is fun!

Integrate my financial data, perhaps? Good to have my envelope-tracking system reflected in my personal dashboard, and maybe I can cross-reference this with time and feelings.

*Integrate automated tests for life.* Write more tests.

Build a web interface for tracking time, etc. That way, I can quickly jump over and enter a note.

Integrate my data from ThinkUp: I’ve been using ThinkUp to archive and analyze my tweets. Maybe I can overlay tweets onto my timeline and convert my current TapLogRecordsController into something more generic.

Learn how to use Mongo: Mongo’s freeform storage might make it much easier for me to stash all sorts of data in a rough timeline. Exploration needed!

So many good things to build. =)

Related:

• Your someday list can change your life (The Simple Dollar)

How many posts did I post per month, not including this or future posts? (See the geek appendix below to find out how I got to the point of being able to run code snippets like this:)

posts = WpBlogPost.published.posts.year(2011)
posts.count(:id, :group => 'month(post_date)').sort { |a,b| a[0].to_i <=> b[0].to_i }

Result: [["1", 32], ["2", 34], ["3", 33], ["4", 33], ["5", 34], ["6", 39], ["7", 33], ["8", 33], ["9", 31], ["10", 33], ["11", 31], ["12", 8]]

This is a straightforward SQL query to write, but ActiveRecord and scopes make it more fun, and I can easily slice the data in different ways. Becuase I’ve connected Rails with my WordPress data, I can use all sorts of other gems. For example, Lingua::EN::Readability can give me text statistics. It’s not a gem, but it’s easy to install with the provided install.rb. Code tends to throw off my word count, so let’s get rid of HTML tags and anything in pre tags, then calculate some text statistics:

include ActionView::Helpers::SanitizeHelper
require 'lingua/en/readability'
# Needs lots of memory =)
post_bodies = posts.map { |x| strip_tags(x.post_content.gsub(/<pre.+?<\/pre>/m, '')) }
all_text = post_bodies.join("\n").downcase
report = Lingua::EN::Readability.new(all_text)

According to this, my writing should be readable by high school seniors, although they’ll probably have to be geeks in order to be interested in the first place.

The Readability library has other handy functions, like occurrences for finding out how frequently a word shows up in your text.

I want to get better at clear, specific descriptions. That means avoiding adjectives like ‘nice’ and hedging words like ‘really’.

Let’s look at feelings:

I recently used the N-Gram gem to analyze the text of Homestar reviews looking for recurring phrases. I suspected that one of the contractors we were considering had salted his reviews, and unusual recurring phrases or spikes in frequency might be a tip-off. I can use the same technique to identify any pet phrases of mine.

csv = FasterCSV.open('ngrams.csv', 'w')
n_gram = NGram.new(all_text, :n => [2, 3])
csv << "NGRAM 2"
n_gram.ngrams_of_all_data[2].sort { |a,b| a[1] <=> b[1] }.map { |a| csv << a };
csv << "NGRAM 3"
n_gram.ngrams_of_all_data[3].sort { |a,b| a[1] <=> b[1] }.map { |a| csv << a };
csv.close

The ten most common 3-word phrases on my blog tend to be related to planning and explaining. It figures. I can stop saying “a lot of”, though.

Some frequent two-word phrases:

Two-word phrases starting with “I’m…”

I wonder what other questions I might ask with this data…

Geek appendix: Using the Rails Console to work with WordPress data

The Rails console is awesome. You can do all sorts of things with it, like poke around your data objects or run scripts. With a little hacking, you can even use it as a smarter interface to other databases.

For example, I decided to get rid of all the syntax formatting that Org-mode tried to do with my blog posts when I published them to WordPress. Fortunately, this was the only use of span tags in my post content, so I could zap them all with a regular expression… if I could confidently do regular expressions in the MySQL console.

In the past, I might have written a Perl script to go through my database. If desperate, I might have even written a Perl script to do a regular expression replacement on my database dump file.

Rails to the rescue! I decided that since I was likely to want to use data from my WordPress blog in my Rails-based self-tracking system anyway, I might as well connect the two.

I found some code that created ActiveRecord models for WordPress posts and comments, and I modified it to connect to a different database. I added some scopes for easier queries, too.

class WpBlogPost < ActiveRecord::Base
  establish_connection Rails.configuration.database_configuration["wordpress"]

  set_table_name "wp_posts"
  set_primary_key "ID"

  has_many :comments, :class_name => "WpBlogComment", :foreign_key => "comment_post_ID"

  def self.find_by_permalink(year, month, day, title)
    find(:first,
         :conditions => ["YEAR(post_date) = ? AND MONTH(post_date) = ? AND DAYOFMONTH(post_date) = ? AND post_name = ?",
                         year.to_i, month.to_i, day.to_i, title])
  end

  scope :posts, where("post_type='post'")
  scope :published, where("post_status='publish'")
  scope :year, lambda { |year| where("year(post_date)=?", year) }
end

# http://snippets.dzone.com/posts/show/1314
class WpBlogComment < ActiveRecord::Base
  establish_connection Rails.configuration.database_configuration["wordpress"]

  # if wordpress tables live in a different database (i.e. 'wordpress') change the following
  # line to set_table_name "wordpress.wp_comments"
  # don't forget to give the db user permissions to access the wordpress db
  set_table_name "wp_comments"
  set_primary_key "comment_ID"

  belongs_to :post , :class_name => "WpBlogPost", :foreign_key => "comment_post_ID"

  validates_presence_of :comment_post_ID, :comment_author, :comment_content, :comment_author_email

  def validate_on_create
    if WpBlogPost.find(comment_post_ID).comment_status != 'open'
      errors.add_to_base('Sorry, comments are closed for this post')
    end
  end

end

I specified the database configuration in config/database.yml, and granted my user access to the tables:

wordpress:
  adapter: mysql
  encoding: utf8
  database: wordpress_database_goes_here
  username: rails_username_goes_here

After I rigged that up, I could then run this little bit of code in Rails console to clean up all those entries.

WpBlogPost.where('post_content LIKE ?', '%<span style="color:%').each do |p|
  s = p.post_content.gsub /<span style="color:[^>]+>/, ''
  s.gsub! '</span>', ''
  p.update_attributes(:post_content => s)
end

Cleaning up subscripts (accidental use of underscore without escaping):

WpBlogPost.where('post_content LIKE ?', '%<sub>%').each do |p|
  s = p.post_content.gsub /<sub>/, '_'
  s.gsub! '</sub', ''
  p.update_attributes(:post_content => s)
end

Now I can use all sorts of other ActiveRecord goodness when generating my statistics, like the code above.

From November:

We got the mail information for our Rails site, so I reconfigured the mail server. We’re doing a lot of development, and testing mail is much easier when you can send example mail addresses to one bucket of mail. Here’s how I set up the server to redirect everything addressed to @example.org to a test mail address.

First, I set the mail server up for local delivery only, and I confirmed that I could send mail from a local user to another local user account. Then I experimented with rewriting, using virtual_alias_maps to rewrite addresses to a specific account. I confirmed that worked. Then I checked the database to make sure we didn’t have live e-mail addresses, reconfigured the mail server to deliver Internet mail, and crossed my fingers. A few quick tests showed that mail was being delivered as planned – example.org mail routed to our temporary address, everything else delivered normally. We’ll see how this works!

Here’s our /etc/postfix/main.cf

smtpd_banner = $myhostname ESMTP $mail_name
biff = no
append_dot_mydomain = no
readme_directory = no
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
myhostname = domain.goes.here
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = domains.go.here, example.org
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
default_transport = smtp
relay_transport = smtp
virtual_alias_maps = hash:/etc/postfix/virtual
inet_protocols = ipv4

And /etc/postfix/virtual:

@example.org change_to_real_email@example.com

Don’t forget to run postmap /etc/postfix/virtual; /etc/init.d/postfix reload after changing /etc/postfix/virtual and your configuration.

At Rails Pub Nite the other week, Andrew Louis showed me his timestamped-based self-tracker inspired by Your Flowing Data and similar semi-structured text entry systems. He had a year of data in his system, and had built a fascinating dashboard. 

Nudged by the idea, I moved from using Time Recording to capturing timestamped data, and started using Tap Log for data entry because it lets me set up categories for quick entry.

I wanted to use Tap Log to capture the following types of data:

• activities (work, sleep, etc.) so that I can do weekly and long-term time analysis
• meals, so that I can get a sense of what I eat and when
• thoughts, so that I can jot quick ideas, keep track of things I need to follow up on, and copy information into other systems
• moods / feelings

I’ve set up my Tap Log to start with high-level categories. From this screen, I can quickly enter text or choose an activity.

The “Text” button is in yellow, which means that the next screen will have a text entry screen. “Sleep” is in red, which means that it needs no additional input – it creates a log entry, and that’s it. Entries like “Discretionary” lead to other menus, which are set up similarly.

For example, if I’m planning to write, I’d start by tapping “Discretionary”, then “Writing”, and I’d add a note about writing for my blog. 

You can have up to three levels of categories. Timestamped entries can be associated with numbers, ratings (1-5), or text. You can review log entries in the application itself, or you can export the log as a CSV and process it yourself.

The CSV will give you the following data:

• latitude, longitude, altitude, accuracy, gpstime, street, city, state, country, zip
• samples, _id
• timestamp, DayOfYear, DayOfMonth, DayOfWeek, TimeOfDay
• catOne, catTwo, catThree
• number, rating, note

For my use, I focus on _id (for updates), timestamp, catOne, catTwo, catThree, number, and note. I usually keep GPS off on my Google Nexus One in order to save battery. Besides, GPS performance on that phone isn’t very good. W- wins our GPS battles all the time thanks to his trusty N8. You might find the GPS tagging handy, though.

Because I’m interested in activity tracking, I need to infer ending timestamps from the data. Some of my records are associated with activities. Some of them capture other data, such as thoughts. Here’s the basic idea behind my code:

• Read each timestamp and copy it into my database, updating the record if it already exists
• Re-sort the records by timestamp
• Delete time records for this period
• For each timestamp in the updated period:
  • If this is an activity (based on the category)
    • Close the last activity and save it as an activity record
    • Set the last activity to the current activity

I also added some text analysis to look for notes tagged with !memory, !todo, !private, and other tags I’m starting to find handy.

Tada! http://quantifiedawesome.com/tap_log_records:

and activity graphs at http://quantifiedawesome.com/time/graph:

Observations: I like the ability to capture text and ratings quickly, and I look forward to taking advantage of that. I’ve been tracking activities just as much as I did on Time Recording, so the lack of on-phone activity summaries hasn’t gotten in my way.

What would make this even better? I’d like built-in time tracking, although that might mean that people would need to indicate which buttons correspond to activities. I’d like to have full Tasker integration so that I can automatically create entries based on different events, but I can get around that by logging the information separately and then merging it based on time. Sometimes I wish I could have four or five levels of categories, but I can use text for now. Mostly, I just need to keep adding to the analysis tools I’ve been building on Quantified Awesome: tags, activities, summaries, paging, and so on.

So that’s where I am. Let’s see where this helps me go!

One of the risks for this Rails project that I’m working on is that new users won’t have enough ramp-up time before we finish the project. We’re planning to wrap up in December, which is the end-users’ busiest time of year. The project also highly depends on external factors we can’t control, so it might be weeks or even a few months before people get a chance to try the most important parts of the application.

To make it easier for people to get started, we decided to build an interactive tutorial into the system. When people log in, the tutorial should create an offer that they can respond to, walk them through the process of working with it, and then tell them about the next steps they can take. People should be able to stop the tutorial at any time, and they should be able to start the tutorial from the beginning whenever they want a refresher.

People will be on a system that other people use and that generates reports, so all the tutorial information needs to be hidden from reporting and from people who are not in tutorial mode.

I started off by writing a Cucumber test that described how things should work: what people should see, what they could do, and so on.

To keep all the tutorial-related methods in one place, I put them in a file called tutorial_methods.rb and I included these methods in my controller. I added a conditional div to my application.html.erb that displayed the tutorial in a consistent spot if a tutorial was specified for the current page. Then I defined a function that took the current page and figured out what needed to be done for a tutorial. This function created a sample offer at the beginning of a tutorial, performed the behind-the-scenes work to approve the offer once people finished the first step, and loaded the tutorial text from the localization file into an instance variable.

I decided to use Rails’ built-in internationalization support instead of putting the tutorial in the database so that it could easily support multiple languages, although I might use a gem to support internationalization of database values if we need to.

To make things easier on the reporting side, I extended ActiveRecord::Base with my own association methods that filtered the queries depending on whether or not the user was in tutorial mode. These custom association methods made it much easier to make sure all the relevant queries were filtered.

I really liked adding an interactive tutorial to this project, and I think I’ll use that technique for Quantified Awesome as well. Online help is good, but it’s even better if people can practice on something and know it won’t mess up anything else.

One of the things I really like about Rails is the ability to add to existing classes so that your code can be cleaner. For example, in the app we’re working on, I need to be able to display a list of offers associated with an organization. I also need to filter that list of offers by different criteria. If the user is not in tutorial mode, I need to filter out any tutorial-related offers. I want to show offers with different workflow states, too.

If I had to do this in straight SQL, I would need to write many queries to cover the different cases, or write my own query-building engine that takes conditions into account. In the Drupal world, I might try to build a View with lots of arguments, and then use a views_pre_execute hook to monkey around with the generated SQL.

In the Rails world, things are much simpler. I started off by chaining queries, because you can add conditions to the end of an ActiveRelation and go from there. That gave me code that looked like this:

base = Offer.includes(:donation).where("organization_id = ? AND (donations.deadline IS null OR donations.deadline >= ?) AND (NOT (offers.workflow_state IN (?, ?, ?)))", @organization.id, Time.now, Offer::DRAFT, Offer::ALLOCATED, Offer::CONFIRMED).order('offers.deadline')
@direct_offers = base.where("offers.workflow = ?", Donation::DIRECT)
@open_offers = base.where("offers.workflow = ?", Donation::OPEN)

Then I asked myself: How can I make this code even cleaner? I thought about adding instance methods. For example, in my Organization class, I could define the following:

class Organization
  # Other stuff goes here
  def current_offers
    self.offers.includes(:donation).where("(donations.deadline IS null OR donations.deadline >= ?) AND (NOT (offers.workflow_state IN (?, ?, ?)))", Time.now, Offer::DRAFT, Offer::ALLOCATED, Offer::CONFIRMED).order('offers.deadline')
  end
  def current_offers_by_workflow(workflow)
    self.current_offers.where("offers.workflow = ?", Donation::OPEN)
  end
end

That would allow me to replace the code above with something like this:

@direct_offers = @organization.current_offers_by_workflow(Donation::DIRECT)
@open_offers = @organization.current_offers_by_workflow(Donation::OPEN)

… so if I wanted to filter out tutorial entries, I could do that in def current_offers by adding a where clause for the tutorial column.

But it seemed clunky to have to specify all these instance methods in order to filter by different ways. What I really wanted was to be able to chain my custom filters together, so that I could write code like this:

@direct_offers = @organization.offers.filter(current_user).direct
@open_offers = @organization.offers.filter(current_user).open

and then eventually be able to do things like:

list = @organization.offers.filter(current_user).current.direct.allocated

(If I really wanted to.)

I couldn’t figure out where to add the methods so that they’d be defined in the right place. If I added the methods to the Organization class, they couldn’t be called on the ActiveRecord relations. A little bit of searching, and I figured out how to do it in Rails. It turns out that you can extend ActiveRecord relations with your own methods! Here’s how.

You’ll need to extend ActiveRecord::Base with your own methods. I put this in config/initializers/activerecord_extensions.rb.

module ProjectNameActiveRecordExtensions
  def filter(control)
    exclude_tutorial = true
    # Include the tutorial offers for users in tutorial mode
    if control.is_a? User and control.tutorial
      exclude_tutorial = false
    # You can also pass filter(false) to turn off these filters for testing
    elsif !control 
      exclude_tutorial = false
    end    
    if exclude_tutorial
      scoped.joins(:donation).where('donations.tutorial=?', false)
    else
      scoped
    end
  end
  # other methods go here...
end
ActiveRecord::Base.extend ProjectNameActiveRecordExtensions

The trickiest part was figuring out how to do a conditional filter, and that’s what scoped is for. I wanted to include the tutorials if the user was in tutorial mode, so my function should be a pass-through in that case. I couldn’t return self or nil, because that broke the associations. scoped turned out to be the magic keyword that refers to the current scope of the query.

What if you want to use the same words in different contexts? For example, “pending” might need to result in two different queries depending on whether you’re asking for pending offers or pending requests. ActiveRecord::Base is used for all classes, but you can use self to find out what class is being used for scoping. For example:

def pending
  if self == StandingRequest
    scoped.where("standing_requests.workflow_state=?", StandingRequest::PENDING)
  else
    # Replace with other cases as I find the need for them
    raise "Undefined behaviour"
  end
end

I love the fact that Rails lets you modify so much in order to make building sites easier. It’s like Emacs for the Web, and it makes my brain happy.

We’d like to learn more about 3D modeling, so W- checked out Beginning Google SketchUp for 3D Printing (Sandeep Singh, Apress.

Here’s the model I made following the first tutorial in the book and 2.8 hours of learning with W-. It’s fun!

Google SketchUp has a Ruby interface. I wonder if I can use that to visualize my data…