Category Archives: quantified

Experimenting my way to an awesome life

“The question I really want to answer is: How can I live a fuller life, a happier life, a more productive life?” said someone in a recent e-mail about Quantified Self.

This made me think: The ideal life differs from person to person. What kind of awesome life am I moving towards? What motivates my choices and experiments, and how can I explore and learn more effectively?

I have role models for this, so I can imagine what it looks like. I can look at the differences between our lives to get a better understanding of the gaps and divergences.

My parents have full, happy, productive, and significant lives (although I think my mom thinks that what she’s doing isn’t as awesome or as significant as what my dad does). They make things happen. In particular, my dad touches lots of people’s lives. He has this really big scope.

W- lives a full, happy, productive life. I think he focuses on doing a good job at work, doing the right thing, knowing (and applying!) all sorts of good knowledge, and being a great husband and dad. We’re probably not going to get added to any tribal epics or history books, but that’s okay. I tend to think of his scope as smaller, more local, and he’s totally awesome within it. He sometimes reaches beyond that scope to support interesting things, like Kickstarters for well-designed products.

I think I live a decently full, happy, and productive life as well. Definitely yes to the happy bit; yay for high genetic set-points for happiness, good coping mechanisms, and a tremendous amount of luck. I keep some slack in my life, so I don’t feel like it’s super-full or super-productive. But people tell me that I do a lot, so maybe this is like the way my mom’s not as sure about her own contributions. I could probably do more, but this is as good a start as any.

My scope tends to be similar to W-‘s, focusing on our little world. But I also have these odd outgrowths for things like Emacs, visual thinking, social business, Hacklab… These aren’t as driven as my dad’s initiatives or my friends’ startups. They’re more… curiosity-based, maybe? I enjoy exploring those playgrounds and sharing what I’m learning. I think W- is like this too – he follows his curiosity into new areas.

So, if that helps me understand a little of who I am now, what does that tell me about the future Sacha I’m gradually inching towards, and what experiments can help me learn more?

I imagine Awesome Sacha to be this capable, curious person with lots of skills, including practical DIY stuff. Her equanimity and optimism lets her handle whatever life throws at her (and learn from it!). Maybe she’s more involved in the community now, helping her favourite causes, but probably more from a position of lifting people up rather than going on crusades. She takes plenty of notes and shares them, helping other people learn faster and see the connections among different ideas.

If that’s a potentially interesting Future Sacha I could become, what can I track to measure my progress along the way, and what kinds of experiments could stretch me a little bit more towards that?

  • I can pick up and practise more skills: Cooking, sewing, electronics, DIY repair, etc. I can track this through journal entries, blog posts, comfort level, and decisions to do things myself versus asking or paying someone else to do things. For example, I now feel comfortable cooking, and I remember feeling a lot more uncertain about it before. I feel moderately okay about repairing small appliances and doing simple woodworking, but could use more practice. I have hardly any experience with plumbing or tiling.
  • I can observe more, and write about more of what I’m learning: The little hiccups and challenges in my life feel so much smaller than the ones that other people go through, and I usually don’t write about them. Keeping a journal (even for the small stuff) might result in interesting reading later on, though. I already bounce back pretty quickly. It might be interesting to see how I respond to larger and larger changes, though, so deliberately taking on more commitments and more risks can help me develop this part of my life.
  • I can help out more. I think it’s okay even if I don’t try to maximize utility on this for now. I’ll start with the things that resonate with me. It’s easy enough to track hours and money for this; maybe later I can add stories too.
  • I can get better at taking, organizing, and sharing my notes. I can see the gap in my note-taking by noticing when I’m annoyed that I can’t find my old notes (either because I hadn’t written them up properly or I didn’t make them findable enough). As for organizing and sharing my notes, perhaps I can track the number of longer guides I put together, and whether I can get the hang of working with outlines and pipelines…

Most of my little experiments come from looking at ideas that are close by and saying, “Hmm, that’s interesting. Maybe I can explore that.” Sometimes it helps to look a little further ahead–to sketch out an ideal life, or even just a slightly-better-than-this life–and to plan little steps forward, going roughly in the right direction. Some ideals fit you better than others do, and some ideals just won’t resonate with you. For example, I currently don’t wish to be a highly-paid jetsetting public speaker. Thinking about this helps you figure out what kind of future you might want, and maybe figure out a few ways to try it on for size and track your progress as you grow into it.

What kind of person would Awesome You be like, and how can you inch a little closer?

When you feel like you’re spending a lot of time on low-impact activities

Alan Lin asked:

One issue I have is prioritization. I sometimes find myself spending a lot of time on low-impact activities. How do you tackle this in your life? What’s the most important thing you’re working on right now?

It’s easy to feel that most of your time is taken up with trivial things. There’s taking care of yourself and the household. There are endless tasks to check off to-do lists. There’s paperwork and overhead. Sometimes it feels like you’re making very little progress.

Here are some things I’ve learned that help me with that feeling:

  1. Understand and embrace your constraints.
  2. Lay the groundwork for action by understanding yourself.
  3. Act in tune with yourself.
  4. Accumulate gradual progress.

1. UNDERSTAND AND EMBRACE YOUR CONSTRAINTS

Many productivity and time management books seem to have the mindset where your Real Work is what matters and the rest of your life is what gets in the way. Sometimes it feels like the goal is to be able to work a clear, focused 60-hour or 120-hour week, to squeeze out every last bit of productivity from every last moment.

For me, the unproductive time that I spend snuggling with W- or the cats – that’s Real Life right there, for me, and I’m often all too aware of how short life is. The low-impact stuff is what grounds me and makes me human. As Richard Styrman points out in this comment, if other people can focus for longer, it’s because the rest of their lives don’t pull on them as much. I like the things that pull on me.

Instead of fighting your constraints, understand and embrace them. You can tweak them later, but when you make plans or evaluate yourself, do so with a realistic acceptance of the different things that pull on you. Know where you’re starting from. Then you can review commitments, get rid of ones that you’ve been keeping by default, and reaffirm the ones that you do care about. You might even find creative ways to meet your commitments with less time or effort. In any case, knowing your constraints and connecting them to the commitments behind them will make it it easier to remember and appreciate the reason why you spend time on these things.

One of my favourite ways of understanding constraints is to actually track them. Let’s look at time, for example. I know I spend a lot of my time on the general running of things. A quick summary from my time-tracking gives me this breakdown of the 744 hours in Oct 2014, a fairly typical month:

Hours Activity
255.0 sleep
126.3 consulting, because it helps me make a difference and build skills
91.9 doing other business-related things
80.5 chores and other unpaid work
86.2 taking care of myself
38.3 playing, relaxing
30.4 family-related stuff
12.6 socializing
10.3 writing, because it helps me learn and connect with great people
7.4 working on Emacs, because it helps me learn and connect with great people
1.5 gardening
1.0 reading
0.5 tracking
1.7 woodworking

Assuming that my consulting, writing, and working on Emacs are the activities that have some impact on the wider world, that’s 144 hours out of 744, or about 19% of all the time I have. This is roughly 4.5 hours a day. (And that’s a generous assumption – many of the things I write are personal reflections of uncertain value to other people.)

Even with tons of control over my schedule, I also spend lots of time on low-impact activities. And this is okay. I’m fine with that. I don’t need to turn into a value-creating machine entirely devoted to the pursuit of one clear goal. I don’t think I even can. It works for other people, but not for me. I like the time I spend cooking and helping out around the house. I like the time I spend playing with interesting ideas. I like the pace I keep.

So I’m going to start with the assumption that this is the time that I can work with instead of being frustrated with the other things that fill my life.

An average of 4.5 hours a day is a lot, even if it’s broken up into bits and pieces. It’s enough time for me to write a deep reflection, sketch one or two books, work on some code… And day after day, if I add those hours up, that can become something interesting. Of course, it would probably add up to something more impressive if I picked one thing and focused on that. But I tend to enjoy a variety of interests, so I might as well work with that instead of against it, and sometimes the combinations can be fascinating.

Accepting your constraints doesn’t mean being locked into them. You can still tweak things. For example, I experiment with time-saving techniques like bulk-cooking. But starting from the perspective of accepting your limits lets you plan more realistically and minimize frustration, which means you don’t have to waste energy on beating yourself up for not being superhuman. Know what you can work with, and work with that.

You might consider tracking your time for a week to see where your time really goes. You can track your time with pen and paper, a spreadsheet, or freely-available tools for smartphones. The important part is to track your time as you use it instead of relying on memory or perception. Our minds lie to us about constraints, often exaggerating what we’re dealing with. Collect data and find out.

2. LAY THE GROUNDWORK FOR ACTION BY UNDERSTANDING YOURSELF

When I review my constraints and commitments, I often ask myself: “Why did I commit to this? Why is this my choice?” This understanding helps me appreciate those constraints and come up with good ways to work within them.

My ideal is to almost always work on whatever I feel like working on. This sounds like a recipe for procrastination, an easy way for near-term pleasurable tasks to crowd out important but tedious ones. That’s where preparing my mind can make a big difference. If I can prepare a list of good things to do that’s in tune with my values, then I can easily choose from that list.

Here are some questions that help me prepare:

  • Why do I feel like doing various things? Is there an underlying cause or unmet need that I can address? Am I avoiding something because I don’t understand it or myself well enough? Do I only think that I want something, or do I really want it? I do a lot of this thinking and planning throughout my life, so that when those awesome hours come when everything’s lined up and I’m ready to make something, I can just go and do it.
  • Can I deliberately direct my awareness in order to change how I feel about things by emphasizing positive aspects or de-emphasizing negative ones? What can I enjoy about the things that are good for me? What can I dislike about the things that are bad for me?
  • What can I do now to make things better later? How can I take advantage of those moments when I’m focused and everything comes together? How can I make better use of normal moments? How can I make better use of the gray times too, when I’m feeling bleah?
  • How can I slowly accumulate value? How can I scale up by making things available?

I think a lot about why I want to do something, because there are often many different paths that can lead to the same results. If I catch myself procrastinating a task again and again, I ask myself if I can get rid of the task or if I can get someone else to do it. If I really need to do it myself, maybe I can transform the task into something more enjoyable. If I find myself drawn to some other task instead, I ask myself why, and I learn a little more about myself in the process.

I plan for small steps, not big leaps. Small steps sneak under my threshold for intimidation – it’s easier to find time and energy for a 15-minute task than for a 5-day slog.

I don’t worry about whether I’m working on Important things. Instead, I try to keep a list full of small, good things that take me a little bit forward. Even if I proceed at my current pace–for example, accumulating a blog post a day–in twenty years, I’ll probably be somewhere interesting.

In addition to the mental work of understanding yourself and shifting your perceptions by paying deliberate attention, it’s also good to prepare other things that can help you make the most of high-energy, high-concentration times. For example, even when I don’t feel very creative, I can still read books and outline ideas in preparation for writing. I sketch screens and plan features when I don’t feel like programming. You can probably find lots of ways you can prepare so that you can work more effectively when you want to.

2014-12-03 Motivation and understanding 3. ACT IN TUNE WITH YOURSELF

For many people, motivation seems to be about forcing yourself to do something that you had previously decided was important.

If you’ve laid the groundwork from step 2, however, you probably have a list of many good things that you can work on, so you can work on whatever you feel like working on now.

Encountering resistance? Have a little conversation with yourself. Find out what the core of it is, and see if you can find a creative way around that or work on some other small thing that moves you forward.

4. ACCUMULATE GRADUAL PROGRESS.

So now you’re doing what you want to be doing, after having prepared so that you want to do good things. But there’s still that shadow of doubt in you: “Is this going to be enough?”

It might not seem like you’re making a lot of progress, especially if you’re taking small steps on many different trails. This is where keeping track of your progress becomes really important. Celebrate those small accomplishments. Take notes. Your memory is fuzzy and will lie to you. It’s hard to see growth when you look at it day by day. If you could use your notes (or a journal, or a blog) to look back over six months or a year, though, chances are you’ll see that you’ve come a long way. And if you haven’t, don’t get frustrated; again, embrace your constraints, deepen your understanding, and keep nibbling away at what you want to do.

For me, I usually use my time to learn something, writing and drawing along the way. I’ve been blogging for the past twelve years or so. It’s incredible how those notes have helped me remember things, and how even the little things I learn can turn out to be surprisingly useful. Step by step.

So, if you’re feeling frustrated because you don’t seem to be making any progress and yet you can’t force yourself to work on the things that you’ve decided are important, try a different approach:

  1. Understand and embrace your constraints. Don’t stress out about not being 100% productive or dedicated. Accept that there will be times when you’re distracted or sick, and there will be times when you’re focused and you can do lots of good stuff. Accepting this still lets you tweak your limits, but you can do that with a spirit of loving kindness instead of frustration.
  2. Lay the groundwork for action. Mentally prepare so that it’s easier for you to want what’s good for you, and prepare other things so that when you want to work on something, you can work more effectively.
  3. Act in tune with yourself. Don’t waste energy forcing yourself through resistance. Use your preparation time to find creative ways around your blocks and come up with lots of ways you can move forward. That way, you can always choose something that’s in line with how you feel.
  4. Accumulate gradual progress. Sometimes you only feel like you’re not making any progress because you don’t see how far you’ve come. Take notes. Better yet, share those notes. Then you can see how your journey of a thousand miles is made up of all those little steps you’ve been taking – and you might even be able to help out or connect with other people along the way.

Alan has a much better summary of it, though. =)

To paraphrase, you start by examining your desires because that’s the only way to know if they’re worthwhile pursuits. This thinking prepares you and gives you with a set of things to spend time on immediately whenever you have time, and because you understand your goals & desires and the value they add to your life, you are usually satisfied with the time you do spend.

Hope that helps!

Related posts:

Thanks to Alan for nudging me to write and revise this post!

Quantified Awesome: Added sparklines and percentages

As I was answering the standard question of “Who are you and what do you do?”, I thought it might be interesting to come up with the percentages for what I actually do based on my time records. After all, I have the data. In the past, I used to export my records to a spreadsheet and do some easy number-crunching. Why bother, though, if I can program the system to do this for me?`

I ended up spending 3.5 hours adding percentages and sparklines to Quantified Awesome, updating my RSpec tests along the way. (100% coverage, yay!) Here’s what the result looks like:

2014-06-11 13_34_46-quantified awesome

The sparklines let me easily see trends and exceptions, while percentages can be easily multiplied by 168 hours to get weekly estimates or 24 hours to get daily ones. For example, sleep took up 35.8% of my time from 2012-02-17 to 2014-06-11, or an average of 8.6 hours a day. Activities directly related to earning money took up 10.9% of my time, or roughly 18.3 hours a week. The sprints and spikes are easier to see with sparklines than with tabular data, and they were easy to implement with JQuery Sparklines.

So now I can be more accurate when answering the question: “What do you do?” It doesn’t make sense to include all the minutiae. People don’t really answer that question with “Sleep,” even though that takes up much of people’s time. However, I can pick a threshold (1%?) and look at the activities above that. That is, for the ~28 months of this 5-year experiment so far, I:

  • consult (10.3%)
  • connect with people (5.0%)
  • write (3.0%)
  • draw (2.5%)
  • cook (2.3%)
  • bike (2.0%)
  • work on Emacs (1.2%) and
  • work on Quantified Awesome and other tracking tools (1%)

Data! Mwahahaha…

Hmm, maybe I’m not slacking off after all

Even though I’ve got the steady accumulation of DONE tasks showing my slow-but-constant progress, I still sometimes feel like I’m leaving something on the table when it comes to how I use my time. I feel like I’m living with a more relaxed pace, especially compared with the world of work around me or my fuzzed-by-time recollections of pre-experiment and early-experiment days.

Top line = All tasks excluding cancelled ones, bottom line = DONE

Top line = All tasks excluding cancelled ones, bottom line = DONE

I was thinking about how my time use has shifted over the past few years. I compared my percentages in different categories for 2012, 2013, and for 2014 to date. But the numbers say I’m actually spending more time on work and personal projects, and I do seem to manage to check off lots of things on my TODO list. =) So maybe I’m doing okay with this after all, even though sometimes I think I’m slacking off.

Top-level categories:

  • Sleep: Pretty consistent (34.5-36.6%) – this works out to 8.3-8.8 hours a day.
  • Business: Down, then up lately – 24%, 21%, 26%; but I expect this to be a little lower this year, since I’m taking three months off. =) I’ll probably focus on even more writing, drawing, and Emacs geekery then. (And maybe a crash course in a useful skill…)
    Avg hours per week 2012 2013 2014 to date
    Earn 20 15 17
    Build 12 13 18
    Connect 8 7 9
    Total 40 35 44
  • Discretionary: Up, then down – 18%, 22%, 16%
  • Personal care: Pretty consistent (13-14%)
  • Chores/unpaid work: Pretty consistent (7-8%)

As before, the business/discretionary trade-off is really the main thing that moves. The rest of my life stays pretty much the same. The second level of categories is worth looking at too:

  • Writing is pretty consistent at 3%, or roughly 5 hours a week. Still, I think I’d like to write more. What should get reduced? Ah, video games have been soaking up a little time – although they’re exercise too. Hmm, I could intensify that exercise so that I get more out of it. Oh! I’ve been spending more time gardening lately; that could be another reason. I like both of those alternative activities too, and I think they’ll taper off after a while. That’s okay, there’ll be time enough to write more. Besides, some of my writing is filed under Emacs-related time instead. =)
  • Trending up:
    • Drawing (2.0-4.1%): This is good.
    • Planning (0.2-1.5%): Hmm, this is interesting. Am I running into diminishing returns here? Maybe less time planning, more time experimenting.
    • Emacs (0.4-2.8%), and I’m looking forward to spending even more time on this.
    • Relaxing (0.6-2.0%)
  • Trending down:
    • Tidying up, cleaning the kitchen (2.3-1.6%) – about 3 hours a week? I should do more around the house (or maybe I am, and I’m not tracking it properly)
    • Working on Quantified Awesome (1.4-0.8%) – steady-state since I’m happy with the code so far?
    • Reading fiction (1.2-0.4%) – subsumed into other activities
    • Socializing (8.0-1.4%) – big drop here; winter, becoming more selective?
    • Networking (4.7-1.7%) – big drop here too; not networking as actively
    • Biking (2.4-0.7%) – but then it’s still early in the biking season, and I work fewer days too

I’ll continue to focus on gardening for a bit until the garden is more established. I want to exercise and bike more as well. And there’s all sorts of Emacs coolness to learn about and share! =) Writing will have to be content with these little snippets–thinking out loud, sharing what I learn, and other things like that–until I can spend more time focusing on developing ideas. Mostly, the increase in time on other activities seems to be coming from the time I used to spend socializing. I actually like this new balance. The stuff I make and share online seems to lead to more ongoing conversations than those hi-hellos at tech events, and I’m still happy to spend a few hours getting to know people or going somewhere.

I got the time numbers from http://quantifiedawesome.com and a bit of spreadsheet number-crunching, and the task numbers from Emacs + Org Mode + R. =) Yay data!

Update on time tracking with Quantified Awesome and with Emacs

With another Quantified Self Toronto meetup in a few weeks and a conversation with fellow self-trackers, it’s time for me to think about time again.

I’ve been fixing bugs and adding small pieces of functionality to Quantified Awesome, and I spent some time improving the integration with Emacs. Now I can type ! to clock in on a task and update Quantified Awesome. Completing the task clocks me out in Emacs and updates Beeminder if appropriate. (I don’t update Quantified Awesome when finishing a task, because I just clock into the next activity.) This allows me to take advantage of Org’s clock reports for project and task-level time, at least for discretionary projects that involve my computer. I’m not going to get full coverage, but that’s what Quantified Awesome’s web interface is for. It takes very little effort to track things now, if I’m working off my to-do list. Even if I’m not, it still takes just a few taps on my phone to switch activities.

Most of my data is still medium-level, since I’m still getting the hang of sorting out my time in Emacs. Looking at data from 2014 so far, dropping partial weeks, and doing the analysis on April 14 (which is when I’m drafting this), here’s what I’ve been finding.

  • I sleep a little more than I used to: an average of 8.9 hours a day, or 37% of the time. This is up from 8.3 hours last year.
  • It takes me about an hour to get ready in the mornings. If I have a quick breakfast instead of having rice and fried egg, I can get out the door in 30-45 minutes.
  • It takes me 50-60 minutes to get downtown, whether this is by transit or bicycle. Commuting takes 3% of my time.

Little surprises:

  • I’ve spent almost twice as much time on business building or discretionary productive activities (19%) as I have earning (11%) – good to see decisions in action!
  • I’ve spent more time drawing than writing this year (5% vs 3%). Next to writing, Emacs is the productive discretionary activity I spend most of my time on (2%).
  • I’ve spent 10% of my time this year on connecting with people, a surprisingly high number for me. E-mail takes 1% of my overall time.
  • It turns out that yes, coding and drawing are negatively correlated (-0.63 considering all coding-related activities). But writing and drawing are positively correlated (0.44), which makes sense – I draw, and then I write a blog post to glue sketches together and give context. Earning is slightly negatively correlated with building business/skills (-0.15), but connecting is even more negatively correlated with time spent building business/skills (-0.35). So it’s probably not that consulting takes me away from building skills. Sleep is slightly negatively correlated with all records related to socializing (-0.14), but strongly negatively correlated with productive discretionary activities (-0.55). Hmm. Something to tinker with.

Some things I’m learning from tracking time on specific tasks:

  • Outlining doubles the time I take to write (and drops me from about ~30wpm to about 9wpm), but I feel that it makes things more structured.
  • Drawing takes longer too, but it makes blog posts more interesting.
  • Trying to dictate posts takes me way more time than outlining or typing it, since I’m not as used to organizing my thoughts that way.
  • Encoding litter box data takes me about a minute per data point. So spending a lot of time trying to figure out computer vision and image processing in order to partially automate the process doesn’t strictly make sense, but I’m doing it out of curiosity.
  • I generally overestimate the time I need for programming-related tasks, which is surprising. That could just be me padding my estimates to account for distractions or to make myself feel great, though.
  • I generally underestimate the time I need to write, especially if I’m figuring things out along the way.

This post took me 1:20 to draft (including data analysis), although to be fair, part of that involved a detour checking electricity use for an unrelated question. =)

Getting R and ggplot2 to work in Emacs Org Mode Babel blocks; also, tracking the number of TODOs

I started tracking the number of tasks I had in Org Mode so that I could find out if my TODO list tended to shrink or grow. It was easy to write a function in Emacs Lisp to count the number of tasks in different states and summarize them in a table.

(defun sacha/org-count-tasks-by-status ()
  (interactive)
  (let ((counts (make-hash-table :test 'equal))
        (today (format-time-string "%Y-%m-%d" (current-time)))
        values output)
    (org-map-entries
     (lambda ()
       (let* ((status (elt (org-heading-components) 2)))
         (when status
           (puthash status (1+ (or (gethash status counts) 0)) counts))))
     nil
     'agenda)
    (setq values (mapcar (lambda (x)
                           (or (gethash x counts) 0))
                         '("DONE" "STARTED" "TODO" "WAITING" "DELEGATED" "CANCELLED" "SOMEDAY")))
    (setq output
          (concat "| " today " | "
                  (mapconcat 'number-to-string values " | ")
                  " | "
                  (number-to-string (apply '+ values))
                  " | "
                  (number-to-string
                   (round (/ (* 100.0 (car values)) (apply '+ values))))
                  "% |"))
    (if (called-interactively-p 'any)
        (insert output)
      output)))
(sacha/org-count-tasks-by-status)

I ran this code over several days. Here are my results as of 2014-05-01:

Date DONE START. TODO WAIT. DELEG. CANC. SOMEDAY Total % done + done +canc. + total + t – d – c Note
2014-04-16 1104 1 403 3 1 104 35 1651 67%
2014-04-17 1257 0 114 4 1 171 107 1654 76% 153 67 3 -217 Lots of trimming
2014-04-18 1292 0 74 4 5 183 100 1658 78% 35 12 4 -43 A little bit more trimming
2014-04-20 1305 0 80 4 5 183 100 1677 78% 13 0 19 6
2014-04-21 1311 1 78 4 4 184 99 1681 78% 6 1 4 -3
2014-04-22 1313 2 75 4 4 184 99 1681 78% 2 0 0 -2
2014-04-23 1369 4 66 4 5 186 101 1735 79% 56 2 54 -4 Added sharing/index.org
2014-04-24 1371 3 69 4 5 186 101 1739 79% 2 0 4 2
2014-04-25 1379 3 60 3 5 189 103 1742 79% 8 3 3 -8
2014-04-26 1384 3 65 3 5 192 103 1755 79% 5 3 13 5
2014-04-27 1389 2 66 3 5 192 103 1760 79% 5 0 5 0
2014-04-28 1396 3 67 3 5 192 103 1769 79% 7 0 9 2
2014-04-29 1396 3 67 3 5 192 103 1769 79% 0 0 0 0
2014-04-30 1404 4 70 4 5 192 103 1782 79% 8 0 13 5
2014-05-01 1413 4 80 3 4 193 103 1800 79% 9 1 18 8

Here’s the source for that table:

#+NAME: burndown
#+RESULTS:
|       Date | DONE | START. | TODO | WAIT. | DELEG. | CANC. | SOMEDAY | Total | % done | + done | +canc. | + total | + t - d - c | Note                       |
|------------+------+--------+------+-------+--------+-------+---------+-------+--------+--------+--------+---------+-------------+----------------------------|
| 2014-04-16 | 1104 |      1 |  403 |     3 |      1 |   104 |      35 |  1651 |    67% |        |        |         |             |                            |
| 2014-04-17 | 1257 |      0 |  114 |     4 |      1 |   171 |     107 |  1654 |    76% |    153 |     67 |       3 |        -217 | Lots of trimming           |
| 2014-04-18 | 1292 |      0 |   74 |     4 |      5 |   183 |     100 |  1658 |    78% |     35 |     12 |       4 |         -43 | A little bit more trimming |
| 2014-04-20 | 1305 |      0 |   80 |     4 |      5 |   183 |     100 |  1677 |    78% |     13 |      0 |      19 |           6 |                            |
| 2014-04-21 | 1311 |      1 |   78 |     4 |      4 |   184 |      99 |  1681 |    78% |      6 |      1 |       4 |          -3 |                            |
| 2014-04-22 | 1313 |      2 |   75 |     4 |      4 |   184 |      99 |  1681 |    78% |      2 |      0 |       0 |          -2 |                            |
| 2014-04-23 | 1369 |      4 |   66 |     4 |      5 |   186 |     101 |  1735 |    79% |     56 |      2 |      54 |          -4 | Added sharing/index.org    |
| 2014-04-24 | 1371 |      3 |   69 |     4 |      5 |   186 |     101 |  1739 |    79% |      2 |      0 |       4 |           2 |                            |
| 2014-04-25 | 1379 |      3 |   60 |     3 |      5 |   189 |     103 |  1742 |    79% |      8 |      3 |       3 |          -8 |                            |
| 2014-04-26 | 1384 |      3 |   65 |     3 |      5 |   192 |     103 |  1755 |    79% |      5 |      3 |      13 |           5 |                            |
| 2014-04-27 | 1389 |      2 |   66 |     3 |      5 |   192 |     103 |  1760 |    79% |      5 |      0 |       5 |           0 |                            |
| 2014-04-28 | 1396 |      3 |   67 |     3 |      5 |   192 |     103 |  1769 |    79% |      7 |      0 |       9 |           2 |                            |
| 2014-04-29 | 1396 |      3 |   67 |     3 |      5 |   192 |     103 |  1769 |    79% |      0 |      0 |       0 |           0 |                            |
| 2014-04-30 | 1404 |      4 |   70 |     4 |      5 |   192 |     103 |  1782 |    79% |      8 |      0 |      13 |           5 |                            |
| 2014-05-01 | 1413 |      4 |   80 |     3 |      4 |   193 |     103 |  1800 |    79% |      9 |      1 |      18 |           8 |                            |
#+TBLFM: @3$11..@>$11=$2-@-1$2::@3$13..@>$13=$9-@-1$9::@3$14..@>$14=$13-$11-($7-@-1$7)::@3$12..@>$12=$7-@-1$7

I wanted to graph this with Gnuplot, but it turns out that Gnuplot is difficult to integrate with Emacs on Microsoft Windows. I gave up after a half an hour of poking at it, since search results indicated there were long-standing problems with how Gnuplot got input from Emacs. Besides, I’d been meaning to learn more R anyway, and R is more powerful when it comes to statistics and data visualization.

Getting R to work with Org Mode babel blocks in Emacs on Windows was a challenge. Here are some of the things I ran into.

The first step was easy: Add R to the list of languages I could evaluate in a source block (I already had dot and ditaa from previous experiments).

(org-babel-do-load-languages
 'org-babel-load-languages
 '((dot . t)
   (ditaa . t) 
   (R . t)))

But my code didn’t execute at all, even when I was trying something that printed out results instead of drawing images. I got a little lost trying to dig into org-babel-execute:R with edebug, eventually ending up in comint.el. The real solution was even easier. I had incorrectly set inferior-R-program-name to the path of R in my configuration, which made M-x R work but which meant that Emacs was looking in the wrong place for the options to pass to R (which Org Babel relied on). The correct way to do this is to leave inferior-R-program-name with the default value (Rterm) and make sure that my system path included both the bin directory and the bin\x64 directory.

Then I had to pick up the basics of R again. It took me a little time to figure out that I needed to parse the columns I pulled in from Org, using strptime to convert the date column and as.numeric to convert the numbers. Eventually, I got it to plot some results with the regular plot command.

dates <- strptime(as.character(data$Date), "%Y-%m-%d")
tasks_done <- as.numeric(data$DONE)
tasks_uncancelled <- as.numeric(data$Total) - as.numeric(data$CANC.)
df <- data.frame(dates, tasks_done, tasks_uncancelled)
plot(x=dates, y=tasks_uncancelled, ylim=c(0,max(tasks_uncancelled)))
lines(x=dates, y=tasks_uncancelled, col="blue", type="o")
lines(x=dates, y=tasks_done, col="green", type="o")

r-plot

I wanted prettier graphs, though. I installed the ggplot2 package and started figuring it out. No matter what I did, though, I ended up with a blank white image instead of my graph. If I used M-x R instead of evaluating the src block, the code worked. Weird! Eventually I found out that adding print(...) around my ggplot made it display the image correctly. Yay! Now I had what I wanted.

library(ggplot2)
dates <- strptime(as.character(data$Date), "%Y-%m-%d")
tasks_done <- as.numeric(data$DONE)
tasks_uncancelled <- as.numeric(data$Total) - as.numeric(data$CANC.)
df <- data.frame(dates, tasks_done, tasks_uncancelled)
plot = ggplot(data=df, aes(x=dates, y=tasks_done, ymin=0)) + geom_line(color="#009900") + geom_point() + geom_line(aes(y=tasks_uncancelled), color="blue") + geom_point(aes(y=tasks_uncancelled))
print(plot)

 r-graph

The blue line represents the total number of tasks (except for the cancelled ones), and the green line represents tasks that are done.

Here’s something that looks a little more like a burn down chart, since it shows just the number of things to be done:

library(ggplot2)
dates <- strptime(as.character(data$Date), "%Y-%m-%d")
tasks_remaining <- as.numeric(data$Total) - as.numeric(data$CANC.) - as.numeric(data$DONE)
df <- data.frame(dates, tasks_remaining)
plot = ggplot(data=df, aes(x=dates, y=tasks_remaining, ymin=0)) + geom_line(color="#009900") + geom_point()
print(plot)

r-graph-2

The drastic decline there is me realizing that I had lots of tasks that were no longer relevant, not me being super-productive. =)

As it turns out, I tend to add new tasks at about the rate that I finish them (or slightly more). I think this is okay. It means I’m working on things that have next steps, and next steps, and steps beyond that. If I add more tasks, that gives me more variety to choose from. Besides, I have a lot of repetitive tasks, so those never get marked as DONE over here.

Anyway, cool! Now that I’ve gotten R to work on my system, you’ll probably see it in even more of these blog posts. =D Hooray for Org Babel and R!

Update 2014-05-09: Stephen suggested http://blogs.neuwirth.priv.at/software/2012/03/28/r-and-emacs-with-org-mode/ for more tips on setting up Org Mode with R and Emacs Speaks Statistics (ESS).