Tags: graph

RSS - Atom - Subscribe via email

Quantified Self: Learning from a year of time data and planning what to tweak in 2013

| quantified

Last year, I decided to move from tracking my time using off-the-shelf applications (Time Recording, then Tap Log Record) to building my own system using Ruby on Rails so that I could tailor it to my quirks. Quantified Awesome now has more than a year of time data, and I wanted to see the patterns in how I use my time.

How I categorize my time

For ease in comparison with OECD time studies, I use the following high-level categories:

  • Sleep: What it says it is. Important!
  • Discretionary: Hobbies, socializing; anything optional or chosen
  • Personal: Morning and evening routines, personal care, exercise
  • Work: Working on IBM projects
  • Business: I split this out from work because I wanted to see how much time I was spending on building my business or improving my skills
  • Unpaid work: Commuting and other unpaid work/business-related activities; also, tidying up, getting groceries, cooking, doing laundry, and any household tasks that I could theoretically outsource
  • Within the categories, I have one or two levels of detail, which I’ll discuss later.
  • image

    This graph shows the major changes in how I used time this year. To account for the varying numbers of days in a month, I’ve expressed each category as a percentage of the time available for the month. The major change was the swap between working with IBM and experimenting with running my own business, but all my other categories are surprisingly stable.

    Here are some basic statistics looking at the monthly and weekly variation. There’s a bit more variation on the weekly level, but it smoothens out a lot when it gets to the monthly level. Also, the overall numbers tell me I should probably work less and spend more time on discretionary activities.

    OECD 2011 – Canada   Mean ~ total hours / week Monthly STDEV Weekly STDEV
      Sleep 35% 58 2% 3%
    22% Business + work 28% 47 5% 6%
    21% Discretionary 16% 28 5% 7%
      Personal 14% 23 2% 4%
    14% Unpaid work 7% 12 2% 3%

    Sleep + personal for me = 49%; OECD 2011 stats for Canada: 42%

    Sleep

    I get an average of 8.3 hours of sleep per day, which is a familiar and fairly stable number, and right in line with the OECD 2011 leisure time study’s findings. Looking at the inter-day statistics for sleep, I see a standard deviation of 1.63 hours, which means my sleep pattern is a little jagged. Here’s a daily chart that shows the variation.

    image

    It doesn’t look so irregular on a weekly scale, though. I tend to be pretty good at taking it easy after I catch myself getting tired due to lack of sleep.

    image

    Business

    Business-wise, I was thrilled to have a running start. Here’s billable time as a percentage of total time (out of 7 days a week). May was a little crazy because I was helping out two clients at the same time. I took time off in September and December to focus on other interests, and I’m generally scaling back consulting because I need to make myself learn how to do other kinds of business too.

    image

    Here’s some more information in a table, showing that while I don’t reach the utilization ratios I remember from my performance review days, I still do okay.

      Billable time  
      % of total time % of business time
    Mar 2012 17% 61%
    Apr 2012 16% 71%
    May 2012 24% 76%
    Jun 2012 17% 60%
    Jul 2012 21% 66%
    Aug 2012 18% 66%
    Sep 2012 1% 5%
    Oct 2012 11% 37%
    Nov 2012 14% 42%
    Dec 2012 2% 10%

    Going forward, I should probably plan for a 25% billable : 75% marketing/overhead mix (or even more weighted towards marketing).

    On average, I spend about 10 hours a week connecting with people for business, which is a surprisingly large chunk of time. It’s good, though. I’m learning a ton and helping lots of people along the way. The weekly standard deviation for this is 7.8 hours, which probably points to “introvert overload” kicking in – after an intensely social week, I’ll hibernate for a while in order to recharge.

    Discretionary

    All work and no play makes for a boring sort of life, so this is where discretionary activities come in. Discretionary – Social is by far the juggernaut of this category, with 46% of all discretionary time use (average per week: 13.7 hours, stdev 11.9 hours – same introvert overload kicking in). Business networking + discretionary socializing works out to an average of 20.8 hours per week, with a standard deviation of 15.1 hours. Here’s the sparkline, with a spike around the September trip where I went to a conference and hung out with family.

    image

    The graph below shows that I’m not necessarily substituting business connecting for discretionary socializing. There’s actually a very slight positive correlation between them. I do need my breaks afterwards, though.

    image

    On to other things I do with my discretionary time. Because the Social sub-category is so much bigger than the other categories, these sparklines all use different vertical axes instead of using a shared axis for inter-category comparison. They show percentage of discretionary time, with the peak time highlighted. (Remember, we can’t compare heights across categories!) The third column shows the total percentage of discretionary time spent doing that activity.

    image

    The sparklines show that my interests tend to shift. They also show some categories that I’ve forgotten to use, such as Discretionary – Family which tends to get lumped under Discretionary – Social, and Discretionary – Read – Blogs, which has become more of either Personal – Routines or Unpaid work – Travel. Looking at this, I can see that LEGO games tend to give us about three months of obsession time, which may not be a good thing. Winking smile Fortunately, W- plays them too, so it’s actually “sit on the couch and chat” time, with bonus scritching of kitties who like sitting in our laps.

    Unpaid Work

    Duty comes before pleasure, though, so I need to make sure chores are taken care of before I settle in for some writing. Here’s how the chores worked out.

    image

    For scale: I spend about 3.3 hours a week cooking, which is really spending maybe 6-7 hours every other week or so cooking a whole batch of things. Or at least that’s what I think it works out to. The weekly data shows me that I tend to cook in cycles (mean = 3.5 hours, STDEV = 2.3 hours):

    image

    Other interesting things: Why, yes, biking and subway time are negatively correlated (coeff = -0.53). Yay biking! The weather’s been decent, actually, so I should totally break out the bicycle and bike some more. (Biking: 209 hours this year, average of 7.2 hours per week during biking season)

    % of total time Personal – Bike Unpaid work – Subway
    Nov 2011   2.4%
    Dec 2011   0.6%
    Jan 2012   0.9%
    Feb 2012 0.1% 1.6%
    Mar 2012 0.1% 4.5%
    Apr 2012 7.3% 0.3%
    May 2012 5.9% 0.2%
    Jun 2012 4.6% 0.3%
    Jul 2012 3.6% 0.7%
    Aug 2012 2.5% 1.2%
    Sep 2012 0.8%  
    Oct 2012 3.7% 1.1%
    Nov 2012   6.9%
    Dec 2012   3.6%

    Personal

    The personal category includes all the little things that keep life running, like having breakfast and brushing my teeth. On average, I spend 2.1 hours a day dressing up, eating, brushing my teeth, and so on. That’s 815 hours over the last 386 days! Biking, walking, and exercising account for 408 hours over that time span, which works out to be an hour a day. Not bad.

    So, what does this mean for 2013?

    I’m planning to:

    • Spend less time commuting; spend more time biking and exercising – extend biking season earlier and later (November was totally bikeable, but I chickened out and got a Metropass!), and ramp up personal exercise to ~4 hours a week.
    • Spend less time working as a whole (and yes, trying to not panic about this shift either); spend more time writing and doing other discretionary activities – keep business-related time to ~40 hours a week
    • Spend less time working on billable projects; spend more time marketing/selling/learning (and trying not to panic about this shift) – shift to 25-30% billing as a percentage of total business time

    Glad to have the numbers! You can actually see my time data on Quantified Awesome.  I’ve just added a “Split by midnight” option that makes analysis a little easier for me and other people who use the system to track their own data.

    Onward!

    Analyzing my Lotus Notes sent mail since January 2011

    | analysis, ibm, lotus

    Today is my penultimate day at IBM! Having successfully turned my projects over to another developer (hooray for the habit of organizing project-related files in Lotus Connections Activities), I’ve been focusing on getting things ready for the traditional goodbye e-mail, which I plan to send tomorrow.

    I dug around in the Lotus Connections Profiles API to see if I could get a list of my contacts’ e-mail addresses. I fixed a small bug in the feed exporter of the Community Toolkit (w4.ibm.com/community for people in the IBM intranet) and exported my contacts, giving me a list of 530 IBMers who had accepted or sent me an invitation to connect.

    Not everyone participates in that Web 2.0 network, though, so I wanted to analyze my sent mail to identify other people to whom I should send a note. I couldn’t find a neat LotusScript to do the job, and I couldn’t get the NSF to EML or mbox converter to work. Because I didn’t need all the information, just the recipients, subjects, and times, I wrote my own script (included at the end of this blog post).

    I used the script to summarize the messages in my sent mail folder, and crunched the numbers using PivotTables in Microsoft Excel. I worked with monthly batches so that it was easier to find and fix errors. I decided to analyze all the mail going back to the beginning of last year in order to identify the people I mailed the most frequently, and to come up with some easy statistics as well.

    image

    Spiky around project starts/ends, I’d guess.

    image

    I wanted to see which roles I tended to e-mail often, so I categorized each recipient with their role. I distinguished between people I’d worked with directly on projects (coworkers) and people who worked with IBM but with whom I didn’t work on a project (colleagues). The numbers below count individual recipients.

    Role

    Number of people

    Number of individual
    e-mails sent

    Average e-mails sent
    per person

    colleague 407 827 2.0
    coworker 50 562 11.2
    client 21 387 18.4
    manager 4 109 27.3
    partner 9 51 5.7
    system 9 21 2.3
    other 8 11 1.4
    self 1 5 5.0
    Grand Total 509 1973 3.9

    As it turns out, I sent a lot of mail to a lot of people throughout IBM, mostly in response to questions about Lotus Connections, Idea Labs, or collaboration tools.

    Now I can sort my summarized data to see whom I e-mailed the most often, and add more names to my don’t-forget-to-say-goodbye list. If all goes well, I might even be able to use that mail merge script. =)

    The following agent processes selected messages and creates a table with one row per recipient, e-mailing the results to the specified mail address. It seems to choke on calendar entries and other weird documents, but if you go through your sent mail box in batches (Search This View by date is handy), then you should be able to find and delete the offending entries.

    Option Public
    Dim TempNitem As NotesItem
    Dim TempNm As NotesName
    Dim session As  NotesSession
    Dim db As NotesDatabase
    Sub Initialize
    	mailAddress = "YOUR_ADDRESS@HERE"
    	
    	Dim ws As New NotesUIWorkspace
    	Dim uidoc As NotesUIDocument
    	Dim partno As String
    	Dim db As NotesDatabase
    	Dim view As NotesView
    	Dim doc As NotesDocument
    	Dim collection As NotesDocumentCollection
    	Dim memo As NotesDocument
    	Dim body As NotesRichTextItem
    	Dim range As NotesRichTextRange
    	Dim count As Integer
    	
    	Set session = New NotesSession
    	Set db = session.CurrentDatabase
    	Set collection = db.UnprocessedDocuments
    	
    	Dim FldTitles(3) As String
    	FldTitles(0) = "E-mail"
    	FldTitles(1) = "Subject"
    	FldTitles(2) = "Date sent"
    	
    	Set maildoc = db.CreateDocument
    	maildoc.Form = "Memo"
    	maildoc.Subject = "Summary"
    	maildoc.SendTo = mailAddress
    	Dim ritem As NotesRichTextItem
    	Set ritem=New NotesRichTextItem(maildoc,"body") 
    ' passing the rich text item & other relevant details
    	Set ritem = CreateTable(FldTitles, collection, ritem, "Sent items", "Summary created on " + Format(Now, "YYYY-MM-DD"))
    	maildoc.send(False)
    End Sub
    Function CreateTable(FldTitles As Variant, doccoll  As NotesDocumentCollection, rtitem As NotesRichTextItem,msgTitle As String,msgBody As String ) As NotesRichTextItem
    	'http://searchdomino.techtarget.com/tip/0,289483,sid4_gci1254682_mem1,00.html
    	'Takes  Documentcollection & creates tabular information on to the passed   rtitem (rich text item)
    	
    	Set ritem=rtitem
    	Set rtnav = ritem.CreateNavigator
    	Set rstyle=session.CreateRichTextStyle 
    	
    	'===================================================
    	'heading in the body section of the mail
    	rstyle.Bold=True
    	rstyle.NotesColor=COLOR_RED 
    	rstyle.Underline=True
    	rstyle.NotesFont=FONT_COURIER
    	rstyle.FontSize=12
    	Call  ritem.AppendStyle(rstyle)
    	ritem.AppendText(msgTitle)
    	
    	rstyle.Underline=False
    	rstyle.NotesColor=COLOR_BLACK
    	ritem.AddNewline(2)
    	
    	rstyle.FontSize=10
    	rstyle.Bold=False
    	rstyle.NotesColor=COLOR_BLACK
    	Call  ritem.AppendStyle(rstyle) 
    	ritem.AppendText(msgBody)
    	ritem.AddNewline(1)
    	
    	'===================================================
    	rows=doccoll.Count +1
    	cols=CInt(UBound(FldTitles)) 
    	
    	Call ritem.AppendTable(1, cols)
    	Dim rtt As NotesRichTextTable
    	Call rtnav.FindFirstElement(RTELEM_TYPE_TABLE)
    	Set rtt = rtNav.GetElement 
    	'=================================================
    	'heading of the table
    	rstyle.Bold=True
    	rstyle.NotesColor=COLOR_BLUE 
    	rstyle.FontSize=10
    	Call  ritem.AppendStyle(rstyle)
    	
    	For i=0 To UBound(FldTitles) - 1
    		Call rtnav.FindNextElement(RTELEM_TYPE_TABLECELL)
    		Call ritem.BeginInsert(rtnav) 
    		Call ritem.AppendText(FldTitles(i))
    		Call ritem.EndInsert
    	Next
    	
    	'=================================================
    	rstyle.FontSize=10
    	rstyle.Bold=False
    	rstyle.NotesColor=COLOR_BLACK
    	Call  ritem.AppendStyle(rstyle)
    	Dim count As Integer
    	count = 0
    	Set  doc=doccoll.GetFirstDocument
    	While Not (doc Is Nothing)
    		subject = doc.GetFirstItem("Subject").values(0)
    		posted = doc.GetFirstItem("PostedDate").values(0)
    		Set sendTo = doc.getFirstItem("SendTo")
    		For i = 0 To UBound(sendTo.values)
    			Call rtt.AddRow(1)
    			Call rtnav.FindNextElement(RTELEM_TYPE_TABLECELL)   
    			Call ritem.BeginInsert(rtnav)
    			ritem.appendText(sendTo.values(i))
    			Call ritem.EndInsert
    			Call rtnav.FindNextElement(RTELEM_TYPE_TABLECELL)   
    			Call ritem.BeginInsert(rtnav)
    			ritem.appendText(subject)
    			Call ritem.EndInsert
    			Call rtnav.FindNextElement(RTELEM_TYPE_TABLECELL)   
    			Call ritem.BeginInsert(rtnav)
    			ritem.appendText(posted)
    			Call ritem.EndInsert
    		Next   
    		count = count + 1
    		Set doc=doccoll.GetNextDocument(doc)
    	Wend
    	Set CreateTable=ritem
    	MsgBox "E-mails summarized: " & count	
    End Function

    I find it helpful to save it as the "Summarize Recipients" agent and assign it to a toolbar button that runs @Command([RunAgent]; "Summarize Recipients").

    Waking up: looking at my data

    | geek, life, quantified, sketches

    Whenever I manage to wake up early a few days in a row, I feel great about it. But I don’t do it consistently. I spend a couple of days waking up before 6 AM and enjoying a good spurt of writing, and then I find myself slipping back into later bedtimes and later wake-up times (~ 7 AM) or hitting the snooze. Clearly there are some things I still need to tweak about my system.

    Time-tracking means I’ve got a way to see what my current sleep patterns are like:

    image

    • Average sleep length when waking up before 6 AM: 7:09
    • Average sleep length when waking up after 6 AM: 8:47
    • Average sleep time for wake-up times before 6 AM: 9:45 PM, which is a bit of a stretch but is doable.

    Here are the questions I’m thinking about:

    • Is it a matter of getting to bed earlier?
    • Would it help to disable snooze entirely?
    • Is it a matter of setting my alarm clock even earlier? (Ex: Set it for 4 AM so that I eventually get out of bed at 5 AM.)
    • Would it help to set our programmable thermostat warmer in the morning, or promise myself a hot cup of tea when I get up?
    • Would it help to set my snooze interval to 5 minutes instead of 10?
    • How about if I find a way to turn my Android into a light clock? (Using Tasker to bring up a bright app, maybe…)
    • What if I give up on waking up early and instead shift to more of a night owl schedule? Advantage: can sync up with W-. I’ll need to figure out how to give my personal pursuits the creative energy they need, though.

    Hmm. More things to hack…

    Find your wall

    | life, reflection, sewing, sketches

    The Sewing Hype Cycle
    The Sewing Hype Cycle
    (Apologies to Gartner ;) )

    I like sewing because it frustrates me.

    I start optimistically enough. I pick out a pattern. I choose fabric. I tweak the pattern. I cut out pieces. I start sewing them together.

    Seams don’t quite line up. Threads break. Pins prick. I hit my lowest point: the facings are flapping about, the clothes don’t quite fit, and I’ve just sewed a seam that I have to rip out. I wonder why I put myself through this agony when I could buy better-made clothes for less than what I would spend on fabric.

    I stop and put my work away. The next day, I take the unfinished pieces out and keep going. Somehow, it turns into something that looks okay.

    I’ve never had a “flow” moment during sewing. It’s a struggle all the way to the end. That frustration is important. It’s why I do it.

    It’s a good kind of frustration. It’s not a “life is unfair” kind of frustration. It’s not a “people suck” kind of frustration. It’s the frustration of knowing that there’s something I don’t know, or something I’ve skipped, or something I haven’t figured out.

    It’s the frustration that accompanies learning things that don’t come easy to me, like a wall with hardly any handholds.

    I learn, and I learn how to deal with that frustration. I learn when frustration and fatigue push me into making mistakes. I learn the value of sleeping on it. I learn how to keep thinking about how to do things better even when I’d rather do something easier or more fun. I learn how to experiment. I learn that I can find a way over, under, around, or through things that frustrate me.

    I learn how great it feels to climb that wall.

    I learn that there’s always going to be another wall, and another, and another – and that’s okay, because the walls help me learn.

    I learn not to fear walls by trying them, just as I learned not to fear falling by intentionally doing so.

    It’s tempting to spend your time on easy escapes. Find your walls. Deal with that frustration, and keep going.

    Living in the sweet spot

    | kaizen, life, reflection, sketches

    The sweet spot is the intersection of what you’re good at, what you love doing, and what the world needs. This idea shows up in lots of career books because it’s so powerful. Find your sweet spot, and you can make great things happen.

    I’ve written about changing the world before, and it becomes more real every day. I do more and more of the things that make me happy in life and at work. This is what my current diagram looks like:

    intersections

    There are more skills I can include in these, though, but these are the most important ones.

    I love what I do, I get better and better at it, and I create value by doing what I do. How did I get to be so lucky? =)

    If you look at the posts I’ve shared on my blog through the years, you’ll notice that I frequently think about what I love doing and how I can do those things even better. Interests blossom into passions through practice and experience. The more I learn about something, the deeper I appreciate it. I share what I’ve learned at work, too. That almost always results in people finding some way to take advantage of my skills and passions, which is how I end up getting paid for all these things I love to do. If the company ever decided to phase out my group, I can see myself creating a business around these core skills.

    How did I get to this point? One idea led to another. It started with coding. I taught myself how to program in grade school. I joined competitions throughout high school and college, and I learned a lot in the process. My interest in programming led to open source software, which got me interested in Emacs and personal information management. That led me to blog, which resulted in a new interest in writing. I’d never enjoyed writing essays for English class, but I loved writing about what I was learning. This turned into public speaking when I found out that the things I was learning also interested other people. The more I learned, the more I could help people brainstorm new ideas. The more I wrote, the more I found myself connecting with others, which also helped me brainstorm. The more I wrote and connected, the more people asked me to coach them on how to do the same. I started playing around with drawing when a friend asked me to explain something, and that kicked off yet another interest. I picked up other hobbies like photography, sewing, and cooking along the way. Then I was asked to facilitate sessions on emerging technologies, and here I am. And paperwork, well, everyone has to do that. =)

    Where do I go from here? With a strong foundation like this, I can see opportunities to grow almost everywhere. I’m looking forward to improving my facilitation skills. I’m not bad at facilitation. I’m not consistently good yet, and someday, I might be. I love working on my core skills and adding new ones. I can’t wait to figure out what I’ll learn how to do next, and how I can share that with everyone!

    Digraphs with Graphviz

    | geek

    And for the geeks, here’s the Graphviz dot file that created the graph in How to do a lot. Posting here because I know I’m going to forget, and also because it’s so cool…

    digraph {
      label = "Do things that complement each other";
      subgraph {
        rank=same
        experimenting
        programming
      }
      writing
      presenting
      programming -> writing  [label="new experience"]
      experimenting -> writing [label="new experience"]
      programming -> experimenting [label="automation"]
      experimenting -> programming [label="improvements"]
      writing -> presenting [label="content,\nopportunity"]
      presenting -> writing [label="content"]
      writing -> programming [label="reflection,\nideas"]
      writing -> experimenting [label="reflection,\nideas"]
      presenting -> experimenting [label="ideas"]
      experimenting -> presenting [label="improvements"]
    }
    

    I created it with the command:

    dot -Nfontsize=10 -Efontsize=11 FILENAME -o OUTPUTFILENAME -Tpng
    

    The result:

    Directed graph

    LifeCampTO social graph

    Posted: - Modified: | connecting, social

    After LifeCampTO, I asked people to give me the list of people they wanted to talk to (or, well, those people’s primary keys ;) ). I’m still figuring out how to do a great little mail merge that reminds people of the keywords, but along the way, I thought I might I’d learn more about network visualization.

    Here’s the resulting graph: (click on it for a larger version)

    LifeCampTO social graph

    So, what does this graph say?

    You can see that most people have quite a lot of follow-up conversations ahead. It wasn’t the kind of event where most people walked away with only two or three conversations, although they might have smaller follow-up conversations with different groups of people. It might be interesting to do some cluster analysis around topics, and maybe someday I’ll figure out how to encode the data in order to make that analysis easier. ;) Based on this, our on-the-fly decision to have three big conversations turned out to have made sense, although it would also be interesting to try having small conversations about both popular and niche topics, and then having people come together at the end (or on a wiki).

    Getting to this graph (and to the individualized graphs I’ve just figured out how to produce – it highlights each person’s connections) involved a lot of bubblegum and string.

    1. I typed in the data people had written down, using OpenOffice.org to form the upper triangle of an adjacency matrix. Two people’s sheets were missing, and one person didn’t have any connections incoming or outgoing. =( Thank you, programming competitions, for all those lovely data structures.
    2. I copied the adjacency matrix and pasted it onto itself using OOo’s Paste Special – Transpose, Skip Empty Cells. This gave me a full adjacency matrix.
    3. I used a really long and hairy OOo formula to concatenate the cells into Emacs Lisp code as an associative list, with extra information and an edge list.
    4. I copied that into Emacs and processed the associative list’s edges. I needed to do that anyway in order to be able to e-mail people personalized e-mail with all of their introductions, instead of sending one e-mail per edge. Along the way, I got the idea of visualizing the network diagram, so I spun off some code to output a full edge list in DOT format for visualization with circo.
    5. I used a command like
      circo -Gsplines=true < lifecampto.dot -Tpng > lifecampto.png

      to generate the graph shown.

    6. Then I thought it would be cool to personalize the graphs, too, so I wrote some more Emacs Lisp to generate personalized DOT files that highlighted the recipient in green and the recipient’s requested links/nodes in green, too. I used a Bash for loop to turn all those personalized DOT files into PNG files.

    Example of a personalized image:

    Tomorrow, I’ll work on the mail merge. =)

    A little computer science is a dangerous, dangerous thing.