Category Archives: business

What it’s like to work with data

How did I learn to work with data?

I learned the basics of SQL in high school, I think. In university, I got most of my kicks from the extracurricular projects I worked on because doing so let me hang out with interesting people. As those people graduated, I moved to handling those systems on my own. Blogging have me another reason to explore data analysis, since I was curious about my stats. Eventually, with Quantified Self, I started collecting and scraping my own data.

I do a lot of data analysis and report creation as part of my social business consulting. It has deepened my appreciation of database indexes, subqueries, common table expressions, recursive queries, caching tables, arrays, partitioned queries, string manipulation with regular expressions, and visualization tools. I’d love to get together with other social business data geeks so that we could swap analysis questions and techniques, but we’d need to get approval for sharing data or set up a sanitization protocol that my clients would be comfortable with. We’re doing some pretty cool stuff.
What is it like when my clients ask me data questions, or when I think of a question I’d like to explore?
I start by thinking of whether we have the data to answer that question, or how I can collect the data we need. I think about whether there are similar questions that are easier to answer. Then I start thinking about how to bring everything together: which tables, which joins, which conditions. Sometimes I have to use subqueries to combine the data. I’m getting into the habit of using common table expressions to make those easier to read. I feel satisfied when I can connect everything in a way that makes sense to me. I also like seeing the common threads among different questions, and turning those insights into parameterized reports.
Sometimes the first report I make fits the situation perfectly. Other times, we go back and forth a little to figure out what the real question is. I really appreciate it when other people help me sanity-check the numbers, because I occasionally overlook things. I’d like to get better at catching those errors.
Once the report settles down, I can think about the performance. Sometimes it’s as simple as adding an index or creating a table that caches complex calculations. Other times, I might need to modify the presentation or the question a little.
In addition to making my reports more reliable, I’d like to get better at visualizing the data so that people can get an intuitive feel for what’s going on.
I also want to get better at making inferences based on the data, especially when it comes to teasing out time-delayed or multivariate factors. I think my data sets are usually too small for things like that, though.
Anyway, that’s what it’s like to enjoy crunching the numbers. I love being able to do it, and I like exploring the kinds of questions that people imagine. =)

Embracing the fuzziness

I’m feeling a little more clear-headed at the moment — not as fuzzy-brained as before. Well, there’s the slight matter of my ongoing cough and congestion, so I’m not quite all the way there, but I can think more easily than I did last week.

Cycling through different mental states (normal, squirrel, fuzzy, etc.) in quick succession has been helping me get better at differentiating among them, and I’ve been thinking about how I can make the most of them. In particular, I’ve been thinking about the consequences of considering one or the other as my “default” state, or of getting rid of the notion of a default state altogether.

2015-05-03e Default or majority state and fuzziness -- index card #fuzzy

2015-05-03e Default or majority state and fuzziness – index card #fuzzy

If I think of clarity as my normal state and fuzziness as a short interruption, then when I’m sharp, it makes sense to make the most of it, and when I’m fuzzy, it makes sense to do the background work that will help me make the most of my next sharp period.

If I think of fuzziness as my normal state and clarity as a gift, then when I’m clear, it makes sense to invest a lot into building the systems, habits, and skills that would make fuzzy times even better. I’m not sure how probable this is, but I’m leaning towards it being likely, even though I tend to remember my past as clear. I’m basing this on the fact that many people around me have reported being relatively slower compared to their younger selves, possibly due to age and circumstances. On the other hand, I know a few people who are older than I am and who seem to be accelerating, so there’s something to be said for that.

A mix of both strategies seems to make sense. I can spend some time putting the infrastructure in place to do well during fuzzy times, and I can also take advantage of quick sprints to make things happen when I have a clear idea.

So that might translate into the following:

2015-05-02e When I'm sharp, when I'm fuzzy -- index card #fuzzy

2015-05-02e When I’m sharp, when I’m fuzzy – index card #fuzzy

When I’m sharp, that’s the time to lean in on health and systems and skills, the time to act on ideas and explore questions, the time to come up with lots of ideas and define tasks that I can do while fuzzy. It’s important to write, too, so that I can remember what it’s like to be sharp.

When I’m fuzzy and don’t feel particularly like pushing, that’s a good time to focus on self-care and close relationships. It’s okay to relax, to observe, to explore.

One of the nice things about being fuzzy is that it’s easier to explore alternatives and develop skills; it’s all right to do something you’re mediocre at when you feel mediocre at everything, or too fuzzy to do things you’re normally excellent at. <laugh>

And there’s always the philosophical practice at being patient and dealing with challenges… It’s good.

Hmm. What’s a good way for me to tell whether I’m under-preparing, over-preparing, or getting the balance right?

2015-05-05f Balancing preparation and action in different mental states -- index card #fuzzy #sharp

2015-05-05f Balancing preparation and action in different mental states – index card #fuzzy #sharp

  • If I under-prepare for fuzziness while I’m sharp, then when I’m fuzzy, I’m stuck with low-value activities like video games. This is okay, but with some thought, I might be able to tweak it to get more value.
  • If I over-prepare for fuzziness while I’m sharp, then I’m missing the opportunity to build momentum and get stuff done.
  • If I under-prepare for sharpness while I’m fuzzy, then when I’m sharp, I end up spinning my wheels or doing things that I could have done when I was fuzzy anyway.
  • If I over-prepare for sharpness while I’m fuzzy, I might prolong my fuzziness or feel bleah.

Hmm. It might be interesting to revisit my notes on the kinds of things I can do when I’m sharp and when I’m fuzzy (High energy and low energy activities). That might help me detect if I’m using my sharp time well.

Also, it’s okay not to totally optimize this. =) I can be a little inefficient.

There’s definitely more fuzziness in my future. It might be interesting to graph this to see when I’m majority-sharp and when I’m majority-fuzzy, but even without those patterns, it can help to start slowly thinking about how I can make this better. Hmm…

Enjoy the prep work

In sewing, the actual sewing–the stitching of layers of fabric together–seems to be such a small part of it.

There’s choosing a sewing pattern, or making one up.

There’s choosing the fabric, thinking of how it feels, drapes, stretches, handles, and washes.

There’s tracing, marking, and adjusting the pattern.

There’s cutting the fabric. Accuracy in this step makes everything else easier.

There’s marking and pinning. How you pin it together depends on how you want the finished piece to feel like.

Sometimes there’s lining, sometimes there’s interfacing. I haven’t learned how to really work with these yet.

And then – sewing! But just a little at a time, because…

There’s pressing, which is a little like ironing except with more time and pressure. This is what creates those creases and curves. It’s important to do this. If you skip it, your seams might pucker or puff. Things just don’t hang right without it.

Sometimes you trim things with scissors, or carefully rip threads and resew if needed.

And then more pinning, more sewing, more pressing, more cutting… So much of sewing is outside that narrow definition of “sewing.”

Cooking is like that too: choosing tastes, recipes, ingredients; preparing the ingredients; cooking!; adjusting the taste; serving.

What are the writing equivalents? Researching, outlining, writing, revising, packaging.

Coding? Planning, coding, testing… Maybe testing is like pressing – it helps make different pieces of code fit together neatly.

It’s like these activities have a part where you can say “This is it! I’m sewing/cooking/writing/coding!” But all the other parts might be even more important than that one bit. Hmm.

The trick, perhaps, is to enjoy those other parts as well, even if you don’t feel you’re making good progress, even if it’s not the part with the pay-off or the glamour. Doing the prep work well makes the main work more effective and enjoyable.

I wonder if life works the same way too…

Tell the difference between diminishing returns and compounding growth when it comes to investing in skills

When is it worth improving a skill you’re already good at, and when should you focus on other things?

I started thinking about this after a conversation about what it means to master the Emacs text editor. Someone wondered if the additional effort was really worth it. As I explored the question, I noticed that skills respond differently to the investment of time, and I wondered what the difference was.

For example, going from hunt-and-peck typing to touch-typing is a big difference. Instead of having to think about typing, you can focus on what you want to communicate or do. But after a certain point, getting faster at typing doesn’t give you as much of a boost in productivity. You get diminishing returns: investing into that skill yields less over time. If I type a little over 100 words per minute, retraining bad habits and figuring out other optimizations so that I can reach a rate of 150 words per minute isn’t going to make a big difference if the bottleneck is my brain. (Just in case I’m wrong about this, I’d be happy to hear from people who type that fast about whether it was worth it!)

Some skills seem shallow. There’s only so much you can gain from them before they taper off. Other skills are deeper. Let’s take writing, for instance. You can get to the point of being able to competently handwrite or type. You can fluently express yourself. But when it comes to learning how to ask questions and organize thoughts I’m not sure there’s a finish line at which you can say you’ve mastered writing. There’s always more to learn. And the more you learn, the more you can do. You get compounding growth: investing into that skill yields more over time.

I think this is part of the appeal of Emacs for me. Even after more than a decade of exploring it and writing about it, I don’t feel I’m at the point of diminishing returns. In fact, even the small habits that I’ve been focusing on building lately yield a lot of value.

No one can objectively say that a skill is shallow or deep. It depends on your goals. For example, I think of cooking as a deep skill. The more you develop your skills, the wider your possibilities are, and the more enjoyable it becomes. But if you look at it from the perspective of simply keeping yourself fueled so that you can concentrate on other things, then it makes sense to find a few simple recipes that satisfy you, or outsource it entirely by eating out.

It’s good to take a step back and ask yourself: What kind of value will you get from investing an hour into this? What about the value you would get from investing an hour in other things?

Build on your strengths where building on those strengths can make a difference. It can make a lot of sense to reach a professional level in something or inch towards becoming world-class. It could be the advantage that gets you a job, compensates for your weakness, opens up opportunities, or connects you to people. On the other hand, you might be overlearning something and wasting your time, or developing skills to a level that you don’t actually need.

When you hit that area of diminishing returns – or even that plateau of mediocrity – you can think about your strategies for moving forward. Consider:

  • What kind of return are you getting on your time? (understanding the value)
  • Is there a more effective way to learn? (decreasing your input)
  • Can you get more value out of your time from this skill or other skills? (increasing your output)
  • If you learn something else first,
    • will that make more of a difference in your life?
    • will that help you when you come back to this skill?

These questions are helping me decide that for me, learning more about colours is worthwhile, but drawing more realistic figures might not be at the moment; learning more about basic Emacs habits is better than diving into esoteric packages; and exploring questions, doing research, and trying things out is likely to be more useful than expanding my vocabulary. I’ll still flip through the dictionary every now and then, but I can focus on developing other skills.

How about you? What are you focusing on, and what helps you decide?

Related:

Different dimensions of scaling up

When I was coming up with a three-word life philosophy, learn – share – scale felt like a natural fit for me (Nov 2012). Learning and sharing were pretty straightforward. I thought of scaling in terms of sharing with more people, sharing more effectively, building tools to help people save time, connecting the dots among people and ideas, and getting better at getting better.

A recent conversation got me thinking about scale and the different dimensions that you can choose to scale along. For example, startups often talk about scaling up to millions of users; that’s one kind of scale. There’s saving people five minutes and there’s launching people into space; that’s another kind of scale.

What kinds of scale I see myself exploring? Here’s a rough categorization. (With ASCII art!)

Category Left Where I am Right
Size “This might save someone five minutes” X--------- “I’m going to help people get into space.”
People “This might help 1,000 people.” -X-------- “I want to help 1 billion people.”
Time “This might help 1,000 people over ten years.” X--------- “I want to help 1,000 people tomorrow.”
Team “I’m going to gradually develop my skills.” -X-------- “I’m going to build a team of people.”
Performance “We’ll start by doing it manually.” -X-------- “I want to get to sub-second response.”
Focus “I’m going to explore and see what comes up.” X--------- “I’m going to focus on one idea and knock it out of the park.”
Variety “I’ll put lots of things out there and people can tell me what they value.” --X------- “I’ll choose what to put out there and connect with people who need that.”
Demand “I’ll come up with the idea and find the market.” ----X----- “I’ll find the market and then come up with an idea.”
Pace “If I grow slowly and steadily, I’ll build a solid foundation.” --X------- “If I grow quickly, I’ll have momentum.”
Time/money tradeoff “I’m going to make my time more valuable.” ---------X “I’m going to make something outside the time=money equation.”
Risk “If I mess up, things are still okay.” X--------- “If I mess up, people die.”
Empowerment “I’m going to do things myself.” -------X-- “I’m going to support other people.”
Teaching “I will build systems so that I can catch fish for more people.” --------X- “I’m going to teach more people how to catch their own fish.”

Hmm. This is similar to those visions of wild success I occasionally sketch out for myself as a way to test my ideas and plans. Wild success at scaling up for me (at least along my current interests and trajectory) probably looks like:

  • Learning about a wide variety of interesting things
  • Writing, drawing, and publishing useful notes
  • Getting better at organizing them into logical chunks like books and courses so that I can help more people (including people who don’t have the patience to wade through fifty blog posts)
  • Reaching more people over time through good search and discovery in my archives
  • Getting updates to more people through subscriptions and interest-based filters

What would an Alternate Universe Sacha be like? I’d probably keep a closer eye out for problems I run into or that people I care about run into, and practise building small websites, tools, systems, and businesses to solve those problems. I might start with trying to solve a problem for ten people, then a hundred, then a thousand, then ten thousand and more. I might look for medium-sized annoyances so that it’s worth the change. I might build tools instead of or in addition to sharing my notes. (After all, The $100 Startup points out that most people don’t want to learn how to fish, they just want to eat fish for dinner and get on with the rest of their lives.)

Hmm. Alternate Universe Sacha makes sense too. Since I’m doing fine in terms of Normal Universe Sacha and scaling up here is mostly a matter of gradual accumulation, it might be interesting to experiment with Alternate Universe Sacha sometime. Maybe during the next two years of this 5-year experiment, or in a new experiment after that?

It’s good to break down a word like “scale” and figure out the different dimensions along which you can make decisions. Are you working on scaling up? If so, what kind of scale are you working towards?

Intentionally interrupting momentum and limiting flow

You know how when you get going on something, you want to keep going? It’s a great feeling. You’re in the flow, you’re in the zone. Time passes unnoticed. You’re getting stuff done.

I don’t trust that feeling. At least not all the way.

Here’s what got me thinking about this: I had just finished sketchnoting a book. It was fun. I felt accomplished. I wanted to do another sketchnote. In fact, I had already returned the previous book, picked another book from the shelf, and settled in for more drawing.

Then I stopped and asked myself, Is this really what I should be doing next? I was basking in the glow of people’s appreciation on Twitter and I already had all my tools set up for doing the next book, so it made sense to do another sketchnote. But was that really the best use of that moment?

More of the same, or something else?

I still stay up too late programming sometimes. I still spend hours reading. I still write my way past lunch, snapping out of the trance, suddenly starving, late in the afternoon. But I’m getting better at paying attention when part of me pipes up with weird questions.

I dug deeper and found these sub-questions that help me evaluate whether to continue or whether to switch, and what to do next:

  • Am I at the point of diminishing returns or temporary saturation? It’s like the way that if you’re eating your favourite food, there’s a point after which you don’t enjoy it as much. Sometimes giving it a bit of a rest lets you appreciate it more.
  • What could I be neglecting if I focus on this, both in terms of things I need to do and things I want to do? Am I better off spending time with W-, taking care of things around the house, or learning about things that don’t currently give me the same thrill?
  • Is there value in letting this simmer and blend? I can crank out a lot of similar things quickly. Or I can give myself time to learn from people’s feedback and my reflections on process, so that I improve more with each step. Sometimes different things mixed together result in interesting flavours and textures, like the difference between a purée and a stew.
  • It’s easy to do more, but what would enable me to do better? How can I step back and improve the infrastructure for future work? Infrastructure is not exciting, but it’s good to do. It helps to think about specific ways to make something better. What could better mean?
    • Faster?
    • Deeper?
    • Broader?
    • More consistent?
    • More focused?
    • More aligned?
    • More engaging?
    • With better chunking or flow?

Sure, sometimes I’ll lose myself coding or writing or drawing. But sometimes it’s good to interrupt my momentum and ask: What’s important to do, even if it’s not currently as shiny or as fun as what I’m doing?

Do you do this too? What have you learned? What questions do you ask yourself to help you decide what to do next?

Related posts: