Emacs Org Mode and the power of plain text

When I build a tool for other people to use and I want to store data, I usually have to think in terms of relational databases: tables, fields, and queries. There are other kinds of databases out there, like ones with flexible documents or ones that are optimized for graphs, but I haven’t gotten the hang of them yet.

When I build a tool for myself and I want to store data, I usually use plain text. (Or maybe a spreadsheet, but now that I’m getting the hang of Org Mode tables, I’m leaning more and more towards text.)

2015-02-02 Plain text -- index card #data #organization #pkm #org

2015-02-02 Plain text – index card #data #organization #pkm #org

I like the flexibility of plain text. Sometimes I want to organize my thoughts in an outline or an index. Sometimes I want to make a graph, like the way I wanted to visualize how my goals are related to each other. Sometimes I change my mind about what I want. (All the time, actually. =) ) Plain text lets me add structure the way I want to. It’s all in my text editor, so I can move things around or reorganize things using the tools in Emacs.

Sure, sometimes I mess up because of formatting mistakes or the lack of validation. For example, typos in my personal ledger show up when the numbers don’t match my bank balances or there’s a new category with a misspelled name. But these are easy enough to catch and fix, and I can’t completely guard against them with a database anyway. And it’s nice to know that version control can let me visually step through the changes or recover from mistakes.

What about speed? Databases can be much faster than plain text for large quantities of data, for sure. I tend to work with pretty small quantities of data. For example, my blog index has 3257 lines, and the file that I’m drafting this in is under a megabyte. Even with whatever Emacs Lisp I’ve written to extract or cross-reference data, I’m still mostly bottlenecked by my brain instead of my computer. Sure, it took me a little longer to figure out how to do table calculations using Org Mode, but now that I have some notes on that, I should be able to come up with future calculations more easily. Besides, if I need to analyze things quickly, I can export and then crunch the numbers using a different tool.

Speaking of tools, staying with lightly-structured plain text lets me build a toolkit of text manipulation techniques. When I’m editing things in Emacs, scripting with Emacs Lisp, searching with grep, or writing Javascript/Ruby/Perl code to work with text, I’m developing skills that I can use in a wide range of situations.

If you’re interested in keeping your data in plain text with Org Mode, here are some tips that can help you learn how to work with your information.

Start with tables

  1. Learn how to use keyboard shortcuts to create, move, or delete rows and columns.
  2. Learn how to sort tables.
  3. Learn how to use the column with specifier (ex: <10>) to limit the displayed size of your column while still being able to add more information.
  4. Use Org Mode’s support for calculations to do math or perform other operations on your table.

Consider using properties

Org tables don’t do well with paragraphs or more complex information, so you might want to use Org subtrees with properties.

You can use Org columns to display property values, or use Org dynamic blocks to put a summary of the values into your Org Mode file. See org-collector.el for a propview report.

If you want more control, you can work with the information using Emacs Lisp. You can use org-entry-get, org-entry-get-multivalued-property, or org-entry-get-with-inheritance to get the value of the property. If you want to go through all the subtrees (or a subset of them), use org-map-entries to call your own function at each of the matching headlines in the scope. org-heading-components will give you the information from the current heading, and you can use org-end-of-subtree to give you the boundary of the subtree if you want to process it further.

You can parse Org Mode lists with org-list-struct. I haven’t dug into this deeply yet, but it looks interesting.

Parse free-form text

In addition to working with tables and properties, you can write functions that use regular expressions or other techniques to extract data from text. re-builder can be useful for visual feedback while you’re figuring out the right regular expression to build. Remember, you’re in Emacs, so you don’t have to come up with the perfect regular expression that extracts all the data in one go. You can search for a regular expression, use a command like forward-line, save something to a variable, and so forth. Try thinking about how you would do something by hand, and then using repeat-complex-command to see what functions Emacs called when you did that.

save-restriction, narrow-to-region, and save-excursion are very useful when it comes to limiting the scope of your processing or saving your position, so check them out in the Emacs Lisp manual.

I find plain text to be really useful when I’m figuring things out (so, all the time), since I don’t have to build a complex interface for working with it. As I learn more about Org Mode’s features, I find myself using it for more and more of my data. Org’s slogan is “Organize your life in plain text!” – and I think it just might be serious about that!

Getting better at writing other-directed posts

Would you like to help me get better at helping you through blogging? I’d love to hear your feedback. =)

I’m working on writing more posts that people might find useful instead of filling every week with my ruminations. One of my challenges is that posts make sense to me (of course), but I’m sure there are many people for whom the posts don’t make sense. Sometimes there are unanswered questions, or extraneous material that I could move to a separate post. Maybe the flow doesn’t make sense to other people.

How can I learn how to step outside myself and read my writing with a stranger’s eyes? One way is to ask for help: if you can share your thoughts (both content and style) on my posts through comments or e-mail, that would be great. (Consider it an open invitation!) Another way is to pay for help, especially if I can be clear about the kind of feedback I want. (More logic than typos, please!) A third way is to develop a list of questions that I can use to evaluate my own work.

Since this list of questions is useful for both paying for assistance and editing things myself, I worked on drafting this list:

  • Title
    1. Does the post title start with a verb? Can you think of a more vivid verb to start it with?
      • Bad: Do stuff
    2. Does the post title help people decide whether this post is relevant to them?
      • Good: specific problem or tip
      • Bad: generic or mysterious post title; title not clearly related to content
  • Body
    1. Is the first paragraph focused on “you” (the reader)?
    2. Does the first paragraph or two help the reader quickly decide whether this post is relevant to them?
      • Good: Can tell right away whether this will be too introductory, too advanced, or covering something they already know
    3. Do I share my background (context, etc.) in a way that helps the reader understand what I’ve learned or how I can identify with them?
    4. Do I share a useful tip that I have researched or personally experienced?
    5. Is the next step clear for the reader?
  • How it’s said
    1. In your own words, what is the key point of this post?
    2. Does each paragraph have a key point? Do the paragraphs flow logically?
    3. Does each sentence flow logically to the one following it?
    4. Is the key point sufficiently supported by the post? What’s missing?
    5. Is there anything here that does not support the key point and that can be removed?
    6. After reading this, what questions will the reader likely have? What will they want to know next?
  • Bonus
    1. Do I include research or links to other sources (not my blog)?
    2. Are there related links, and do they look interesting or useful?
    3. What else would you suggest to improve this post?

This particular post does not count as an other-directed post. It’s me trying to figure things out. =) The other-directed version of this might be called something like “Use checklists to get better at writing posts that other people will find useful” or “Pay for perspective by hiring editors for your blog”. My other-directed posts start with a verb, and I usually schedule them for Thursdays. But if you’ve got some ideas on how I can write better, I’d love to hear your suggestions in the comments. Thanks!

Related sketches:

2015-01-13 Evaluating posts written for others -- index card #writing #questions #blogging

2015.01.13 Evaluating posts written for others – index card #writing #questions #blogging

2015-01-11 What makes a good other-focused post -- index card #writing

2015.01.11 What makes a good other-focused post – index card #writing

 

Sketched Book – The Checklist Manifesto: How to Get Things Right – Atul Gawande

Atul Gawande’s The Checklist Manifesto: How to Get Things Right (2009) emphasizes the power of checklists for improving reliability. Errors creep in when we forget things entirely or skip over things we should have done. In medicine, these errors can be fatal.

Gawande draws on his experience as a surgeon, the research he conducted with the World Health Organization, and insights from construction, finance, and other industries that take advantage of checklists to improve processes.

The book discusses ways to address the cultural resistance you might encounter when introducing a checklist. It recommends making sure that checklists are precise, efficient, short, easy to use, and practical. You need to develop a culture of teamwork where people feel that they can speak up as part of a team. You may even need to modify supporting systems to make the checklist doable.

I’ve sketched the key points of the book below to make it easier to remember and share. Click on the image for a larger version that you can print if you want.

2014-12-31 Sketched Book - The Checklist Manifesto - How to Get Things Right - Atul Gawande

I like the reminders that you should design your checklists around logical “pause points,” keep checklists focused on the essentials, and treat people as smart instead of making the checklist too rigid.

The book distinguishes between “Do-Confirm” checklists, which allow experienced people to work quickly and flexibly with a confirmation step that catches errors, and “Read-Do” checklists, which walk people step-by-step through what they need to do. I’m looking forward to applying the book’s tips towards systematizing my sharing. For example, I’m working on a YASnippets in Emacs that will not only display a “Read-Do” checklist for doing these sketched notes, but will also assemble the links and code to do the steps easily. Sure, no one will die if I miss a step, but I think discipline and thoroughness might yield dividends. I also want to develop a good “Do-Confirm” process for writing and committing code; that could probably save me from quite a few embarrassing mistakes.

I’m interested in the diffusion of ideas, so I was fascinated by the book’s coverage of the eight-hospital checklist experiment the WHO conducted. The book discussed the challenges of getting other people to adopt checklists, and adapting the checklists to local conditions. Here’s an excerpt:

… By the end, 80 percent reported that the checklist was easy to use, did not take a long time to complete, and had improved the safety of care. And 78 percent actually observed the checklist to have prevented an error in the operating room.

Nonetheless, some skepticism persisted. After all, 20 percent did not find it easy to use, thought it took too long, and felt it had not improved the safety of care.

Then we asked the staff one more qusetion. “If you were having an operation,” we asked, “would you want the checklist to be used?”

A full 93 percent said yes.

There’s a comparison to be made between the reluctance of doctors to accept checklists and the committed use of checklists by pilots and builders. I came across a quote from Lewis Schiff’s Business Brilliant in this comment by Rich Wellman:

The following quote sums up the essential difference between a checklist for a doctor and a checklist for a pilot.

“How can I put this delicately? Pilots are seated in the same planes as their passengers. Surgeons are not under the same knives as their patients. To paraphrase an old joke, surgeons may be interested in safety, but pilots are committed.”

So checklists are a good idea when you’re dealing with people’s lives, but what about the rest of us? Checklists are good for catching errors and building skills. They’re also great for reducing stress and distraction, because you know that the checklist is there to help you think. That’s why packing lists are useful when you travel.

Already a fan of checklists? Tell me what you have checklists for

Want the book? You can buy it from Amazon (affiliate link).

Like this sketch? Check out sketchedbooks.com for more. Feel free to share – it’s under the Creative Commons Attribution License, like the rest of my blog. Enjoy!

Somewhat related:

Monthly review: February 2015

Last month, I wrote that in February, I was looking forward to:

  • Preparing for some projects: Done!
  • Hosting another Emacs Hangout: Done!
  • Learning more about Helm (in Emacs): Customized!

What a month for cooking, drawing, writing, sewing, and tweaking Emacs. Somehow I managed to make more progress on personal projects and do lots of consulting.

2015-03-08a Monthly review - February 2015 -- index card #monthly output

Time

Category Last month (%) This month (%) Avg h per week Delta
Business – Earn 4.1 7.8 13 3.7
Unpaid work 8.5 9.9 17 1.4
Discretionary – Productive 11.9 12.6 21 0.7
Discretionary – Social 0.5 1.2 2 0.7
Discretionary – Family 4.4 5.1 9 0.7
Sleep 35.3 35.3 59 0.0
Personal 15.1 14.7 25 -0.4
Discretionary – Play 3.8 2.8 5 -1.0
Business – Connect 5.4 2.8 5 -2.6
Business – Build 11.1 7.8 13 -3.3

As an experiment, I tried treating consulting as a “dessert” task to encourage me to work on my personal projects. I also stayed late at the office when I happened to have evening get-togethers scheduled downtown. I made decent progress on my other projects, I got more consulting in, and my clients were happy for the help. Still, my personal projects and interests are starting to be more motivating for me, so I think I might be able to rely on their intrinsic motivation soon. =)

This month, I spent some time thinking about clothes, shopping, sewing, and how I can make this experience better. It was a little stressful trying to find clothes for a family event (I ended up going in my office clothes), so I’ve resolved to try:

  • shopping more frequently
  • giving myself permission to spend more money
  • investing in learning how to sew

So far, so good. I’ve found a simple pattern for tops and am working on filling my wardrobe with them. Some people like variety. I think I get a kick out of consistency.

I’ve been cooking a lot too, yay!

I expect next month to focus even more on personal projects. I’d like to:

  • finish a few more tops in this season’s colours
  • bake a few pies for Pi Day
  • work on a few personal projects

Blog posts

Sketches

Weekly review: Week ending March 6, 2015

I decided to take it easy this week, since my energy levels were pretty low. I still managed to squeeze in going to a couple of social events, baking lots of things, levelling up in terms of personal finance, and reflecting on freedom, yay!

2015-03-07a Week ending 2015-03-06 -- index card #weekly #review output

Blog posts

Sketches

Link round-up

Focus areas and time review

  • Business (26.1h – 15%)
    • Earn (8.1h – 31% of Business)
      • Earn: E1: 1-2 days of consulting
    • Build (8.0h – 30% of Business)
      • Drawing (4.0h)
      • Delegation (0.0h)
      • Packaging (0.0h)
      • Paperwork (4.0h)
    • Connect (9.9h – 38% of Business)
  • Relationships (5.0h – 2%)
    • Co-host Quantified Self Toronto meetup
    • Go to house party
    • Work on project F
  • Discretionary – Productive (5.8h – 3%)
    • Emacs (1.0h – 0% of all)
    • Attempt to sort out 2008 again
    • Do paperwork for mom
    • Buy ETFs in SDRSP account
    • Writing (2.5h)
  • Discretionary – Play (3.4h – 2%)
  • Personal routines (39.9h – 23%)
  • Unpaid work (21.0h – 12%)
  • Sleep (66.9h – 39% – average of 9.6 per day)

Alternatives to sitting meditation: How I clear my mind

The Lifehacker article “Try a Writing Meditation If Sitting Still Isn’t Working For You” reminded me of how I’ve never quite resonated with the popular advice to clear your mind, meditate, and be mindful. When people ask me if I meditate, the word makes me think of doing yoga or sitting zazen or taking deep breaths – none of which I do. Maybe I’ll come around to those ways eventually, but in the meantime, let me share a few of the ways that work for me just in case they might help you too. =)

2015-01-16 Thinking about why I don't meditate -- index card #reason #meditation

2015-01-16 Thinking about why I don’t meditate – index card #reason #meditation

I generally keep my life low-stress. Frugality gives me a buffer from most of life’s financial stressors. Low expectations and personal responsibility make happiness easier. Stoicism helps me focus on what I can control.

I do like relaxing and being more appreciative. Everyday activities like doing the dishes, cuddling cats, spending time with W-, and following my curiosity give me that sense of abundance and fortune. They also help me slow down my thoughts and bump into interesting ideas. This reminds me of the relaxing side of meditation.

As for becoming aware of and addressing my self-talk or my thoughts, writing and drawing do a great job of bringing those thoughts out there so that I can acknowledge them or do something with them. I think this is like the self-awareness side of meditation.

Still, sometimes something perturbs my calm more than I’d like. When I’m miffed at something, that’s really more about me than about something else. It’s a good opportunity to take a look at my thoughts to see where I was lax or mistaken. This is like the clarifying side of meditation.

So I guess I do meditate, but I don’t do it in the stock-photography-meditation sort of way. Here are some other tools I use to shift my mental state:

2015-01-27 Resetting my mental state -- index card #emotions

2015-01-27 Resetting my mental state – index card #emotions

Activities that move my body or my mind make it easy for me to move my thoughts, too. For example, walking gets me breathing fresh air and looking around. Cooking immerses me in tastes and lets me enjoy doing something tangibly productive. Reading takes me inside someone else’s experiences. Helping someone shifts my focus from myself to someone else.

2015-01-25 Walking in High Park -- index card #relax

2015-01-25 Walking in High Park – index card #relax

(In particular, walking to a nearby park will almost certainly result in seeing lots of really happy dogs. There’s something about seeing a dog with a big grin and an even bigger stick.)

What is it that I’m really doing when I choose these activities? I think I’m quieting my brain enough so that I can think with less distraction. Then I can pay attention to the thoughts that I find odd or that I’d like to address, to see if I can resolve them.

In addition to responding to life as it comes, I sometimes think ahead about the way I’d like to respond to life. This is because my life has so far been pretty awesome. I don’t want to take it for granted, and I also don’t want to be blindsided by challenges. From time to time, I think about more difficult situations that I could find myself in so that I can try out different responses. This is the contemplative side of meditation, I think.

2015-01-29 Being okay with unfair -- index card #stoicism

2015-01-29 Being okay with unfair – index card #stoicism

So that’s how I “meditate,” I guess. No relaxing music, no super-deep thoughts. Mostly just everyday activities and the occasional bit of reflection. Seems to be working for me so far. =)

If you don’t meditate with a capital M, what do you enjoy doing instead? =)