Emacs Org Mode and the power of plain text
Posted: - Modified: | emacs, orgWhen I build a tool for myself and I want to store data, I usually use plain text. (Or maybe a spreadsheet, but now that I'm getting the hang of Org Mode tables, I'm leaning more and more towards text.)
I like the flexibility of plain text. Sometimes I want to organize my thoughts in an outline or an index. Sometimes I want to make a graph, like the way I wanted to visualize how my goals are related to each other. Sometimes I change my mind about what I want. (All the time, actually. =) ) Plain text lets me add structure the way I want to. It's all in my text editor, so I can move things around or reorganize things using the tools in Emacs.
Sure, sometimes I mess up because of formatting mistakes or the lack of validation. For example, typos in my personal ledger show up when the numbers don't match my bank balances or there's a new category with a misspelled name. But these are easy enough to catch and fix, and I can't completely guard against them with a database anyway. And it's nice to know that version control can let me visually step through the changes or recover from mistakes.
What about speed? Databases can be much faster than plain text for large quantities of data, for sure. I tend to work with pretty small quantities of data. For example, my blog index has 3257 lines, and the file that I'm drafting this in is under a megabyte. Even with whatever Emacs Lisp I've written to extract or cross-reference data, I'm still mostly bottlenecked by my brain instead of my computer. Sure, it took me a little longer to figure out how to do table calculations using Org Mode, but now that I have some notes on that, I should be able to come up with future calculations more easily. Besides, if I need to analyze things quickly, I can export and then crunch the numbers using a different tool.
Speaking of tools, staying with lightly-structured plain text lets me build a toolkit of text manipulation techniques. When I'm editing things in Emacs, scripting with Emacs Lisp, searching with grep, or writing Javascript/Ruby/Perl code to work with text, I'm developing skills that I can use in a wide range of situations.
If you're interested in keeping your data in plain text with Org Mode, here are some tips that can help you learn how to work with your information.
Start with tables
- Learn how to use keyboard shortcuts to create, move, or delete rows and columns.
- Learn how to sort tables.
- Learn how to use the column with specifier (ex:
<10>) to limit the displayed size of your column while still being able to add more information. - Use Org Mode's support for calculations to do math or perform other operations on your table.
Consider using properties
Org tables don't do well with paragraphs or more complex information, so you might want to use Org subtrees with properties.
You can use Org columns to display property values, or use Org dynamic blocks to put a summary of the values into your Org Mode file. See org-collector.el for a propview report.
If you want more control, you can work with the information using Emacs Lisp. You can use org-entry-get, org-entry-get-multivalued-property, or org-entry-get-with-inheritance to get the value of the property. If you want to go through all the subtrees (or a subset of them), use org-map-entries to call your own function at each of the matching headlines in the scope. org-heading-components will give you the information from the current heading, and you can use org-end-of-subtree to give you the boundary of the subtree if you want to process it further.
You can parse Org Mode lists with org-list-struct. I haven't dug into this deeply yet, but it looks interesting.
Parse free-form text
In addition to working with tables and properties, you can write functions that use regular expressions or other techniques to extract data from text. re-builder can be useful for visual feedback while you're figuring out the right regular expression to build. Remember, you're in Emacs, so you don't have to come up with the perfect regular expression that extracts all the data in one go. You can search for a regular expression, use a command like forward-line, save something to a variable, and so forth. Try thinking about how you would do something by hand, and then using repeat-complex-command to see what functions Emacs called when you did that.
save-restriction, narrow-to-region, and save-excursion are very useful when it comes to limiting the scope of your processing or saving your position, so check them out in the Emacs Lisp manual.
I find plain text to be really useful when I'm figuring things out (so, all the time), since I don't have to build a complex interface for working with it. As I learn more about Org Mode's features, I find myself using it for more and more of my data. Org's slogan is "Organize your life in plain text!" – and I think it just might be serious about that!