Emacs Org Mode and the power of plain text
Posted: - Modified: | emacs, orgWhen I build a tool for other people to use and I want to store data, I usually have to think in terms of relational databases: tables, fields, and queries. There are other kinds of databases out there, like ones with flexible documents or ones that are optimized for graphs, but I haven't gotten the hang of them yet.
When I build a tool for myself and I want to store data, I usually use plain text. (Or maybe a spreadsheet, but now that I'm getting the hang of Org Mode tables, I'm leaning more and more towards text.)
I like the flexibility of plain text. Sometimes I want to organize my thoughts in an outline or an index. Sometimes I want to make a graph, like the way I wanted to visualize how my goals are related to each other. Sometimes I change my mind about what I want. (All the time, actually. =) ) Plain text lets me add structure the way I want to. It's all in my text editor, so I can move things around or reorganize things using the tools in Emacs.
Sure, sometimes I mess up because of formatting mistakes or the lack of validation. For example, typos in my personal ledger show up when the numbers don't match my bank balances or there's a new category with a misspelled name. But these are easy enough to catch and fix, and I can't completely guard against them with a database anyway. And it's nice to know that version control can let me visually step through the changes or recover from mistakes.
What about speed? Databases can be much faster than plain text for large quantities of data, for sure. I tend to work with pretty small quantities of data. For example, my blog index has 3257 lines, and the file that I'm drafting this in is under a megabyte. Even with whatever Emacs Lisp I've written to extract or cross-reference data, I'm still mostly bottlenecked by my brain instead of my computer. Sure, it took me a little longer to figure out how to do table calculations using Org Mode, but now that I have some notes on that, I should be able to come up with future calculations more easily. Besides, if I need to analyze things quickly, I can export and then crunch the numbers using a different tool.
Speaking of tools, staying with lightly-structured plain text lets me build a toolkit of text manipulation techniques. When I'm editing things in Emacs, scripting with Emacs Lisp, searching with grep, or writing Javascript/Ruby/Perl code to work with text, I'm developing skills that I can use in a wide range of situations.
If you're interested in keeping your data in plain text with Org Mode, here are some tips that can help you learn how to work with your information.
Start with tables
- Learn how to use keyboard shortcuts to create, move, or delete rows and columns.
- Learn how to sort tables.
- Learn how to use the column with specifier (ex:
<10>) to limit the displayed size of your column while still being able to add more information. - Use Org Mode's support for calculations to do math or perform other operations on your table.
Consider using properties
Org tables don't do well with paragraphs or more complex information, so you might want to use Org subtrees with properties.
You can use Org columns to display property values, or use Org dynamic blocks to put a summary of the values into your Org Mode file. See org-collector.el for a propview report.
If you want more control, you can work with the information using Emacs Lisp. You can use org-entry-get, org-entry-get-multivalued-property, or org-entry-get-with-inheritance to get the value of the property. If you want to go through all the subtrees (or a subset of them), use org-map-entries to call your own function at each of the matching headlines in the scope. org-heading-components will give you the information from the current heading, and you can use org-end-of-subtree to give you the boundary of the subtree if you want to process it further.
You can parse Org Mode lists with org-list-struct. I haven't dug into this deeply yet, but it looks interesting.
Parse free-form text
In addition to working with tables and properties, you can write functions that use regular expressions or other techniques to extract data from text. re-builder can be useful for visual feedback while you're figuring out the right regular expression to build. Remember, you're in Emacs, so you don't have to come up with the perfect regular expression that extracts all the data in one go. You can search for a regular expression, use a command like forward-line, save something to a variable, and so forth. Try thinking about how you would do something by hand, and then using repeat-complex-command to see what functions Emacs called when you did that.
save-restriction, narrow-to-region, and save-excursion are very useful when it comes to limiting the scope of your processing or saving your position, so check them out in the Emacs Lisp manual.
I find plain text to be really useful when I'm figuring things out (so, all the time), since I don't have to build a complex interface for working with it. As I learn more about Org Mode's features, I find myself using it for more and more of my data. Org's slogan is "Organize your life in plain text!" – and I think it just might be serious about that!
11 comments
undisclosed location
2015-03-11T15:01:02Zorg mode looks great unless you need to use speech recognition. another example of disabled user unfriendly free software.
Miguel Piña
2015-03-12T22:18:41Zwhy do you need to use speech recognition?
undisclosed location
2015-03-12T23:37:40ZI have nerve damage in my hands (tremors, loss of fine motion control, chronic pain that gets worse with use)
it is called an invisible disability.
Pingpong
2015-03-13T15:11:25ZYou may be interested in this video of Tavis Rudd controlling Emacs with his voice:
https://www.youtube.com/wat...
You could use the same idea to control org-mode with unique voice commands.
undisclosed location
2015-03-14T05:18:36Zyea, I know about his work and it is not a UI I would ever adopt. I've been using speech since the mid 90's and Tavis's model has been tried and discarded many times.
sachac
2015-03-16T01:16:41ZThanks for taking the time to read and comment on my blog, then, considering the effort that it takes. =) I'm still not quite happy with speech recognition even with more polished applications (such as Dragon Naturally Speaking + Microsoft Word), but that might be more because I haven't trained it enough or I just haven't gotten used to talking out loud. Given the proprietary nature of decent speech recognition engines, I doubt Emacs will have full support for them any time soon. Anyway, the good thing about software is that people can choose what fits them. Are there any applications you find particularly useful with a mostly-speech interface?
undisclosed location
2015-03-17T05:14:35ZI'm not happy with any apps and speech (which may be more telling than I want to admit :-) the fundamental reason for my dissatisfaction is the cognitive chasm between a GUI and a SUI. the disconnect is made worse by rigidity in the gui and an inability to automate tasks.
there is a good simple word processor, jarta, that works well with NS. *except* that I can not use speech to setup and process text within the editor.
simple example: logging work notes. the setup is: bring up only one instance of the editor, open the log file according to the week or month, place the cursor at the end of the file. I could never get this to work reliably.
speech interfaces really can be that simple. intent is edit/add text to a log file. which file to open is predetermined by context (date, customer, task), all the file navigating, opening, closing is not needed.
wrt the conflict between free software and accessibility, ask yourself, is it fair to ask disabled people to waste a life waiting for a free solution to their accessibility needs? is it fair to ask them accept exclusion from an on-line life like in the physical world? must they be forced into the false choice of all free or all non-free software?
as an uppity crip, I say that the ultimate intent of a disabled user is to accomplish a task, using a computer as a tool. the intent is not to maintain political purity.
the compassionate choice is to accept a hybrid system mixing the minimum non-free solutions to make free software accessible and valuable. In other words, adding a speech recognition interface to emacs make emacs more valuable, not NaturallySpeaking.
it is also a false choice to say that you can't make free software accessible because you don't have a free SR system. the free software world has a huge amount of work ahead of it just to be ready for speech interfaces. Don't burden down the process with reinventing the wheel when you have a good wheel at hand. build the vehicle and then replace the wheels.
sachac
2015-03-22T21:45:29ZIn the most recent Emacs Hangout (http://sachachua.com/blog/2..., somewhere around 1:34:55, Eric S. Johansson talked a little about his ideas for speech user interfaces and Emacs. It's a long way off, but it's interesting to see people are thinking about this. We'll just have to see how things work out!
Israel R
2017-05-15T20:01:24ZSacha, first of all thanks for your great resource of a blog, I find myself here a lot. I've been trying to find out more about multivalued properties, the emacs help and google seaches haven't been very forthcoming on how to actully set the properties up. What is the syntax for such properties? My use case would be something like having a property, :attendees: "attendee 1" "attendee 2" "etc.", which is my current setup, however, doing it this way I lose the ability of using a :attendees_all: property with a predefined list of attendees. Unfortunately, I'm not very familiar with emacs_lisp, so some help would be much appreciated. Thanks!
sachac
2017-05-19T03:35:35ZSure, try this as an example. Source at https://gist.githubusercont... in case Disqus messes the formatting up. =)
I wonder if a checklist might be a convenient alternative. It'd require more finagling to get a summary, though, so maybe your idea of using properties is a good way to go!
Israel R
2017-05-20T07:09:37ZThanks Sacha! I like your solution, I actually toyed with the idea of using #+columns while thinking about this, only I'd never used them before so not very familiar with them. I'll play around with your solution soon.