Improving how I organize notes with Org Mode

Posted: - Modified: | emacs, kaizen, org

Let me think about how I organize my Org Mode files, and how I might improve that. =)

Separate files

You can put different things in different files, of course. I use a few large Org files instead of lots of small ones because I prefer searching within files rather than searching within directories. Separate files make sense when I want to define org-custom-agenda-commands that summarize a subset of my tasks. No sense in going through all my files if I only want the cooking-related ones.

What would help me make better use of lots of files? I can practise on my book notes, which I've split up into one file per book. It's easy enough to open files based on their titles (which I put in my filenames). But I don't have that overall sense of it yet. Maybe #+INDEX: entries, if I can get them to generate multiple hyperlinks and I have a shortcut to quickly grep across multiple files (maybe with a few lines of context)? Maybe a manual outline, an index like the one I've been building for my blog posts? I can work with that as a starter, I think.

Okay. So, coming at it from several directions here:

  • A manual map based on an outline with lots of links, with some links between topics as well – similar to my blog outline or to my evil plans document
  • Quick way to grep? helm-do-grep works, but my long filenames are hard to read.
  • Links between notes and to blog posts
  • TODOs, agenda views

Outlines

Within each file, outlines work really well. You can create any number of headings by using *, and you can use TAB to collapse or expand headings. You can promote or demote subtrees, move them around, or even sort them.

I generally have a few high-level headings, like this:

* Projects
** One heading per current project
*** TODO Project task
* Reference
Information I need to keep track of
* Other notes
* Tasks
** TODO Lots of miscellaneous tasks go here
** TODO Lots of miscellaneous tasks go here
** TODO Lots of miscellaneous tasks go here

Every so often, I do some clean-up on my Org files, refiling or archiving headings as needed. This makes it easier to review my current list of projects. I keep this list separate from the grab-bag of miscellaneous tasks and notes that might not yet be related to particular projects.

I use org-refile with the C-u argument (so, C-u C-c C-w) to quickly jump to headings by typing in part of them. To make it easy to jump to the main headings in any of my agenda files, I set my org-refile-targets like this:

(setq org-refile-targets '((org-agenda-files . (:maxlevel . 6))))

How can I get better at organizing things with outlines? My writing workflow is a natural place to practise. I've accumulated lots of small ideas in my writing file, so if I work on fleshing those out even when I don't have a lot of energy–breaking things down into points, and organizing several notes into larger chunks–that should help me become more used to outlines.

Tags

In addition to organizing notes in outlines, you can also use tags. Tags go on the ends of headings, like this:

** Heading title     :tag:another-tag:

You can filter headings by tags using M-x org-match-sparse-tree (C-c \) or M-x org-tags-view (C-c a m).

Tags are interesting as a way to search for or filter out combinations. I used tags a lot more before, when I was using them for GTD contexts. I don't use them as much now, although I've started tagging recipes by main ingredient and cooking method. (Hmm, maybe I should try visualizing things as a table…) I also use tags to post entries under WordPress blog categories.

How can I get better at using tags? I can look for things that don't lend themselves well to outlines, but have several dimensions that I may want to browse or search by. That's probably going to be recipe management for now. If I figure out a neat way to add tags to my datetree journal notes and then visualize them, that might be cool too.

Links

Org Mode links allow me to refer not only to web pages, files, headings, and text searches, but to things like documentation or even executable code. When I find myself jumping between places a lot, I tend to build links so that I don't have to remember what to jump to. My evil plans Org Mode file uses links to create and visualize structure, so that's pretty cool, too. But there's still a lot more that I could probably do with this.

How can I use links more effectively? I can link to more types of things, such as Lisp code. I can go back over my book notes and fill in the citation graph out of curiosity. Come to think of it, I could do that with my writing as well. My writing ideas rarely fit in neat outlines. I often feel like I'm combining multiple threads, and links could help me see those connections.

In addition to explicit links, I can also define “radio targets” that turn any instance of that text into a hyperlink back to that location. Only seems to work within a single file, though, and I've never actually used this feature for something yet.

Properties

You can set various properties for your Org Mode subtrees and then display those properties in columns or filter your subtrees by those properties. I've used Effort to keep track of effort estimates and I have some agenda commands that use that. I also use a custom Quantified property to make it easier to clock into tasks using my Quantified Awesome system.

I could track energy level as either tags or properties. Properties allow for easier sorting, I think. Can I define a custom sort order, or do I have to stick with numeric codes? Yeah, I can sort by a custom function, so I can come up with my own thing. Okay. That suggests a way I can learn to use properties more effectively.

There are even more ways to organize Org Mode notes in Emacs (agenda views, exports, etc.), but the ones above look like good things to focus on. So much to try and learn!

You can view 19 comments or e-mail me at sacha@sachachua.com.

19 comments

I'm quite the opposite. I used to use outlines to maintain my content, but now I'm moving everything to tags. I've written a lot of code to make tag browsing/searching more convenient too, I might even publish it one day :)

Day by day I'm more and more convinced that tags are the future of organization of data of any kind. Plus, you can easily simulate hierarchy with tags if you ever need that. I even started using tagsistant (fuse tag-based fs) for my stuff outside emacs.

Interesting! What kinds of tag combinations do you often find yourself using?

I have very little "fixed structure", I usually tag with whatever feels apropriate. I have a couple obvious tags like book, music, youtube, bug, note, linux, emacs, ..., then I have a bunch for languages (EN, DE, FR, IT, LA, ...) and then all sorts of topic tags (genres, activities, locations...). Auto-completion (IDO everywhere!) helps a lot when assigning them, so I don't have 20 variations of the same "semantic" tag.

My bookmark/notes/knowledge-base org file is now totally flat with the deepest headline being one star deep, just holding the tags.

I still keep my projects under hierarchy because of all the code I've written to maintain that... plus that is actually an area where trees make sense (for me, each subheader defines a subproject, and subprojects can be nested). They still have thematic/semantic tags sometimes, but I mostly use those for my "content" org files.

A classical situation where hierarchy failed me is when you have a "history" book and you want to keep it under "german" (because, well..., it's written in german :D). Do you now organize "books/history/german" or "books/german/history" or "german/books/history"? What if I now want to see all books *and* movies in german? With trees, filtering that is a PITA, with tags, it's amazingly simple. Now each book/movie can have multiple genres/authors/languages and you get into a mess that's impossible to solve without tags.

One of the reasons I split up my book notes into multiple Org files was because it was taking me a while to search, expand and collapse trees in that one file (about 1.8MB of notes). How do you find Org Mode performance with your knowledge base Org file?

Indeed, editing performance is quite bad. The tag search/regexp search is actually fast enough I don't even notice, but editing is awfuly slow. My solution is to spawn an indirect buffer, narrow to the headline and edit it there. I think this is because org installs tons of overlays/TPs to hide headlines you shouldn't see, and that slows the rendering quite a lot.

Grant Rettke

2014-12-04T03:03:51Z

It gets real slow real fast.

My solution is `#+STARTUP: showeverything'

Grant: Oh! That’s brilliantly simple and appears to work. =D Maybe I’ll consolidate all my notes into one file again…

Jason John Wells

2016-02-10T20:11:56Z

That is indeed, very useful. I had left '#+STARTUP: hidestars hideblocks' and did my large zettelkasten.org file bogged down...and hiding '*'s no longer makes sense after using org-bullets. After my big knowledge system/ontology/commonplace book I wanted to get modular (as in LaTeX), and interoperable dividing files into semantic units.

I thought it would be cool to have .org files within directories divided into /d (domain) c/ (class) and /i (instance). This reduces the, "Where does this go?" decision lag / "paradox of choice" load and makes for a consistent namespace in ontology engineering matched with analogy "directories" of transparent binder sheets. .... but even with my basic Helm via Prelude, I still end up spending to much time switching directories, org time management becomes a pain.

I was thinking that each domain, class, and instance could be a separate repo with most shared publicly..a kind of open academic note-taking.. But then I heard computer science prof Cal Newport interviewed about his now book, Deep Work: Rules for Focused Success in a Distracted World. He has me questioning opportunity costs of "social" sharing.

Anyway, I think moving .org into a single directory and exporting out to a domain/class/instance directory as needed is simpler.

Any tips on a "flat file" -ish structure for "literate ontology"/ethnography? If you wanted to use Closure tawnly-owl library syntax source code blocks as a metadata alternative to :PROPERTIES: how would you go about it?

Tawny-owl is amazing for bringing Web Ontology Language (OWL) to Emacs! I'm no Clojure or Emacs expert but am much impressed with Cider being a REPL conduit to a descriptive logic reasoner AI. You can make your own Emacs Doctor that is actually useful by making a graph database.

Grant Rettke

2016-02-11T18:01:52Z

Jason, after switching to `nlinum' I didn't have the slowdown anymore. Now Emacs collapses big files without any slowdown, for me. https://www.wisdomandwonder...

Grant Rettke

2016-02-11T18:07:43Z

> Anyway, I think moving .org into a single directory and exporting
> out to a domain/class/instance directory as needed is simpler.

Agreed. I didn't get to the detail you did, but I tried. I found that what I made Org into was a "Lispy" document. It is about exploration. I never quite know where it will lead. What helps me find my way, is being able to narrow, and to easily re-structure both the document and my own cognitive scope. That is too techie... Org for me is about exploration and discovery, and that is why like you said, export to an ontology, don't start there in Org, or make a newly supported document type.

I tried doing an onology'ish document in Freemind, once. It was fun, and powerful, but lacked the editing features of Emacs, and everything of Org mode. My failing there was that I started cross linking, because I didn't invest enough in the domain. The solution for me, there, is to use Org's Literate Features, to tangle new documents for my own consumption. Very raw and even hacky, but, it works well for me.

I tend to use tags primarily to filter and search in my agenda view. Since tagging every task by hand is tedious, I make heavy use of tag inheritance, org-clock-in-switch-to-state, org-clock-out-switch-to-state etc. in combination with org-todo-state-tags-triggers. This is great to set context based tags and even define project tasks depending on the outline structure automatically. So I use hierarchy for projects and tags for everything else.

i use the deft package to view org files. 'Deft is an Emacs mode for quickly browsing, filtering, and editing directories of plain text notes'. i have configured deft to view the directory of files that i use for org mode. and made deft default to .org files. deft gives you a very quick text filter (rather like helm) of all the text in all the files.

using deft and org mode together makes it pretty easy to search through lots of files.

jack

jack99999: I’ve heard good things about deft and should definitely try it, especially for my book notes. =) Could be more structured than helm-do-grep.

Jason John Wells

2015-11-13T20:36:11Z

I just started using deft as an alternative to nvALT and am really
enjoying it. The "Notational Velocity" UI is popular among mac power
users for good reason. It's quick, simple. I have been overwhelmed by
the arbitrary hierarchy of huge outlines in Org but breaking notes down
into smaller atomic/semantic makes for synergy with other programs. I'm
wondering how to use use tags in my deft file-system-as-database and
how it may work with bookmarks plus. I saw an Emacs for non-programmers
tutorial (and just learned about indirect buffers and using an alias to
toggle line truncating for tables OMG!!!!) in which the guy said
bookmarks plus can scale to 100,000s of bookmarks and bookmark other
content types. .....sounds interesting....
https://www.youtube.com/pla...

Thanks for sharing something that works well for you! =)

Felix Albrecht

2016-07-03T11:39:24Z

Hey. Just a quick question. How did you generate the awesome svg of your org-file in your Evil-plan? I've been looking for this kind of visualization all over the place.

The Emacs Lisp code at the bottom of https://github.com/sachac/s... may be helpful.

It's a little complicated, since there's code to analyze the text in order to create the graph, and there's code to transform that into graphviz for rendering. You may want to check out http://doc.norang.ca/org-mo... as well for graphviz examples.

Felix Albrecht

2016-07-04T06:46:24Z

Wow. Great. Thank you.
I was looking for a way to visualize my outlines like that as I can more easily think in mindmaps usually. (But org-mode is just so damns useful. ;-) )