Treemap visualization of an Org Mode file
| org, visualization
One of the challenges with digital notes is that
it's hard to get a sense of volume, of mass, of
accumulation. Especially with Org Mode, everything
gets folded away so neatly and I can jump around
so readily with C-c j (org-goto) or C-u C-c
C-w (org-refile) that I often don't stumble
across the sorts of things I might encounter in a
physical notebook.
Treemaps are a quick way to visualize hierarchical data using nested rectangles or squares, giving a sense of relative sizes. I was curious about what my main organizer.org file would look like as a treemap, so I wrote some code to transform it into the kind of data that https://github.com/danvk/webtreemap wants as input. webtreemap creates an HTML file that uses Javascript to let me click on nodes to navigate within them.
For this treemap prototype, I used
org-map-entries to go over all the headings and
make a report with the outline path and the size
of the heading. To keep the tree visualization
manageable, I excluded done/cancelled tasks and
archived headings. I also wanted to exclude some
headings from the visualization, like the way my
Parenting subheading has lots of personal
information underneath it. I added a :notree:
tag to indicate that a tree should not be
included.
Reflections
The video and the screenshot above show the
treemap for my main Org Mode file,
organizer.org. I feel like the treemap makes it
easier to see projects and clusters where I'd
accumulated notes, both in terms of length and
quantity. (I've omitted some trees like
"Parenting" which take up a fairly large chunk of
space.)
To no one's surprise, Emacs takes up a large part of my notes and ideas. =)
When I look at this treemap, I notice a bunch of
nodes I need to mark as DONE or CANCELLED
because I forgot to update my organizer.org. That
usually happens when I come up with an idea, don't
remember that I'd come up with it before, put it
in my inbox.org file, and do it from there or from
the organizer.org location I've refiled it to
without bumping into the first idea. Once in a
blue moon, I go through my whole organizer.org
file and clean out the cruft. Maybe a treemap like
this will make it easier to quickly scan things.
Interestingly, "Explore AI" takes up a disproportionately large chunk of my "Inactive Projects" visualization, even though I spend more time and attention on other things. Large language models make it easy to generate a lot of text, but I haven't really done the work to process those. I've also collected a lot of links that I haven't done much with.
It might be neat to filter the headings by timestamp so that I can see things I've touched in the last 6 months.
Hmm, looking at this treemap reminds me that I've
got "organizer.org/Areas/Ideas for things to do
with focused time/Writing/", which probably should
get moved to the posts.org file that I tend to
use for drafts. Let's take look at the treemap for
that file. (Updated: cleared it out!)
Unlike my organizer.org file, my posts.org
file tends to be fairly flat in terms of
hierarchy. It's just a staging ground for ideas
before I put them on my blog. I usually try to
keep posts short, but a few of my posts have
sub-headings. Since the treemap makes it easy to
see nodes that are larger or more complex, that
could be a good nudge to focus on getting those
out the door. Looking at this treemap reminds me
that I've got a bunch of EmacsConf posts that I
want to finish so that I can document more of our
processes and tools.
My inbox.org is pretty flat too, since it's
really just captured top-level notes that I'll
either mark as done or move somewhere else
(usually organizer.org). Because the treemap
visualization tool uses / as a path separator,
the treemap groups headings that are plain URLs
together, grouped by domain and path.
My Emacs configuration is organized as a hierarchy. I usually embed the explanatory blog posts in it, which explains the larger nodes. I like how the treemap makes it easy to see the major components of my configuration and where I might have a lot of notes/custom code. For example, my config has a surprising amount to do with multimedia considering Emacs is a text editor, and that's mostly because I like to tinker with my workflow for sketchnotes and subtitles. This treemap would be interesting to colour based on whether something has been described in a blog post, and it would be great to link the nodes in a published SVG to the blog post URLs. That way, I can more easily spot things that might be fun to write about.
The code
This assumes https://github.com/danvk/webtreemap is installed with npm install -g webtreemap-cli.
(defvar my-org-treemap-temp-file "~/treemap.html") ; Firefox inside Snap can't access /tmp (defvar my-org-treemap-command "treemap" "Executable to generate a treemap.") (defun my-org-treemap-include-p (node) (not (or (eq (org-element-property :todo-type node) 'done) (member "notree" (org-element-property :tags node)) (org-element-property-inherited :archivedp node 'with-self)))) (defun my-org-treemap-data (node &optional path) "Output the size of headings underneath this one." (let ((sub (apply 'append (org-element-map (org-element-contents node) '(headline) (lambda (child) (if (my-org-treemap-include-p child) (my-org-treemap-data child (append path (list (org-no-properties (org-element-property :raw-value node))))) (list (list (- (org-element-end child) (org-element-begin child)) (string-join (cdr (append path (list (org-no-properties (org-element-property :raw-value node)) (org-no-properties (org-element-property :raw-value child))))) "/") nil)))) nil nil 'headline)))) (append (list (list (- (org-element-end node) (org-element-begin node) (apply '+ (mapcar 'car sub)) ) (string-join (cdr (append path (list (org-no-properties (org-element-property :raw-value node))))) "/") (my-org-treemap-include-p node))) sub))) (defun my-org-treemap () "Generate a treemap." (interactive) (save-excursion (goto-char (point-min)) (let ((file (expand-file-name (expand-file-name my-org-treemap-temp-file))) (data (cdr (my-org-treemap-data (org-element-parse-buffer))))) (with-temp-file file (call-process-region (mapconcat (lambda (entry) (if (elt entry 2) (format "%d %s\n" (car entry) (replace-regexp-in-string org-link-bracket-re "\\2" (cadr entry))) "")) data "") nil my-org-treemap-command nil t t)) (browse-url (concat "file://" (expand-file-name my-org-treemap-temp-file))))))
There's another treemap visualization tool that can produce squarified treemaps as coloured SVGs, so that style might be interesting to explore too.
Next steps
I think there's some value in being able to look
at and think about my outline headings with a
sense of scale. I can imagine a command that shows
the treemap for the current subtree and allows
people to click on a node to jump to it (or maybe
shift-click to mark something for bulk action), or
one that shows subtrees summing up :EFFORT:
estimates or maybe clock times from the logbook,
or one limited by a timestamp range, or one that
highlights matching entries as you type in a
query, or one that visualizes s-exps or JSON or
project files or test coverage.
It would probably be more helpful if the treemap were in Emacs itself, so I could quickly jump to the Org nodes and read more or mark something as done when I notice it. boxy-headings uses text to show the spatial relationships of nested headings, which is neat but probably not up to handling this kind of information density. Emacs can also display SVG images in a buffer, animate them, and handle mouse-clicks, so it could be interesting to implement a general treemap visualization which could then be used for all sorts of things like disk space usage, files in project modules, etc. SVGs would probably be a better fit for this because that allows increased text density and more layout flexibility.
It would be useful to browse the treemap within Emacs, export it as an SVG so that I can include it in a webpage or blog post, and add some Javascript for web-based navigation.
The Emacs community being what it is (which is
awesome!), I wouldn't be surprised if someone's
already figured it out. Since a quick search
for treemap in the package archives and various
places doesn't seem to turn anything up, I thought
I'd share these quick experiments in case they
resonate with other people. I guess I (or someone)
could figure out the squarified treemapping
algorithm or the ordered treemap algorithm in
Emacs Lisp, and then we can see what we can do
with it.
I've also thought about other visualizations that can help me see my Org files a different way. Network graphs are pretty popular among the org-roam crew because org-roam-ui makes them. Aside from a few process checklists that link to headings that go into step-by-step detail and things that are meant to graph connections between concepts, most of my Org Mode notes don't intentionally link to other Org Mode notes. (There are also a bunch of random org-capture context annotations I haven't bothered removing.) I tend to link to my public blog posts, sketches, and source code rather than to other headings, so that's a layer of indirection that I'd have to custom-code. Treemaps might be a good start, though, as they take advantage of the built-in hierarchy. Hmm…
%20--%20VisualBookNotes%20habits.png){kind=link}
