Categories: pkm

RSS - Atom - Subscribe via email

How sketchnotes fit into my personal knowledge management

| pkm, drawing

Text from sketch
  • worth doing even if you don't feel like you can draw well
    • really, I just draw stick figures
  • good for your own thoughts and other people's
  • own thoughts:
    • non-linear
    • visual metaphors & organizers can be helpful
    • can be a launchpad for more details
  • other people's thoughts: distill key points from a talk, book, etc. using my understanding
  • visual cues make it easy to see important things first
  • doodling is fun
  • IDs help with linking (ex: 2024-10-17-02)
  • How I use sketchnotes:
    • Flesh out an idea, especially during non-computer time
    • Sketch talks or books to make them easier to review
    • Optical character recognition (Google Cloud Vision API, etc.) to blog text: I edit this to provide a good text alternative in blog posts
  • My evil plan
    • Sketchnotes are very shareable
      • People are always looking for visuals to add.
    • When people share them, they usually tell me about it
    • I get to find out what else people are thinking about & learning from.
    • More learning! More fun!
    • It's also a nice way to give back to people who've shared what they learned
      • Then they might share more!

I've been enjoying using sketchnotes as an idea launchpad for audio braindumps or blog posts, as a quick way to review the key points of a book or talk, and as a way to participate in the larger conversation. It's easy for me to link to sketches and extract the text within them.

Someday I'll probably improve my ability to search for the text within sketches. Right now, I just go by filenames and the text in my blog posts. I can probably make something that goes through the text annotations in the JSON files from Google Cloud Vision, or maybe I can turn them into a text file that can be updated when I write a blog post. Hmm, that actually sounds pretty straightforward, I should go do that…

Examples of my evil plan working:

Mwahaha!

View org source for this post

Karl Voit's 2023 talk: The Art of Personal Information Management

| pkm

I'm starting to dig into what other people have shared about personal information management and personal knowledge management. Karl Voit is one of my favourite people in this space, and I've enjoyed

I thought I'd sketchnote the recording of his talk at Worklab 2023: "The Art of Organizing Yourself and Your Data". Here it is:

Text from sketch

The Art of Organizing Yourself and Your Data - 2023 presentation by Karl Voit

My focus

  • personal information
  • methods, not so much tools
  • developing your own methods

Vocabulary problem

  • water, water bottle, bottle, drink, beverage, container
  • If you ask different people to list words to describe it, you have to go far down the list to find shared words.
  • Everyone has a different mental model, even past you vs. present you

Navigation, search

Desktop metaphor

  • Things in the real world can have only one specific location
    • Hierarchies
  • In the virtual world, you can have multiple ways to find what you want: tags, search, …

Tag trees, filter

  • ex: sports, hardware

Tagging tip: Controlled vocabulary: develop a short list of preferred words

Everything is Miscellaneous - David Weinberger

  • physical order: only one order at a time (can change)
  • index: library index catalogue
  • no order

It nudges me to think about:

  • where I can use tags to connect ideas that I file in different places, such as Embark-related context menus in my Emacs configuration
  • reviewing my tags to see how I can consolidate terms or develop further distinctions
  • creating maps and linking notes to improve navigation
  • improving search for my personal notes so that it's easier for me to find things

I also edited some captions for it, because captions are nice. Enjoy!

Links:

View org source for this post

Thinking about 12 aspects of personal information/knowledge management

| pkm, org

Here is a totally rough list of aspects that I came up with to start thinking about how I do personal information/knowledge management and how I want to explore other people's systems.

(text from sketch duplicated as headings below)

Quick reflections on my setup:

Use: What do you want to use it for, and how?

I mostly work on code, so I need to keep things like TODOs and setup instructions.

I also want to organize resources and refer people to them.

It's important to me to get things out of my head because unfinished thoughts in my head are intrusive (Ovsiankina effect). They get in the way of being able to enjoy time with the kiddo. I need to be able to get them out into a system that I can trust, so that I can stop thinking about it until it's time to think about it again. I don't have a lot of computer time, so I want to be able to pick things up quickly when I do.

Capture: How do you get stuff in?

Most of the time, I add quick questions or ideas using Orgzly Revived on my phone because I'm not close to a computer. Sometimes I look up web pages that relate to something, and then I can share that with Orgzly using the Android share menu. If I'm close to a computer, then I can use org-capture.

I also use my Supernote to sketch/write ideas.

I use my phone for audio braindumps.

Challenge: I want to write down more context because I occasionally come across notes that don't make sense to me.

Retrieval: How do you get stuff out?

I usually tend to work on things that I've recently thought about, so I'm working out of my inbox or out of a few active projects. Either the relevant items I've captured are still there in my inbox or in the project's tree, or I can quickly organize them before I dive into my work.

Sometimes I need to retrieve something that's a lot older, such as when I want to recommend something I remember seeing a year or two ago. This is challenging because I often don't remember the exact words that will bring it up. I can help that a little bit by adding my own words when I create the note, but I don't feel like that's a solid solution yet. I think that this is a challenge that's going to get worse as my brain gets fuzzier. Finding things using approximate matches could be interesting. Most of the time, I end up relying on an Internet search, because then I can take advantage of the variety of words used in other people's descriptions of the thing.

Blog posts (and funneling my toots and sketches into blog posts) makes things slightly more findable. I've come across things I've completely forgotten writing about.

Challenge: When I'm trying to move too quickly instead of writing things down, then there's nothing to retrieve years later when I'm picking a project back up again. For example, when I finally dusted off my time-tracking project so that I could upgrade the Rails version, I had to do a lot of figuring out. That tells me I need to write more notes. As I run into things that I didn't write down well enough (or as I bump into things I could've sworn I wrote about but I just can't find my notes), I try to write down what I've figured out, where I looked, and what words I used in order to look for it. Maybe that will make it more findable in the future.

Priorities: How do you get the right stuff out?

I tend to work on a few recent thoughts, so I can generally schedule them for the day that I think I'll be able to work on them. Then I can use my Org Mode agenda to get a short list of the things that I want to work on. When that's done, I can then go through the more general things–still biased towards what's recent, what's in my inbox, what I've been thinking about lately. It takes extra time to context shift back into older things.

My life generally doesn't have a lot of urgent commitments, so it's mostly a matter of thinking: What do I feel like working on? What's the most annoying thing I need to work around? What am I curious about? Then I can go to that project or thought.

Sometimes I'll use the TODO status to distinguish between things that I want to do someday versus things that I could do sooner. Pushing things off to SOMEDAY is especially handy for ideas that are not very fleshed out yet. My newly created tasks default to SOMEDAY so that it takes me an active effort to say, okay, this stuff is on my list of things to focus on.

Sometimes I use the [#A] and [#C] priority marker in Org Mode to move things to the top or bottom of my list.

In general, I don't worry too much about making sure that I'm working on the absolute best thing at the time, because that stuff takes planning, too.

Time: How do you deal with dates/times/conditions?

Scheduling something on a particular day is how I pick a short list of things that I want to do. These things don't always happen. Sometimes I end up procrastinating something for another few weeks out or a month out. If I do that too often, I usually end up cancelling it, because clearly there are other things I want to do.

There are also the things I've got to schedule once in a while that I don't actively think about until the reminder pops up, like renewing my passport. The Org agenda takes care of that.

I like to keep journal entries so that I can look back and see the progress I've made.

Revision: How do you add to or refine things?

I might start off with just a quick question or idea. Depending on what I have time for, I might flush out that idea in an audio braindump or a sketch. I can convert either of those things into text and dump them into my note for editing, or I can sit down and flesh out the idea further by writing it, with the eventual goal of turning it into either a toot or a post. Maybe some of them will get turned into videos. So that's how I gradually refine things.

I would like to get better at this. Maybe I can keep track of which thoughts could benefit from sketching or doing a brain dump, or refining those sketches or brain dumps into posts. Which posts are almost there and just need a little bit more work? Which ones do I want to turn into a video?

Since the sketching and the braindumping can happen in parallel, it's probably more about tags rather than TODO states.

One improvement could be showing me where these ideas are in the pipeline so that if I'm at my computer and I want to get something out the door, I can make a list of posts that are almost there. If I'm heading out for a walk to the store, then I can make a list of the things to think about out loud. Then I can have my system do the transcript and stick it back into the pipeline so I can edit it.

How do I take those fragments of thoughts, put them together, and turn them into a finished chunk?

When it comes to refining sketches, I can just flip open my supernote and I add more stuff to it. It's very easy to pick up and put down again. I like that.

Audio is harder to work with in terms of refining an idea, but maybe I'll figure out the workflow for that someday. The draft for this post came from a sketch and an audio braindump.

There's also this idea of refining a project. When I do my first pass through my inbox, I'm just basically throwing things in the rough direction of where I'm probably going to want them. I'll refile things very roughly into Consulting or EmacsConf or whatever else. Refining in that context would be collecting several resources and putting them under one subtree, or making sense of something, mapping out the resources for a topic, or summarizing.

If I've saved a web page, it becomes a lot easier to learn from and find again if I use my own words to describe what I'm learning from it. That's another area that I could definitely do better in.

Refining is easier to do when I'm on my computer, but when I'm on my computer, I tend to want to make stuff rather than edit stuff. If I'm refining something with the goal of making it a post, that sometimes happens. But if I want to review a page whose link I saved, sometimes that ends up very low on my priority list. I'm throwing all these things into my SOMEDAY list and not actually getting around to them yet. Maybe someday!

Connection: How do you link things together?

Most of the time, I refile things so they're roughly close to where other things I need are. I can just scroll to find connected items.

I don't have many things that need to be in multiple places in my In my outline. When I do, I tend to use links to connect the ideas. I like linking between blog posts and sketches.

I don't have a good facility for backlinks yet. I should make this easier for myself, either by just opening the blog post that I'm referring to so that I can quickly add a link to it going the other way–a manual backlink that lets me provide the context–or maybe adding some backlink support to my static site generator.

Anyhow, at least the forward links are fine. I've got some completion to help me with that. Web searches are helpful just in case my completion doesn't work, as right now my completion only works with title searches. If I am a little fuzzier about what I've called something, then I will search the Internet, grab the URL, and pop in the link.

Most of the linking happens in my blog posts because the blog posts live outside my outline. They are just roughly organized by date and category. So if I want to build on another thought, I've got to link to it. Fortunately, I've got the URL, so it's easy to link to things.

I can link to things within Org Mode. I probably should more often, and it will probably involve getting the hang of Org IDs. It hasn't been as big a need for me for now because I try to push things into blog posts as much as possible.

Sometimes it makes sense to have a URL or a link that works for both the exported version and my own internal notes. I want some things to open up in Emacs instead. Then I might have a custom link type to make that easier.

Externals: How do you refer to things outside your system?

There are a lot of things that I want to think about or refer to that aren't within my Org Mode files. Fortunately, Org Mode makes it super easy to link to the things, so that part is fairly solid.

There are some kinds of things that I don't have an easy way of thinking about or working with yet, like audio.

Work ideas are harder for me to link to now that I can't access the company's WebEx chat on my personal phone, so I just write down a couple of keywords to remind myself what to think about or search for. I also tend to read my e-mail on my phone, so I don't have Org Mode's fancy linking. I write down or copy a few keywords and tag the note with "email" to help me remember where to look. Life would be much easier if I could do all of these things within Emacs so that I could just create a task and it would automatically be annotated with the link to the original stuff, but we've got to work with what we've got.

Sharing: How do you share with others?

I've been gradually refining my workflow for turning my notes ito blog posts. Org Mode is fantastic for this. I can have source blocks, I can export to various formats, it's all good. I'm also exploring the idea of turning some things into richer text–adding diagrams or sketches, or narrating it, or turning it into a video.

My main thing is I want to get thoughts, ideas, and questions from my notes into some kind of public chunk. Toots are nice because I can get smaller thoughts out instead of waiting until I've fleshed them out further. Blog posts are ideal.

I want to experiment with this by using audio braindumps and sketches to explore ideas faster and use non-computer time to help with writing.

Maintenance: How do you tidy or trim?

Part of maintenance is figuring out what's out of date and what I can archive to make it easier for me to just see the current stuff. I periodically go through my inbox and archive things or refile things into projects. I am slowly getting the hang of archiving things instead of deleting things, since disk space is cheap. Once in a while, I'll go through my Org file to archive inactive projects and neaten things up.

On the public side, I could probably do automated things like link-checking, but it's been pretty low priority. Most of the time, I end up updating posts when I look up them up in order to link to them or when people ask me about them. I have a snippet that makes it a little easier to note an update, but I should probably improve it to handle adding an update to a post that's already been updated before.

I don't have a list of recently modified but not newly posted posts, which might be a good idea for exposing that to blog readers.

I also want to create more evergreen pages that organize resources, kind of like my blog outline but more granular. I still want to have the last modified date as text in the page itself, but it doesn't have to be part of the permalink.

Discovery: How do you stumble upon things?

I have a lot in my notes that I've completely forgotten about. One of the benefits of keeping most of my notes online is that when people come across those notes, their links or comments help me find them again.

I've also added a random blog post button on my blog, and I'm trying to shift some doom-scrolling to use that instead.

For my personal notes, I don't bump into things as much because org-refile is very efficient for getting to just the thing I want to look at. For the most part, things get hidden away under their sub-trees until I feel like working on that particular area, so it might be years before I touch something again, if at all.

I could probably add some kind of randomness thing, but I don't really struggle with finding things to work on when I'm on my computer. There's usually something else more pressing that I want to work on, so it hasn't been an issue.

I do want to add a random sketch thing, though. I think it could be fun to cycle my background through the files in my public sketches on my desktop or my phone lockscreen.

Longevity: How do you keep it around?

Using plain text and free and open source software is really important to me because I want it to be easy to back up and I want to be able to trust that it's going to be around. Having seen many things get bought up or taken down… Yeah, I want to have my own notes. I feel reasonably confident, based on other people's experiences, that if I want to keep using my notes in another 20 years or more, it'll probably still be there as long as I don't do anything silly with the data.

For my sketches, I put titles and tags in the filenames. I've been using Google Cloud Vision to do handwriting recognition so that I have some kind of text that presumably I could search, although I haven't built that part yet.

Audio is a bit more ephemeral, but it might still be interesting to hear archived audio.

One of these days, I should make an organized backup of the things that I've shared on YouTube and other places. Videos take much more space.

Another thing that I'm thinking of long-term, once in a while, is how to keep going into this, how to keep it easy for me to access, use, add to, and share as I get older. I hear menopause might really do a number on my brain. People report having a hard time remembering words and thinking thoughts. It would be nice to have approximate search in place by then so that I can still find things, or at least have shared as much as possible.

My long-term plan (in case stuff happens) is to have whatever notes might be helpful be publicly available already so that theoretically someone could use the Internet Archive or a static mirror or to get back to it. Even in the case where I die and my hosting stops being paid for, the core things about it, I think, have been well-demonstrated and can be easily picked up by somebody else if they want to.

Planet Emacslife is a blog aggregator. The idea of Emacs News is fairly straightforward and somebody else could step into it easily. The ideas are not dependent on me, whic his nice.

My posts and code are out there too. They're not immortal, and they don't have to do be. If they're useful in the moment, that's already enough. If somebody comes across them months or years later and finds them useful, that's a bonus. I use them to think through something, so that's already a win.

Wrapping up

I'd love to hear about your personal information/knowledge management systems, whether you want to think about it using these aspects or your own framework. Let's share notes!

View org source for this post

Mapping knowledge

Posted: - Modified: | pkm, learning

I chatted with someone about maps and personal knowledge management, so I thought I'd write an extended reflection.

2015-05-13e Mapping knowledge for yourself and others -- index card #mapping #pkm #sharing

Mapping is useful for myself and for others. For managing my own learning:

  • Scope: What's included, and what's not? How does this relate to other things I've learned or I'm learning?
  • Landmarks and destinations: Role models, motivation, tracking progress…
  • Main path, detours: How do you get from A to B? Are there interesting places in the neighbourhood?
  • Here there be dragons, places under construction: Managing appropriate difficulty; tracking areas to explore or revisit

When helping other people learn, mapping lets me:

  • Define scope: Define a manageable chunk, and link to related maps: zooming in, zooming out, going to other places
  • Provide landmarks
  • Main path, detours: Organize a reasonable path (particularly based on someone's interests) and nice detours
  • Here there be dragons / construction: Warn newbies, encourage intermediate/advanced exploration

So here's my current workflow:

2015-05-13f Mapping what I know -- index card #workflow #blogging #index-cards #mapping #pkm

2015-05-08c Managing my structured information -- index card #pkm #knowledge #sharing

Using index cards, outlines, and chunks seems to be working well for me in terms of current thinking, although I haven't been turning my attention to organizing, fleshing out knowledge, and filling in gaps.

Here are some notes from 2013 on mapping forwards (plans) and backwards (guides for other people). I've figured out ways around some of the challenges I encountered before:

  • Rough categorization of blog posts: I've written some Emacs Lisp code to help me update my blog post index monthly.
  • Hundreds of sketches with few links: Now I have more than a thousand sketches! But that's okay, I have metadata in the filename, integration in my outline, and eventual chunking into blog posts.
  • Duplicate metadata entry, no synchronization: Tags in the filename and a NodeJS script that sets the same tags on Flickr upload, yay
  • No clear picture of follow-up questions, ideas, or actions: Outline still needs work; maybe also a quick way to review open sketches?
  • No clear role models: Found historical and contemporary ones, yay!

Mostly I've been focusing on little explorations rather than map-making. It's like collecting nature specimens so that I can start to classify them, since you don't see that order until later. Sometimes I look back and retrace my path. That's when I can try to figure out where things are and how people might go a little faster or in a better order. Other times, when I'm looking forward, I'm trying to see what's close by and how to get there. I remind myself of the landmarks in the distance, too, and what progress might look like. But I can only walk the routes until I reach a height that lets me review the paths ahead, so sometimes it's just the accumulation of steps…

2015-05-12d What do I want to get from my blog archive, looking back twenty years from now -- index card #blogging #pkm #archive

2015-05-12e What do I want from my archive of index cards -- index card #pkm #archive #drawing #index-cards

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

Writing: Open loops, closed loops, and working with forgetfulness

Posted: - Modified: | pkm, blogging, writing

I think I’ve written about something before, but I can’t find it. I have thirteen tabs open with Google search results from my blog. I’ve tried countless keywords and synonyms. I’ve skimmed through posts I only half-remember writing. (Was that blog post really that short? I thought I wrote more details.) I still haven’t found the post I want.

I wonder: Did I really publish it? Or did I just outline or sketch it? Am I confusing it with something similar that I wrote, or someone else’s post that I admired?

Ah, well, time to write it from scratch. It’s a little like writing code. Sometimes it would take so long to find an appropriate open source module that you’re better off just writing the code yourself. Sometimes it would take so long to find an existing post that it’s better to just write it from scratch.

I was looking for that particular post because of a conversation with Flavian de Lima where I mentioned the benefits of blogging while you’re learning something. He resonated with the idea of sharing your notes along the way so that other people can learn from them, even if you’ve moved on to different topics.

Despite having a clear memory of writing about this topic, when I went to the post that I thought was related to it (spiral learning), it didn’t mention blogging at all. “Share while you learn” didn’t quite address it, either. After trying lots of searches, I gave up and started writing a new post. After all, memories are fallible; you could have full confidence in an imagined event.

The reason this came up was because Flavian described how he often took advantage of open loops when working on writing. He would stop with an incomplete thought, put the draft away, and let his subconscious continue working on it. Sometimes it would be days or weeks before he got back to working on the article. He mentioned how other authors might take years to work on novels, dusting off their manuscripts and revising scenes here and there.

Keeping loops open by stopping mid-sentence or mid-task is a useful technique often recommended for writing or programming. Research describes this as the Zeigarnik effect: an interrupted task stays in your memory and motivates you to complete it.

But after reading David Allen’s Getting Things Done, I had become a convert of closed loops: getting tasks, ideas, notes out of your head and into a trusted system so that you don’t have to waste energy trying to remember them. I noticed that if I kept too many loops open, my mind felt buzzy and distracted. To work around this, I got very good at writing things down.

In fact, I took closing loops one step further. Publishing my notes on my blog helped me get rid of the guilt and frustration I used to feel whenever I found myself wanting to move on to a different project. Because my notes were freely available for anyone who was trying to figure out the same thing, I could go ahead and follow the butterflies of my interest to a different topic. My notes could also help me pick things up again if I wanted to.

I didn’t stop mid-sentence or mid-thought, but I published in the middle of learning instead of waiting until I finished. Even my review posts often included next steps and open questions. So I got a little satisfaction from posting each small chunk, but I still left dangling threads for me to follow up on. I closed the loops enough so that the topics didn’t demand my attention.

Writing helped me clear my mind of strong open loops–but it worked a little too well. I tried to close things off quickly, so that I could revisit them when I wanted to. The trick was remembering that they were there. Sometimes I forgot the dangling threads for a year or more. I never followed up on others. Even with my regular review processes, I often forgot what I had written, as in the search that prompted this post.

Writing and memory have an ancient trade-off. Even Socrates had something to say about it, quoting an ancient Egyptian king in Plato’s The Phaedrus:

“…for this discovery of yours will create forgetfulness in the learners’ souls, because they will not use their memories; they will trust to the external written characters and not remember of themselves.”

as quoted in On writing, memory, and forgetting: Socrates and Hemingway take on Zeigarnik

In 2011, Sparrow, Liu, and Wegner showed that people remember less if they think a computer will keep their notes for them, and they tend to remember how to get to the information rather than the information itself. Having written the words, published the posts, and indexed the titles, I’ve forgotten the words; and now I can’t find my way back.

Hence my immediate challenge: sometimes I forget how to get to the information I’ve stored, like a squirrel stashing nuts. (More research: tree squirrels can’t find 74% of the nuts they bury. So I’m doing slightly better than a squirrel, I think.)

Google helps if I can remember a few words from the post, but since it tends to search for exact words, I have to get those words right. Hah, maybe I need to use search engine optimization (SEO) techniques like writing with different keywords – not for marketing, but for my own memory. It reminds me of this SEO joke:

How many SEO copywriters does it take to change a lightbulb, light bulb, light, bulb, lamp, bulbs, flowers, flour…?

My blog index is helpful, but it isn’t enough. I need to write more descriptive titles. Perhaps I should summarize the key point as well. Maps can help, as can other deliberate ways of connecting ideas.

Let me take a step back and look at my goals here. Linking to posts helps me save time explaining ideas, build on previous understanding, and make it easy for people to dig into more detail if they want. But I can also accomplish these goals by linking to other people’s explanations. With so many people writing on the Web, chances are that I’ll find someone who has written about the topic using the words I’m looking for. I can also write a new post from scratch, which has the advantages of being tailored to a specific question and which possibly integrates the forgotten thoughts even without explicit links.

It’s an acceptable trade-off, I think. I’ll continue writing, even with the increased risk of forgetting. If I have to write from scratch even when I think I’ve probably written about the same topic before, I can accept that as practice in writing and thinking.

Other writers have better memories. Flavian told me how he can remember articles he wrote in the 1990s, and I’ve heard similar accounts from others. Me, I’ve been re-reading this year’s blog posts in preparation for my annual review, and I’ve come across ones that pleasantly surprised me. Posts two or three years back are even fuzzier in my memory. I can try to strengthen my memory through exercises and processes. The rest of the time, I can work with the brain that I have. In fact, I’m inclined to build more memory scaffolds around myself, moving more of my memory outside my mind.

[I do not] carry such information in my mind since it is readily available in books. …The value of a college education is not the learning of many facts but the training of the mind to think.

And really, how much difference would perfect memory make? I might add more links, include more citations, cover more new ground. I can still learn and share without it.

Forgetful squirrels have their uses. Forgotten acorns grow into oaks for others to enjoy. From time to time, I hear from people who’ve come across old posts through search engines, or I come across old posts in a review. Loops re-open, dangling threads are taken up again, and we continue.

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

Visualizing the internal citation network of my blog

Posted: - Modified: | pkm, visual, wordpress

I’m curious about the internal citation of my blog. Which thoughts have been developed over a long chain of posts? Which posts do I often link to? Where are there big jumps in time? Where have I combined threads?

2014-12-03 Internal citation network

I’ll probably need to build my own data extractor so that it can:

and I’ll probably want to create something that I could eventually plot as an SVG or imagemap using Graphviz, or maybe analyze using Gephi.

It would be super-interesting to create some kind of output that I could fold into my blog outline or into individual posts. I would need a static dump for that one, I think.

How would I build something like this? One time, I used Ruby to analyze the text of my blog. That might work. I might be able to pull out all the link hrefs, do lookups…

As of Dec 3, 2014, there are 87 posts in 2014 that link to previous posts, out of 259 non-review posts (so roughly 34%). I used this SQL query to get that:

SELECT post_title FROM wp_posts WHERE post_content LIKE ‘%<a href=”https://sachachua.com/blog/20%’ AND post_date >= ‘2014-01-01’ AND post_title NOT LIKE ‘%review:%’ AND post_state=’publish’;

Hmm. I might even be able to do some preliminary explorations with Emacs and text processing instead of writing a script to analyze this, if I focus on 2014 and ignore the special cases (short permalinks, redirection, and sneak previews), just to see what the data looks like. Rough technical notes:

perl -i -p -e s/href/\nhref/gi 2014-manip.html
grep http://sachachua.com/blog/20 2013-manip.html > list-2013
perl -i -p -e "s/(<\/a>(<\/h2>)?).*/$1/gi" list-2013
(defun sacha/misc-clean-up-reviews ()
  (interactive)
  (while (re-search-forward "\\(Monthly\\|Weekly\\) review: .*</h2>" nil t)
    (let ((start (line-beginning-position)))
      (re-search-forward "</h2>")
      (delete-region start (line-beginning-position))
      (goto-char (line-beginning-position)))))

(defun sacha/org-tabulate-links ()
  (interactive)
  (goto-char (point-min))
  (let (main-link edges nodes)
    (while (not (eobp))
      (if (looking-at "^href=\"\\(.*?\\)\".*?</a></h2>")
          ;; Main entry
          (progn
            (setq nodes (cons (match-string 1) nodes))
            (setq main-link (match-string 1)))
        (if (looking-at "^href=\"\\(.*?\\)\"")
            (setq edges (cons (concat 
                               main-link  ;; from
                               "\t"
                               (match-string 1)   ;; to 
                               ) edges))))
      (forward-line 1))
    (kill-new (mapconcat 'identity edges "\n"))))

Ooooh. Pretty. Gephi visualization of the edge list formed by links, using the Yifan Hu layout. That big thread in the middle, that’s the one about taskmasters and choice and productivity, which is indeed the core theme running through this year of the experiment. The cluster on the right is a year in review. We see lots of little links too.

Internal links for entries posted in 2014

Internal links for entries posted in 2014

Now I’m curious about what happens when we add the posts and links from 2013 and 2012, too. I’ve colour-coded this by year, with It ties together posts on sketchnoting, blogging, choice, learning, writing, plans… Neat.

blog-graph

 

What does this say? It says that even though I write about lots of different things, there are connections between the different topics, and the biggest tangle in the middle has more than 320 nodes. I have lots of blog posts that build on an idea for three or four posts, sometimes spanning several years, even if I don’t think about it in advance. There are 90 such clumps, and it might be good to revisit some of these 2- and 3-post chains to see if I can link them up even further.

Also, it could be interesting to do this analysis with tags, not just year. Hmm… Also, I should dust off my data structures and algorithms, and make my own connected-component analyzer so that I can get a list of the clumps of topics. Good ideas to save for another day!

View or add comments (Disqus), or e-mail me at sacha@sachachua.com

Thoughts in context: Connecting posts to my blog post index

Posted: - Modified: | pkm, wordpress

I’ve been thinking about how to improve the inter-post organization of my blog so that I can write more effectively and so that other people can find things faster. I often link to posts I’ve previously written, but I rarely update old posts to link forward to other related posts. There are quite a few internal linking plugins for WordPress, but they seem more slanted towards SEO and keywords than I’d like.

I wanted to come up with another approach that could take advantage of the big outline of my blog posts at https://sachachua.com/index that I update every month. I’ve found this to be pretty handy for organizing things into finer categories instead of going back and updating lots of posts in WordPress. I can search this easily on my computer by using helm-swoop, and I can move things around using Org Mode.

It got me thinking: Is there a way I can make it easier to connect posts to the index so that if people find an old post useful, they can explore related posts?

So here’s what I came up with: a small See in index link at the end of the post.

2014-12-08 16_45_13-sacha chua __ living an awesome life - Page 2 of 1159 - learn - share - scale

Index link on old posts

Not all the posts are indexed yet, but for those that are, clicking on that link will open up the blog index and scroll it to the right post, highlighting the match, so people can see what else is in the neighbourhood.

2014-12-08 16_45_33-sacha chua __ blog

Matching link

To make this work, I added the following HTML code to my blog index:

<script type="text/javascript">
// from http://www.jquerybyexample.net/2012/06/get-url-parameters-using-jquery.html
function getURLParameter(sParam)
{
    var sPageURL = window.location.search.substring(1);
    var sURLVariables = sPageURL.split('&');
    for (var i = 0; i < sURLVariables.length; i++)
    {
        var sParameterName = sURLVariables[i].split('=');
        if (sParameterName[0] == sParam)
        {
            return sParameterName[1];
        }
    }
}

function sachaScrollToBlog() {
  if (getURLParameter("url")) {
    var link = $('a[href="' + getURLParameter("url") + '"]');
    if (link.length > 0) { 
      $('html, body').scrollTop(link.offset().top - 100);
      link.addClass("highlighted");
    } else {
      alert("Sorry, could not find post in index.");
    }
  }
}

$(document).ready(sachaScrollToBlog);
</script>
<style type="text/css">
a.highlighted { background-color: yellow; padding: 10px }
</style>

Then I added this to the post.php and single.php for my WordPress theme:

<?php
if (get_the_date('Y') >= '2008' && get_the_date('Y-m') < date('Y-m', time() - 60 * 60 * 24 * 7 * 2)) {
  print '<a href="http://pages.sachachua.com/sharing/blog.html?url=' . get_permalink() . '">See in index</a>';
}
?>

The date condition is there to minimize frustration. I’ve indexed posts published 2008 or later, and I usually post an updated index within the first half of the month. I might spend some time indexing older posts; if I do, I’ll update the starting position.

It would be interesting to refine my writing workflow so that my blog index is always up to date. That way, even new posts will have this magic indexing power. It might be difficult to get that squared up with my scheduling, though, since I sometimes write a few weeks in advance. Anyway, neat, huh? This should make the archives marginally more useful. =) Good for me too!

View or add comments (Disqus), or e-mail me at sacha@sachachua.com