I've been thinking a lot about how to manage a large archive to encourage discovery and serendipity, and to make it easier to fish out articles so that I can send them to people. I started in 2001-ish and have more than 6,500 posts. There's not a lot of information on how to manage a large archive. Most blogging-related advice focuses on helping people get started and get going. Few people have a large personal archive yet. I love coming across other bloggers who have been at this for more than ten years, because information architecture is fascinating. Here's what I do, in case it gives you any ideas.

1.  I set up Google Chrome quick searches for my blog, categories, and tags. This means I can quickly dig up blog posts if I remember roughly where they are. (Gear > Settings > Search > Manage Search Engines):
   • Blog (b): https://www.google.ca/search?q=site%3Asachachua.com+%s
   • Blog category (bc): https://sachachua.com/blog/category/%s
   • Blog tag (bt): https://sachachua.com/blog/tag/%s
2. I create pages with additional notes and lists of content. I use either Display Posts Shortcode or WP Views, depending on what I need. See the Emacs page as an example.
3. I've started using Organize Series to set up trails through my content. It's more convenient than manually defining links, and it allows people to page through the posts in order too. Read my notes to find examples. I'm also working on maps, outlines, and overviews.
4. I've also started packaging resources into PDFs and e-books. It makes sense to organize things in a more convenient form.
   
5. I converted all the categories with fewer than ten entries to tags. Categories can get unwieldy when you create them organically, so I use categories for main topics and tags for other keywords that might graduate to become categories someday. I think I used Categories to Tags Converter or Taxonomy Converter for this. Hah! Similar Posts reminded me that I used Term Management Tools. Awesome.
6. I manually maintain a more detailed categorical index at sach.ac/index. This makes it easier for me to see when many blog posts are piling up in a category, and to organize them more logically.
7. I set up short URLs for frequently-mentioned posts. The Redirection plugin does a decent job at this. For example, people often ask me about the tools I use to draw, and it's great to just be able to type in http://sach.ac/sketchtools as an answer.
8. I post weekly and monthly reviews. The weekly review includes links to that week's blog posts, and the monthly review includes a categorized list. I've also set up daily, weekly, and monthly subscriptions based on the RSS feeds. This is probably overkill (more choices = lower subscriptions), but I want to give people options for how frequently they want updates. The weekly and monthly reviews are also helpful for me in terms of quickly getting a sense of the passage of time.
9. I use Similar Posts to recommend other things people might be interested in. There are a number of similar plugins, so try different ones to see which one you like the most. I tried nRelate and the one from Zemanta, but I wasn't happy with the way those looked, so I'm back to plain text.
10. I show recent comments. People often comment on really old posts, and this is a great way for other people to discover them.
11. I use post titles in my next/previous navigation, and I labelled them “Older” and “Newer”. I think they're more interesting than
12. I customized my theme pages to make it easier to skim through posts or get them in bulk. For example, https://sachachua.com/blog/2014/02 lists all the posts for February. https://sachachua.com/blog/2014/?bulk=1 puts all the posts together so that I can copy and paste it into a Microsoft Word file. https://sachachua.com/blog/2014/?org=1 puts it in a special list form so that I can paste it into Org Mode in Emacs. You can also pass the number of posts to a category page: https://sachachua.com/blog/category/drawing/?posts_per_page=-1 displays all the posts instead of paginating them. These tweaks make it easier for me to copy information, too.
13. I give people the option to browse oldest posts first. Sometimes people prefer starting from the beginning, so I've added a link that switches the current view around.
14. I have an “On this day” widget. Sometimes I notice interesting things in it. I used to put it at the end of a post, but I moved it to the sidebar to make the main column cleaner.
15. For fun, I have a link that goes to a random post. I used to display random post titles in the sidebar, which might be an interesting approach to return to.
16. I back up to many different places. I mirror my site as a development environment. I back up the database and the files to another web server and to my computer, and I duplicate the disk image with Linode too. I should set up incremental backups so that it's easier to go back in time, just in case.
17. I rated my posts and archived my favourite ones as a PDF so that I'll still have them even if I mess up my database. Besides, it was a good excuse to read ten years of posts again.

Hope that gives you some ideas for things to experiment with! I'm working on organizing more blog posts into trails and e-books. I'm also getting better at planning what I want to write about and learn. If you're curious about any of the techniques I use or you want to bounce around ideas, feel free to e-mail me at sacha@sachachua.com or set up a chat.

Do you have a large blog? How do you manage it?

I want to learn about more than I can fit into my working memory, so I need to take notes and I need to relate those notes to each other. My sketches, blog posts, and Evernote entries are great for remembering things, but I also want to see overviews so that I don’t miss the forest for the trees. This is where mapping comes in. Mapping is about organizing topics so that I can see the relationships, find the gaps, and keep moving forward.

For example, I mapped out what I wanted to learn about learning, and I frequently refer to it while planning my next steps.

I also have a few other maps at lower levels of detail. For example, this is a rough map of topics related to taking notes:

Mapping helps me look ahead, and it also gives me a framework for connecting what I learn to what I’ve learned before.

One way to practise mapping is by mapping what you know. This helps you review your notes, identify any gaps, see how far you’ve come, and connect ideas (and discover interesting relationships you might not have come across before). Your maps can also help other people learn.

To map what you know, you can start from the bottom level (detailed notes answering specific questions) and work your way up to overviews. Alternatively, you can start from the top (an outline) and then work your way down to the specifics. Combining these strategies can help you get around mental blocks.

I’ve been working on mapping what I know. I’m still trying to find a good set of tools to help me do this. Instead of getting intimidated by the task, I’ve decided I’m going to start in the middle, mapping out things I recently learned and things that I’m learning next. Once I get the hang of doing that, I can start adding older entries like my blog posts.

I haven’t quite found the perfect tool yet. Evernote is great for personal notes, but even though it has public notebooks (see my sketchbook and my sketchnotes), people aren’t used to following or discussing new notes there. Flickr is good for exposure and a little discussion, but it’s not as easy to search or back up. Neither tool is good for overall non-linear organization.

Most note-taking systems focus on indexes for paper notes, either with straightforward tables of contents or mindmaps that refer to pages by IDs. Evernote and OneNote have been around for a long time, so I’ll probably be able to find people who have thought about how to organize lots of information using those systems. In the meantime, I’ve been experimenting with using mindmaps to organize hyperlinks and next actions. I’m testing Mindjet MindManager, Xmind, and Freeplane. So far, I like Freeplane the most because:

• the GTD add-in lets me quickly get a list of next actions
• customizable keyboard shortcuts let me tweak the interface
• filtering lets me focus on a selected branch
• and hey, it’s free and open source

To make this, I added each item in my current public and private sketchbooks to my map, creating nodes when necessary. I don’t have a lot of topics in my sketchbook yet, but if I find myself with more than twenty or so items in a single category, it’s probably time to split that. I’ve split the categories in my blog index a few times, and I’m long overdue for splitting some of the others. It’s much easier to reorganize things in a map instead of editing each item, although I should come up with some kind of bulk interface so that I can update the categorizations.

The arrows are hyperlinks to either my blog posts or my Evernote entries (using evernote:// URLs from Copy Note Shortcut). I included private notes as well, so the map works only for me. Sorry! In the future, I’d love to make a version of this that omits private URLs and information. Freeplane supports lots of export formats, so maybe I’ll be able to process XML or HTML and make something that’ll help people browse too.

I’d like to get to the point of having a smooth workflow for drawing, scanning, publishing, organizing, browsing through, and following up on these thoughts. Do you know of anyone who’s doing something similar to this? This kind of visual thinking isn’t quite like the visual recording that most sketchnoters do. There are plenty of mindmaps at Biggerplate, but they look more like templates rather than thoughts-in-progress.

Reaching further back, parts of Leonardo da Vinci’s notebooks are online, and Prof. Carlo Pedretti’s introduction to the Codex Arundel has some notes on the structure: unnumbered loose sheets, usually one page per thought. That’s encouraging, although it doesn’t tell me much about overall structure. Part of Galileo Galilei’s notebooks are online, too. He numbered his pages and usually left plenty of whitespace. There are plenty of examples of note-takers throughout history, but it’s hard to find ones explicitly talking about how they map the connections between ideas. How to Read a Book briefly mentions building a syntopicon, so I checked out the resulting volume (A Syntopicon: An Index to The Great Ideas). It’s a huge project, but it didn’t give me a lot of clues about the process of building such a thing. More recently, there’s How to Make a Complete Map of Every Thought You Think, which described the process of making and updating a global subject map of contents. Books on study skills also talk about how to condense a lesson into successively smaller cheat sheets until you can fit everything onto one page (or an index card, or whatever the teacher permits), and that’s somewhat related to the kind of summarization and overview I want to do.

On the non-graphical front, I’ve also had fun making a huge outline of blog posts I want to write (and therefore things I want to learn about). Org Mode outlines work better than Freeplane maps for large amounts of text or fine-grained detail, so I’ll probably switch over to outlines when I’m drafting the post and then update the map with the hyperlink to the post when I’m done. It’s all tied together.

Anyway, here we are. I think the sketches and maps I’m making are promising, and I’m looking forward to digging deeper. If you happen to have put a lot of thought into a similar system, I’d love to hear from you!

I don’t write about everything. I think it might be interesting to write about more, to dig deeper into the things that people rarely write about—but since there’s so much to learn and share even in terms of topics you can talk about with complete strangers, I end up focusing on that instead. Less risky. If I’m writing about drawing, Emacs or other uncontroversially useful topics, I’m less likely to upset or offend (or bring out other odd tendencies in people). So I write more posts about that instead.

This is a bit of a pity because I’m learning so much about interesting things that I don’t yet know how to write publicly about. Decision-making in the face of uncertainty? Finances and semi-retirement? Making stuff happen? I’d love to write about what I’m figuring out. Maybe if I felt safer. (You never know what will bring down the wrath of the Internet – see Kathy Sierra and death threats.) Maybe if I cared less or worried less. People have done that before. Some writers are driven to write, even if the dynamics of relationships are a little bit odd. (A. J. Jacobs manages to pull this off well; I like how he did that in The Guinea Pig Diaries.)

I’m also learning a lot about interesting public things, so it’s not all that bad. =) Again, there’s tons to write about. But it also means that the things I’m learning about interesting, non-public things are more likely to be wasted.

I don’t think keeping an anonymous blog is enough. People get de-anonymized pretty often, and I don’t want to worry about slipping up.

Journals would probably be good, except that my track record of keeping paper notebooks is terrible and they are nowhere near as searchable as digital notes. Private notes in huge text files can get unwieldy and hard to review. Maybe I should use Evernote more often, and just work out some way to tag and organize the notes so that I can do the same kind of search and review that I use for my blog.

Hmm, maybe a private blog, since I already have the backup strategy for that one sorted out? Maybe a private part of the current blog?

Maybe that’s a good skill to figure out: how to keep good enough private notes so that I can build on them for future decisions or learning, or maybe even for time-delayed posting.

How do other people manage it? How do you manage it? How do you remember well enough to be able to build on that instead of wasting the time? How do you organize notes so that they don’t disappear after you’ve forgotten about them?

I’m curious about how to take more effective notes, so I’ve been researching different systems. I came across Lion Kimbro’s experiment with mapping out his thoughts years ago. I finally sat down and condensed the free 131-page e-book on How to Make a Complete map of Every Thought You Think (2003) into this one-page summary. You can click on it for a larger version and print it out if you want.

Feel free to print, share, or modify this image! (Creative Commons Attribution License)

The book describes a system for taking quick notes and integrating them into subject-based sections in a larger binder, with some notes on managing your archive. It also gives tips on splitting large subjects and mapping out the connections between topics.

I’ve been using some of the book’s ideas on colour and icons, indicating TODOs with green boxes and structure with blue ink. Maps are handy too, although I tend to use computers so that I can link and rearrange easily. Both visual thinking and technology have come a long way since 2003, when Lion Kimbro wrote this book. With Evernote and the Fujitsu ScanSnap, I can scan my sketches and file them along with my blog posts and other notes. I back up my data so that I’m less worried about losing my archive, and I also keep my paper notes by date. I’ve started using Freeplane to build my global subject map, which I’ll cover in a future blog post. There are still some tech gaps, but things are pretty cool.

Want to learn more? You can check out my blog posts about learning. Tell me what you think or what you’re curious about!

I’m celebrating my 30th birthday this August. Milestone birthdays are great excuses to look behind and look ahead. I don’t know how other people do it. I can barely remember what happened last week, much less ten years ago. Me, I cheat. I have blog archive, which 18-year-old me had the foresight to experiment with (although back then, I was just looking for a way to remember all those class notes and Emacs tidbits I was picking up). I’ve written more than six thousand blog posts in the last eleven years. (See Quantifying my blog posting history for a nifty visualization of my blog posting history.) My published posts probably include well over two million words. This is awesome.

Since not a lot of people have the same experience of blogging consistently over more than a decade, I thought I’d share what I’ve been learning along the way.

Have your own domain name. One of my first websites was on Geocities. Another was on Veranda.com.ph (hosted by I-Manila, which was our ISP then). Both services are long gone. I registered sachachua.com in 2006 and moved everything over to that. Since my name can be hard to spell, I registered LivingAnAwesomeLife.com in 2008. I‘ve started experimenting with my own URL shortening domains, sach.ac and liv.gd . While domain names are a recurring expense, they’ve been well worth it.

Move your data instead of starting from scratch. I changed blogging platforms (Emacs Planner Mode to WordPress) and moved web hosts, but I’d taken pains to move my data instead of starting fresh. Now I’m enjoying the benefits of having that archive handy.

Back up, back up, back up. I want this to be around in another sixty years. I like backing up the data in many different ways: database, files, HTML dumps, PDFs, even paper. I lost a bunch of photos and drawings when my Gallery2 setup got hacked, but I restored a number of them from files I found elsewhere. I look forward to being able to review decades and decades of notes.

Weekly, monthly, and yearly reviews go a long way towards making it easier to remember what happened. Day-to-day living makes it hard to see what’s important. A week seems to be the most natural chunk of time for my reviews. I wrote a little bit of code that auto-summarizes my task list and accomplishments. Every month, I

Search is your friend. If it takes a lot of digging to find something, make it more findable. I often use Google Search or my blog’s built-in search to find posts based on keywords that I remember. If it takes me a while to find something, I edit the post and add categories or tags to make it easier to find in the future. I sometimes write a new post that shares what I’ve learned since then, linking to the previous post for history.

Comments on older posts are awesome. Search engines are a wonderful, wonderful thing. I love it when people comment on old posts – it’s nice to know those posts are still helpful. Sometimes people comment on things I’ve completely forgotten writing, so it’s a great way to refresh my memory as well.

Check your analytics once in a while. I don’t really care about the number of visitors or the bounce rate, but I’m curious about what people are reading and where they’re coming from.

Indexes are good, too. Every month, I update this categorical index of my blog posts. I probably should go back and make sure that the WordPress categories match this as well, although in WordPress, I tend to use categories more like tags (I file a post in multiple categories).

Cultivate synchronicity and randomness. WordPress plugins help recommend similar posts, other posts that were written on the same day, and random posts. It might mean that my pages are overloaded with links… but it might also spark an aha! or an interesting conversation with someone browsing around, so I think it’s worth it. Besides, at this point, a computer will often be better than I could be at recommending other things that people should check out, so I use those features myself when I’m browsing my blog.

Write about the small stuff. I used to wonder whether the weekly reviews were worth posting on my blog, seeing as they’re mostly my task lists. Reviewing my blog years later, I was surprised to find that the weekly reviews were excellent at helping me remember what was going on. They were also great for filling in the blanks in my records – When did I fly out? What did I do? Whatever happened to that thing? Hooray for the small stuff.

Revise and summarize. It’s okay to write about something you’ve written before. In fact, it can be a great excuse to learn more and get closer to understanding the big picture.

If you’re starting out today, don’t worry. Stick with it, and in ten years, you’ll have something pretty darn awesome too.

Out of curiosity, do I know anyone else who’s got a big archive? How do you manage yours?

I’ve been blogging for almost ten years. I started with notes from my university classes and snippets of open source code, and became comfortable enough to share decisions I’m puzzling through and things I’m learning about life. There’s a lot of stuff in my archive, and I want to be able to review things again.

Categories would probably make this easier, but I use categories liberally and sometimes inconsistently. I use them like tags, quick keywords that I add so that people might explore a category and bump into other posts. I probably should split it out so that I assign posts to one category and leave everything else as tags. Someday.

In the meantime, it’s easy enough to maintain a manual topical index of my blog posts, and it’s a good opportunity to review what I’ve been writing as well.

I use Emacs Org Mode to manage a large text file divided into headings. Every month, I copy a list of titles into my topical index. I hacked Org-friendly output into my WordPress theme – you can see April’s blog posts as an example (sachachua.com/blog/2012/04/?org=1). I manually organize the list items under different headings, splitting off new headings when I can see a pattern. Working with two windows viewing the same buffer makes it easy to move information around, and org-refile is handy too. I use a checklist structure so that Org can automatically update the number of posts under each heading (C-u M-x org-update-statistics-cookies). When I’m happy with the structure, I use org-publish-current-file to publish it using the settings I’ve configured. The files are in my public Dropbox folder, so they’re automatically published to the Web. It takes me about 10 minutes to add a month of posts to my index and publish the page.

I like seeing how much I’ve written about different topics, and it encourages me to write and organize more posts. Maybe the index might be handy for other people too!

• Topical index as a webpage
• Org source for my blog index

I was thinking about information management and how I could get a better sense of what’s in my blog archive. I’ve written a lot over the years, enough that I’m surprised by what I find in here.

Topical index of blog posts from 2008-2012

I’ll add 2007 and earlier posts over the next week. I’m also looking forward to revisiting the map of things I want to learn, and consciously planning what to write.

To create this index, I used the Org-compatible output that I built into my WordPress theme (it outputs post titles in list format). I copied and pasted the list into an Org file, temporarily changed all the list items to headings, and used org-refile to move items under categories as needed. Afterwards, I converted the link headings back into list items and used org-export to export the HTML. The process was fairly easy, but it took me about four hours to process slightly over three years of blog posts.