sacha chua living an awesome life

I screwed up my self-hosted WordPress blog three ways in one evening.

First, I had an infinite loop thanks to Display Post Shortcode and include_content in a post that had the same tag that I was looking for. Right. Don’t do that. Remember to remove include_content or exclude the post right before publishing.

Then I published a post without checking several times that my site was still up. I checked twice, which was apparently not enough. I should probably check five times. Or ten. Or at least a few times after restarting PHP. Since I hadn’t checked, I spent a couple of hours playing a video game with W- instead of, say, stressing out about my site. Could’ve solved the problem sooner.

And then when I stressed out about my site, I “fixed it” in entirely the wrong way. I reduced php5-fpm‘s max_children instead of increasing it. This made it worse.

I had been worrying about running out of memory when I should’ve been worrying about running out of processes.

I felt that panic-induced haze setting in, scrambling through Google and adding all sorts of bits to my config, ripping out all sorts of plugins, and wondering how I could get a coredump. When I noticed I was making things worse, I made myself stop. I took a deep breath, and started untangling what was going on. I tried the opposite of what I had been trying. That worked.

Good timing, actually. Well, it could have been better timing, but that’s one of the nice things about minimizing commitments and making good stuff for free; it lowers the risk of trying things out and learning something new.

I’m still getting SIGSEGV errors. It happens even if there are few active server processes. But at least it’s sporadic instead of constant, and 1:59 AM is not the best time to dig into something like that. After sleep and reading, perhaps.

I’m a little bit nervous about my setup, and I’ll probably set aside some time this week to dig into system administration and build my skills. I really should have a good plan for downtime, and a better plan for learning the essentials outside of a fire. But mistakes are great because they show you multiple holes in your system, so this is not too bad. Better now than when I’m managing a client site.

On the plus side, I did have the presence of mind to temporarily redirect sach.ac to the static URL, switching it back after the PHP issues seemed to have cleared. Good strategy. Should do that first in the future.

With any luck, the blog is still up today. If you’re reading this, yay!

I’m still happy that I self-host, even if I make mistakes like this. =) Good time to make mistakes and learn from them.

So, what am I going to change for next time?

• Prepare a contingency plan for Stuff Happening, possibly involving throwing up a quick maintenance page that collects e-mails so that I can send abject apologies and link updates. Make this static so that it loads quickly, and have an external copy (ex: Dropbox) just in case my server is down and I have to redirect at the domain level.
• Check several times after restarting PHP. Just because it loads once doesn’t mean it’s going to load again.
• Consider redirecting URLs to static sketchnotes or external pages. Links and comments are nice, but viewing is essential.
• Start screen right away instead of trying to juggle different commands. Set up logs and go to different configuration directories in order to minimize typing.
• Don’t panic. Yes, website failures are embarrassing, but the nice thing about making this a gift is that I don’t have to worry about racking up the business losses. When I notice that I’m panicking (making wild guesses in terms of changes, for example), I should slow down and remind myself that This Is Not The End of The World.

Everything’s going to be all right.

Over ten years, my WordPress blog had ballooned to more than 500 categories. Part of it was because Org2Blog makes creating new categories super-easy, so I just piled them on (occasionally mispelling a few). Part of it was because I don’t really know what I’ll write a lot about until I write, so I had categories with one or two posts and then I moved on. Part of it was because I hadn’t decided what I’m going to use categories for and what I’m going to use tags for–yes, even after all those years.

I wanted to revamp my categories so that the Life, Geek, and Visual categories and their corresponding feeds might be useful to people who find my daily posts awesome-but-overwhelming and who would prefer a slice of my blog tailored to their interests. (There are even more categories on my archive page.) This meant organizing the categories into a hierarchy, but first I wanted to cut the number down to something more manageable.

The WordPress interface for managing categories leaves much to be desired when it comes to bulk actions. Fortunately, the Term Management Tools plugin makes it easy to merge categories or convert them to tags using additional Bulk Actions on the standard Categories screen.

I merged a few of the common typos, then converted any category with fewer than 10 posts into a tag. The original version failed silently when converting a category if a tag with the same name already existed, so I patched my version to silently merge the terms.

The Screen Options menu let me change the number displayed on screen to 100 items, which made it much easier for me to weed out most of my categories. Term Management Tools also provides a bulk action for setting a category parent, which was great for quickly reorganizing my categories into a hierachy.

I still had almost 3,000 uncategorized posts. Since I haven’t quite found or written an automatic N-gram text classifier for WordPress posts (if you have one, please share!), I decided to see if I could make a dent in this manually. I started by prioritizing the posts with comments. I assigned categories using the Posts screen, but that took a while and too many mouseclicks. The Categorized plugin automatically unchecks the default category once you select a different one, which saved me one click per post, but it still wasn’t enough. I ended up extracting a list of posts from my database with the following SQL command:

SELECT p.id, p.post_title, p.post_date, p.comment_count FROM wp_posts p 
INNER JOIN wp_term_relationships r ON (p.id=r.object_id AND r.term_taxonomy_id=1) 
WHERE p.post_type='post' AND p.post_status='publish' into outfile '/tmp/published.txt';

and another list of terms and taxonomy IDs:

SELECT t.*, tt.term_taxonomy_id FROM wp_terms t INNER JOIN wp_term_taxonomy tt ON (t.term_id=tt.term_id AND tt.taxonomy='category')
INTO OUTFILE '/tmp/terms.txt';

After a little spreadsheet manipulation involving VLOOKUP-ing the category name that I manually entered for each one, I copied the term taxonomy ID and post IDs into an Emacs buffer and used a keyboard macro to change it into the form:

UPDATE wp_term_relationships SET term_taxonomy_id=? WHERE object_id=? AND term_taxonomy_id=1;

where 1 was the term_taxonomy_id corresponding to Uncategorized.

Since I was on a roll, I decided to categorize everything from 2007 onwards, which is farther back than my manual index goes. That got me through about a thousand items before I decided it was enough filing for one day. As of the time of writing, there were 6512 posts on my blog. 4,536 posts (70%) belong to various categories, while 1,976 are still uncategorized.

I hope this work pays off! =) I expect that it will make my blog a little easier to browse.

Emacs keyboard shortcuts often mystify beginners because they’re not the same as the shortcuts for other applications (C-w instead of C-x for cutting text, etc.), and they’re long (what do you mean, C-x 5 f?!). I hope this guide will help break down the learning process for you so that you can pick up the keyboard shortcuts step by step. It’s under the Creative Commons Attribution License, so feel free to share!

Click on the image to view or save a larger version. It should print out fine on 8.5×11 paper in landscape mode, and you might even be able to go up to 11×17.

This is actually my second version of the guide. In the first one, I got a little sidetracked because I wanted to address common frustrations that get in people’s way. Here’s the Grumpy Guide to Learning Emacs Keyboard Shortcuts:

The #emacs channel on Freenode was totally awesome in terms of feedback and encouragement. Special thanks go to agumonkey, aidalgol, Fuco, ijp, JordiGH, nicferrier, pkkm, rryoumaa, and webspid0r for suggestions. =)

If you like this, you might also like the similar hand-drawn one-page guide I made on How to Learn Emacs, or my other Emacs-related posts. Enjoy!

For your convenience, you can find this page at http://sach.ac/emacs-keys.

Update 2013-08-30: By the way, here’s the link to Christopher Olah’s first post using org2blog. Neato!

The LaTeX logo, typeset with LaTeX (Photo credit: Wikipedia)

Org2Blog is an awesome geek-friendly way of writing posts and publishing them to a WordPress-based blog. When Christopher Olah told me that he’d gotten convinced to try Org Mode thanks to the enthusiastic recommendations from Michael Nielsen and me, we figured that getting him sorted out with taking notes in Org and publishing them through Org2Blog to his WordPress.com-hosted blog would be an excellent way to start – especially with inline images and LaTeX.

Chris has promised to write a blog post about what he’s learned, but he’ll probably find these notes useful. Here’s what we ran into when getting org2blog working on Ubuntu.

Need to install the files and set up the load paths

We downloaded the following and added them to an ~/.elisp directory

• https://github.com/punchagan/org2blog
• https://github.com/punchagan/metaweblog.el
• https://launchpad.net/xml-rpc-el

We also set up his ~/.init.d/emacs.el to load the libraries, set the blog list, and load ido-mode and icomplete-mode.

Emacs / Org too old

Metaweblog and Org2blog didn’t work well with Emacs 23 and Org 6. We upgraded to Emacs 24 with apt-get install emacs24 in order to get Org 7.

Can’t find library org

It turns out that you also need to apt-get install emacs24-el in order to include those libraries.

Org still too old for org2blog

We were having some problems with the version detection of org2blog, so we replaced the org2blog he downloaded with the version I forked at https://github.com/sachac/org2blog , which I’ve been using with Org 7.

Then we tested it with org2blog/wp-post-subtree, and that worked. Inline images with org-toggle-inline-images worked too, yay!

Next step: viewing LaTeX fragments, since Chris does a lot of math.

dvipng required

org-preview-latex-fragment wanted dvipng to be installed, so we apt-get install dvipng.

LaTeX fragment preview showed blank images

We looked at the *Messages* buffer and found that the .tex files in /tmp could not be rendered by dvipng because marvosym.sty could not be found. We fixed that with apt-get install texlive-fonts-recommended.

(Doing this on my own, I found that I also needed apt-get install texlive-latex-extra .)

… and then we could see and publish LaTeX fragments, which was awesome. =D

WordPress.com double-interpreted LaTeX fragments

Chris was having problems with LaTeX fragments when org-export-with-LaTeX-fragments was set to dvipng. It turns out that WordPress.com also interprets LaTeX, so it was getting confused by the alt tags. To solve this, use M-x customize org-export-with-LaTeX-fragmentsand choose Leave math verbatim. Then the LaTeX fragments are passed to WordPress, which renders them as PNGs.

Michael Jones wanted to know how I organized my Org Mode files. Here’s how I do things!

Org Mode for Emacs is an outliner that lets you add a little structure to plain text files. Not only can you use it to move around, hide, and show sections of your outline, but you can also:

• schedule tasks and mark them as complete,
• add hyperlinks and formatting,
• estimate effort and track time,
• export to HTML and other formats,
• and even include code that you can run in-line.

I started with a single Org Mode file (appropriately called organizer.org), but I’ve gradually fleshed this out into a number of files. My goals for organizing my files this way are to be able to:

• Publish some files while keeping other files private,
• Add or remove groups of tasks from my agenda, or focus my agenda/search on the current file,
• Simplify processing my weekly review (categorizing accomplishments/tasks),
• Get a quick overview of important things, and
• Have file-specific options, like columns.

I often use Org agenda custom commands to jump around. For example, one agenda command lists projects, and pressing RET on an agenda line will take me to that project. I also use org-capture to take a note from anywhere, and I use org-goto to navigate my files. For jumping to a specific file, I use ido-find-file.

I use several Org Mode files. The six files below have a little more than 1.3MB of text in total – tiny! – but they help me tremendously. I also have lots of other Org files like my Emacs configuration and my blog index (I often use Org for publishing), but these are my main files.

Personal tasks and notes: organizer.org

This is the catch-all for any tasks or notes that don’t belong to the files below. Here’s the rough structure:

• Quick notes: Tidbits that might not make it into their own blog posts, but which can be included in weekly reviews
• Reference: Hours, license keys, etc.
• Open loops: Anything I need to check on every so often
• Projects: High-level things I’m focusing on
• Financial goals: Things to save up for
• Someday/maybe: Projects to do someday
• Weekly review: Divided by year
• Monthly review: More summaries
• Plans: Personal plans
• 2011, 2012, 2013…: I use org-capture to quickly jot down notes. The datetree option automatically files them by day, which makes older ones easier to archive.
• Tasks: A bucket for miscellaneous tasks

Anything to do with business: business.org

I organize these by the types of tasks I focus on and the notes I want to keep.

• Earn
  • Clients
  • Leads
• Build
  • Projects
  • Research
  • Business ideas
  • Blog
  • Delegation
  • Planning
  • Business hygiene (accounting, etc.)
  • Learning
• Connect
  • Meetups
  • Hangouts
  • Other
• Reference
• Tasks

Relationships: people.org

I organize these by relationships so that I can remember who’s out there.

• Family
• Extended family
• Canada friends
• Hacklab
• Barkada
• Letters
• Meetups
• Bloggers
• Family friends
• Other tasks

Regular tasks: routines.org

I organize these by frequency and omit the tasks from my weekly review. This also contains my “In case of…” scenarios and my backup documentation.

• Every day
• Once a week
• Once a month
• Once a quarter
• Once a year
• When…

Outline for future blog posts: sharing/index.org

I organize this by topic. See http://sach.ac/outline for the published version

Decision review: decisions.org

I organize these by status. I also use org-choose markers (ex: CHOSEN, MAYBE) inside the categories, but the headings make it easier to review.

• Pending
• Current
• For review
• Someday / maybe
• Archive

Personal finance: ledger.org

I use John Wiegley’s command-line Ledger program to manage my finances. My financial data is in separate ledger-mode files, and I use an Org file with org-babel to make it easier for me to answer some questions about my finances. For example:

• Given my average monthly expenses and the amount of money I’ve set aside, how long can I sustain my early-retirement experiment?
• Am I ahead or behind in terms of household contributions?
• What did I spend on last month?
• Are my virtual envelopes balanced?

How do you organize your Org files or outlines?

Everyone’s got different ways of organizing outlines, and people also also change over time. How do you organize yours?

Update Aug 22 2013: See presentation at the end of this post.

I built library-book tracking into Quantified Awesome in October 2011, hard-coding the patterns used by the Toronto Public Library system. I regularly hit the 50-book checkout limit and sometimes have to check items out on my husband’s account, so it really helps to have a system that can tell me what’s due and when on all the library cards that we have.

Crunching the numbers

In the 668 days (or 95.5 weeks) between October 1, 2011 and the time I exported my data for number-crunching, we checked out 1,252 items, or an average of 13 items a week. That included 250 movies and 44 other videos (TV series, documentaries, and so on). I’m boggled to find out that I checked out only 8 science fiction books. There were 152 other fiction books, including graphic novels and manga.

… 250 movies borrowed from the library results in saving of $150+ a month assuming we snag DVDs at $15. Not that we would watch 2.6 movies a week if we had to pay for them. In November 2011, I tracked the retail prices and page count of the books I read: $1,075 and 10,671 pages in a month, boggle. I don’t read all of those pages thoroughly, mind you; I tend to skim books looking for just what I need. Still, there’s no denying that the Toronto Public Library saves me a heck of a lot of book and entertainment money.

I figured I’d probably want to take a look at my reading list at some point, so I had programmed the system to record titles and Dewey Decimal System classifications as well. Fiction books and feature movies tend to have generic codes, but nonfiction books show me interesting patterns in my reading habits.

Here are my top categories:

Here are my top sub categories:

Here they are, broken down by Dewey decimal group:

I read a lot of management, personal finance, and psychology books. I enjoy reading them. I read them at breakfast, over lunch, before bed, on weekend afternoons. I’m not surprised by the proportions, although I’m a little surprised by the number – have I really checked out an average of eight nonfiction books a week? Gotten through more than 300 management-related books? Neat.

Time

The time data in my current system goes back to November 29, 2011. Excluding the nonfiction books that were returned before then (although still including the books I currently have checked out that I haven’t read yet), there are 727 nonfiction books checked out. Let’s assume I’ve read or skimmed through 80% of those (I’m probably closer to 90%) – that’s ~580 books. I’ve tracked 123.3 hours as “Discretionary – Productive – Nonfiction”. This undercounts the number of hours because I tend to read things over meals and during subway commutes, so let’s double that time to be in the right ballpark for multitasking. That’s a little less than half an hour per book… which is actually quite reasonable, considering I skim through most books in 10-15 minutes each and spend maybe two hours reading selected books in depth (the ones that I take notes on, for example).

This is what my nonfiction reading habit looks like, with the dark boxes indicating when I read more. (This doesn’t take into account reading while doing other things.)

That’s interesting… I read a lot more frequently when I was starting up my business in January/February 2012 (although I wonder what happened in April!). I read more sporadically now. I think it’s because I’m re-figuring-out my strategies for taking notes and applying ideas to my life.

How do I pick books to read?

The library releases lists of new books on the 15th or 16th of every month. I’ve written a small script that extracts the titles, authors, and IDs of the book into a text file that I can review. I delete the lines that I’m not interested in, and my script then requests the books that remain on the list. I monitor the new releases because I don’t want to wait for the usual press

When I ‘m learning about a topic, I tend to check out six or more books related to it. A wide variety of books lets me see different viewpoints, and I can focus on books of better quality.

I occasionally look at Amazon’s recommendations for other ideas, although the books are often not yet available at the library.

Sketchnoting a new release can have high impact, which is another reason why I monitor the new releases. I sometimes reach out to publishers for review copies as well.

The library doesn’t carry everything. I usually add other interesting releases to my Amazon wishlist. I rarely buy books, though, because there’s just such an interesting backlog that I haven’t yet gotten through. I buy books if there are clever illustrations that I’d like to use for ongoing inspiration, or if I want to give the book to a friend, or if it’s an older book that someone has recommended to me and the library doesn’t stock it. Now that I have a business, I usually file those books as business expenses.

So much for quantity. If I’m reading all that, what am I doing with it?

I use books to:

• Learn about different viewpoints and approaches, especially scenarios that I might not anticipate on my own
• Learn how to organize and communicate complex ideas
• Get shortcuts on explaining ideas – for example, I don’t have to explain outsourcing from scratch, because I can point people to the 4-Hour Work Week for starters

Many of the ideas I pick up from books resurface in my blog posts and experiments. Books help me recognize what’s going on in real life, because the authors have already come up with words for them. I also like applying the advice from books – much to learn.

I frequently recommend books to other people. Visual book reviews make that easier. I try to slow down and recognize books that I’ll probably recommend to others so that I can make visual book reviews of them while I have the book. Sometimes I’ll take quick text notes for myself and then use that for reference. If I find myself recommending the book frequently, then I’ll check it out again and make better notes.

I don’t review my book notes much, relying instead on situations to trigger my memories. When I come across something that’s related to a book I’ve read, or I talk to someone who could benefit from a book recommendation, I dig through my visual and text-based notes.

Next steps

“Better” isn’t about reading more books – it’s about being able to apply, organize, and share what I’m learning from those books. I want to learn more about doing good research: identifying a topic to explore, synthesizing insights from multiple sources, and adding personal experiences or ideas. The process might look like this:

• Outline a topic for research
• Search the library catalogue for resources; also check bibliographies, Amazon, and other recommendations
• File properly-cited notes so that I can give credit later on
• Compare and organize ideas from different sources
• Test ideas in small experiments
• Write blog posts, then articles, then e-books

Visual book reviews are another way for me to grow. In addition to making visual book reviews of interesting new releases, I’d like to revisit the books that were a big influence on me in order to make visual reviews of them too.

Yay for Quantified Self and tracking. Onward!

(Also, if you’re curious about tracking your own library use: I can probably extend Quantified Awesome to support other libraries with online interfaces, but you’re going to need to walk me through how your library works.)

Update from August 22, 2013: Here’s a presentation I’m putting together for Quantified Self Toronto.

Here are my notes from Damian Conway’s talk “Fun With Dead Languages”. =) I heard him give an older version of this talk years ago, and I’m amused to find that my Latin dabbling gave me a much deeper appreciation of this talk.

As always, click on the image to view a larger version, which you can print out if you want.

Please feel free to share this under the Creative Commons Attribution License! =)

If you like this, check out Damian Conway’s site or this paper on Lingua::Latina::Perligata. Like these sketches? Check out my other sketchnotes and visual book reviews.