Category Archives: org

Cobbling together a semi-auto-responder using Emacs, Gnus, and org-contacts

It turns out that lots of people are interested in an e-mail-based course for learning Emacs Lisp. Yay! =) Maybe it’s the idea of bite-size chunks. Maybe it’s the ease of asking questions. Maybe it’s the regular reminders to work on something. Who knows? Whatever the reason, it’s awesome to see so many people willing to join me on this experiment.

Since this is my first time to venture into the world of teaching people online, I wanted to see how far I could push actually doing all the mails myself, instead of just signing up for an Aweber account and handing everyone off to an impersonal autoresponder. I dusted off Gnus, offlineimap, and org-contacts, and started figuring out my workflow. I’ll share how that workflow’s evolving so that you can get a sense of how someone might write little bits of Emacs Lisp to make something repetitive easier.

For the first little while, I got by with using C-x r s (copy-to-register) and C-x r i (insert-register) to store the text that I needed.
Sometimes I needed to paste in the welcome message and checklist, and sometimes I needed to paste in the first lesson. By using registers, I could insert whatever I wanted instead of going through the kill ring. I also had another bit of templated code in yet another register so that I could easily create an org-contacts entry for the person whose mail I was replying to. In the beginning, I used tasks under each person’s heading to indicate that I had sent them the checklist or that I had sent them the first lesson. Eventually, I changed my org-contacts notes so that the TODO state of each person showed which lesson I was going to send them next, or CHECKLIST if I was waiting for their reply to the checklist. I also set up Org so that it would automatically log when the TODO state was changed.

#+TODO: TODO | DONE
#+TODO: CHECKLIST(c!) BEGINNER1(1!) BEGINNER2(2!) BEGINNER3(3!) BEGINNER4(4!) FULL(f!) | FINISHED(x!)
#+TODO: | CANCELLED

* Who
** CHECKLIST Jane Smith ...
** BEGINNER1 John Smith
   SCHEDULED: <2014-05-28 Wed>
   :PROPERTIES:
   :EMAIL: [email protected]
   :END:
(notes from the messages, etc.)

I wrote some code to make it easier to send someone a checklist and create a note for them in my org-contacts file. I bound it to C-c e c for convenience.
(The bind-key function is defined by a package.)

(setq sacha/elisp-course-checklist-body "... really long text here...")
(defun sacha/elisp-course-checklist ()
  "Copy this message and put it at the end as a checklist item. 
Start a message with the checklist."
  (interactive)
  (gnus-summary-scroll-up 1)
  (with-current-buffer gnus-article-buffer
    (let ((message (buffer-substring-no-properties (point-min) (point-max)))
          (email (cadr (org-contacts-gnus-get-name-email))))
      (with-current-buffer "elisp-course.org"
        (save-excursion
          (goto-char (point-max))
          (save-excursion
            (insert "\n** " message)
            (org-set-property "EMAIL" email)
            (org-todo "CHECKLIST"))))))
  (gnus-summary-followup-with-original nil)
  (goto-char (point-max))
  (insert sacha/elisp-course-checklist-body))
(bind-key "C-c e c" 'sacha/elisp-course-checklist)

This made it easier for me to read the starred messages from my inbox and use C-c e c to get a head start on processing people’s introductory messages.
Yay! I used the register trick to help me reply to people who were ready for the first lesson. After the first few replies, I noticed that the attachment code was fine even if I put that in the register too, so I added it as well.

Things got more complicated when I started processing lesson 2. I didn’t want to have to set up and remember lots of different registers, and I didn’t want to manually update the TODO states either. So I started defining functions that I could call with keyboard shortcuts:

(defun sacha/elisp-course-1 ()
  (interactive)
  (let ((marker (org-contacts-gnus-article-from-get-marker)))
    (if marker
        (org-with-point-at marker
          (org-todo "BEGINNER2"))))
  ;; Find the person's contact record
  (gnus-summary-scroll-up 1)
  (gnus-summary-followup-with-original nil)
  (message-goto-subject)
  (message-delete-line)
  (insert (concat "Subject: " sacha/elisp-course-1-subject "\n"))
  (goto-char (point-max))
  (insert sacha/elisp-course-1-body))
(bind-key "C-c e 1" 'sacha/elisp-course-1)
(defun sacha/elisp-course-2 ()
  (interactive)
  (let ((marker (org-contacts-gnus-article-from-get-marker)))
    (if marker
        (org-with-point-at marker
          (org-todo "BEGINNER3"))))
  ;; Find the person's contact record
  (gnus-summary-scroll-up)
  (gnus-summary-followup-with-original nil)
  (goto-char (point-max))
  (insert sacha/elisp-course-2-body))
(bind-key "C-c e 2" 'sacha/elisp-course-2)

Really, though, it doesn’t make sense to have a lot of duplicated code. So I wrote some code that would use the person’s TODO keyword to look up the message to send them, and then move them to the next keyword. Now I don’t need sacha/elisp-course-1 or sacha/elisp-course-2 any more.

(setq sacha/elisp-course-info
      `(("CHECKLIST" nil ,sacha/elisp-course-checklist-body)
        ("BEGINNER1" ,sacha/elisp-course-1-subject ,sacha/elisp-course-1-body)
        ("BEGINNER2" ,sacha/elisp-course-2-subject ,sacha/elisp-course-2-body)))

(defun sacha/elisp-course-process (subject body &optional state)
  "Process this course entry."
  (if (derived-mode-p 'org-mode)
      (progn
        ;; Move this node to the next state and compose a message
        (if state (org-todo state))
        (org-todo 'right)
        (message-mail (org-entry-get (point) "EMAIL") subject)
        (goto-char (point-max))
        (insert body))
    ;; Doing this from Gnus; find the person's info
    (let ((marker (org-contacts-gnus-article-from-get-marker)))
      (if marker (org-with-point-at marker
                   (if state (org-todo state))
                   (org-todo 'right)))
      ;; Compose a reply
      (gnus-summary-scroll-up 1)
      (gnus-summary-followup-with-original nil)
      (message-goto-subject)
      (message-delete-line)
      (insert (concat "Subject: " subject "\n"))
      (goto-char (point-max))
      (insert body))))

(defun sacha/elisp-course-guess-and-process (&optional state)
  (interactive (list (if current-prefix-arg (read-string "State: "))))
  (let ((current-state
         (or state (elt
                    (if (derived-mode-p 'org-mode)
                        (org-heading-components) 
                      (let ((marker (org-contacts-gnus-article-from-get-marker)))
                        (if marker (org-with-point-at marker (org-heading-components)))))
                    2))))
    (sacha/elisp-course-process
     (elt (assoc current-state sacha/elisp-course-info) 1)
     (elt (assoc current-state sacha/elisp-course-info) 2)
     state)))
(bind-key "C-c e e" 'sacha/elisp-course-guess-and-process)

Come to think of it, I should totally have it schedule the next update for the next Wednesday, too. ;) That’s just (org-schedule "+wed"). Neat, huh?
And I’m sure there are all sorts of ways the code can be simpler, but it works for me at the moment, so hooray!

I really like this approach. It lets me pull in standard information while also letting me customize the messages and how it fits into my task tracking. I can’t get that with Gmail (even with canned responses), and I’m not sure any CRM is going to be quite as awesome as this. I can’t wait to see how else we’ll tweak this as we go through more conversations. I’d like to get better at:

  • having a consistent place where I can process all the messages and make sure nothing falls through the cracks; I currently star messages to make sure I process them, since the Gmail label folder in IMAP seems to be missing some messages
  • seeing all Gnus conversations related to an org-contacts entry
  • reaching out to people proactively with the next lesson, even if they haven’t e-mailed me (or maybe I should wait for them?)

Anyway, that’s an example of writing a little bit of Emacs Lisp in order to connect different packages. Gnus handles mail, Org handles notes, org-contacts links the two together, and with a little bit of custom code, I can make the combination fit what I want to do. I read the source code of org-contacts to find out how I could look up the appropriate note, and I looked at org-shiftright to find out how to move things to the next TODO state. If you know something that works roughly like what you want it to work, you can find out how it does things and then copy that.

As for the course itself: I’ve been sending people links to the HTML output, attached .txt files (with -*- mode: org -*-) so they can open it in Emacs if they want, and inline text so that they can skim it briefly in their e-mail client if they want to. I’m not perfectly happy with the plain-text formats, but it seems to be a reasonable compromise, and so far people have been able to deal with it. I’ve been improving pieces of it based on feedback on clarity, suggestions for good examples, and so on. I didn’t take all the feedback; after thinking about some of the suggestions, I still preferred it my way. It’s shaping up quite nicely, though!

If you’re curious about the beginner’s course on reading Emacs Lisp, e-mail me at [email protected] and we’ll see how this works out. I’m certainly learning a lot. =)

How to update the Org 7 that comes with Emacs to Org 8 (more configuration! better exports!)

Update 2014-05-12: Simplified thanks to Sebastian’s note that Org 8 is available in the built-in package repository, yay!

The Org Mode included in Emacs 24 is version 7. Version 8 has lots of new configuration variables and the exporting mechanism has been rewritten. However, it needs to be installed in an Emacs that has not yet loaded any Org code or files. Here’s how you can upgrade your Org:

  1. Start Emacs with emacs -q. This skips your personal configuration.
  2. You will need an Internet connection for this step. Type M-x package-install, and type in org. This will install the latest version of Org from the built-in package repository.
  3. Edit your ~/.emacs.d/init.el (or ~/.emacs, if you’re using that instead). Add the following code to the beginning of the file:
    (package-initialize)
    (setq package-enable-at-startup nil)
    

    This will load the installed packages when you start Emacs, overriding the buit-in Org 7 with the Org 8 version that you installed.

    Advanced note: If you’ve downloaded Emacs Lisp code that should override code already installed through packages, you need to change this to (package-initialize nil) instead, and add (package-initialize t) after your load-path settings.

  4. Check your configuration for references to the older version of Org. In particular, look for any configuration related to exporting (ex: (require 'org-html)). You can change those lines to their Org 8 equivalents (ex: (require 'ox-html)), but it’s probably easier to just comment them out for now. You can comment out lines by adding ; to the beginning.
  5. Save your init.el and restart Emacs (this time, without the -q option). M-x org-version should now start with Org-mode version 8.
  6. Review your Emacs configuration for any changes that you will need to make. You can ask the Org Mode mailing list for help if you get stuck.

Good luck!

Getting R and ggplot2 to work in Emacs Org Mode Babel blocks; also, tracking the number of TODOs

I started tracking the number of tasks I had in Org Mode so that I could find out if my TODO list tended to shrink or grow. It was easy to write a function in Emacs Lisp to count the number of tasks in different states and summarize them in a table.

(defun sacha/org-count-tasks-by-status ()
  (interactive)
  (let ((counts (make-hash-table :test 'equal))
        (today (format-time-string "%Y-%m-%d" (current-time)))
        values output)
    (org-map-entries
     (lambda ()
       (let* ((status (elt (org-heading-components) 2)))
         (when status
           (puthash status (1+ (or (gethash status counts) 0)) counts))))
     nil
     'agenda)
    (setq values (mapcar (lambda (x)
                           (or (gethash x counts) 0))
                         '("DONE" "STARTED" "TODO" "WAITING" "DELEGATED" "CANCELLED" "SOMEDAY")))
    (setq output
          (concat "| " today " | "
                  (mapconcat 'number-to-string values " | ")
                  " | "
                  (number-to-string (apply '+ values))
                  " | "
                  (number-to-string
                   (round (/ (* 100.0 (car values)) (apply '+ values))))
                  "% |"))
    (if (called-interactively-p 'any)
        (insert output)
      output)))
(sacha/org-count-tasks-by-status)

I ran this code over several days. Here are my results as of 2014-05-01:

Date DONE START. TODO WAIT. DELEG. CANC. SOMEDAY Total % done + done +canc. + total + t – d – c Note
2014-04-16 1104 1 403 3 1 104 35 1651 67%
2014-04-17 1257 0 114 4 1 171 107 1654 76% 153 67 3 -217 Lots of trimming
2014-04-18 1292 0 74 4 5 183 100 1658 78% 35 12 4 -43 A little bit more trimming
2014-04-20 1305 0 80 4 5 183 100 1677 78% 13 0 19 6
2014-04-21 1311 1 78 4 4 184 99 1681 78% 6 1 4 -3
2014-04-22 1313 2 75 4 4 184 99 1681 78% 2 0 0 -2
2014-04-23 1369 4 66 4 5 186 101 1735 79% 56 2 54 -4 Added sharing/index.org
2014-04-24 1371 3 69 4 5 186 101 1739 79% 2 0 4 2
2014-04-25 1379 3 60 3 5 189 103 1742 79% 8 3 3 -8
2014-04-26 1384 3 65 3 5 192 103 1755 79% 5 3 13 5
2014-04-27 1389 2 66 3 5 192 103 1760 79% 5 0 5 0
2014-04-28 1396 3 67 3 5 192 103 1769 79% 7 0 9 2
2014-04-29 1396 3 67 3 5 192 103 1769 79% 0 0 0 0
2014-04-30 1404 4 70 4 5 192 103 1782 79% 8 0 13 5
2014-05-01 1413 4 80 3 4 193 103 1800 79% 9 1 18 8

Here’s the source for that table:

#+NAME: burndown
#+RESULTS:
|       Date | DONE | START. | TODO | WAIT. | DELEG. | CANC. | SOMEDAY | Total | % done | + done | +canc. | + total | + t - d - c | Note                       |
|------------+------+--------+------+-------+--------+-------+---------+-------+--------+--------+--------+---------+-------------+----------------------------|
| 2014-04-16 | 1104 |      1 |  403 |     3 |      1 |   104 |      35 |  1651 |    67% |        |        |         |             |                            |
| 2014-04-17 | 1257 |      0 |  114 |     4 |      1 |   171 |     107 |  1654 |    76% |    153 |     67 |       3 |        -217 | Lots of trimming           |
| 2014-04-18 | 1292 |      0 |   74 |     4 |      5 |   183 |     100 |  1658 |    78% |     35 |     12 |       4 |         -43 | A little bit more trimming |
| 2014-04-20 | 1305 |      0 |   80 |     4 |      5 |   183 |     100 |  1677 |    78% |     13 |      0 |      19 |           6 |                            |
| 2014-04-21 | 1311 |      1 |   78 |     4 |      4 |   184 |      99 |  1681 |    78% |      6 |      1 |       4 |          -3 |                            |
| 2014-04-22 | 1313 |      2 |   75 |     4 |      4 |   184 |      99 |  1681 |    78% |      2 |      0 |       0 |          -2 |                            |
| 2014-04-23 | 1369 |      4 |   66 |     4 |      5 |   186 |     101 |  1735 |    79% |     56 |      2 |      54 |          -4 | Added sharing/index.org    |
| 2014-04-24 | 1371 |      3 |   69 |     4 |      5 |   186 |     101 |  1739 |    79% |      2 |      0 |       4 |           2 |                            |
| 2014-04-25 | 1379 |      3 |   60 |     3 |      5 |   189 |     103 |  1742 |    79% |      8 |      3 |       3 |          -8 |                            |
| 2014-04-26 | 1384 |      3 |   65 |     3 |      5 |   192 |     103 |  1755 |    79% |      5 |      3 |      13 |           5 |                            |
| 2014-04-27 | 1389 |      2 |   66 |     3 |      5 |   192 |     103 |  1760 |    79% |      5 |      0 |       5 |           0 |                            |
| 2014-04-28 | 1396 |      3 |   67 |     3 |      5 |   192 |     103 |  1769 |    79% |      7 |      0 |       9 |           2 |                            |
| 2014-04-29 | 1396 |      3 |   67 |     3 |      5 |   192 |     103 |  1769 |    79% |      0 |      0 |       0 |           0 |                            |
| 2014-04-30 | 1404 |      4 |   70 |     4 |      5 |   192 |     103 |  1782 |    79% |      8 |      0 |      13 |           5 |                            |
| 2014-05-01 | 1413 |      4 |   80 |     3 |      4 |   193 |     103 |  1800 |    79% |      9 |      1 |      18 |           8 |                            |
#+TBLFM: @3$11..@>$11=$2-@-1$2::@3$13..@>$13=$9-@-1$9::@3$14..@>$14=$13-$11-($7-@-1$7)::@3$12..@>$12=$7-@-1$7

I wanted to graph this with Gnuplot, but it turns out that Gnuplot is difficult to integrate with Emacs on Microsoft Windows. I gave up after a half an hour of poking at it, since search results indicated there were long-standing problems with how Gnuplot got input from Emacs. Besides, I’d been meaning to learn more R anyway, and R is more powerful when it comes to statistics and data visualization.

Getting R to work with Org Mode babel blocks in Emacs on Windows was a challenge. Here are some of the things I ran into.

The first step was easy: Add R to the list of languages I could evaluate in a source block (I already had dot and ditaa from previous experiments).

(org-babel-do-load-languages
 'org-babel-load-languages
 '((dot . t)
   (ditaa . t) 
   (R . t)))

But my code didn’t execute at all, even when I was trying something that printed out results instead of drawing images. I got a little lost trying to dig into org-babel-execute:R with edebug, eventually ending up in comint.el. The real solution was even easier. I had incorrectly set inferior-R-program-name to the path of R in my configuration, which made M-x R work but which meant that Emacs was looking in the wrong place for the options to pass to R (which Org Babel relied on). The correct way to do this is to leave inferior-R-program-name with the default value (Rterm) and make sure that my system path included both the bin directory and the bin\x64 directory.

Then I had to pick up the basics of R again. It took me a little time to figure out that I needed to parse the columns I pulled in from Org, using strptime to convert the date column and as.numeric to convert the numbers. Eventually, I got it to plot some results with the regular plot command.

dates <- strptime(as.character(data$Date), "%Y-%m-%d")
tasks_done <- as.numeric(data$DONE)
tasks_uncancelled <- as.numeric(data$Total) - as.numeric(data$CANC.)
df <- data.frame(dates, tasks_done, tasks_uncancelled)
plot(x=dates, y=tasks_uncancelled, ylim=c(0,max(tasks_uncancelled)))
lines(x=dates, y=tasks_uncancelled, col="blue", type="o")
lines(x=dates, y=tasks_done, col="green", type="o")

r-plot

I wanted prettier graphs, though. I installed the ggplot2 package and started figuring it out. No matter what I did, though, I ended up with a blank white image instead of my graph. If I used M-x R instead of evaluating the src block, the code worked. Weird! Eventually I found out that adding print(...) around my ggplot made it display the image correctly. Yay! Now I had what I wanted.

library(ggplot2)
dates <- strptime(as.character(data$Date), "%Y-%m-%d")
tasks_done <- as.numeric(data$DONE)
tasks_uncancelled <- as.numeric(data$Total) - as.numeric(data$CANC.)
df <- data.frame(dates, tasks_done, tasks_uncancelled)
plot = ggplot(data=df, aes(x=dates, y=tasks_done, ymin=0)) + geom_line(color="#009900") + geom_point() + geom_line(aes(y=tasks_uncancelled), color="blue") + geom_point(aes(y=tasks_uncancelled))
print(plot)

 r-graph

The blue line represents the total number of tasks (except for the cancelled ones), and the green line represents tasks that are done.

Here’s something that looks a little more like a burn down chart, since it shows just the number of things to be done:

library(ggplot2)
dates <- strptime(as.character(data$Date), "%Y-%m-%d")
tasks_remaining <- as.numeric(data$Total) - as.numeric(data$CANC.) - as.numeric(data$DONE)
df <- data.frame(dates, tasks_remaining)
plot = ggplot(data=df, aes(x=dates, y=tasks_remaining, ymin=0)) + geom_line(color="#009900") + geom_point()
print(plot)

r-graph-2

The drastic decline there is me realizing that I had lots of tasks that were no longer relevant, not me being super-productive. =)

As it turns out, I tend to add new tasks at about the rate that I finish them (or slightly more). I think this is okay. It means I’m working on things that have next steps, and next steps, and steps beyond that. If I add more tasks, that gives me more variety to choose from. Besides, I have a lot of repetitive tasks, so those never get marked as DONE over here.

Anyway, cool! Now that I’ve gotten R to work on my system, you’ll probably see it in even more of these blog posts. =D Hooray for Org Babel and R!

Update 2014-05-09: Stephen suggested http://blogs.neuwirth.priv.at/software/2012/03/28/r-and-emacs-with-org-mode/ for more tips on setting up Org Mode with R and Emacs Speaks Statistics (ESS).

Thinking about my TODO keywords

It’s been twelve years since David Allen published Getting Things Done, with its geek-friendly flowcharts and processes for handling tasks in an interrupt-driven life. The way I manage my tasks is heavily influenced by GTD. I think in terms of next actions, waiting, and someday, and I have weekly reviews. I modified the TODO states a little to reflect what I need. It’s time to think about those states again to see what I can tweak and what reports I could use.

I use Org Mode in Emacs to manage my tasks and my notes. I can customize it to give me different kinds of reports, such as showing me all of my unscheduled tasks, or all tasks with a specific category, or even projects that are “stuck” (no next actions defined). Thinking about my processes will help me figure out what reports I want and how I want to use them.

Here are different types of tasks and how I track them:

  • Things I can work on right now (next actions): TODO
  • Things that I can work on after a different task is finished: currently WAITING, but probably better to implement with org-depend
  • Things I will revisit at a certain date, but I don’t need to think about them until then: TODO, scheduled (I used to use POSTPONED)
  • Things that would be nice to do someday, but maybe are incompletely specified or understood: SOMEDAY
  • Things I have decided not to work on: CANCELLED
  • Things I have asked someone else to do: DELEGATED
  • Things I can ask someone else to do: TODELEGATE
  • Things I am waiting for (usually not based on date) and that I need to follow up on: WAITING
  • Things I can write about: TOBLOG. These are pretty optional, so I don’t want them in my TODO list…
  • If something is a duplicate of something else – remove TODO keyword and add link?

I use the following code for an agenda view of unscheduled tasks:

(defun sacha/org-agenda-skip-scheduled ()
  (org-agenda-skip-entry-if 'scheduled 'deadline 'regexp "\n]+>"))

(add-to-list 'org-agenda-custom-commands
   '("u" "Unscheduled tasks" alltodo ""
     ((org-agenda-skip-function 'sacha/org-agenda-skip-scheduled)
     (org-agenda-overriding-header "Unscheduled TODO entries: "))))

So the to-do process looks like this:

  • Every week, review my evil plans and projects. Check my agenda without the routine tasks to see what new things I’m working on. Schedule a few tasks to encourage me to make regular progress.
  • Every day, go through my Org agenda (C-c a a) and do all the tasks that are scheduled.
  • When I’m done or if I feel like working on something else:
    • What do I feel like doing? If there’s a specific activity that I feel like:
      • Go to the relevant project/section of my TODO list, or check the TODOs by context (drawing, writing, etc.)
      • Clock in on that task.
    • If there’s a specific task I feel like working on:
      • Find the task, maybe with C-u C-c C-w (org-refile) and work on it.
    • If there’s a new idea I want to work on:
      • Use org-capture to create the task, file it in the appropriate project, and then clock in.
  • If I have an idea for a task, use org-capture to create the task and file it in the appropriate project.

How do I want to improve this?

  • Maybe get more used to working with contexts? I have all these Org Agenda commands and I hardly ever use them. I tend to work with projects instead. Actually, working with projects makes sense too, because that minimizes the real context shift.
  • Get better at reviewing existing tasks. I started tracking the number of tasks in each state (DONE, TODO, etc.), which nudged me to review the tasks and cross old tasks off. If I streamline my process for capturing tasks, filing them, and reviewing them by project/context/effort, then I can get better at choosing good tasks to work on from my existing TODO list.
  • Estimate effort for more tasks, and use that more often I have some reports that can filter or sort by estimated effort. I don’t use effort that much, though. Does it makes sense to get into the habit of choosing tasks by estimated time as an alternative approach? I usually have fairly large, flexible blocks of time…
  • Tag things by level of energy required? I want to take advantage of high-energy times. So, when I feel alert and creative, I want to focus on coding and writing. I can save things like paperwork for low-energy times. I can tag some tasks as :lowenergy: and then filter my reports.

Hmm…

How Org Mode helps me deal with an ever-growing backlog

If you’re like me, you probably have a to-do list several miles long. I like thinking of this as the backlog from agile programming. It’s a list of tasks that I could choose to work on, but I haven’t committed to doing everything on the list. This means I don’t have to waste energy feeling guilty about not getting everything done. Instead, I can treat it like a buffet of projects to choose from depending on what I feel like working on.

2014-04-28 Dealing with an ever-growing backlog

2014-04-28 Dealing with an ever-growing backlog

I think I add tasks faster than I cross tasks out. (Hmm, I should track this!) It never ends. Most tasks suggest next steps I could take after I finish the first ones. You might think that an ever-growing to-do list is a bad thing. This is okay. In fact, this is good. It means that I’ll always have a variety of tasks to choose from.

People manage tasks in different ways. For my personal tasks, I use several large text files in Org Mode for Emacs. Org Mode is an outline-based tool which makes it easy for me to organize my tasks into projects and projects into themes. It also supports tagging, links, agendas, dynamic views, and all sorts of other great ways to slice-and-dice my task list. Here’s how I deal with some of the common challenges people face with a large task backlog:

  • Making sure important, urgent tasks don’t fall through the cracks
  • Making sure you don’t neglect important but not urgent tasks
  • Keeping track of what you’re waiting for
  • Catching procrastination

Making sure important, urgent tasks don’t fall through the cracks

If something has a deadline, I add the deadline in Org using C-c C-d (org-deadline). This means that reminders will appear on my daily agenda for the 14 days before the deadline, counting down to the deadline itself. (The number of days is controlled by org-deadline-warning-days.) In addition, I usually schedule the task for a day that I want to work on it, so that I can get the task out of the way.

I’m careful about what I commit to, erring on the side of under-committing rather than over-committing. I’m selective about my client work and my volunteering. I keep my schedule as open as I can, and I’m not afraid to reschedule if I need to. Hardly anything I work on could be considered urgent. If an urgent request does come in, I ask questions to determine its true urgency, including potential alternatives and consequences of failure.

You might not have as much choice about what to work on, but you might also be surprised by how much you can push back. Be careful about what you allow to be urgent in your life.

Making sure you don’t neglect important but not urgent tasks

I have plenty of space to work on things that are important but not urgent because I manage my commitments carefully. This means that I can usually finish a few important-but-not-urgent tasks every day.

Which tasks do I consider important? I like thinking in terms of projects. Important tasks tend to be associated with projects instead of standing in isolation. Important tasks move me toward a specific goal. I have many goals and projects, but because they’re fewer than the number of tasks I have, I can prioritize them more easily. I can decide that some projects are in the background and some are in focus. Important tasks are the tasks that help me make more progress on the projects I consider important.

Because I like having two or three projects on the go, it helps to make sure that I make regular progress on those projects instead of getting carried away on just one. Tracking my time helps me stay aware of that balance. I also review my projects every week and schedule specific tasks for each of them, so I can make a little progress at least. Once I switch context and start thinking about a project, it’s easy to pick another couple of tasks in that area and get even more done.

If you’re struggling with creating enough space to work on important but not urgent tasks, you might be able to partner up with someone so that you can block off time to work on non-urgent things. Many teams have a rotating schedule for dealing with customer requests or urgent issues. One person covers the requests for a day, allowing the rest of the team to focus. Then the next person takes on that duty, and so on.

Keeping track of what you’re waiting for

One of the useful tips I picked up from David Allen’s Getting Things Done book was the idea of marking a task as WAITING. I usually add a description of what I’m waiting for, who’s responsible, and when I want to follow up. This makes it easier to follow up. When I’m waiting for a specific date (ex: the library makes a DVD hold-able after a certain date), I schedule the task for then.

I use the Boomerang for Gmail extension when I’m waiting for an e-mail reply. Boomerang lets me pop the message back into my inbox if I haven’t received a reply by a specific date, so I don’t have to keep track of that myself.

Handling less-important but still useful things

There are tasks on my to-do list that have been on that list for years. This is okay.
I’m getting better at noting names and contact information in my tasks so that I can follow up with people even after some time. This is particularly useful for book recommendations. I get a lot of book recommendations and I get most of my books from the library, so there’s usually a delay of a few weeks. Because Org Mode lets me add notes and links to the body of a task, I can look up information easily.

I work on less-important tasks when I don’t feel like working on my major tasks, or when I’m looking for small tasks so I can fill in the gaps of my day. Org Mode gives me plenty of ways to look up tasks. I usually look for tasks by projects, navigating through my outline. I can also look for tasks by effort estimate, so I can see everything that will probably take me less than 15 minutes. Context is useful too – I can search for various tags to find tasks I can do while I’m on the phone, or out on errands, or when I feel like writing or drawing.

I like thinking in terms of low-hanging fruit, so I often choose tasks that require little time or effort and have good impact. It can be overwhelming to look at a long list of tasks and decide which ones have good return. It’s easier to tag these tasks when you create the task, or to think in terms of projects instead.

Some tasks grow in importance or urgency over time. If I want to make sure that I revisit a task on a certain date, I schedule it for then.

Catching procrastination

I still end up rescheduling tasks multiple times. (I’ve been putting off redoing my business cards for a few months now!) I’ve noticed that there are different kinds of procrastination, including:

  • Procrastinating because you don’t have time today: It’s easy to reschedule things a few weeks or a month in advance. In fact, Org has a built-in command for bulk-scattering tasks. From the agenda view, you can type m to mark multiple tasks, then type B and then S to scatter tasks randomly over the next N days. (Call it with a prefix argument as C-u B S to limit it to weekdays.) If I catch myself procrastinating because I don’t have enough time, that’s usually a sign to be more cautious about my estimates and commitments, so I adjust those too.
  • Procrastinating because it’s less important than other tasks: This is related to the time reason. I have no qualms about pushing less-important tasks forward.
  • Procrastinating because you don’t feel like working on it: Is the task actually important? If it’s not, I usually get rid of it without feeling guilty. If it’s still useful, I might unschedule it so that I see it only if I’m looking for tasks in that project or in that context. Alternatively, I can just mark the task as CANCELLED or SOMEDAY. If the task is important, I think about whether I’m likely to feel like working on it at some point in the future. If I’m likely to not feel any different about it, I might delegate it, or I might just sit down and do it since procrastination doesn’t add value. On the other hand, if I’m likely to feel like working on it at some point, then I tag it with that context and push it out to some other date.
  • Procrastinating because you forgot about it: I usually check my agenda every day and Org shows forgotten things in a different colour, so I catch these quickly. If the tasks are more important than the tasks I’ve already scheduled, I might work on those first. Alternatively, I might schedule it for sometime later.

I procrastinate based on my to-do list, not based on my inbox. The inbox is a terribly unstructured way to manage your tasks. I use Boomerang for Gmail to defer some mail to a later date, but that’s usually so that I can pop it back into my inbox the day that I meet someone so that I have context and so that I don’t have to copy the link into the calendar entry or my TO-DO list.

Wrapping up

So that’s how I deal with having a large backlog. I focus first on the stuff that I need to do, and I make sure that shows up on my agenda. Then I make it easy to look for stuff that I want to do using Org’s support for projects, tags, time estimates, and so on. I don’t feel guilty about having lots of tasks to choose from. I view my backlog positively. It lets me do good stuff without worrying too much about how I spend my time.

How do you deal with your backlog? =)

Sneak peek! Writing this post prompted me to start tracking whether my backlog grew or shrank each day. Check out my preliminary results and the code I used to analyze my TODOs.
2014-04-27: Fixed typo in keybinding – thanks, Sujith Abraham!

Emacs tweaks: Export Org checkboxes using UTF-8 symbols

UPDATE 2014-03-28: Newer versions of org have the org-html-checkbox-type variable, which you can set to unicode. Use M-x customize-variable org-html-checkbox-type to see if you have it.

This snippet turns - [X] into ☑ and - [ ] into ☐.

(defun sacha/org-html-checkbox (checkbox)
  "Format CHECKBOX into HTML."
  (case checkbox (on "<span class=\"check\">&#x2611;</span>") ; checkbox (checked)
        (off "<span class=\"checkbox\">&#x2610;</span>")
        (trans "<code>[-]</code>")
        (t "")))
(defadvice org-html-checkbox (around sacha activate)
  (setq ad-return-value (sacha/org-html-checkbox (ad-get-arg 0))))

To find this code, I searched ox-html.el for [. Eventually I found org-html-checkbox, which is directly called by org-html-format-list-item instead of being a function variable that you can change. So that meant I needed to override the behaviour of org-html-checkbox through defadvice. You can see above how I wrap advice around org-html-checkbox and replace the return value with my own function. For more about advice, read the Emacs Lisp Intro manual.

To find the hex codes for the UTF-8 characters, I searched Google for UTF-8 checkbox and found BALLOT BOX WITH CHECK. I used the hex code so that I didn’t have to worry about encoding issues. I tested it by updating one of my weekly reviews. Tada!

Inspired by Grant from Wisdom and Wonder.