Category Archives: org

Using your own Emacs Lisp functions in Org Mode table calculations: easier dosage totals

UPDATE 2015-06-17: In the comments below, Will points out that if you use proper dates ([yyyy-mm-dd] instead of yyyy-mm-dd), Org will do the date arithmetic for you. Neato! Here’s what Will said:

Hi Sacha. Did you know you can do date arithmetic directly on org’s inactive or active timestamps? It can even give you an answer in fractional days if the time of day is different in the two timestamps:

| Start                  | End                    | Interval |
|------------------------+------------------------+----------|
| [2015-06-16 Tue]       | [2015-06-23 Tue]       |        7 |
| <2015-06-13 Sat>       | <2015-06-15 Mon>       |        2 |
| [2015-06-10 Wed 20:00] | [2015-06-17 Wed 08:00] |      6.5 |
#+TBLFM: $3=$2 - $1 

Here’s my previous convoluted way of doing things… =)
—-

I recently wrote about calculating how many doses you need to buy using an Org Mode table. On reflection, it’s easier and more flexible to do that calculation using an Emacs Lisp function instead of writing a function that processes and outputs entire tables.

First, we define a function that calculates the number of days between two dates, including the dates given. I put this in my Emacs config.

(defun my/org-days-between (start end)
  "Number of days between START and END.
This includes START and END."
  (1+ (- (calendar-absolute-from-gregorian (org-date-to-gregorian end))
         (calendar-absolute-from-gregorian (org-date-to-gregorian start)))))

Here’s the revised table. I moved the “Needed” column to the left of the medication type because this makes it much easier to read and confirm.

| Needed | Type         | Per day |      Start |        End | Stock |
|--------+--------------+---------+------------+------------+-------|
|     30 | Medication A |       2 | 2015-06-16 | 2015-06-30 |     0 |
|      2 | Medication B |     0.1 | 2015-06-16 | 2015-06-30 |   0.2 |
#+TBLFM: @2$1..@>$1='(ceiling (- (* (my/org-days-between $4 $5) (string-to-number $3)) (string-to-number $6)))

C-c C-c on the #+TBLFM: line updates the values in column 1.

@2$1..@>$1 means the cells from the second row (@2) to the last row (@>) in the first column ($1).  '  tells Org to evaluate the following expression as Emacs Lisp, substituting the values as specified ($4 is the fourth column’s value, etc.).

The table formula calculates the value of the first column (Needed) based on how many you need per day, the dates given (inclusive), and how much you already have in stock. It rounds numbers up by using the ceiling function.

Because this equation uses the values from each row, the start and end date must be filled in for all rows. To quickly duplicate values downwards, set org-table-copy-increment to nil, then use S-return (shift-return) in the table cell you want to copy. Keep typing S-return to copy more.

This treats the calculation inputs as strings, so I used string-to-number to convert some of them to numbers for multiplication and subtraction. If you were only dealing with numbers, you can convert them automatically by using the ;N flag, like this:

| Needed | Type         | Per day | Days | Stock |
|--------+--------------+---------+------+-------|
|      6 | Medication A |       2 |    3 |     0 |
|      1 | Medication B |     0.1 |    3 |   0.2 |
#+TBLFM: @2$1..@>$1='(ceiling (- (* $3 $4) $5)));N

Providing values to functions in org-capture-templates

Over at the Emacs StackExchange, Raam Dev asked how to define functions for org-capture-templates that could take arguments. For example, it would be useful to have a function that creates a Ledger entry for the specified account. Functions used in org-capture-templates can’t take any arguments, but you can use property lists instead. Here’s the answer I posted.

You can specify your own properties in the property list for the template, and then you can access those properties with plist-get and org-capture-plist. Here’s a brief example:

Here’s a brief example:

(defun my/expense-template ()
  (format "Hello world %s" (plist-get org-capture-plist :account)))
(setq org-capture-templates '(("x" "Test entry 1" plain
                               (file "~/tmp/test.txt")
                               (function my/expense-template)
                               :account "Account:Bank")
                              ("y" "Test entry 2" plain
                               (file "~/tmp/test.txt")
                               (function my/expense-template)
                               :account "Account:AnotherBank")))

I hope that helps!

Using Emacs Org Mode tables to calculate doses to buy

I got tired of manually calculating how many I needed to buy based on a daily protocol and how many I had in stock, so I wrote a little bit of Emacs Lisp to figure it out. You can specify the type, daily dose, start and end dates (inclusive; defaults to the last specified date if blank), and how many you have in stock.

First, define a table of this form, and give it a name.

#+NAME: input
| Type         | Per day |      Start |        End | Stock |
|--------------+---------+------------+------------+-------|
| Medication A |       2 | 2015-06-09 | 2015-06-16 |     5 |
| Medication B |       1 |            |            |     0 |
| Medication C |     0.1 | 2015-06-12 | 2015-06-16 |   0.2 |
Type Per day Start End Stock
Medication A 2 2015-06-09 2015-06-16 5
Medication B 1 0
Medication C 0.1 2015-06-12 2015-06-16 0.2

To call the code from the bottom of this post, use something like:

#+CALL: calculate-meds-needed(meds=input) :hlines yes :colnames yes
Type Total In stock Needed
Medication A 16 5 11
Medication B 8 0 8
Medication C 0.5 0.2 1

Here’s the code that processes it:

#+name: calculate-meds-needed :var meds=meds :colnames yes :hlines yes
#+begin_src emacs-lisp
(let (start end)
  (append
   (list (list "Type" "Total" "In stock" "Needed"))
   (list 'hline)
   (sort (delq nil (mapcar
                    (lambda (row)
                      (unless (or (eq row 'hline) (string= (elt row 0) "Type"))
                        (let (total)
                          (setq start (if (string< "" (elt row 3)) (elt row 3) start)
                                end (if (string< "" (elt row 2)) (elt row 2) end)
                                total (* (elt row 1)
                                         (- (calendar-absolute-from-gregorian (org-date-to-gregorian start))
                                            (calendar-absolute-from-gregorian (org-date-to-gregorian end))
                                            -1)))
                          (list
                           (elt row 0)
                           total
                           (elt row 4)
                           (max 0 (ceiling (- total (elt row 4))))))))
                    meds)) (lambda (a b) (string< (car a) (car b))))))
#+end_src

Adding calculations based on time to the Org Agenda clock report

Duplicating this answer on my blog in case StackOverflow goes away. =)

Leo asked:

I’m trying to make the Agenda Clockreport show how many pomodoros I’ve invested in a task. A Pomodoro is 25 minutes. For example, 1:15 hours of work is 3 pomodoros.

I’m trying to customize org-agenda-clockreport-paramater-plist, and I would like to extract “Time” and convert it to a pomodoro. I.e., (time in minutes / 25) = pomodoro.

I wrote:

This will create a column in your clocktable report that sums the hours from columns 3 and 4, and then another column that shows you the round number of pomodoros that took up.

(setq org-agenda-clockreport-parameter-plist
      '(:link t :maxlevel 2 :formula "$5=$3+$4;t::$6=ceil($5*60/25);N"))

If you don’t want in-between columns, here’s a totally hackish approach:

(defun my/org-minutes-to-clocksum-string (m)
  "Format number of minutes as a clocksum string.
Shows the number of 25-minute pomodoros."
  (format "%dp" (ceiling (/ m 25))))
(fset 'org-minutes-to-clocksum-string 'my/org-minutes-to-clocksum-string)

Alternatively, you can use :formatter, but the formatting function looks very long and annoying to change.

Leo eventually configured it with:

(setq org-agenda-clockreport-parameter-plist
 '(:fileskip0 t :link t :maxlevel 2 :formula "$5=($3+$4)*(60/25);t"))

(He didn’t mind the decimals, I guess! =) )

Emacs Org Mode and the power of plain text

When I build a tool for other people to use and I want to store data, I usually have to think in terms of relational databases: tables, fields, and queries. There are other kinds of databases out there, like ones with flexible documents or ones that are optimized for graphs, but I haven’t gotten the hang of them yet.

When I build a tool for myself and I want to store data, I usually use plain text. (Or maybe a spreadsheet, but now that I’m getting the hang of Org Mode tables, I’m leaning more and more towards text.)

2015-02-02 Plain text -- index card #data #organization #pkm #org

2015-02-02 Plain text – index card #data #organization #pkm #org

I like the flexibility of plain text. Sometimes I want to organize my thoughts in an outline or an index. Sometimes I want to make a graph, like the way I wanted to visualize how my goals are related to each other. Sometimes I change my mind about what I want. (All the time, actually. =) ) Plain text lets me add structure the way I want to. It’s all in my text editor, so I can move things around or reorganize things using the tools in Emacs.

Sure, sometimes I mess up because of formatting mistakes or the lack of validation. For example, typos in my personal ledger show up when the numbers don’t match my bank balances or there’s a new category with a misspelled name. But these are easy enough to catch and fix, and I can’t completely guard against them with a database anyway. And it’s nice to know that version control can let me visually step through the changes or recover from mistakes.

What about speed? Databases can be much faster than plain text for large quantities of data, for sure. I tend to work with pretty small quantities of data. For example, my blog index has 3257 lines, and the file that I’m drafting this in is under a megabyte. Even with whatever Emacs Lisp I’ve written to extract or cross-reference data, I’m still mostly bottlenecked by my brain instead of my computer. Sure, it took me a little longer to figure out how to do table calculations using Org Mode, but now that I have some notes on that, I should be able to come up with future calculations more easily. Besides, if I need to analyze things quickly, I can export and then crunch the numbers using a different tool.

Speaking of tools, staying with lightly-structured plain text lets me build a toolkit of text manipulation techniques. When I’m editing things in Emacs, scripting with Emacs Lisp, searching with grep, or writing Javascript/Ruby/Perl code to work with text, I’m developing skills that I can use in a wide range of situations.

If you’re interested in keeping your data in plain text with Org Mode, here are some tips that can help you learn how to work with your information.

Start with tables

  1. Learn how to use keyboard shortcuts to create, move, or delete rows and columns.
  2. Learn how to sort tables.
  3. Learn how to use the column with specifier (ex: <10>) to limit the displayed size of your column while still being able to add more information.
  4. Use Org Mode’s support for calculations to do math or perform other operations on your table.

Consider using properties

Org tables don’t do well with paragraphs or more complex information, so you might want to use Org subtrees with properties.

You can use Org columns to display property values, or use Org dynamic blocks to put a summary of the values into your Org Mode file. See org-collector.el for a propview report.

If you want more control, you can work with the information using Emacs Lisp. You can use org-entry-get, org-entry-get-multivalued-property, or org-entry-get-with-inheritance to get the value of the property. If you want to go through all the subtrees (or a subset of them), use org-map-entries to call your own function at each of the matching headlines in the scope. org-heading-components will give you the information from the current heading, and you can use org-end-of-subtree to give you the boundary of the subtree if you want to process it further.

You can parse Org Mode lists with org-list-struct. I haven’t dug into this deeply yet, but it looks interesting.

Parse free-form text

In addition to working with tables and properties, you can write functions that use regular expressions or other techniques to extract data from text. re-builder can be useful for visual feedback while you’re figuring out the right regular expression to build. Remember, you’re in Emacs, so you don’t have to come up with the perfect regular expression that extracts all the data in one go. You can search for a regular expression, use a command like forward-line, save something to a variable, and so forth. Try thinking about how you would do something by hand, and then using repeat-complex-command to see what functions Emacs called when you did that.

save-restriction, narrow-to-region, and save-excursion are very useful when it comes to limiting the scope of your processing or saving your position, so check them out in the Emacs Lisp manual.

I find plain text to be really useful when I’m figuring things out (so, all the time), since I don’t have to build a complex interface for working with it. As I learn more about Org Mode’s features, I find myself using it for more and more of my data. Org’s slogan is “Organize your life in plain text!” – and I think it just might be serious about that!

Getting Helm and org-refile to clock in or create tasks

I’ve been thinking about how to improve the way that I navigate to, clock in, and create tasks in Org Mode. If the task is one of the ones I’ve planned for today, I use my Org agenda. If I know that the task exists, I use C-u C-c C-w (org-refile) to jump to it, and then ! (one of my org-speed-commands-user options) to clock in and track it on Quantified Awesome. If I want to resume an interrupted task, I use C-u C-c j (my shortcut for org-clock-goto). For new tasks, I go to the appropriate project entry and create it, although I really should be using org-capture instead.

2015-01-30 Org Mode jumping to tasks -- index card #emacs #org

2015-01-30 Org Mode jumping to tasks – index card #emacs #org

I thought about how I can reduce some of these distinctions. For example, what if it didn’t matter whether or not a task already exists? I can modify the org-refile interface to make it easier for me to create tasks if my description doesn’t match anything. To make things simpler, I’ll just reuse one of my org-capture-templates, and I’ll pre-fill it with the candidate from Helm.

(ert-deftest sacha/org-capture-prefill-template ()
  (should
   ;; It should fill things in one field at a time
   (string=
    (sacha/org-capture-prefill-template
     "* TODO %^{Task}\nSCHEDULED: %^t\n:PROPERTIES:\n:Effort: %^{effort|1:00|0:05|0:15|0:30|2:00|4:00}\n:END:\n%?\n"
     "Hello World")
    "* TODO Hello World\nSCHEDULED: %^t\n:PROPERTIES:\n:Effort: %^{effort|1:00|0:05|0:15|0:30|2:00|4:00}\n:END:\n%?\n"
    ))
  (should
   (string=
    (sacha/org-capture-prefill-template
     "* TODO %^{Task}\nSCHEDULED: %^t\n:PROPERTIES:\n:Effort: %^{effort|1:00|0:05|0:15|0:30|2:00|4:00}\n:END:\n%?\n"
     "Hello World" "<2015-01-01>")
    "* TODO Hello World\nSCHEDULED: <2015-01-01>\n:PROPERTIES:\n:Effort: %^{effort|1:00|0:05|0:15|0:30|2:00|4:00}\n:END:\n%?\n"))
  (should
   (string=
    (sacha/org-capture-prefill-template
     "* TODO %^{Task}\nSCHEDULED: %^t\n:PROPERTIES:\n:Effort: %^{effort|1:00|0:05|0:15|0:30|2:00|4:00}\n:END:\n%?\n"
     "Hello World" "<2015-01-01>" "0:05")
    "* TODO Hello World\nSCHEDULED: <2015-01-01>\n:PROPERTIES:\n:Effort: 0:05\n:END:\n%?\n")))

(defun sacha/org-capture-prefill-template (template &rest values)
  "Pre-fill TEMPLATE with VALUES."
  (setq template (or template (org-capture-get :template)))
  (with-temp-buffer
    (insert template)
    (goto-char (point-min))
    (while (re-search-forward
            (concat "%\\("
                    "\\[\\(.+\\)\\]\\|"
                    "<\\([^>\n]+\\)>\\|"
                    "\\([tTuUaliAcxkKInfF]\\)\\|"
                    "\\(:[-a-zA-Z]+\\)\\|"
                    "\\^\\({\\([^}]*\\)}\\)"
                    "?\\([gGtTuUCLp]\\)?\\|"
                    "%\\\\\\([1-9][0-9]*\\)"
                    "\\)") nil t)
      (if (car values)
          (replace-match (car values) nil t))
      (setq values (cdr values)))
    (buffer-string)))

(defun sacha/helm-org-create-task (candidate)
  (let ((entry (org-capture-select-template "T")))
    (org-capture-set-plist entry)
    (org-capture-get-template)
    (org-capture-set-target-location)
    (condition-case error
        (progn
          (org-capture-put
           :template
           (org-capture-fill-template
            (sacha/org-capture-prefill-template (org-capture-get :template)
                                                candidate)))
          (org-capture-place-template
           (equal (car (org-capture-get :target)) 'function)))
      ((error quit)
       (if (get-buffer "*Capture*") (kill-buffer "*Capture*"))
       (error "Capture abort: %s" error)))) t)

Next, I want to add this to the way that Helm prompts me to refile. That means that my creation task should return something ready for org-refile. Actually, maybe I don’t have to do that if I know I’m always going to call it when I want to jump to something. I might as well add that bit of code that sets up clocking in, too.

(defvar sacha/helm-org-refile-locations nil)

(defun sacha/helm-org-clock-in-and-track-from-refile (candidate)
  (let ((location (org-refile--get-location candidate sacha/helm-org-refile-locations)))
    (save-window-excursion
      (org-refile 4 nil location)
      (sacha/org-clock-in-and-track)
      t)))

(defun sacha/helm-org-refile-read-location (tbl)
  (setq sacha/helm-org-refile-locations tbl)
  (helm
   (list
    (helm-build-sync-source "Refile targets"
      :candidates (mapcar 'car tbl)
      :action '(("Select" . identity)
                ("Clock in and track" . sacha/helm-org-clock-in-and-track-from-refile))
      :history 'org-refile-history)
    (helm-build-dummy-source "Create task"
      :action (helm-make-actions
               "Create task"
               'sacha/helm-org-create-task)))))

(defun sacha/org-refile-get-location (&optional prompt default-buffer new-nodes no-exclude)
  "Prompt the user for a refile location, using PROMPT.
PROMPT should not be suffixed with a colon and a space, because
this function appends the default value from
`org-refile-history' automatically, if that is not empty.
When NO-EXCLUDE is set, do not exclude headlines in the current subtree,
this is used for the GOTO interface."
  (let ((org-refile-targets org-refile-targets)
        (org-refile-use-outline-path org-refile-use-outline-path)
        excluded-entries)
    (when (and (derived-mode-p 'org-mode)
               (not org-refile-use-cache)
               (not no-exclude))
      (org-map-tree
       (lambda()
         (setq excluded-entries
               (append excluded-entries (list (org-get-heading t t)))))))
    (setq org-refile-target-table
          (org-refile-get-targets default-buffer excluded-entries)))
  (unless org-refile-target-table
    (user-error "No refile targets"))
  (let* ((cbuf (current-buffer))
         (partial-completion-mode nil)
         (cfn (buffer-file-name (buffer-base-buffer cbuf)))
         (cfunc (if (and org-refile-use-outline-path
                         org-outline-path-complete-in-steps)
                    'org-olpath-completing-read
                  'org-icompleting-read))
         (extra (if org-refile-use-outline-path "/" ""))
         (cbnex (concat (buffer-name) extra))
         (filename (and cfn (expand-file-name cfn)))
         (tbl (mapcar
               (lambda (x)
                 (if (and (not (member org-refile-use-outline-path
                                       '(file full-file-path)))
                          (not (equal filename (nth 1 x))))
                     (cons (concat (car x) extra " ("
                                   (file-name-nondirectory (nth 1 x)) ")")
                           (cdr x))
                   (cons (concat (car x) extra) (cdr x))))
               org-refile-target-table))
         (completion-ignore-case t)
         cdef
         (prompt (concat prompt
                         (or (and (car org-refile-history)
                                  (concat " (default " (car org-refile-history) ")"))
                             (and (assoc cbnex tbl) (setq cdef cbnex)
                                  (concat " (default " cbnex ")"))) ": "))
         pa answ parent-target child parent old-hist)
    (setq old-hist org-refile-history)
    ;; Use Helm's sources instead
    (setq answ (sacha/helm-org-refile-read-location tbl))
    (if (and (stringp answ)
             (setq pa (org-refile--get-location answ tbl)))
        (progn
          (org-refile-check-position pa)
          (when (or (not org-refile-history)
                    (not (eq old-hist org-refile-history))
                    (not (equal (car pa) (car org-refile-history))))
            (setq org-refile-history
                  (cons (car pa) (if (assoc (car org-refile-history) tbl)
                                     org-refile-history
                                   (cdr org-refile-history))))
            (if (equal (car org-refile-history) (nth 1 org-refile-history))
                (pop org-refile-history)))
          pa)
      (if (and (stringp answ) (string-match "\\`\\(.*\\)/\\([^/]+\\)\\'" answ))
          (progn
            (setq parent (match-string 1 answ)
                  child (match-string 2 answ))
            (setq parent-target (org-refile--get-location parent tbl))
            (when (and parent-target
                       (or (eq new-nodes t)
                           (and (eq new-nodes 'confirm)
                                (y-or-n-p (format "Create new node \"%s\"? "
                                                  child)))))
              (org-refile-new-child parent-target child)))
        (if (not (equal answ t)) (user-error "Invalid target location"))))))

(fset 'org-refile-get-location 'sacha/org-refile-get-location)

Hooray! Now C-u C-c C-w (org-refile) also lets me use TAB or F2 to select the alternative action of quickly clocking in on a task. Mwahaha.

You can check out this code in my config to see if anything has been updated. Want to learn more about modifying Helm? Check out these posts by John Kitchin and Rubikitch.

I think I’m getting the hang of tweaking Helm.  Yay!