# Category Archives: sharing

## Python, Org Mode, and writing Org tables to CSVs so that I can read them back

I’ve been getting deeper into Python so that I can model our personal finances. I really like using the pandas library to manipulate data. All those years I spent trying to juggle increasing complex spreadsheets… Working with Python code in Org Babel blocks is just so much more fun. I like being able to keep my assumptions in tables without having to fuss around with naming cells for easy-to-read formulas, slice and summarize parts of my data frames, organize my notes in outlines and add commentary, and define more complicated functions that I don’t have to squeeze into a single line.

I haven’t quite been able to tempt W- into the world of Org Babel Python blocks. Still, I don’t want to give up the awesomeness of having pretty tables that I can easily edit and use. So I have a bunch of named tables (using #+NAME:), and some code that exports my tables to CSVs:

#+NAME: tables
| Table         | Key                 |
|---------------+---------------------|
| assets_w      | Description         |
| assets_s      | Description         |
| tax_rates     |                     |
| disposition   | Asset               |
| probate_rates | Asset               |
| basic         | Client information  |
| base_expenses | Category            |
| general       | General assumptions |

#+begin_src emacs-lisp :results silent :var tables=tables :tangle no
  (defun my-tbl-export (row)
    "Search for table named NAME and export."
    (interactive "s")
    (save-excursion
      (goto-char (point-min))
      (let ((case-fold-search t))
        (when (search-forward-regexp (concat "#\\+NAME: +" (car row)) nil t)
          (next-line)
          (org-table-export (format "%s.csv" (car row)) "orgtbl-to-csv")))))
  (mapc 'my-tbl-export tables)
#+end_src


and some code that imports them back in, and formats tables nicely if I’m displaying them in Org. The in_org block doesn’t get tangled into index.py, so I don’t clutter command-line use with Org table markup.

#+begin_src python :results silent :tangle no
  in_org=1
#+end_src

#+begin_src python :results silent :exports code
  import pandas as pd
  import numpy as np
  import orgbabelhelper as ob
  def out(df, **kwargs):
    if 'in_org' in globals():
      print(ob.dataframe_to_orgtable(df, **kwargs))
    else:
      print(df)
    return df
#+end_src

#+begin_src python :results silent :var tables=tables :colnames yes
  for row in tables:
    table = row[0]
    index = row[1] 
    if row[1] == '':
      index = None
    globals()[table] = pd.read_csv(table + '.csv', index_col=index).apply(pd.to_numeric, errors='ignore')
    # print(globals()[table])
#+end_src


Then I can use C-c C-v C-b (org-babel-execute-buffer) to update everything if I change the table in my Org file, and I can use C-c C-v C-t (org-babel-tangle) to create an index.py that W- can read through or run without needing Org.

## Making an 8-page 7″x4.25″ captioned photo book with Org Mode and LaTeX

Here’s another technique that makes a simple photo book. I wanted to
make an 8-page book that could be printed 4 pages to a 8.5″x14″ sheet
(duplex, flip along the short edge), with a final page size of
7″x4.25″.

Sample with my own photos:

## Prerequisites

• ImageMagick
• Texlive (probably)
• latex-beamer
• Org Mode and Emacs

## Process

We can define the labels and their captions in a named table like this:

 Let’s Go for a Walk Caption for photo 1 placeholder.png Caption for photo 2 placeholder.png Caption for photo 3 placeholder.png Caption for photo 4 placeholder.png Caption for photo 5 placeholder.png

Note that the first page is row #1 this time, instead of starting with
the last page.

Then we generate the LaTeX code with some Emacs Lisp, like so:

#+begin_src emacs-lisp :var pages=story :results value latex :exports results
(mapconcat (lambda (row) (format "\\graphicframe{%s}{%s}" (cadr row) (org-export-string-as (car row) 'latex t))) pages "\n")
#+end_src


I put that in a subtree for easier exporting with C-c C-e C-s l b (org-export-dispatch, subtree, LaTeX, Beamer).

## Process

• Set up Org Mode export to Beamer
(eval-after-load "ox-latex"
;; update the list of LaTeX classes and associated header (encoding, etc.)
;; and structure
("beamer"
,(concat "\\documentclass[presentation]{beamer}\n"
"[DEFAULT-PACKAGES]"
"[PACKAGES]"
"[EXTRA]\n")
("\\section{%s}" . "\\section*{%s}")
("\\subsection{%s}" . "\\subsection*{%s}")
("\\subsubsection{%s}" . "\\subsubsection*{%s}"))))


This file gets included in the LaTeX file for the children’s book.
Tweak it to change the appearance. In this example, I use black serif
text text on the left side of a picture, both occupying roughly half
of the page. I also experimented with using a different font this time, which you might need to install (for me, I did apt-get install texlive-fonts-extra).

\geometry{paperwidth=7in,paperheight=4.25in,left=0.5in,right=0.5in}
\usepackage{etoolbox}
\usepackage[T1]{fontenc}
\usepackage{Alegreya}
\usefonttheme{serif}
\setbeamercolor{normal text}{fg=black,bg=white}
\setbeamercolor{structure}{fg=black,bg=white}
%% \setbeamertemplate{frametitle}
%% {
%%   \begin{center}
%%   \noindent
%%   \insertframetitle
%%   \end{center}
%% }
\newcommand{\graphicframe}[2] {
{
%% \if #1\empty
%% \usebackgroundtemplate{}
%% \fi
\ifstrempty{#1}{
\begin{frame}[plain]
\begin{center}
\noindent
\textbf{\huge{#2}}
\end{center}
\end{frame}
}{
\begin{frame}[plain]
\begin{columns}
\begin{column}{0.48\textwidth}
\huge{#2}
\end{column}
\begin{column}{0.48\textwidth}
\includegraphics[height=\textheight,width=\textwidth,keepaspectratio=true]{#1}
\end{column}
\end{columns}
\end{frame}
}
}
}
\usepackage[noframe]{showframe}
\renewcommand{\maketitle}{}

• Create the PDF
pdflatex index.tex

• Create one PNG per page
mkdir pages
convert -density 300 index.pdf -quality 100 pages/page%d.png

• Create the 4-up imposition

The diagram at https://pressnostress.com/iw/howto/booklets/1.html was helpful.

montage $$page4.png -rotate 180$$ $$page3.png -rotate 180$$ page7.png page0.png -tile 2x2 -mode Concatenate front.png
montage $$page2.png -rotate 180$$ $$page5.png -rotate 180$$ page1.png page6.png -tile 2x2 -mode Concatenate back.png
convert front.png back.png -density 300 ../print.pdf


## Other notes

Placeholder image from https://en.wikipedia.org/wiki/File:Portrait_placeholder.png – public domain.

## Story

• It’s time to make a smoothie!
• I pour blueberries into the blender.
• Mama blends it all with some water.
• I peel and add a banana.
• Mama blends it again.
• Yum yum!

## Process

• Prerequisites

• ImageMagick
• Texlive (probably)
• latex-beamer
• Org Mode and Emacs
• Set up Org Mode export to Beamer

(eval-after-load "ox-latex"
;; update the list of LaTeX classes and associated header (encoding, etc.)
;; and structure
("beamer"
,(concat "\\documentclass[presentation]{beamer}\n"
"[DEFAULT-PACKAGES]"
"[PACKAGES]"
"[EXTRA]\n")
("\\section{%s}" . "\\section*{%s}")
("\\subsection{%s}" . "\\subsection*{%s}")
("\\subsubsection{%s}" . "\\subsubsection*{%s}"))))


• Set up image directories

mkdir text-pages blank-spreads drawn drawn-pages


 text-pages Will contain one image per page of just the plain text. blank-spreads Will contain text spreads ready for drawing drawn Export one image per spread (without the text layers) from your drawing program drawn-pages Will contain one image per page combining text and drawing

This file gets included in the LaTeX file for the children’s book. Tweak it to change the appearance. In this example, I use black serif text at the bottom of the page.

\geometry{paperwidth=7in,paperheight=8.5in,left=0.5in,top=0.5in,right=0.5in,bottom=0.5in}
\setbeamercolor{normal text}{fg=black,bg=white}
\setbeamercolor{structure}{fg=black,bg=white}
\usefonttheme{serif}
\setbeamertemplate{frametitle}
{
\begin{center}
\vspace{0.7\textheight}
\noindent
\insertframetitle
\end{center}
}
\usepackage[noframe]{showframe}
\renewcommand{\maketitle}{}


• Write the story

I used Org Mode to make it easy to write the story.

Some considerations:

• Because we’re printing this as a saddle-stitched booklet, the number of lines should be a multiple of four. Sixteen is probably a good maximum.
• The first heading is actually for the last page.
• The second heading is for the cover page.
• The third heading is for the first inner page, the fourth heading is for the second inner page, and so on.
#+OPTIONS:   TeX:t LaTeX:t skip:nil d:nil todo:t pri:nil tags:not-in-toc author:nil date:nil
#+OPTIONS: H:1
#+startup: beamer
#+LaTeX_CLASS: beamer
#+LaTeX_CLASS_OPTIONS: [20pt]
#+BEAMER_FRAME_LEVEL: 1

* Story

**
**
**
** It's time to make a smoothie!
** I pour blueberries into the blender.
** Mama blends it all with some water.
** I peel and add a banana.
** Mama blends it again.
** Yum yum!


• Make the tex, PDF, page PNGs, and spread PNGs

1. Go to the subtree for the story and use M-x org-export-dispatch (C-c C-e) with the subtree option (C-s) to export it as a Beamer file (option l b).
2. Use pdflatex to convert the .tex to PDF.

pdflatex index.tex


3. Create one PNG per text page with:

convert -density 300 index.pdf -quality 100 text-pages/page%02d.png


4. Create spreads to draw on with:

montage text-pages/page*.png -tile 2x1 -mode Concatenate blank-spreads/spread%d.png


5. Optionally, create a layered PSD with:

convert blank-spreads/spread*.png $$-clone 1,0 -background white -flatten -alpha off$$ -reverse spreads-for-drawing.psd


• Draw

I imported the PNG layers into MediBang Paint on a Samsung Note 8 Android phone, and then:

• imported photos
• traced them
• hid the text layers
• exported one PNG per spread to QuickPic, renamed them, and uploaded them to Dropbox, because I couldn’t figure out how to export to Dropbox directly

Layer folders were handy for organizing spread-related images. I couldn’t seem to move all of the layers in a layer folder together on Android, but the iPad was able to do so. If I didn’t have the iPad handy, I combined the layers by exporting a PNG and then importeing it back into MediBang Paint.

This was a decent setup that allowed me to draw and paint even when I was in bed nursing A- and waiting for her to fall asleep. I held the phone with one hand and rotated the canvas as needed so that it was easier for me to draw lines with my right. Because of the awkward position and the small screen size, the lines are not as smooth as I might like, but the important thing is that they’re there. Whee! =)

It turns out to be possible to use the free MediBang Pro drawing program under Wine on Linux to import the PSD and save it to the cloud. I was also sometimes able to switch to drawing with iPad Pro with Pencil, but it was harder to find time to do that because that usually made A- want to draw too.

Anyway, after I drew and exported the PNGs, the next step was to…

• Convert the drawn spreads back to pages and combine them with the text

Here’s some code that combines the drawing and the text. Keeping the drawing and the text separate until this stage (instead of exporting the PNGs with the text) makes it easier to change the text later by recreating the text PNGs and running this step.

(defun my/combine-spread-drawing-and-text (page num-pages)
(let ((gravity (if (= (% page 2) 1) "West" "East"))
(spread (/ (% page num-pages) 2)))
(shell-command
(format
(concat "convert \$$" "drawn/spread%d.png -gravity %s " "-chop 50%%x0 +repage \$$ "
"text-pages/page%02d.png -compose darken "
"-composite drawn-pages/page%02d.png")

(cl-loop for i from 0 to (1- pages) do


This code pairs up the drawn pages into a PDF that can be printed duplex. Make sure to choose the option to flip along the short edge. I hard-coded the page orders for 4-, 8-, 12-, and 16-page booklets.

(let* ((page-order
'((0 1 2 3)   ; hard-coded page sequences
(0 1 2 7 6 3 4 5)
(0 1 2 11 10 3 4 9 8 5 6 7)
(0 1 2 15 14 3 4 13 12 5 6 11 10 7 8 9)))
(sequence
(mapconcat (lambda (d) (format "drawn-pages/page%02d.png" d))
(elt page-order (1- (/ pages 4))) " ")))
(shell-command
(format
"montage %s -tile 2x1 -mode Concatenate print-duplex-short-edge-flip.pdf"
sequence)))


• Print and bind

After printing and folding the book, I used tape to make the book hold together. Tada!

• Create on-screen PDF for reading

A little bit of manipulation so that the last page is in the right place:

(shell-command
(format "convert %s onscreen.pdf"
(mapconcat 'identity (cl-loop for i from 1 to pages
collect (format "drawn-pages/page%02d.png" (% i pages))) " ")))


(cl-loop
for i from 0 to (1- (/ pages 2)) do
(shell-command
(format
(concat "convert "
"\$$blank-spreads/spread%d.png " "drawn/spread%d.png " "-compose darken " "-resize %dx -flatten \$$ "
"\$$+clone -background black -shadow 50x1+%d+%d \$$ "
"+swap -compose src-over -composite "


• Ideas for next steps

• Better thumbnails for easy previews
• More layout possibilities (photos, verses, etc.)
• Smaller books:

• crop marks on a full-page print, or
• the right imposition rules to print more pages on a sheet

## Making books for A-

A- loves being read to. She picks up new words and ideas from the books we read, requests both favourites and new books again and again, and can identify objects in photographs and drawings. I borrowed a few children’s books from the library in case reading about upcoming changes or challenges helps her understand. The books were okay, but didn’t quite fit the words we use or the way we like to handle things. So this week, I decided to make my own books for A-, especially since there are few books that cover things like microphthalmia.

The first book I made was about night weaning, since we might have to do that in preparation for dental surgery under anaesthesia. I sketched it using ZoomNotes on my iPad, exported the SVG, tinkered with it in Inkscape, exported PNGs, combined the PNGs with ImageMagick, and created a 12-page PDF with 7″x8.5″ pages. That let me print the book out on legal-size paper (8.5″x14″), 2 pages per sheet, duplex printing set to flip on the short side, using this page order:
12, 1, 2, 11, 10, 3, 4, 9, 8, 5, 6, 7. I folded each sheet in half. Instead of hand-sewing the binding, I just taped the pages together. And just like that, I had a book that I could page through properly: “No More Nursing, Time to Sleep.”

I read the new book to A-. She asked me to reread it several times. She pointed to the book and said, “A-!” She pointed to the stick figure for me and said, “Mama!” Success!

The next thing I wanted to try was printing in colour. We recently replaced our printer with an HP M277dw colour laser printer that could print duplex, so I was looking forward to giving that a try. I wanted to make a book about the conformer in A-‘s little eye. This time, I drew the pages of the book using layers in Medibang Paint. I drew on the bus home from Riverdale Farm, working around a sleeping A- snuggled in my carrier. I exported each layer as a PNG, used ImageMagick to convert pairs of pages into what I needed to print (page order: 8, 1, 2, 7, 6, 3, 4, 5), combined those into a PDF. I couldn’t figure out how to get the HP app to properly scale the document and print in duplex, but printing from Linux worked fine. I quickly had another book in my hands: “My Conformer.”

She’s starting to echo phrases from the to books, and it’s been only a few days. Wow!

I’m working on a third book now. Time for something fun: “Let’s Make a Smoothie,” since she enjoys making and drinking them. She already knows all the words, so this is more about enjoyment. This time, I’m going to make a workflow that lets me draw on two-page spreads. I don’t have any wide drawings planned yet, but it could be handy for later. I made an Inkscape template to help me keep margins in mind. I learned how to use Medibang Paint’s folders to organize all the layers, and I’m getting the hang of digitally tracing and painting the photos I took.

I’m looking forward to making even more books and refining my workflow along the way. Here are a few things I want to try:

• Flat colour
• Painting
• Printing photos
• Two-page drawing
• Parametric templates
• Smaller format by cutting
• Heavier-weight paper
• Board book replacement
• Printing at Staples or similar
• Print-on-demand book
• Handstitching
• Binding with a cover
• Smaller format by folding and gluing
• Mobile workflow
• Vector drawing

And a few quick ideas for possible next books:

• Potty Time
• Brushing Teeth
• Feelings
• When I Feel Nervous
• Going to Sleep at the Dentist
• My Life
• My Day
• Going Out
• At Home
• I Can…
• I Can Draw
• Waiting
• Try Again
• Dressing Up
• Alimango sa Dagat
• Leron Leron Sinta

Journaling considerations:

• A- nurses a lot in bed. I keep my phone handy and I write when she doesn’t want to let me go.
• I also jot quick notes throughout the day so that I don’t have to keep them in my head. These go into the nearest synchronized device.
• It’s hard to remember the context for those notes if too much time passes. A daily verbal recap for W- and a weekly summary for my blog seem to be just the right balance. Anything older than a week gets too fuzzy, while writing detailed notes every day takes too much time away from other things I’d like to do.
• Monthly reviews give me a better perspective on big changes. It’s hard to keep enough in my head when I’m reading or writing on my phone, so I need help summarizing a month’s worth of highlights.

Here are the technical details:

I set up Memento Database on my phone and on a backup Android phone. I picked it because it can synchronize between phones in the background, and it can also sync with Google Sheets so that I can process things further.

My journal database has the following fields:

• Date: defaults to current date
• Note
• Category: single value from a list. Most of my entries go into Gross Motor, Fine Motor, Language, Self-care, Other, or Us, and I add other categories as needed.
• Highlight: a number indicating the level of review this should be included in: 1 – weekly, 2 – monthly, 3 – yearly. I display this field as the status, so that it shows up on the right side.

I have a shortcut on my home screen so that I can quickly add a journal entry.

I normally sort the list by date, with recent entries on top.

As part of my weekly review, I look at recent entries, fill in any categories I skipped, and choose a few to highlight. For example, last week, I wrote 17 entries and I chose 13 to include in the weekly review.

I configured Memento’s default export formatting to include only the Note field and to export that without the field label.

I filtered the database to show only the entries within a given date range where the highlight value was greater than 0.5.

I grouped it by category so that similar entries were together. This was better than fiddling with the sorting, since this takes fewer taps to set back to my default view.

After filtering and grouping the entries, I used the “Send all > Send as text” command to send it to Tasks Free, which is a task manager that synchronizes with Google Tasks. I like the way I can drag-and-drop tasks to reorder them, which makes prioritizing so much easier on my phone. I edit the text in Tasks Free, turning the keywords into paragraphs and moving things around for better flow.

After drafting the body of the post (and possibly switching between phones, if my battery ran low), I select all the text, copy it into the WordPress app, set the categories and the title, and post the entry.

The monthly review process is quite similar. I start with a filtered view that shows all entries for last month (133 entries in November), and I group it by category. I skim all the entries, not just the ones included in the weekly review, because sometimes little moments turn out to be significant or part of a bigger pattern. After setting the highlight values for the things I’d like to include in my monthly review, I switch to another filter that shows me last month’s entries with a highlight value greater than 1.5 (28 entries in November). I send it all to Tasks Free, edit the post, copy it into WordPress, and publish.

If I manage to squeeze in some computer time, I use Google Tasks to copy the text into Emacs and then use my regular Org Mode review/publish processes.

I’ve been thinking about how I can improve this workflow. Sending text to the WordPress app doesn’t seem to work (the text disappears after I save or publish), and it’s kinda nice being able to move my weekly review task around on my task list in order to accommodate other priorities. I also like the way Google Tasks keeps the data from completed tasks, which has come in handy a few times. Tasks Free editing is more responsive, too. Synchronizing with Tasks Free seems to be more robust than synchronizing with Orgzly, since I only have to watch out for editing the same task on two devices instead of watching out for the whole file.

I’d like to get back to drawing the weekly and monthly reviews, but maybe that can wait until A-‘s sleep is more settled and my discretionary time is more consolidated. The visual journals are more fun to flip through, but the bulk and chronological views I hacked into my WordPress theme are reasonable workarounds.

## What do I want to think through when writing on my phone?

My first priority is to get thoughts out of my head and into a form I can work with. This often becomes an explosion of lists and keywords, which can be handy for thinking and drawing, but not for my blog posts. It helps me clear my mind and be less distracted. It lets me make progress instead of covering the same ground. It also helps me make better decisions. I can see the costs and benefits more clearly when they’re written down. I can brainstorm options and compare them. I can review the decisions afterwards, too.

Prose makes things slower to capture and harder to read on my phone, but also a little easier to search or reflect on after a while, and possibly good for review or conversation when turned into a blog post.

And then there’s stuff I’d flesh out in writing anyway, like my weekly/monthly/yearly reviews. Those start off as sketches on my computer, then lists of keywords, and then paragraphs on my phone.

If my list is cluttered, it’s hard to pick one thought and follow it through. I end up adding little bits here and there, and I’m not sure how useful that is. Things below a couple of screens get ignored. It might be helpful to dedicate some time to processing. I can either flesh out the top idea on the stack, or move it to my inbox if it needs computer time.

I don’t have to worry too much about writing for other people’s benefit. That can come later, when I have more focused time. People can always choose what to read and what to skim or skip. What’s important is that I think things through and then capture them for later review.

So, how can I do this more effectively? Where are my gaps?

• Motivating questions, not just a scratchpad of ideas: It’s useful to ask questions when I’m going to change something based on the answer. For example, I can describe the gaps and points of friction, then explore why those are so, and then come up with ideas and actions.
• Reviews are less motivating, but I can get the ball rolling by focusing on tidbits rather than overviews. It’s hard to see context on a small screen.
• I don’t have to spend a lot of time picking the best thing to write about. Practically everything will move me forward, even if the distances vary. I can write and write and write, and then reflect.
• If I don’t feel like writing, I can always snuggle or sleep. No need to force it.

The best time to write is when I’m nursing A- and she’s drifting off to sleep, but is awake enough to root if I try to move away. Trade-offs / other things I could be doing during that time:

• Sleeping: Good up to a certain point. It’s handy to use A-‘s first nap to catch up on sleep if needed, but I don’t want to sleep too much.
• Browsing social media: Tempting time sink. Occasionally useful or interesting, but best in small doses.
• Answering email: Nice thing to do, although expectations are low. Some things can only be answered when I’m at my computer.
• Prioritizing my to-do list and adding items to it: Helps me hit the ground running during computer time.
• Reading research and tech stuff: Gives me ideas to explore when I’m at my computer. Less efficient than reading when I’m on my laptop, but it’s okay to just index things for later exploration.
• Reading e-books: sometimes useful, especially if I pick my questions and titles carefully.
• Reading fiction: occasionally entertaining, although often bleh. Maybe I should try library recommendations.
• Playing games: my interest comes and goes. Exercises problem-solving. Artificial sense of progress.

I think it makes sense to prioritize sleep, then writing, with maybe a notification-based pomodoro for social media during breaks. If I don’t feel like writing, I can use the time to learn more about tech or parenting. Let’s see if I can find a pomodoro app that’s compatible with a sleeping baby, or if I can make something using Tasker…