Categories: sharing » writing

RSS - Atom - Subscribe via email

Compiling selected blog posts into HTML and EPUB so I can annotate them

| blogging, 11ty, nodejs, supernote

[2023-01-04 Wed] Added a screenshot showing annotation.

I was thinking about how to prepare for my next 10-year review, since I'll turn 40 this year. I've been writing yearly reviews with some regularity and monthly reviews sporadically, and I figured it would be nice to have those posts in an EPUB so that I can read them on my e-reader and annotate them as I do my review.

I use the 11ty static site generator to publish my blog as HTML files, since I currently can't keep more than Emacs Lisp, Javascript, and Python in my brain. (No Hugo or Jekyll for me at the moment.) I briefly thought about getting 11ty to create that archive for me, but I realized it might be easier to just write it as an external script instead of trying to figure out how to get 11ty to export one thing conditionally.

One of the things I've configured 11ty to make is a JSON file that includes all of my posts with dates, titles, permalinks, and categories. It was easy to then parse this list and filter it to get the posts I wanted. I parsed the HTML out of the _site directory that 11ty produces instead of fetching the pages from my webserver. I got the images from my webserver, though, and I made a local cache and rewrote the URLs. That way, the EPUB conversion could include the images.

Download blog.js

blog.js
const blog = require('/home/sacha/proj/static-blog/_site/blog/all/index.json');
const cheerio = require('cheerio');
const base = '/home/sacha/proj/static-blog/_site';
const fs = require('fs');
const path = require('path');

function slugify(p) {
  return p.permalink.replace('/blog', 'post-').replace(/\//g, '-');
}

async function processPost(p) {
  console.log('Processing '+ p.permalink);
  let $ = cheerio.load(fs.readFileSync(base + p.permalink + 'index.html'));
  $('#comment').remove();
  let images = $('article img');
  await Promise.all(images.map((i, e) => {
    let url = $(e).attr('src');
    const outputFileName = 'images/' + path.basename(url).replace(/ |%20|%23/g, '-');
    $(e).attr('src', outputFileName);
    $(e).attr('style', 'max-height: 100%; max-width: 100%; ' + ($(e).attr('style') || ''));
    $(e).attr('srcset', null);
    $(e).attr('sizes', null);
    $(e).attr('width', null);
    $(e).attr('height', null);
    if (!fs.existsSync(outputFileName)) {
      console.log('fetch', outputFileName);
      return fetch(url).then(res => res.arrayBuffer()).then(data => {
        const buffer = Buffer.from(data);
        return fs.createWriteStream(outputFileName).write(buffer);
      });
    } else {
      console.log(outputFileName, 'exists');
      return null;
    }
  }));
  console.log('Done ' + p.permalink);
  let slug = slugify(p);
  $('article h2').attr('id', slug);
  let header = $('article header').html();
  let entry = $('article .entry').html();
  return `<article>${header}${entry}</article>`;
}

let last10 = blog.filter((p) => p.date >= '2013-08-01');
let posts = last10.filter((p) => p.categories.indexOf('yearly') >= 0)
    .concat(blog.filter((p) => p.title == 'Turning 30: A review of the last decade'))
    .concat(last10.filter((p) => p.categories.indexOf('monthly') >= 0));

let toc = '<h1>Table of Contents</h1><ul>' + posts.map((p) => {
  return `<li><a href="#${slugify(p)}">${p.title}</a></li>\n`;
}).join('') + '</ul>';

let content = posts.reduce(async (prev, val) => {
    return await prev + await processPost(val);
  }, '');
content.then((data) => {
  fs.writeFileSync('archive.html',
                   `<html><body>${toc}${data}</body></html>`);

});

This created an archive.html with my posts, using the images/ directory for the images. Then I used my shell script for converting and copying files to convert it to EPUB and copy it over.

On the SuperNote, I can highlight text by drawing square brackets around it. If I tap that text, I can write or draw underneath it. Here's what that looks like:

20230104_090739.png
Figure 1: Writing an annotation

These notes are collected into a "Digest" view, and I can export things from there. (Example: archive.pdf)

2023-01-04_09-23-57.png
Figure 2: Here's what that digest is like when exported.

(Hmm, maybe I should ask them about hiding the pencil icon…)

Anyway, I think that might be a good starting point for my review.

Writing my blog posts by hand

| blogging, supernote

A- complains if I get screentime when she doesn't get screentime, so it's hard to find time to write on my laptop or on my phone. I've experimented with dictation before, since Google Recorder can make a half-decent transcript. I'm not used to talking things out, though. I keep correcting false starts, stutters, and mis-recognized words.

Fortunately, I can write on my A5X while waiting for A-. I get more space than I do when writing on my phone, so it's easier for me to think. I can export pages as PNGs, Dropbox, share each page, sync with with Google Photos, and then use Lens to copy the text. I can then paste it into Orgzly, which automatically syncs with Syncthing so that I can edit it on my laptop with Emacs. It needs a little cleanup (capitalization, stray punctuation, missed words, things in the wrong order), but editing it feels easier than dealing with the output of speech recognition, so it seems to be worth the extra time and effort. Besides, it feels less embarrassing to write at the sandbox than it is to talk to myself.

I can edit the text directly on my phone, but I still need my laptop to publish my blog because I haven't set up my static site generator on my server. Some day! In the meantime, this might be a good workflow for getting thoughts out there.

What if I want to refer to sketches while I write? Flipping between pages on the A5X can be challenging if they're not next to each other, but I can keep my current writing page next to my sketch. I could also view the sketch on my phone and balance it on the A5X, or use layers to keep a small version of the sketch as a handy reference. Lots of ideas to play around with…

Statically generating my blog with Eleventy

| blogging

Things will probably be a little strange on my blog for the next few days, as I've decided to experiment with statically generating my blog with Eleventy. It's a little complicated because I wanted to keep as many of my posts and category/tag feeds as possible.

To speed things up, I usually work with a subset of my posts. Generating a partial copy of my site results in 557 files and takes 3.73 seconds. When I generate the full copy of my site, it writes 13358 files in 96.73 seconds.

With any luck, I'll be able to get most of the things working before the next Emacs News post. Let's see!

Sharing more of my discretionary time

Posted: - Modified: | writing

Depending on what time A- finally goes to bed, I might have around 1-2 hours of discretionary time that I can use to focus on a small task and complete it. If I pick something that's too big, I get tempted to stay up late, which often makes me grumpy the next day. So a good approach might be to have a number of reasonably small tasks that give me as quick a payoff as possible, especially if those tasks can result in compounding improvements.

Now that I can post Org Mode headings to my journal from Emacs, it's easier to log finished tasks as journal entries that will get picked up during my weekly and monthly review. The next step might be to figure out how to flesh out those lines into more useful posts. That way, I can find things again by searching my blog. Also, if other people can pick up ideas from my posts, I might be able to benefit from their improvements.

There's a lot of room for growth in terms of my workflow for doing stuff, learning stuff, and sharing stuff. Here's what a possible learning path might be like:

sharing-path.png

  • Planning: I've just started excavating the Org files that I've been tossing ideas into over the last 5+ years of limited discretionary time. Now things are mostly refiled, and I've got quite a few projects on my priority list. I might spend a bit of non-computer time mulling over 1-3 possibilities throughout the day, and then work on the most interesting one after processing my inbox. I find that in the course of a week, I tend to focus on one or two projects in order to take advantage of momentum. It's also good to set aside planning/improvement/review time instead of getting tempted to prioritize coding all the time, as fun as it is to write stuff.

    I don't have to optimize this. Most tasks are good to work on and move me forward, so I don't have to spend a lot of time trying to analyze the best effort/reward ratio. I can usually just go with whatever I feel like working on.

  • Coding: I usually work with Emacs Lisp or Javascript, with a little bit of Python. I have some technical debt in Ruby that I don't have the brainspace to dig into at the moment. It may have to wait until A- goes to in-person school. For Emacs Lisp, my next workflow improvement might be to get the hang of Lispy. For my personal projects, infrastructure is the main thing tripping me up. I need to spend some time setting up a proper development environment and learning more about workflows so that I can reduce risk when I'm working on stop-and-go things.
  • Writing: Dictation is out for now, since I don't feel much like talking at night. It's nice to be quiet after a whole day of talking with a kiddo. When I get the Georgi keyboard I ordered, stenography might be an interesting long-term skill investment. The bottleneck is probably still my thinking speed, though. That means I could probably:
    • embrace lists and outlines as a way of getting fragmented thoughts down (possibly over several sessions) and then shuffling them around into some form of coherence (yay Org Mode)
    • lower my threshold for posting; it's better to think out loud
  • Screenshots: I recently tweaked my shortcuts for inserting screenshots. Now I just need to make them part of muscle memory.
  • Drawings: I can sketch things out on my Lenovo X220 tablet PC, although flipping the screen is a little annoying. One option might be to leave my screen rotated and then use a Bluetooth keyboard to type and use shortcuts. The keyboard isn't as comfortable to type on as my laptop is, though. Hmm… org-krita doesn't quite fit my workflow, so I need to write my own. I want to be able to quickly sketch something. If I like it, I want to convert it, rename it with a caption, and add it to my sketches.

(defun my/org-insert-drawing-as-link ()
  (interactive)
  (let ((file (make-temp-file "/tmp/image" nil ".psd")))
    (copy-file my/index-card-template-file file t)
    (insert (org-link-make-string (format "file:%s" file)))
    (my/open-images-in-krita (list file))))

(defun my/preview-in-other-buffer (file)
  (with-current-buffer (find-file-noselect file)
    (display-buffer (current-buffer))))

(defun my/org-convert-sketch-at-point (&optional two-col)
  (interactive "p")
  (let* ((link (org-element-context))
         (file (org-element-property :path link))
         date
         (intermediate (concat (file-name-sans-extension file) ".png"))
         new-file new-link)
    (unless (eq (org-element-type link) 'link)
      (error "Not at a link"))
    ;; (call-process "krita" nil nil nil file "--export" "--export-filename" intermediate)
    (call-process "convert" nil nil nil (concat file "[0]") intermediate)
    (my/preview-in-other-buffer intermediate)
    (setq
     date (org-read-date)
     caption (read-string "Caption: ")
     new-file (expand-file-name (format "%s %s.png" date caption) my/sketches-directory)
     new-link (concat "#+CAPTION: " date " " caption "\n"
                      (org-link-make-string (concat "sketch:" (file-name-base new-file)))))
    (rename-file intermediate new-file t)
    (delete-region (org-element-property :begin link)
                   (org-element-property :end link))
    (if two-col
        (progn (insert
                (format  "#+begin_columns
#+begin_column50
%s
#+end_column50
#+begin_column50
"
                         new-link))
               (save-excursion (insert "
#+end_column50
#+end_columns
")))
      (insert new-link "\n"))))

(defun my/reload-sketches ()
  (interactive)
  (url-retrieve "https://sketches.sachachua.com/reload" (lambda (&rest args) (message "Updated sketches."))))

What are drawings useful for? Nonlinear thinking, sharing, flipping through, building up, visual shorthand, fun. Text is nicer for searching, linking, and dealing with stop-and-go thoughts.

W-‘s offered to let me use his iPad. Concepts and Procreate are both pretty cool, and I have a reasonable workflow for sending files back to my computer and getting them into my Org file. My X220 is still the fastest for quickly switching between text and drawing. I guess either will do.

Also, graphviz is pretty handy for quick diagrams, and it will probably be even more useful as I dig into it and other text-based diagram tools. The diagram at the beginning of this post was generated with:

#+begin_src dot :file "sharing-path.png" :cmdline -Kdot -Tpng -Nfontname=sachacHand -Nfontsize=30
digraph {
  rankdir=LR;
  node [shape=box];
  "Planning" -> "Coding" -> "Writing" -> "Screenshots" -> "Drawings" -> "GIFs?" -> "Video" -> "Streaming";
  "Planning" -> "Reading" -> "Writing";
  "Planning" -> "Writing";
}
#+end_src

Animated GIFs, videos, and streaming may have to wait until I have more brainspace. Plenty to tweak even now!

Python, Org Mode, and writing Org tables to CSVs so that I can read them back

| emacs, org, writing

I’ve been getting deeper into Python so that I can model our personal finances. I really like using the pandas library to manipulate data. All those years I spent trying to juggle increasing complex spreadsheets… Working with Python code in Org Babel blocks is just so much more fun. I like being able to keep my assumptions in tables without having to fuss around with naming cells for easy-to-read formulas, slice and summarize parts of my data frames, organize my notes in outlines and add commentary, and define more complicated functions that I don’t have to squeeze into a single line.

I haven’t quite been able to tempt W- into the world of Org Babel Python blocks. Still, I don’t want to give up the awesomeness of having pretty tables that I can easily edit and use. So I have a bunch of named tables (using #+NAME:), and some code that exports my tables to CSVs:

#+NAME: tables
| Table         | Key                 |
|---------------+---------------------|
| assets_w      | Description         |
| assets_s      | Description         |
| tax_rates     |                     |
| disposition   | Asset               |
| probate_rates | Asset               |
| basic         | Client information  |
| base_expenses | Category            |
| general       | General assumptions |

#+begin_src emacs-lisp :results silent :var tables=tables :tangle no
  (defun my-tbl-export (row)
    "Search for table named `NAME` and export."
    (interactive "s")
    (save-excursion
      (goto-char (point-min))
      (let ((case-fold-search t))
        (when (search-forward-regexp (concat "#\\+NAME: +" (car row)) nil t)
          (next-line)
          (org-table-export (format "%s.csv" (car row)) "orgtbl-to-csv")))))
  (mapc 'my-tbl-export tables)
#+end_src

and some code that imports them back in, and formats tables nicely if I’m displaying them in Org. The in_org block doesn’t get tangled into index.py, so I don’t clutter command-line use with Org table markup.

#+begin_src python :results silent :tangle no
  in_org=1
#+end_src

#+begin_src python :results silent :exports code
  import pandas as pd
  import numpy as np
  import orgbabelhelper as ob
  def out(df, **kwargs):
    if 'in_org' in globals():
      print(ob.dataframe_to_orgtable(df, **kwargs))
    else:
      print(df)
    return df
#+end_src

#+begin_src python :results silent :var tables=tables :colnames yes
  for row in tables:
    table = row[0]
    index = row[1] 
    if row[1] == '':
      index = None
    globals()[table] = pd.read_csv(table + '.csv', index_col=index).apply(pd.to_numeric, errors='ignore')
    # print(globals()[table])
#+end_src

Then I can use C-c C-v C-b (org-babel-execute-buffer) to update everything if I change the table in my Org file, and I can use C-c C-v C-t (org-babel-tangle) to create an index.py that W- can read through or run without needing Org.

Making books for A-

| drawing, parenting, publishing, writing

A- loves being read to. She picks up new words and ideas from the books we read, requests both favourites and new books again and again, and can identify objects in photographs and drawings. I borrowed a few children's books from the library in case reading about upcoming changes or challenges helps her understand. The books were okay, but didn't quite fit the words we use or the way we like to handle things. So this week, I decided to make my own books for A-, especially since there are few books that cover things like microphthalmia.

The first book I made was about night weaning, since we might have to do that in preparation for dental surgery under anaesthesia. I sketched it using ZoomNotes on my iPad, exported the SVG, tinkered with it in Inkscape, exported PNGs, combined the PNGs with ImageMagick, and created a 12-page PDF with 7″x8.5″ pages. That let me print the book out on legal-size paper (8.5″x14″), 2 pages per sheet, duplex printing set to flip on the short side, using this page order:
12, 1, 2, 11, 10, 3, 4, 9, 8, 5, 6, 7. I folded each sheet in half. Instead of hand-sewing the binding, I just taped the pages together. And just like that, I had a book that I could page through properly: “No More Nursing, Time to Sleep.”

I read the new book to A-. She asked me to reread it several times. She pointed to the book and said, “A-!” She pointed to the stick figure for me and said, “Mama!” Success!

The next thing I wanted to try was printing in colour. We recently replaced our printer with an HP M277dw colour laser printer that could print duplex, so I was looking forward to giving that a try. I wanted to make a book about the conformer in A-‘s little eye. This time, I drew the pages of the book using layers in Medibang Paint. I drew on the bus home from Riverdale Farm, working around a sleeping A- snuggled in my carrier. I exported each layer as a PNG, used ImageMagick to convert pairs of pages into what I needed to print (page order: 8, 1, 2, 7, 6, 3, 4, 5), combined those into a PDF. I couldn't figure out how to get the HP app to properly scale the document and print in duplex, but printing from Linux worked fine. I quickly had another book in my hands: “My Conformer.”

She's starting to echo phrases from the to books, and it's been only a few days. Wow!

I'm working on a third book now. Time for something fun: “Let's Make a Smoothie,” since she enjoys making and drinking them. She already knows all the words, so this is more about enjoyment. This time, I'm going to make a workflow that lets me draw on two-page spreads. I don't have any wide drawings planned yet, but it could be handy for later. I made an Inkscape template to help me keep margins in mind. I learned how to use Medibang Paint's folders to organize all the layers, and I'm getting the hang of digitally tracing and painting the photos I took.

I'm looking forward to making even more books and refining my workflow along the way. Here are a few things I want to try:

  • Flat colour
  • Painting
  • Programmatically adding text
  • Printing photos
  • Two-page drawing
  • Parametric templates
  • Smaller format by cutting
  • Programmatically adding photos
  • Heavier-weight paper
  • Board book replacement
  • Printing at Staples or similar
  • Print-on-demand book
  • Handstitching
  • Binding with a cover
  • Smaller format by folding and gluing
  • Mobile workflow
  • Vector drawing

And a few quick ideas for possible next books:

  • Potty Time
  • Brushing Teeth
  • Feelings
  • When I Feel Nervous
  • When I Feel Sad
  • Going to Sleep at the Dentist
  • My Life
  • My Day
  • Going Out
  • At Home
  • I Can…
  • I Can Draw
  • Waiting
  • Try Again
  • Dressing Up
  • Alimango sa Dagat
  • Leron Leron Sinta

Daily, weekly, and monthly journals: my Memento + Google Sheets + Tasks Free + Google Tasks + WordPress workflow

Posted: - Modified: | android, blogging, writing

Journaling considerations:

  • A- nurses a lot in bed. I keep my phone handy and I write when she doesn’t want to let me go.
  • I also jot quick notes throughout the day so that I don’t have to keep them in my head. These go into the nearest synchronized device.
  • It’s hard to remember the context for those notes if too much time passes. A daily verbal recap for W- and a weekly summary for my blog seem to be just the right balance. Anything older than a week gets too fuzzy, while writing detailed notes every day takes too much time away from other things I’d like to do.
  • Monthly reviews give me a better perspective on big changes. It’s hard to keep enough in my head when I’m reading or writing on my phone, so I need help summarizing a month’s worth of highlights.

Here are the technical details:

I set up Memento Database on my phone and on a backup Android phone. I picked it because it can synchronize between phones in the background, and it can also sync with Google Sheets so that I can process things further.

My journal database has the following fields:

  • Date: defaults to current date
  • Note
  • Category: single value from a list. Most of my entries go into Gross Motor, Fine Motor, Language, Self-care, Other, or Us, and I add other categories as needed.
  • Highlight: a number indicating the level of review this should be included in: 1 – weekly, 2 – monthly, 3 – yearly. I display this field as the status, so that it shows up on the right side.

I have a shortcut on my home screen so that I can quickly add a journal entry.

I normally sort the list by date, with recent entries on top.

As part of my weekly review, I look at recent entries, fill in any categories I skipped, and choose a few to highlight. For example, last week, I wrote 17 entries and I chose 13 to include in the weekly review.

I configured Memento’s default export formatting to include only the Note field and to export that without the field label.

I filtered the database to show only the entries within a given date range where the highlight value was greater than 0.5.

I grouped it by category so that similar entries were together. This was better than fiddling with the sorting, since this takes fewer taps to set back to my default view.

After filtering and grouping the entries, I used the “Send all > Send as text” command to send it to Tasks Free, which is a task manager that synchronizes with Google Tasks. I like the way I can drag-and-drop tasks to reorder them, which makes prioritizing so much easier on my phone. I edit the text in Tasks Free, turning the keywords into paragraphs and moving things around for better flow.

After drafting the body of the post (and possibly switching between phones, if my battery ran low), I select all the text, copy it into the WordPress app, set the categories and the title, and post the entry.

The monthly review process is quite similar. I start with a filtered view that shows all entries for last month (133 entries in November), and I group it by category. I skim all the entries, not just the ones included in the weekly review, because sometimes little moments turn out to be significant or part of a bigger pattern. After setting the highlight values for the things I’d like to include in my monthly review, I switch to another filter that shows me last month’s entries with a highlight value greater than 1.5 (28 entries in November). I send it all to Tasks Free, edit the post, copy it into WordPress, and publish.

If I manage to squeeze in some computer time, I use Google Tasks to copy the text into Emacs and then use my regular Org Mode review/publish processes.

I’ve been thinking about how I can improve this workflow. Sending text to the WordPress app doesn’t seem to work (the text disappears after I save or publish), and it’s kinda nice being able to move my weekly review task around on my task list in order to accommodate other priorities. I also like the way Google Tasks keeps the data from completed tasks, which has come in handy a few times. Tasks Free editing is more responsive, too. Synchronizing with Tasks Free seems to be more robust than synchronizing with Orgzly, since I only have to watch out for editing the same task on two devices instead of watching out for the whole file.

I’d like to get back to drawing the weekly and monthly reviews, but maybe that can wait until A-‘s sleep is more settled and my discretionary time is more consolidated. The visual journals are more fun to flip through, but the bulk and chronological views I hacked into my WordPress theme are reasonable workarounds.