Categories: sharing » writing » blogging

RSS - Atom - Subscribe via email

Compiling selected blog posts into HTML and EPUB so I can annotate them

| blogging, 11ty, nodejs, supernote

[2023-01-04 Wed] Added a screenshot showing annotation.

I was thinking about how to prepare for my next 10-year review, since I'll turn 40 this year. I've been writing yearly reviews with some regularity and monthly reviews sporadically, and I figured it would be nice to have those posts in an EPUB so that I can read them on my e-reader and annotate them as I do my review.

I use the 11ty static site generator to publish my blog as HTML files, since I currently can't keep more than Emacs Lisp, Javascript, and Python in my brain. (No Hugo or Jekyll for me at the moment.) I briefly thought about getting 11ty to create that archive for me, but I realized it might be easier to just write it as an external script instead of trying to figure out how to get 11ty to export one thing conditionally.

One of the things I've configured 11ty to make is a JSON file that includes all of my posts with dates, titles, permalinks, and categories. It was easy to then parse this list and filter it to get the posts I wanted. I parsed the HTML out of the _site directory that 11ty produces instead of fetching the pages from my webserver. I got the images from my webserver, though, and I made a local cache and rewrote the URLs. That way, the EPUB conversion could include the images.

Download blog.js

blog.js
const blog = require('/home/sacha/proj/static-blog/_site/blog/all/index.json');
const cheerio = require('cheerio');
const base = '/home/sacha/proj/static-blog/_site';
const fs = require('fs');
const path = require('path');

function slugify(p) {
  return p.permalink.replace('/blog', 'post-').replace(/\//g, '-');
}

async function processPost(p) {
  console.log('Processing '+ p.permalink);
  let $ = cheerio.load(fs.readFileSync(base + p.permalink + 'index.html'));
  $('#comment').remove();
  let images = $('article img');
  await Promise.all(images.map((i, e) => {
    let url = $(e).attr('src');
    const outputFileName = 'images/' + path.basename(url).replace(/ |%20|%23/g, '-');
    $(e).attr('src', outputFileName);
    $(e).attr('style', 'max-height: 100%; max-width: 100%; ' + ($(e).attr('style') || ''));
    $(e).attr('srcset', null);
    $(e).attr('sizes', null);
    $(e).attr('width', null);
    $(e).attr('height', null);
    if (!fs.existsSync(outputFileName)) {
      console.log('fetch', outputFileName);
      return fetch(url).then(res => res.arrayBuffer()).then(data => {
        const buffer = Buffer.from(data);
        return fs.createWriteStream(outputFileName).write(buffer);
      });
    } else {
      console.log(outputFileName, 'exists');
      return null;
    }
  }));
  console.log('Done ' + p.permalink);
  let slug = slugify(p);
  $('article h2').attr('id', slug);
  let header = $('article header').html();
  let entry = $('article .entry').html();
  return `<article>${header}${entry}</article>`;
}

let last10 = blog.filter((p) => p.date >= '2013-08-01');
let posts = last10.filter((p) => p.categories.indexOf('yearly') >= 0)
    .concat(blog.filter((p) => p.title == 'Turning 30: A review of the last decade'))
    .concat(last10.filter((p) => p.categories.indexOf('monthly') >= 0));

let toc = '<h1>Table of Contents</h1><ul>' + posts.map((p) => {
  return `<li><a href="#${slugify(p)}">${p.title}</a></li>\n`;
}).join('') + '</ul>';

let content = posts.reduce(async (prev, val) => {
    return await prev + await processPost(val);
  }, '');
content.then((data) => {
  fs.writeFileSync('archive.html',
                   `<html><body>${toc}${data}</body></html>`);

});

This created an archive.html with my posts, using the images/ directory for the images. Then I used my shell script for converting and copying files to convert it to EPUB and copy it over.

On the SuperNote, I can highlight text by drawing square brackets around it. If I tap that text, I can write or draw underneath it. Here's what that looks like:

20230104_090739.png
Figure 1: Writing an annotation

These notes are collected into a "Digest" view, and I can export things from there. (Example: archive.pdf)

2023-01-04_09-23-57.png
Figure 2: Here's what that digest is like when exported.

(Hmm, maybe I should ask them about hiding the pencil icon…)

Anyway, I think that might be a good starting point for my review.

Writing my blog posts by hand

| blogging, supernote

A- complains if I get screentime when she doesn't get screentime, so it's hard to find time to write on my laptop or on my phone. I've experimented with dictation before, since Google Recorder can make a half-decent transcript. I'm not used to talking things out, though. I keep correcting false starts, stutters, and mis-recognized words.

Fortunately, I can write on my A5X while waiting for A-. I get more space than I do when writing on my phone, so it's easier for me to think. I can export pages as PNGs, Dropbox, share each page, sync with with Google Photos, and then use Lens to copy the text. I can then paste it into Orgzly, which automatically syncs with Syncthing so that I can edit it on my laptop with Emacs. It needs a little cleanup (capitalization, stray punctuation, missed words, things in the wrong order), but editing it feels easier than dealing with the output of speech recognition, so it seems to be worth the extra time and effort. Besides, it feels less embarrassing to write at the sandbox than it is to talk to myself.

I can edit the text directly on my phone, but I still need my laptop to publish my blog because I haven't set up my static site generator on my server. Some day! In the meantime, this might be a good workflow for getting thoughts out there.

What if I want to refer to sketches while I write? Flipping between pages on the A5X can be challenging if they're not next to each other, but I can keep my current writing page next to my sketch. I could also view the sketch on my phone and balance it on the A5X, or use layers to keep a small version of the sketch as a handy reference. Lots of ideas to play around with…

Statically generating my blog with Eleventy

| blogging

Things will probably be a little strange on my blog for the next few days, as I've decided to experiment with statically generating my blog with Eleventy. It's a little complicated because I wanted to keep as many of my posts and category/tag feeds as possible.

To speed things up, I usually work with a subset of my posts. Generating a partial copy of my site results in 557 files and takes 3.73 seconds. When I generate the full copy of my site, it writes 13358 files in 96.73 seconds.

With any luck, I'll be able to get most of the things working before the next Emacs News post. Let's see!

Daily, weekly, and monthly journals: my Memento + Google Sheets + Tasks Free + Google Tasks + WordPress workflow

Posted: - Modified: | android, blogging, writing

Journaling considerations:

  • A- nurses a lot in bed. I keep my phone handy and I write when she doesn’t want to let me go.
  • I also jot quick notes throughout the day so that I don’t have to keep them in my head. These go into the nearest synchronized device.
  • It’s hard to remember the context for those notes if too much time passes. A daily verbal recap for W- and a weekly summary for my blog seem to be just the right balance. Anything older than a week gets too fuzzy, while writing detailed notes every day takes too much time away from other things I’d like to do.
  • Monthly reviews give me a better perspective on big changes. It’s hard to keep enough in my head when I’m reading or writing on my phone, so I need help summarizing a month’s worth of highlights.

Here are the technical details:

I set up Memento Database on my phone and on a backup Android phone. I picked it because it can synchronize between phones in the background, and it can also sync with Google Sheets so that I can process things further.

My journal database has the following fields:

  • Date: defaults to current date
  • Note
  • Category: single value from a list. Most of my entries go into Gross Motor, Fine Motor, Language, Self-care, Other, or Us, and I add other categories as needed.
  • Highlight: a number indicating the level of review this should be included in: 1 – weekly, 2 – monthly, 3 – yearly. I display this field as the status, so that it shows up on the right side.

I have a shortcut on my home screen so that I can quickly add a journal entry.

I normally sort the list by date, with recent entries on top.

As part of my weekly review, I look at recent entries, fill in any categories I skipped, and choose a few to highlight. For example, last week, I wrote 17 entries and I chose 13 to include in the weekly review.

I configured Memento’s default export formatting to include only the Note field and to export that without the field label.

I filtered the database to show only the entries within a given date range where the highlight value was greater than 0.5.

I grouped it by category so that similar entries were together. This was better than fiddling with the sorting, since this takes fewer taps to set back to my default view.

After filtering and grouping the entries, I used the “Send all > Send as text” command to send it to Tasks Free, which is a task manager that synchronizes with Google Tasks. I like the way I can drag-and-drop tasks to reorder them, which makes prioritizing so much easier on my phone. I edit the text in Tasks Free, turning the keywords into paragraphs and moving things around for better flow.

After drafting the body of the post (and possibly switching between phones, if my battery ran low), I select all the text, copy it into the WordPress app, set the categories and the title, and post the entry.

The monthly review process is quite similar. I start with a filtered view that shows all entries for last month (133 entries in November), and I group it by category. I skim all the entries, not just the ones included in the weekly review, because sometimes little moments turn out to be significant or part of a bigger pattern. After setting the highlight values for the things I’d like to include in my monthly review, I switch to another filter that shows me last month’s entries with a highlight value greater than 1.5 (28 entries in November). I send it all to Tasks Free, edit the post, copy it into WordPress, and publish.

If I manage to squeeze in some computer time, I use Google Tasks to copy the text into Emacs and then use my regular Org Mode review/publish processes.

I’ve been thinking about how I can improve this workflow. Sending text to the WordPress app doesn’t seem to work (the text disappears after I save or publish), and it’s kinda nice being able to move my weekly review task around on my task list in order to accommodate other priorities. I also like the way Google Tasks keeps the data from completed tasks, which has come in handy a few times. Tasks Free editing is more responsive, too. Synchronizing with Tasks Free seems to be more robust than synchronizing with Orgzly, since I only have to watch out for editing the same task on two devices instead of watching out for the whole file.

I’d like to get back to drawing the weekly and monthly reviews, but maybe that can wait until A-‘s sleep is more settled and my discretionary time is more consolidated. The visual journals are more fun to flip through, but the bulk and chronological views I hacked into my WordPress theme are reasonable workarounds.

What do I want from an annual review?

Posted: - Modified: | blogging

I’ve got most of the pieces for an annual review: monthly reviews in visual and text form, my time records, and a recent flip-through of all of my sketches. I’d like to bring my ledger of income and expenses up to date, finish reading all of my blog posts, and draw a couple of yearly summaries (monthly events, differences between 2015 and 2016, analyses). I want to make the most of my computer time, so I should think about what I want from my annual review and how I can get that more efficiently.

Highlights of the year
A month-by-month list of highlights is good for reminding me of events and getting around the fogginess of memory. There’s so much to celebrate and appreciate. This also simplifies longer-term reviews, like the 10-year review I did when I turned 30.
Differences
What did I learn? What did I forget? It can be easier to see the differences when you compare across a longer time period. This can help me solidify growth, revisit things I’ve left behind, watch out for drifting, and choose what to focus on next year.
Patterns and trends
Taking a look at the data can sometimes turn up things I wouldn’t have guessed. Time, finances, and A-‘s data too – so much to explore! This might take a little longer, since it involves code.
Decision review
This is probably better broken up into separate posts, maybe even decoupled from my annual review.
What worked well? Why? How can we make things even better?
Good for continuous improvement. Might not go into as much depth as the decision reviews.

My overall goals are to:

  • remember and celebrate the journey
  • keep improving; remember what I’ve learned and revisit what I might have shelved
  • make it easier for my future self (or other people reading my archive) to get an overview of the year
  • maybe have conversations that grow out of the updates (notes on things I’ve tried, ideas for stuff that might help)

I’ll probably end up doing my annual review in chunks instead of waiting until it’s all done, since otherwise it might take me a few months.

Thinking about my frequency of annual reviews

Posted: - Modified: | blogging

I’ve been doing annual reviews a few times a year: my birthday in August, the new year in January, and experiment-related reflections in February. It’s a little excessive, perhaps. My weekly and monthly reviews make it easy enough to summarize events over 12 months, so it’s not that much more effort to do a new review with a slight offset.

The experiment review has different guide questions, so that’s useful. The birthday and new year reviews have a lot of overlap, though. What happened? How am I different? What did I learn? What did I forget and want to relearn? What worked well? What do I want to focus on next? What could make this even better? The two reviews cover the same ground, especially since I don’t do New Year’s Resolutions. I like the birthday review because it’s anchored on things that are meaningful to me, and paced according to my life.

The new year review would probably be better suited to reflecting on external influences, since that’s synchronized with other people’s reviews, but external events don’t seem to matter that much to me when I reflect on my year.

People often use the Christmas/New Year break to send out family updates and pictures. Both my family and W-‘s family like taking family pictures, so we’re covered there. I feel somewhat odd about the idea of announcing things on behalf of W- or A-, or getting W- to contribute. I’m more comfortable capturing the changes in my own life, noting the occasional highlight from theirs – but with my individual voice, not a collective We. I think of it more for personal note-taking and celebration (and maybe the occasional acquaintance catching up through my archives) rather than pushing updates to a list of people whom I think should hear about our year. Opt-in is more comfortable for me than opt-out. I’m probably making it more complicated than it needs to be, but I wonder if there’s a thought in here that’s worth untangling…

I wonder how I mentally chunk my memories. Do I think of them in terms of ages: my 20s, etc.? Do I think in terms of calendar years? Years come to mind more easily than ages do when I think about milestones such as coming to Canada. So maybe that’s an argument for keeping the new year review…

There’s also the benefit of being able to send people a link to a tidy summary when they wish me a happy new year, although that happens more around birthdays anyway.

Hmm. I guess I’ll try to squeeze another annual review in this month, and then I can reconsider the question in August. More writing is good, anyway.

Posting more thoughts

Posted: - Modified: | blogging

I can write on my phone while nursing, which is probably a far better occupation for my mind instead of scrolling through Facebook or Reddit for the nth time. It’s not my ideal writing setup – I can see around a paragraph or two on the screen at a time, and I don’t have the outlining/linking/figuring-out tools I’m used to on my computer – but it gets me writing in full sentences instead of just jotting down lists. I can capture more thoughts this way, and I don’t have to stay up late to get through my drawing backlog.

It’s important to me to be able to flesh out thoughts a little despite the interruptions of life with a baby. With a place to store these half-finished thoughts, I can make some progress. I’m not trying to write a great novel (or that Emacs book I planned a long time ago) – just exploring thoughts and questions and ideas, and storing hooks for associative memories.

A- nurses a lot, which we’re okay with. More sustenance and comfort for her, and our lives are flexible enough to accommodate it. I focus on her when she wants interaction, and I keep my phone handy for when she seems to be nursing to sleep. It’s a practice I could probably help her get out of, but things are also fine the way they are. I’ll probably let her take the lead on this one, at least for now.

What do I want to think about during these moments?

  • For the present: task lists, decisions, questions, research
  • For weekly and monthly reviews: highlights, memories
  • For future Sacha: sketches of daily life, thoughts, things I’m learning
  • For other people: things I figured out the hard way; counter-intuitive or alternative experiences; ideas and thoughts
  • For family and friends: stories

There are lots of things I can think through and write about, even in small chunks and without tools for structure. I’ll experiment with writing about and posting more of them. After all, my blog started with a few years of random snippets and thoughts. I don’t mind spending a few more years writing about mundane things and incomplete thoughts that might not be of much interest to other people, just in case it might be of interest to my future self. I’ve already set up categories and filtered mailing lists, so people can choose what to read. I can write more for myself, and enjoy what serendipitous conversations come my way. :)