Categories: python

RSS - Atom - Subscribe via email

Updating Planet Venus so that planet.emacslife.com can handle mix-blend-mode in my SVGs

| geek, python

I wanted to easily turn segments from my Yay Emacs livestreams or recorded narration into closed-caption audio and dynamic highlighting of my SVG sketches and text transcripts. That way, people could easily jump around to sections they're interested in.

Not everyone has Javascript turned on, so I wanted to start with something that made sense even in RSS feeds like the one on Planet Emacslife (which strips out <style> and <script>) and was progressively enhanced with captions and highlighting if you saw it on my site.

2024-01-25_08-37-11.png
Figure 1: My SVGs were broken: black fill, no mix-blend-mode

The blog aggregator I'm using, Planet Venus, hasn't been updated in 14 years. It even uses Python 2. I considered switching to a different aggregator, so I started checking out different community planets. Most of the other planets listed in this HN thread about aggregators looked like they were using the same Planet Venus aggregator, although these were some planets that used something else:

I decided I'd stick with Planet Venus for now, since I could probably figure out how to get the attributes sorted out.

I found planet/vendor/feedparser.py by digging around. Adding mix-blend-mode to the list of attributes there was not enough to get it working. I started exploring pdb for interactive Python debugging inside Emacs, although I think dap is an option too. I wrote a short bit of code to test things out:

import sys,os
sys.path.insert(1, os.path.join(os.path.dirname(__file__), 'planet/vendor'))
from feedparser import _sanitizeHTML
assert 'strong' in _sanitizeHTML('<strong>Hello</strong>', 'utf-8', 'text/html')
assert 'mix-blend-mode' in _sanitizeHTML('<svg><path style="fill: red;mix-blend-mode:darken"></path></svg>', 'utf-8', 'text/html')

It was pretty easy to use pdb to start stepping through and into functions, although I didn't dig into it deeply because I figured it out another way. While looking through the pull requests for the Venus repository, I came across this pull request to add data- attributes which was helpful because it pointed me to planet/vendor/html5lib/sanitizer.py. Once I added mix-blend-mode to that one, things worked. Here's my Github branch.

2024-01-25_08-51-33.png
Figure 2: planet.emacslife.com now lets me use mix-blend-mode

On a somewhat related note, I also had to patching shr to handle SVG images with viewBox attributes. I guess SVGs aren't that common yet, but I'm looking forward to playing around with them more, so I might as well make things better (at least when it comes to things I can actually tweak). mix-blend-mode on SVG elements says it's not supported in Safari or a bunch of mobile browsers, but it seems to be working on my phone, so maybe that's cool now. Using mix-blend-mode means I don't have to do something complicated when it comes to animating highlights while still keeping text visible, and improving SVG support is the right thing to do. Onward!

View org source for this post

Summarizing #EmacsConf's growth over 5 years by year, and making an animated GIF

| emacs, emacsconf, python

Of course, after I charted EmacsConf's growth in terms of number of submissions and minutes, I realized I also wanted to just sum everything up by year. So here it is:

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = df.drop('Weeks to CFP', axis=1).groupby(['Year']).sum()
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12,6))
fig1 = df['Count'].plot(kind="bar", ax=ax[0], title='Number of submissions')
fig2 = df['Minutes'].plot(kind="bar", ax=ax[1], title='Number of minutes')
fig.get_figure().savefig('emacsconf-by-year.png')
return df
Year Count Minutes
2019 28 429
2020 35 699
2021 44 578
2022 29 512
2023 39 730
emacsconf-by-year.png

I also wanted to make an animated GIF so that the cumulative graphs could be a little easier to understand.

import pandas as pd
import matplotlib.pyplot as plt
import imageio as io
df = pd.DataFrame(data[1:], columns=data[0])
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12,6))
count = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
minutes = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Minutes', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
ax[0].set_ylim([0, count.max().max()])
ax[1].set_ylim([0, minutes.max().max()])
with io.get_writer('emacsconf-combined.gif', mode='I', duration=[500, 500, 500, 500, 1000], loop=0) as writer:
    for year in range(2019, 2024):
        count[year].plot(ax=ax[0], title='Cumulative submissions')
        minutes[year].plot(ax=ax[1], title='Cumulative minutes')
        ax[0].legend(loc='upper left')
        ax[1].legend(loc='upper left')
        for axis in ax:
            for line in axis.get_lines():
                if line.get_label() == '2023':
                    line.set_linewidth(5)
            for line in axis.legend().get_lines():
                if line.get_label() == '2023':
                    line.set_linewidth(5)        
        filename = f'emacsconf-combined-${year}.png'
        fig.get_figure().savefig(filename)
        image = io.v3.imread(filename)
        writer.append_data(image)
emacsconf-combined.gif
Figure 1: Animated GIF showing the cumulative total submissions and minutes

I am not quite sure what kind of story this data tells (aside from the fact that there sure are a lot of great talks), but it was fun to learn how to make more kinds of graphs and animate them too. Could be useful someday. =)

#EmacsConf backstage: looking at EmacsConf's growth over 5 years, and how to do pivot tables and graphs with Org Mode and the Python pandas library

| emacsconf, emacs, org, python

Having helped organize EmacsConf for a number of years now, I know that I usually panic about whether we have submissions partway through the call for participation. This causes us to extend the CFP deadline and ask people to ask more people to submit things, and then we end up with a wonderful deluge of talks that I then have to somehow squeeze into a reasonable-looking schedule.

This year, I managed to not panic and I also resisted the urge to extend the CFP deadline, trusting that there will actually be tons of cool stuff. It helped that my schedule SVG code let me visualize what the conference could feel like with the submissions so far, so we started with a reasonably nice one-track conference and built up from there. It also helped that I'd gone back to the submissions for 2022 and plotted them by the number of weeks before the CFP deadline, and I knew that there'd be a big spike from all those people whose Org DEADLINE: properties would nudge them into finalizing their proposals.

Out of curiosity, I wanted to see how the stats for this year compared with previous years. I wrote a small function to collect the data that I wanted to summarize:

emacsconf-count-submissions-by-week: Count submissions in INFO by distance to CFP-DEADLINE.
(defun emacsconf-count-submissions-by-week (&optional info cfp-deadline)
  "Count submissions in INFO by distance to CFP-DEADLINE."
  (setq cfp-deadline (or cfp-deadline emacsconf-cfp-deadline))
  (setq info (or info (emacsconf-get-talk-info)))
  (cons '("Weeks to CFP end date" "Count" "Hours")
        (mapcar (lambda (entry)
                  (list (car entry)
                        (length (cdr entry))
                        (apply '+ (mapcar 'cdr (cdr entry)))))
                (seq-group-by
                 'car
                 (sort
                  (seq-keep
                   (lambda (o)
                     (and (emacsconf-publish-talk-p o)
                          (plist-get o :date-submitted)
                          (cons (floor (/ (days-between (plist-get o :date-submitted) cfp-deadline)
                                          7.0))
                                (string-to-number
                                 (or (plist-get o :video-duration)
                                     (plist-get o :time)
                                     "0")))))
                   info)
                  (lambda (a b) (< (car a) (car b))))))))

and then I ran it against the different files for each year, filling in the previous years' data as needed. The resulting table is pretty long, so I've put that in a collapsible section.

(let ((years `((2023 "~/proj/emacsconf/2023/private/conf.org" "2023-09-15")
               (2022 "~/proj/emacsconf/2022/private/conf.org" "2022-09-18")
               (2021 "~/proj/emacsconf/2021/private/conf.org" "2021-09-30")
               (2020 "~/proj/emacsconf/wiki/2020/submissions.org" "2020-09-30")
               (2019 "~/proj/emacsconf/2019/private/conf.org" "2019-08-31"))))
  (append
   '(("Weeks to CFP" "Year" "Count" "Minutes"))
   (seq-mapcat
    (lambda (year-info)
      (let ((emacsconf-org-file (elt year-info 1))
            (emacsconf-cfp-deadline (elt year-info 2))
            (year (car year-info)))
        (mapcar (lambda (o) (list (car o) year (cadr o) (elt o 2)))
                (cdr (emacsconf-count-submissions-by-week (emacsconf-get-talk-info) emacsconf-cfp-deadline)))))
    years)))
Table
Weeks to CFP Year Count Minutes
-12 2023 4 70
-9 2023 2 30
-7 2023 2 30
-5 2023 2 30
-4 2023 2 60
-3 2023 3 40
-2 2023 5 130
-1 2023 10 180
0 2023 8 140
1 2023 1 20
-8 2022 2 25
-5 2022 2 31
-3 2022 2 31
-2 2022 2 17
-1 2022 8 191
0 2022 8 110
1 2022 5 107
-8 2021 4 50
-7 2021 2 17
-6 2021 1 7
-5 2021 2 22
-4 2021 2 19
-3 2021 5 73
-2 2021 1 10
-1 2021 12 163
0 2021 13 197
1 2021 1 10
2 2021 1 10
-5 2020 1 10
-4 2020 1 15
-2 2020 1 30
-1 2020 4 68
0 2020 21 424
1 2020 7 152
-5 2019 2 45
-4 2019 1 21
-2 2019 6 126
-1 2019 9 82
0 2019 9 148
2 2019 1 7

Some talks were proposed off-list and are not captured here, and cancelled or withdrawn talks weren't included either. The times for previous years use the actual video time, and the times for this year use proposed times.

Off the top of my head, I didn't know of an easy way to make a pivot table or cross-tab using just Org Mode or Emacs Lisp. I tried using datamash, but I was having a hard time getting my output just the way I wanted it. Fortunately, it was super-easy to get my data from an Org table into Python so I could use pandas.pivot_table. Because I had used #+NAME: submissions-by-week to label the table, I could use :var data=submissions-by-week to refer to the data in my Python program. Then I could summarize them by week.

Here's the number of submissions by the number of weeks to the original CFP deadline, so we can see people generally like to target the CFP date.

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True)
fig, ax = plt.subplots()
figure = df.plot(title='Number of submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('number-of-submissions.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 4
-9 0 0 0 0 2
-8 0 0 4 2 0
-7 0 0 2 0 2
-6 0 0 1 0 0
-5 2 1 2 2 2
-4 1 1 2 0 2
-3 0 0 5 2 3
-2 6 1 1 2 5
-1 9 4 12 8 10
0 9 21 13 8 8
1 0 7 1 5 1
2 1 0 1 0 0
number-of-submissions.png

Calculating the cumulative number of submissions might be more useful. Here, each row shows the number received so far.

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Count', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax = plt.subplots()
figure = df.plot(title='Cumulative submissions by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('cumulative-submissions.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 4
-9 0 0 0 0 6
-8 0 0 4 2 6
-7 0 0 6 2 8
-6 0 0 7 2 8
-5 2 1 9 4 10
-4 3 2 11 4 12
-3 3 2 16 6 15
-2 9 3 17 8 20
-1 18 7 29 16 30
0 27 28 42 24 38
1 27 35 43 29 39
2 28 35 44 29 39
cumulative-submissions.png
Figure 1: Cumulative submissions by number of weeks to CFP end date

And here's the cumulative number of minutes based on the proposals.

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame(data[1:], columns=data[0])
df = pd.pivot_table(df, columns=['Year'], index=['Weeks to CFP'], values='Minutes', aggfunc='sum', fill_value=0).iloc[::-1].sort_index(ascending=True).cumsum()
fig, ax = plt.subplots()
figure = df.plot(title='Cumulative minutes by number of weeks to the CFP end date', ax=ax)
for line in ax.get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)
for line in plt.legend().get_lines():
    if line.get_label() == '2023':
        line.set_linewidth(5)        
figure.get_figure().savefig('cumulative-minutes.png')
return df
Weeks to CFP 2019 2020 2021 2022 2023
-12 0 0 0 0 70
-9 0 0 0 0 100
-8 0 0 50 25 100
-7 0 0 67 25 130
-6 0 0 74 25 130
-5 45 10 96 56 160
-4 66 25 115 56 220
-3 66 25 188 87 260
-2 192 55 198 104 390
-1 274 123 361 295 570
0 422 547 558 405 710
1 422 699 568 512 730
2 429 699 578 512 730
cumulative-minutes.png
Figure 2: Cumulative minutes by number of weeks to the CFP end date

So… yeah… 730 minutes of talks for this year… I might've gotten a little carried away. But I like all the talks! And I want them to be captured in videos and maybe even transcribed by people who will take the time to change misrecognized words like Emax into Emacs! And I want people to be able to connect with other people who are interested in the sorts of stuff they're doing! So we're going to make it happen. The draft schedule's looking pretty full, but I think it'll work out, especially if the speakers send in their videos on time. Let's see how it all works out!

(…and look, I even got to learn how to do pivot tables and graphs with Python!)

Resetting the Python logger level

| python

I wanted to get extra debugging output from a Python script that I was running, but one of my imports seemed to mess things up and the logger level was 20 (info) instead of debug.

import logging
from llama_index import GPTSimpleVectorIndex, MockLLMPredictor, MockEmbedding, QuestionAnswerPrompt
logging.basicConfig(level=logging.DEBUG)
logging.debug('This is a test.')

As it turns out, there was already a call to basicConfig in github_repository_reader.py which llama_index loaded at some point, and the Python documentation says: "As it’s intended as a one-off simple configuration facility, only the first call will actually do anything: subsequent calls are effectively no-ops."

So this is what I needed to do instead:

logging.getLogger().setLevel(logging.DEBUG)

Rename, recolor, and file my sketches automatically

| geek, supernote, python, drawing

I want to make it easier to process the sketchnotes I make on my Supernote. I write IDs of the form yyyy-mm-dd-nn to identify my sketches. To avoid duplicates, I get these IDs from the web-based journaling system I wrote. I've started putting the titles and tags into those journal entries as well so that I can reuse them in scripts. When I export a sketch to PNG and synchronize it, the file appears in my ~/Dropbox/Supernote/EXPORT directory on my laptop. Then it goes through this process:

  • I use Google Cloud Vision to detect handwriting so that I can find the ID.
    • I retrieve the matching entry from my journal system and rename the file based on the title and tags.
    • If there's no matching entry, I rename the file based on the ID.
  • If there are other tags or references in the sketch, I add those to the filename as well.
  • I recolor it based on the tags, so parenting-related posts are a little purple, tech/Emacs-related posts are blue, and things are generally highlighted in yellow otherwise.
  • I move it to a directory based on the tags.
    • If it's a private sketch, I move it to the directory for my private sketches.
    • If it's a public sketch, I move it to the directory that will eventually get synchronized to sketches.sachachua.com, and I reload the list of sketches after some delay.

The following code does that processing.

Download supernote-daemon

supernote-daemon source code
#!/usr/bin/python3
# -*- mode: python -*-

# (c) 2022-2023 Sacha Chua (sacha@sachachua.com) - MIT License

# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation files
# (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge,
# publish, distribute, sublicense, and/or sell copies of the Software,
# and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:

# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.

# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
# BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.


import os
import json
import re
import requests
import time
from dotenv import load_dotenv
# Import the Google Cloud client libraries
from google.cloud import vision
from google.cloud.vision_v1 import AnnotateImageResponse
import sys
sys.path.append("/home/sacha/proj/supernote/")
import recolor   # noqa: E402  # muffles flake8 error about import
load_dotenv()


# Set the folder path where the png files are located
folder_path = '/home/sacha/Dropbox/Supernote/EXPORT/'
public_sketch_dir = '/home/sacha/sync/sketches/'
private_sketch_dir = '/home/sacha/sync/private-sketches/'

# Initialize the Google Cloud Vision client
client = vision.ImageAnnotatorClient()
refresh_counter = 0

def extract_text(client, file):
    json_file = file[:-3] + 'json'
    # TODO Preprocess to keep only black text
    with open(file, 'rb') as image_file:
        content = image_file.read()
    # Convert the png file to a Google Cloud Vision image object
    image = vision.Image(content=content)

    # Extract handwriting from the image using the Google Cloud Vision API
    response = client.document_text_detection(image=image)
    response_json = AnnotateImageResponse.to_json(response)
    json_response = json.loads(response_json)
    # Save the response to a json file with the same name as the png file
    with open(json_file, "w") as f:
        json.dump(json_response, f)


def maybe_rename(file):
    # TODO Match on ID
    json_file = file[:-3] + 'json'
    with open(json_file, 'r') as f:
        data = json.load(f)

    # Extract the text from the json file
    text = data['fullTextAnnotation']['text']

    # Check if the text contains a string matching the regex pattern
    pattern = r'(?<!ref:)[0-9]{4}-[0-9]{2}-[0-9]{2}-[0-9]{2}'
    match = re.search(pattern, text)
    if match:
        # Get the matched string
        matched_string = match.group(0)
        new_name = matched_string
        from_zid = get_journal_entry(matched_string).strip()
        if from_zid:
            new_name = matched_string + ' ' + from_zid
        tags = get_tags(new_name, text)
        if tags:
            new_name = new_name + ' ' + tags
        ref = get_references(text)
        if ref:
            new_name = new_name + ' ' + ref
        print('Renaming ' + file + ' to ' + new_name)
        # Rename the png and json files to the matched string
        new_filename = os.path.join(os.path.dirname(file), new_name + '.png')
        rename_set(file, new_filename)
        return new_filename


def get_tags(filename, text):
    tags = re.findall(r'(^|\W)#[ \n\t]+', text)
    return ' '.join(filter(lambda x: x not in filename, tags))


def get_references(text):
    refs = re.findall(r'!ref:[0-9]{4}-[0-9]{2}-[0-9]{2}-[0-9]{2}', text)
    return ' '.join(refs)


def get_journal_entry(zid):
    resp = requests.get('https://' + os.environ['JOURNAL_USER']
                        + ':' + os.environ['JOURNAL_PASS']
                        + '@journal.sachachua.com/api/entries/' + zid)
    j = resp.json()
    if j and not re.search('^I thought about', j['Note']):
        return j['Note']


def get_color_map(filename, text=None):
    if text:
        together = filename + ' ' + text
    else:
        together = filename
    if re.search('r#(parenting|purple|life)', together):
        return {'9d9d9d': '8754a1', 'c9c9c9': 'e4c1d9'}  # parenting is purplish
    elif re.search(r'#(emacs|geek|tech|blue)', together):
        return {'9d9d9d': '2b64a9', 'c9c9c9': 'b3e3f1'}  # geeky stuff in light/dark blue
    else:
        return {'9d9d9d': '884636', 'c9c9c9': 'f6f396'}  # yellow highlighter, dark brown


def rename_set(old_name, new_name):
    if old_name != new_name:
        old_json = old_name[:-3] + 'json'
        new_json = new_name[:-3] + 'json'
        os.rename(old_name, new_name)
        os.rename(old_json, new_json)


def recolor_based_on_filename(filename):
    color_map = get_color_map(filename)
    recolored = recolor.map_colors(filename, color_map)
    # possibly rename based on the filename
    new_filename = re.sub(' #(purple|blue)', '', filename)
    rename_set(filename, new_filename)
    recolored.save(new_filename)


def move_processed_sketch(file):
    global refresh_counter
    if '#private' in file:
        output_dir = private_sketch_dir
    elif '#' in file:
        output_dir = public_sketch_dir
        refresh_counter = 3
    else:
        return file
    new_filename = os.path.join(output_dir, os.path.basename(file))
    rename_set(file, new_filename)
    return new_filename


def process_file(file):
    json_file = file[:-3] + 'json'
    # Check if a corresponding json file already exists
    if not os.path.exists(json_file):
        extract_text(client, file)
    if not re.search('[0-9]{4}-[0-9]{2}-[0-9]{2}-[0-9]{2} ', file):
        file = maybe_rename(file)
    recolor_based_on_filename(file)
    move_processed_sketch(file)


def process_dir(folder_path):
    global processed_files
    # Iterate through all png files in the specified folder
    files = sorted(os.listdir(folder_path))
    for file in files:
        if file.endswith('.png') and '_' in file:
            print("Processing ", file)
            process_file(os.path.join(folder_path, file))


def daemon(folder_path, wait):
    global refresh_counter
    while True:
        process_dir(folder_path)
        time.sleep(wait)
        if refresh_counter > 0:
            refresh_counter = refresh_counter - 1
            if refresh_counter == 0:
                print("Reloading sketches")
                requests.get('https://' + os.environ['JOURNAL_USER'] + ':'
                             + os.environ['JOURNAL_PASS']
                             + '@sketches.sachachua.com/reload?python=1')


if __name__ == '__main__':
    # Create a set to store the names of processed files
    processed_files = set()
    if len(sys.argv) > 1:
        if os.path.isdir(sys.argv[1]):
            folder_path = sys.argv[1]
            daemon(folder_path, 300)
        else:
            for f in sys.argv[1:]:
                process_file(f)
    else:
        daemon(folder_path, 300)

It uses this script I wrote to recolor my sketches with Python.

I'm contemplating writing some annotation tools to make it easier to turn the detected text into useful text for searching or writing about because the sketches throw off the recognition (misrecognized text, low confidence) and the columns mess up the line wrapping. Low priority, though.

My handwriting (at least for numbers) is probably simple enough that I might be able to train Tesseract OCR to process that someday. And who knows, maybe some organization will release a pre-trained model for offline handwriting recognition that'll be as useful as OpenAI Whisper is for audio files. That would be neat!

Using Emacs and Python to record an animation and synchronize it with audio

| emacs, emacsconf, python, subed, video

[2023-01-14 Sat]: Removed my fork since upstream now has the :eval function.

The Q&A session for Things I'd like to see in Emacs (Richard Stallman) from EmacsConf 2022 was done over Mumble. Amin pasted the questions into the Mumble chat buffer and I copied them into a larger buffer as the speaker answered them, but I didn't do it consistently. I figured it might be worth making another video with easier-to-read visuals. At first, I thought about using LaTeX to create Beamer slides with the question text, which I could then turn into a video using ffmpeg. Then I decided to figure out how to animate the text in Emacs, because why not? I figured a straightforward typing animation would probably be less distracting than animate-string, and emacs-director seems to handle that nicely. I forked it to add a few things I wanted, like variables to make the typing speed slower (so that it could more reliably type things on my old laptop, since sometimes the timers seemed to have hiccups) and an :eval step for running things without needing to log them. (2023-01-14: Upstream has the :eval feature now.)

To make it easy to synchronize the resulting animation with the chapter markers I derived from the transcript of the audio file, I decided to beep between scenes. First step: make a beep file.

ffmpeg -y -f lavfi -i 'sine=frequency=1000:duration=0.1' beep.wav

Next, I animated the text, with a beep between scenes. I used subed-parse-file to read the question text directly from the chapter markers, and I used simplescreenrecorder to set up the recording settings (including audio).

(defun my-beep ()
  (interactive)
  (save-window-excursion
    (shell-command "aplay ~/recordings/beep.wav &" nil nil)))

(require 'director)
(defvar emacsconf-recording-process nil)
(shell-command "xdotool getwindowfocus windowsize 1282 720")
(progn
  (switch-to-buffer (get-buffer-create "*Questions*"))
  (erase-buffer)
  (org-mode)
  (face-remap-add-relative 'default :height 300)
  (setq-local mode-line-format "   Q&A for EmacsConf 2022: What I'd like to see in Emacs (Richard M. Stallman) - emacsconf.org/2022/talks/rms")
  (sit-for 3)
  (delete-other-windows)
  (hl-line-mode -1)
  (when (process-live-p emacsconf-recording-process) (kill-process emacsconf-recording-process))
  (setq emacsconf-recording-process (start-process "ssr" (get-buffer-create "*ssr*")
                                                   "simplescreenrecorder"
                                                   "--start-recording"
                                                   "--start-hidden"))
  (sit-for 3)
  (director-run
   :version 1
   :log-target '(file . "/tmp/director.log")
   :before-start
   (lambda ()
     (switch-to-buffer (get-buffer-create "*Questions*"))
     (delete-other-windows))
   :steps
   (let ((subtitles (subed-parse-file "~/proj/emacsconf/rms/emacsconf-2022-rms--what-id-like-to-see-in-emacs--answers--chapters.vtt")))
     (apply #'append
            (list
             (list :eval '(my-beep))
             (list :type "* Q&A for Richard Stallman's EmacsConf 2022 talk: What I'd like to see in Emacs\nhttps://emacsconf.org/2022/talks/rms\n\n"))
            (mapcar
             (lambda (sub)
               (list
                (list :log (elt sub 3))
                (list :eval '(progn (org-end-of-subtree)
                                    (unless (bolp) (insert "\n"))))
                (list :type (concat "** " (elt sub 3) "\n\n"))
                (list :eval '(org-back-to-heading))
                (list :wait 5)
                (list :eval '(my-beep))))
             subtitles)))
   :typing-style 'human
   :delay-between-steps 0
   :after-end (lambda ()
                (process-send-string emacsconf-recording-process "record-save\nwindow-show\nquit\n"))
   :on-failure (lambda ()
                 (process-send-string emacsconf-recording-process "record-save\nwindow-show\nquit\n"))
   :on-error (lambda ()
               (process-send-string emacsconf-recording-process "record-save\nwindow-show\nquit\n"))))

I used the following code to copy the latest recording to animation.webm and extract the audio to animation.wav. my-latest-file and my-recordings-dir are in my Emacs config.

(let ((name "animation.webm"))
  (copy-file (my-latest-file my-recordings-dir) name t)
  (shell-command
   (format "ffmpeg -y -i %s -ar 8000 -ac 1 %s.wav"
           (shell-quote-argument name)
           (shell-quote-argument (file-name-sans-extension name)))))

Then I needed to get the timestamps of the beeps in the recording. I subtracted a little bit (0.82 seconds) based on comparing the waveform with the results.

filename = "animation.wav"
from scipy.io import wavfile
from scipy import signal
import numpy as np
import re
rate, source = wavfile.read(filename)
peaks = signal.find_peaks(source, height=1000, distance=1000)
base_times = (peaks[0] / rate) - 0.82
print(base_times)

I noticed that the first question didn't seem to get beeped properly, so I tweaked the times. Then I wrote some code to generate a very long ffmpeg command that used trim and tpad to select the segments and extend them to the right durations. There was some drift when I did it without the audio track, but the timestamps seemed to work right when I included the Q&A audio track as well.

import webvtt
import subprocess
chapters_filename =  "emacsconf-2022-rms--what-id-like-to-see-in-emacs--answers--chapters.vtt"
answers_filename = "answers.wav"
animation_filename = "animation.webm"
def get_length(filename):
    result = subprocess.run(["ffprobe", "-v", "error", "-show_entries",
                             "format=duration", "-of",
                             "default=noprint_wrappers=1:nokey=1", filename],
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT)
    return float(result.stdout)

def get_frames(filename):
    result = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0", "-count_packets",
                             "-show_entries", "stream=nb_read_packets", "-of",
                             "csv=p=0", filename],
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT)
    return float(result.stdout)

answers_length = get_length(answers_filename)
# override base_times
times = np.asarray([  1.515875,  13.50, 52.32125 ,  81.368625, 116.66625 , 146.023125,
       161.904875, 182.820875, 209.92125 , 226.51525 , 247.93875 ,
       260.971   , 270.87375 , 278.23325 , 303.166875, 327.44925 ,
       351.616375, 372.39525 , 394.246625, 409.36325 , 420.527875,
       431.854   , 440.608625, 473.86825 , 488.539   , 518.751875,
       544.1515  , 555.006   , 576.89225 , 598.157375, 627.795125,
       647.187125, 661.10875 , 695.87175 , 709.750125, 717.359875])
fps = 30.0
times = np.append(times, get_length(animation_filename))
anim_spans = list(zip(times[:-1], times[1:]))
chapters = webvtt.read(chapters_filename)
if chapters[0].start_in_seconds == 0:
    vtt_times = [[c.start_in_seconds, c.text] for c in chapters]
else:
    vtt_times = [[0, "Introduction"]] + [[c.start_in_seconds, c.text] for c in chapters] 
vtt_times = vtt_times + [[answers_length, "End"]]
# Add ending timestamps
vtt_times = [[x[0][0], x[1][0], x[0][1]] for x in zip(vtt_times[:-1], vtt_times[1:])]
test_rate = 1.0

i = 0
concat_list = ""
groups = list(zip(anim_spans, vtt_times))
import ffmpeg
animation = ffmpeg.input('animation.webm').video
audio = ffmpeg.input('rms.opus')

for_overlay = ffmpeg.input('color=color=black:size=1280x720:d=%f' % answers_length, f='lavfi')
params = {"b:v": "1k", "vcodec": "libvpx", "r": "30", "crf": "63"}
test_limit = 1
params = {"vcodec": "libvpx", "r": "30", "copyts": None, "b:v": "1M", "crf": 24}
test_limit = 0
anim_rate = 1
import math
cursor = 0
if test_limit > 0:
    groups = groups[0:test_limit]
clips = []

# cursor is the current time
for anim, vtt in groups:
    padding = vtt[1] - cursor - (anim[1] - anim[0]) / anim_rate
    if (padding < 0):
        print("Squeezing", math.floor((anim[1] - anim[0]) / (anim_rate * 1.0)), 'into', vtt[1] - cursor, padding)
        clips.append(animation.trim(start=anim[0], end=anim[1]).setpts('PTS-STARTPTS')) 
    elif padding == 0:
        clips.append(animation.trim(start=anim[0], end=anim[1]).setpts('PTS-STARTPTS'))
    else:
        print("%f to %f: Padding %f into %f - pad: %f" % (cursor, vtt[1], (anim[1] - anim[0]) / (anim_rate * 1.0), vtt[1] - cursor, padding))
        cursor = cursor + padding + (anim[1] - anim[0]) / anim_rate
        clips.append(animation.trim(start=anim[0], end=anim[1]).setpts('PTS-STARTPTS').filter('tpad', stop_mode="clone", stop_duration=padding))
    for_overlay = for_overlay.overlay(animation.trim(start=anim[0], end=anim[1]).setpts('PTS-STARTPTS+%f' % vtt[0]))
    clips.append(audio.filter('atrim', start=vtt[0], end=vtt[1]).filter('asetpts', 'PTS-STARTPTS'))
args = ffmpeg.concat(*clips, v=1, a=1).output('output.webm', **params).overwrite_output().compile()
print(' '.join(f'"{item}"' for item in args))

Anyway, it's here for future reference. =)

View org source for this post

Avoiding automatic data type conversion in Microsoft Excel and Pandas

| coding, python

Automatic conversion of data types is often handy, but sometimes it can mess things up. For example, when you import a CSV into Microsoft Excel, it will helpfully convert and display dates/times in your preferred format–and it will use your configured format when exporting back to CSV, which is not cool when your original file had YYYY-MM-DD HH:MM:SS and someone's computer decided to turn it into MM/DD/YY HH:MM. To avoid this conversion and import the columns as strings, you can change the file extension to .txt instead of .csv and then change each column type that you care about, which can be a lot of clicking. I had to change things back with a regular expression along the lines of:

import re
s = "12/9/21 11:23"
match = re.match('([0-9]+)/([0-9]+)/([0-9]+)( [0-9]+:[0-9]+)', s)
date = '20%s-%s-%s%s:00' % (match.group(3).zfill(2), match.group(1).zfill(2), match.group(2).zfill(2), match.group(4))
print(date)

The pandas library for Python also likes to do this kind of data type conversion for data types and for NaN values. In this particular situation, I wanted it to leave columns alone and leave the nan string in my input alone. Otherwise, to_csv would replace nan with the blank string, which could mess up a different script that used this data as input. This is the code to do it:

import pandas as pd
df = pd.read_csv('filename.csv', encoding='utf-8', dtype=str, na_filter=False)

I'm probably going to run into this again sometime, so I wanted to make sure I put my notes somewhere I can find them later.