A- is really into Santa and Christmas because of the books she's read. Last year, she wanted to set up the GoPro to capture footage during Christmas Eve. I helped her set it up for a timelapse video. After she went to bed, we gradually positioned the presents. I extracted the frames from the video, removed the ones that caught us moving around, and then used Krita's new animation features to animate sparkles so that the presents magically appeared. She mentioned the sparkles a number of times during her deliberations about whether Santa exists or not.

This year, I want to see if I can use green-screen videos like this reversed-spin sparkle or this other sparkle video. I'm going to take a series of images, with each image adding one more gift. Then I'm going to make a mask in Krita with white covering the gift and a transparent background for the rest of the image. Then I'll use chroma-key to drop out the green screen of the sparkle video and mask it in so that the sparkles only happen within the boundaries of the gift that was added. I also want to fade one image into the other, and I want the sparkles to fade out as the gift appears.

ffmpeg -y -loop 1 -i test.jpg -loop 1 -i test2.jpg -filter_complex "[1:v]fade=t=in:d=4:alpha=1[fadein];[0:v][fadein]overlay[out]" -map "[out]" -r 1 -t 4 -shortest test.webm

ffmpeg -y -loop 1 -i test.jpg -loop 1 -i test2.jpg -filter_complex "[1:v][0:v]blend=all_expr='A*(if(gte(T,4),1,T/4))+B*(1-(if(gte(T,4),1,T/4)))" -t 4 -r 1 test.webm

ffmpeg -y -i test.webm -i balloons.mp4 -filter_complex "[1:v]chromakey=0x00ff00:0.1:0.2[ckout];[0:v][ckout]overlay[out]" -map "[out]" -shortest -r 1 overlaid.webm

ffmpeg -y -i test.jpg -i test2.jpg -i mask.png -filter_complex "[1:v][2:v]alphamerge[a];[0:v][a]overlay[out]" -map "[out]" masked.jpg

ffmpeg -y -loop 1 -i mask.png  -r 1 -t 4  mask.webm

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i mask.webm -filter_complex "[1:v][2:v]alphamerge[masked];[0:v][masked]overlay[out]" -map "[out]" -r 1 -t 4 alphamerged.webm

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i mask.webm -filter_complex "[1:v]chromakey=0x00ff00:0.1:0.2[ckout];[ckout][2:v]alphamerge[masked];[0:v][masked]overlay[out]" -map "[out]" -r 1 -t 4 masked.webm

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i mask.webm -filter_complex "[1:v]chromakey=0x00ff00:0.1:0.2,format=rgba,alphaextract[out]" -map "[out]" -r 1 -t 4
chroma-alpha.webm

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i mask.webm -filter_complex "[1:v]chromakey=0x00ff00:0.1:0.2,format=rgba,alphaextract[ckalpha];[ckalpha][2:v]blend=all_mode=and[out]" -map "[out]" -r 1 -t 4 masked-alpha.webm

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i masked-alpha.webm -filter_complex "[2:v]format=rgba[mask];[1:v][mask]alphamerge[masked];[0:v][masked]overlay[out]" -map "[out]" -r 1 -t 4 alphamerged.webm

ffmpeg -loglevel 32 -y -loop 1 -i test.jpg -t 4 -loop 1 -i test2.jpg -t 4 -i balloons.mp4 -loop 1 -i mask.png -t 4 -filter_complex "[1:v][0:v]blend=all_expr='A*(if(gte(T,4),1,T/4))+B*(1-(if(gte(T,4),1,T/4)))'[fade];[2:v]chromakey=0x00ff00:0.1:0.2,format=rgba,alphaextract[ckalpha];[ckalpha][3:v]blend=all_mode=and,format=rgba[maskedalpha];[2:v][maskedalpha]alphamerge[masked];[fade][masked]overlay[out]" -map "[out]" -r 5 -t 4 alphamerged.webm

ffmpeg -loglevel 32 -y -loop 1 -i test.jpg -t 4 -loop 1 -i test2.jpg -t 4 -i balloons.mp4 -loop 1 -i mask.png -t 4 -filter_complex "[1:v][0:v]blend=all_expr='A*(if(gte(T,4),1,T/4))+B*(1-(if(gte(T,4),1,T/4)))'[fade];[2:v]chromakey=0x00ff00:0.1:0.2,format=rgba,alphaextract[ckalpha];[ckalpha][3:v]blend=all_mode=and,format=rgba[maskedalpha];[2:v][maskedalpha]alphamerge[masked];[masked]fade=type=out:st=2:d=1:alpha=1[maskedfade];[fade][maskedfade]overlay[out]" -map "[out]" -r 10 -t 4 alphamerged.webm

for FILE in *.JPG; do convert $FILE -crop 1558x876+473+842 -resize 1280x720 cropped/$FILE; done

files=(*.JPG)
i=0
j=1
len="${#files[@]}"
while [ "$j" -lt $len ]; do
  compare -fuzz 15% cropped/${files[$i]} cropped/${files[$j]} -compose Src -highlight-color White -lowlight-color Black masks/${files[$j]}
  convert -morphology Open Disk -morphology Close Disk -blur 20x5 masks/${files[$j]} processed-masks/${files[$j]}
  i=$((i+1))
  j=$((j+1))
done

import ffmpeg
import glob
files = glob.glob("images/cropped/*.JPG")
files.sort()
fps = 15
crf = 32
out = ffmpeg.input(files[0], loop=1, r=fps)
duration = 3
for i in range(1, len(files)):
    out = ffmpeg.filter([out, ffmpeg.input(files[i], loop=1, r=fps).filter('fade', t='in', d=duration, st=i*duration, alpha=1)], 'overlay')
args = out.output('images.webm', t=len(files) * duration, r=fps, y=None, crf=crf).compile()
print(' '.join(f'"{item}"' for item in args))

import ffmpeg
import glob
files = glob.glob("images/processed-masks/*.JPG")
files.sort()
files = files[:-2]  # Omit the last two, where I'm just turning off the lights
fps = 15
crf = 32
out = ffmpeg.input('color=black:s=1280x720', f='lavfi', r=fps)
duration = 3
for i in range(0, len(files)):
    out = ffmpeg.filter([out, ffmpeg.input(files[i], loop=1, r=fps).filter('fade', t='in', d=1, st=(i + 1)*duration, alpha=1).filter('fade', t='out', st=(i + 2)*duration - 1)], 'overlay')
args = out.output('processed-masks.webm', t=len(files) * duration, r=fps, y=None, crf=crf).compile()
print(' '.join(f'"{item}"' for item in args))

ffmpeg -loglevel 32 -y -f lavfi -i color=black:s=1280x720 -i sparkles4.webm -ss 13 -filter_complex "[1:v]scale=700:392[sparkles];[0:v][sparkles]overlay=x=582:y=194,setpts=(PTS-STARTPTS)*1.05[out]" -map "[out]" -r 15 -t 53 -shortest sparkles-trimmed.webm
ffmpeg -y -stream_loop 2 -i sparkles-trimmed.webm -t 57 sparkles-looped.webm              

ffmpeg -loglevel 32 -y -i images.webm -i sparkles-looped.webm -i processed-masks.webm -filter_complex "[1:v]chromakey=0x0a9d06:0.1:0.2,format=rgba,alphaextract[ckalpha];[ckalpha][2:v]blend=all_mode=and,format=rgba[maskedalpha];[1:v][maskedalpha]alphamerge[masked];[masked]fade=t=out:st=57:d=1:alpha=1[maskedfaded];[0:v][maskedfaded]overlay[combined];[combined]tpad=start_mode=clone:start_duration=4:stop_mode=clone:stop_duration=4[out]" -map "[out]" -r 15 -crf 32 output.webm

I wanted to convert our VTT files to TTML files so that we might be able to use them for training lachesis for transcript segmentation. I downloaded the VTT files from EmacsConf 2021 to a directory and copied the edited captions from the EmacsConf 2022 backstage area (using head -1 ${FILE} | grep -q "captioned" to distinguish them from the automatic ones). I installed the ttconv python package. Then I used the following command to convert the TTML files:

for FILE in *.vtt; do
    BASE=$(basename -s .vtt "$FILE");
    ffmpeg -y -i $FILE $BASE.srt; tt convert -i $BASE.srt -o $BASE.ttml
done           

I haven't gotten around to installing whanever I need in order to get lachesis to work under Python 2.7, since it hasn't been updated for Python 3. It'll probably be a low-priority project anyway, as EmacsConf is fast approaching. Anyway, I thought I'd stash this in my blog somewhere in case I need to make TTML files again!

It turns out that using -crf 56 compressed the EmacsConf a little too aggressively, losing too much information in the video. We wanted to reencode everything, maybe going back to the default value of -crf 32. My laptop would have taken a long time to do all of those videos. Fortunately, one of the other volunteers shared a VM on a machine with 12 cores, and I had access to a few other systems. It was a good opportunity to learn how to use GNU Parallel to send jobs to different machines and retrieve the results.

First, I updated the compression script, compress-video-low.sh:

Q=$1
WIDTH=1280
HEIGHT=720
AUDIO_RATE=48000
VIDEO_FILTER="scale=w=${WIDTH}:h=${HEIGHT}:force_original_aspect_ratio=1,pad=${WIDTH}:${HEIGHT}:(ow-iw)/2:(oh-ih)/2,fps=25,colorspace=all=bt709:iall=bt601-6-625:fast=1"
FILE=$2
SUFFIX=$Q
shift
shift
ffmpeg -y -i "$FILE"  -pixel_format yuv420p -vf $VIDEO_FILTER -colorspace 1 -color_primaries 1 -color_trc 1 -c:v libvpx-vp9 -b:v 0 -crf $Q -aq-mode 2 -tile-columns 0 -tile-rows 0 -frame-parallel 0 -cpu-used 8 -auto-alt-ref 1 -lag-in-frames 25 -g 240 -pass 1 -f webm -an -threads 8 /dev/null &&
if [[ $FILE =~ "webm" ]]; then
    ffmpeg -y -i "$FILE" $*  -pixel_format yuv420p -vf $VIDEO_FILTER -colorspace 1 -color_primaries 1 -color_trc 1 -c:v libvpx-vp9 -b:v 0 -crf $Q -tile-columns 2 -tile-rows 2 -frame-parallel 0 -cpu-used -5 -auto-alt-ref 1 -lag-in-frames 25 -pass 2 -g 240 -ac 2 -threads 8 -c:a copy "${FILE%.*}--compressed$SUFFIX.webm"
else
    ffmpeg -y -i "$FILE" $*  -pixel_format yuv420p -vf $VIDEO_FILTER -colorspace 1 -color_primaries 1 -color_trc 1 -c:v libvpx-vp9 -b:v 0 -crf $Q -tile-columns 2 -tile-rows 2 -frame-parallel 0 -cpu-used -5 -auto-alt-ref 1 -lag-in-frames 25 -pass 2 -g 240 -ac 2 -threads 8 -c:a libvorbis "${FILE%.*}--compressed$SUFFIX.webm"
fi

I made an originals.txt file with all the original filenames. It looked like this:

emacsconf-2020-frownies--the-true-frownies-are-the-friends-we-made-along-the-way-an-anecdote-of-emacs-s-malleability--case-duckworth.mkv
emacsconf-2021-montessori--emacs-and-montessori-philosophy--grant-shangreaux.webm
emacsconf-2021-pattern--emacs-as-design-pattern-learning--greta-goetz.mp4
...

I set up a ~/.parallel/emacsconf profile with something like this so that I could use three computers and my laptop, sending one job each and displaying progress:

--sshlogin computer1 --sshlogin computer2 --sshlogin computer3 --sshlogin : -j 1 --progress --verbose --joblog parallel.log

I already had SSH key-based authentication set up so that I could connect to the three remote computers.

Then I spread the jobs over four computers with the following command:

cat originals.txt | parallel -J emacsconf \
                             --transferfile {} \
                             --return '{=$_ =~ s/\..*?$/--compressed32.webm/=}' \
                             --cleanup \
                             --basefile compress-video-low.sh \
                             bash compress-video-low.sh 32 {}

It copied each file over to the computer it was assigned to, processed the file, and then copied the file back.

It was also helpful to occasionally do echo 'killall -9 ffmpeg' | parallel -J emacsconf -j 1 --onall if I cancelled a run.

It still took a long time, but less than it would have if any one computer had to crunch through everything on its own.

This was much better than my previous way of doing things, which involved copying the files over, running ffmpeg commands, copying the files back, and getting somewhat confused about which directory I was in and which file I assigned where and what to do about incompletely-encoded files.

I sometimes ran into problems with incompletely-encoded files because I'd cancelled the FFmpeg process. Even though ffprobe said the files were long, they were missing a large chunk of video at the end. I added a compile-media-verify-video-frames function to compile-media.el so that I could get the last few seconds of frames, compare them against the duration, and report an error if there was a big gap.

Then I changed emacsconf-publish.el to use the new filenames, and I regenerated all the pages. For EmacsConf 2020, I used some Emacs Lisp to update the files. I'm not particularly fond of wrangling video files (lots of waiting, high chance of error), but I'm glad I got the computers to work together.

I spent the day cutting up the rest of the videos from the Emacs Conference 2015 Twitch.tv stream into individual talks. I’d already cut the set of talks before lunch, but there were quite a few more after. As it turned out, keeping the video data in .ts format instead of converting it to .mp4 is actually better for Youtube processing.

Since Camtasia Studio and Movie Maker were both having problems with the large videos, I used VLC to play the video and find the timestamps at which I needed to cut the segments. I made an Org Mode table with the start and end times, and then I used the ;T flag in a table function to get the duration. A little bit of Emacs Lisp code later, and I had my ffmpeg commands. Here’s the source from my Org file:

#+NAME: emacsconf-c.ts
| Notes                                            |      Start |        End | Duration |
|--------------------------------------------------+------------+------------+----------|
| Emacs configuration                              | 4:02:25.37 | 4:27:09.30 | 00:24:44 |
| Hearing from Emacs Beginners                     |    4:27:27 |    5:01:00 | 00:33:33 |
| Lightning talk: Emacs Club                       | 5:03:19.30 | 5:19:37.83 | 00:16:18 |
| Starting an Emacs Meetup - Harry Schwartz part 1 | 5:31:52.03 |    6:01:20 | 00:29:28 |
#+TBLFM: $4=$3-$2;T

#+NAME: emacsconf-a.ts
| Notes                                                    |   Start |     End | Duration |
|----------------------------------------------------------+---------+---------+----------|
| Starting an Emacs Meetup - Harry Schwartz part 2         |  0:0:00 | 0:20:04 | 00:20:04 |
| Literate Devops - Howard Abrams                          | 1:28:20 | 2:08:15 | 00:39:55 |
| Lightning talk: Wanderlust and other mail clients        | 2:15:04 | 2:26:55 | 00:11:51 |
| Making Emacs a Better Tool for Scholars - Erik Hetzner   | 2:27:00 | 2:57:38 | 00:30:38 |
| Wrapping up and going forward                            | 2:58:09 | 2:59:44 | 00:01:35 |
| Lightning talk: Collaborative coding with tmux and tmate | 3:00:20 | 3:05:53 | 00:05:33 |
| Lightning talk: Cask and Pellet                          | 3:05:56 | 3:09:04 | 00:03:08 |
| Lightning talk: File sharing with Git and save hooks     | 3:09:34 | 3:17:50 | 00:08:16 |
| Lightning talk: Calc                                     | 3:18:42 | 3:33:20 | 00:14:38 |
| Lightning talk: Magit                                    | 3:35:15 | 3:49:42 | 00:14:27 |
| Lightning talk: gist.el                                  | 3:53:50 | 4:01:58 | 00:08:08 |
| Lightning talk: Go                                       | 4:02:45 | 4:16:37 | 00:13:52 |
| Question: Emacs Lisp backtraces                          | 4:16:50 | 4:20:09 | 00:03:19 |
#+TBLFM: $4=$3-$2;T

#+begin_src emacs-lisp :var data=emacsconf-a.ts :var data2=emacsconf-c.ts :colnames t :results output
(let ((format-str "ffmpeg -i %s -ss %s -t %s -c:v copy -c:a copy \"EmacsConf 2015 - %s.ts\"\n"))
  (mapc (lambda (file)
    (mapc (lambda (row) 
      (princ (format format-str (car file) (elt row 1) (elt row 3) (my/convert-sketch-title-to-filename (elt row 0))))) 
     (cdr file)))
    `(("emacsconf-c.ts" . ,data2)
      ("emacsconf-a.ts" . ,data))))
#+end_src

and the output:

ffmpeg -i emacsconf-c.ts -ss 4:02:25.37 -t 00:24:44 -c:v copy -c:a copy "EmacsConf 2015 - Emacs configuration.ts"
ffmpeg -i emacsconf-c.ts -ss 4:27:27 -t 00:33:33 -c:v copy -c:a copy "EmacsConf 2015 - Hearing from Emacs Beginners.ts"
ffmpeg -i emacsconf-c.ts -ss 5:03:19.30 -t 00:16:18 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - Emacs Club.ts"
ffmpeg -i emacsconf-c.ts -ss 5:31:52.03 -t 00:29:28 -c:v copy -c:a copy "EmacsConf 2015 - Starting an Emacs Meetup - Harry Schwartz part 1.ts"
ffmpeg -i emacsconf-a.ts -ss 0:0:00 -t 00:20:04 -c:v copy -c:a copy "EmacsConf 2015 - Starting an Emacs Meetup - Harry Schwartz part 2.ts"
ffmpeg -i emacsconf-a.ts -ss 1:28:20 -t 00:39:55 -c:v copy -c:a copy "EmacsConf 2015 - Literate Devops - Howard Abrams.ts"
ffmpeg -i emacsconf-a.ts -ss 2:15:04 -t 00:11:51 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - Wanderlust and other mail clients.ts"
ffmpeg -i emacsconf-a.ts -ss 2:27:00 -t 00:30:38 -c:v copy -c:a copy "EmacsConf 2015 - Making Emacs a Better Tool for Scholars - Erik Hetzner.ts"
ffmpeg -i emacsconf-a.ts -ss 2:58:09 -t 00:01:35 -c:v copy -c:a copy "EmacsConf 2015 - Wrapping up and going forward.ts"
ffmpeg -i emacsconf-a.ts -ss 3:00:20 -t 00:05:33 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - Collaborative coding with tmux and tmate.ts"
ffmpeg -i emacsconf-a.ts -ss 3:05:56 -t 00:03:08 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - Cask and Pellet.ts"
ffmpeg -i emacsconf-a.ts -ss 3:09:34 -t 00:08:16 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - File sharing with Git and save hooks.ts"
ffmpeg -i emacsconf-a.ts -ss 3:18:42 -t 00:14:38 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - Calc.ts"
ffmpeg -i emacsconf-a.ts -ss 3:35:15 -t 00:14:27 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - Magit.ts"
ffmpeg -i emacsconf-a.ts -ss 3:53:50 -t 00:08:08 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - gist.el.ts"
ffmpeg -i emacsconf-a.ts -ss 4:02:45 -t 00:13:52 -c:v copy -c:a copy "EmacsConf 2015 - Lightning talk - Go.ts"
ffmpeg -i emacsconf-a.ts -ss 4:16:50 -t 00:03:19 -c:v copy -c:a copy "EmacsConf 2015 - Question - Emacs Lisp backtraces.ts"

You can watch the Emacs Conference 2015 playlist on YouTube. At some point, each talk will probably have individual wiki pages and IRC logs at http://emacsconf2015.org/ . =) Enjoy!

Related tech notes: Emacs Conf video tech notes: jit.si, twitch.tv, livestreamer, ffmpeg

Last week’s Emacs Conf was fantastic. There were lots of people at the in-person event in San Francisco, and people could also watch the stream through twitch.tv and ask questions through IRC. There were remote speakers and in-person speakers, and that mix even worked for the impromptu lightning talks sprinkled throughout the day.

This is how the tech worked:

• Before the conference started, the organizers set up a laptop for streaming on twitch.tv/emacsconf. This was hooked up to the main display (a large television with speakers). They also configured the account to record and archive videos. In the free account, recorded videos are available for 14 days.
• Remote speakers were brought in using the Jitsi open source video conferencing system, using the public servers at meet.jit.si. This was on the same computer that did the twitch.tv streaming, so people watching the stream could see whatever was shared through Jitsi. Organizers read out questions from the in-person audience and from the IRC channel. The audio from Jitsi wasn’t directly available through twitch.tv, though. Instead, the audio came in as a recording from the laptop’s microphone.
• Local speakers either used the streaming laptop to go to a specific webpage they wanted to talk about, or joined the Jitsi web conference using Google Chrome or Chromium so that they could share their screen. The organizers muted the second Jitsi client to avoid audio feedback loops.

That worked out really well. There were more than a hundred remote viewers. As one of them, I can definitely rate the experience as surprisingly smooth.

All that’s left now is to figure out how to make a more lasting archive of the Emacs Conf videos. As it turns out, twitch.tv or online tools don’t make it easy to download stream recordings that are longer than three hours. Fortunately, livestreamer can handle the job. Here’s what I did to download the timestream data from one of the recordings of EmacsConf:

livestreamer -o emacsconf-1.ts --hls-segment-threads 4 http://www.twitch.tv/emacsconf/v/13421774 best
ffmpeg -i emacsconf-1.ts -acodec copy -absf aac_adtstoasc -vcodec copy emacsconf-1.mp4

I normally use Camtasia Studio to edit videos, but for some reason, it kept flaking out on me today. After the umpteenth crash, I decided to keep things simple by using ffmpeg to extract the relevant part of the video. To extract a segment, you can use -ss to specify the start time and t to specify the duration. Here’s a sample command:

ffmpeg -i emacsconf-1.mp4 -ss 1:18:06.11 -t 0:03:32.29 -c:v copy -c:a copy emacsconf-engine-mode.mp4

Your version of ffmpeg might have a -to option, which would let you specify the end time instead of using -t to specify duration.

I’m coordinating with the other organizers to see if there’s a better way to process the videos, so that’s why we haven’t released them publicly yet. (Soon!) It would be nice to improve the audio, especially for some of the talks, and maybe it would be good to add overlays or zoom in as well. The on-site organizers captured backup videos and screen recordings, too, so we might want to edit some of those clips into the streamed recording. One of the organizers has access to better video editing tools, so we’ll try that out.

Anyway, those were the commands that helped me get started with command-line conversion and editing of Twitch.tv recorded videos. Hope they come in handy for other people too.

For more info about EmacsConf 2015, check out http://emacsconf2015.org/. There’ll probably be an announcement there once the videos are up. =)

Hat tip to Reddit and superuser.com for tips.

We have three cats. One of our cats occasionally poops outside the litter box. We had our suspicions, but we couldn't pin down who or why. Territorial issues? Finickiness about box hygiene? Sickness? Fear or surprise? What could we do to reduce the frequency of incidents?

We decided that a litter box webcam was an excellent first project for the Raspberry Pi computer that W- just bought. The Pi is a tiny, quiet, inexpensive Linux server. My webcam worked without hassles, and Motion was easy to set up for motion detection. We set it up to capture videos when the computer detected motion. I watched the videos and encoded the data, tracking which cat and which litter box. I figured that exploring this would be a good excuse to work with the Pi and learn a little more about computer vision.

2014-03-24 Litter Box Cam with Raspberry Pi #raspberry #cats

You might think that watching litter box videos would be boring and somewhat icky. It was surprisingly informative. I had no idea that Luke sniffed so many litter boxes before settling on one. Leia usually checked out one or two boxes before doing her thing, but if all the other boxes were used (even if one of them was used only by her), she sniffed everything and then circled around in indecision before finally pooping in the middle of the basement floor. (Watching her try everything made me feel somewhat better.) The two cats cover, but Neko never does. (Territorial dominance marker by the smallest cat?)

We collected a week of baseline data, which showed that box 1 was twice as popular as box 4 and 5. W- hypothesized that it was because box 4 and box 5 were near the furnace, and the strange noises from the furnace might startle the cats occasionally. Leia pooped outside the box twice, both times sniffing all the boxes before going in the middle.

We took to calling Leia our little data-generator.

2014-03-29 Litterbox analysis #quantified

Since the cats often left a little bit of extra food in their bowls and the vet had suggested they needed less food or more exercise, we decided to try reducing the amount of food we gave them. That change seems to be going well.

We also moved box 5 closer to box 1. That led to box 5 being much more popular than it used to be, which was a pleasant surprise. If Leia likes box 5 a lot more now that it's away from the furnace, maybe it'll be easier for her to find a clean box to poop in.

Preliminary cat litter box results

We set the camera up to capture 2 frames per second in order to save space. Watching it in real-time eventually lost its novelty, so I looked up how to speed up the AVIs.

for FILE in video-*.avi; do
  if [ ! -f "fast-$FILE" ]; then
    ffmpeg -i $FILE -vf "setpts=0.10*PTS" -r 30 fast-$FILE
  fi
done

I also started looking into how to use SimpleCV for computer vision and image processing. I had a hard time getting SimpleCV set up in my Ubuntu virtual machine, but the Windows version worked fine after a lengthy install process on my computer. After much learning, I figured out how to identify changed areas, get the largest share over a certain area threshold, find the centroid of that shape, and plot it back on the image. The real challenge is figuring out some kind of visual output that makes sense to me when I look at it. The image below is a step in the right direction, but it's still not quite what I need.

The Raspberry Pi camera module arrived, so we swapped that in and eventually got everything working again after some SD/power-related grumbling.

It would be great if I could get Python to automatically figure out which cat is in the video, distinguishing between multiple cats and flagging it for manual review if the motion detection got confused. Even better if it can track the path that the cats take!

On the other hand, the speeded-up AVIs are now fast enough that the bottleneck isn't waiting for the video to play, it's me typing in the description of the path (since I track not only the litter box they use, but any other litter boxes they check along the way). Maybe this is fine.

While watching me encode data, W- said, “Isn't this something you can have your assistants do?” It's data entry, sure, but I feel embarrassed about assigning people to watch our cats poop. <laugh> Besides, I'm learning a lot from the encoding process. We'll probably treat it as a time-limited experiment.

Pretty cool! =) Next steps: Collect more data, try more experimental changes, learn more about image processing…

Anticipated questions/responses:

• That's so geeky. You're weird. Yup.
• You have too much time on your hands. I like spending my time learning things, and getting better at computer vision and data analysis will come in handy. =) Better than watching TV or reading forgettable things. (See also: Too much time on her hands)
• Just get a self-cleaning litter box. We got a Litter Robot because that was the best-reviewed of the self-cleaning litter boxes, but only Neko likes using it. We're not into fancy litter boxes that require cartridges or special litter. It's hard to tell if Leia will take to a new automated litter box, and returns/refunds for used litter boxes would be a bit weird. We can probably figure out something that works with our current setup, or maybe with an additional regular litter box. We still need to clean daily anyway, so it makes sense to try low-cost approaches first.
• How about tracking both input and output? Too much work at the moment, and not enough interest in the data. (Weigh the food bowls? Weigh the boxes? Distinguish between cats?)
• What about rigging up an automated water sprayer / Nerf gun to fire when cats poo outside the litter box? That will just make them poop outside the furnace room. It's much easier to clean the furnace room than the carpet, so if they're going to poop outside the litter box, that's as good a place as any. We don't want them to get any negative associations.

I use Sametime Unyte for web conferences at work. Unyte allows you to record your teleconferences (slides and audio), and you can download a ZIP containing Flash video after your session.

I usually extract the audio track and publish that as a separate MP3 so that people can listen to it. I can also have the audio file transcribed. The audio track from Sametime Unyte is of lower quality than my voice recorder, but it’s a good backup and it captures both sides of the phone conversation.

Here is one way to extract the audio using Linux:

for FILE in *.swf; do ffmpeg -i $FILE -ab 64k $FILE.wav; done

Then you can concatenate all the WAV files:

sox *.wav all.wav

Then you can use Audacity to edit the resulting file.