I didn't know how to do any of that yet with ffmpeg, so here's how I started figuring things out. First, I wanted to see how to fade test.jpg into test2.jpg over 4 seconds.

ffmpeg -y -loop 1 -i test.jpg -loop 1 -i test2.jpg -filter_complex "[1:v]fade=t=in:d=4:alpha=1[fadein];[0:v][fadein]overlay[out]" -map "[out]" -r 1 -t 4 -shortest test.webm

Here's another way using the blend filter:

ffmpeg -y -loop 1 -i test.jpg -loop 1 -i test2.jpg -filter_complex "[1:v][0:v]blend=all_expr='A*(if(gte(T,4),1,T/4))+B*(1-(if(gte(T,4),1,T/4)))" -t 4 -r 1 test.webm

Then I looked into chromakeying in the other video. I used balloons instead of sparkles just in case she happened to look at my screen.

ffmpeg -y -i test.webm -i balloons.mp4 -filter_complex "[1:v]chromakey=0x00ff00:0.1:0.2[ckout];[0:v][ckout]overlay[out]" -map "[out]" -shortest -r 1 overlaid.webm

I experimented with the alphamerge filter.

ffmpeg -y -i test.jpg -i test2.jpg -i mask.png -filter_complex "[1:v][2:v]alphamerge[a];[0:v][a]overlay[out]" -map "[out]" masked.jpg

Okay! That overlaid test.jpg with a masked part of test2.jpg. How about alphamerging in a video? First, I need a mask video…

ffmpeg -y -loop 1 -i mask.png  -r 1 -t 4  mask.webm

Then I can combine that:

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i mask.webm -filter_complex "[1:v][2:v]alphamerge[masked];[0:v][masked]overlay[out]" -map "[out]" -r 1 -t 4 alphamerged.webm

Great, let's figure out how to combine chroma-key and alphamerge video. The naive approach doesn't work, probably because they're both messing with the alpha layer.

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i mask.webm -filter_complex "[1:v]chromakey=0x00ff00:0.1:0.2[ckout];[ckout][2:v]alphamerge[masked];[0:v][masked]overlay[out]" -map "[out]" -r 1 -t 4 masked.webm

So I probably need to blend the chromakey and the mask. Let's see if I can extract the chromakey alpha.

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i mask.webm -filter_complex "[1:v]chromakey=0x00ff00:0.1:0.2,format=rgba,alphaextract[out]" -map "[out]" -r 1 -t 4
chroma-alpha.webm

Now let's blend it with the mask.webm.

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i mask.webm -filter_complex "[1:v]chromakey=0x00ff00:0.1:0.2,format=rgba,alphaextract[ckalpha];[ckalpha][2:v]blend=all_mode=and[out]" -map "[out]" -r 1 -t 4 masked-alpha.webm

Then let's use it as the alpha:

ffmpeg -loglevel 32 -y -i test.webm -i balloons.mp4 -i masked-alpha.webm -filter_complex "[2:v]format=rgba[mask];[1:v][mask]alphamerge[masked];[0:v][masked]overlay[out]" -map "[out]" -r 1 -t 4 alphamerged.webm

Okay, that worked! Now how do I combine everything into one command? Hmm…

ffmpeg -loglevel 32 -y -loop 1 -i test.jpg -t 4 -loop 1 -i test2.jpg -t 4 -i balloons.mp4 -loop 1 -i mask.png -t 4 -filter_complex "[1:v][0:v]blend=all_expr='A*(if(gte(T,4),1,T/4))+B*(1-(if(gte(T,4),1,T/4)))'[fade];[2:v]chromakey=0x00ff00:0.1:0.2,format=rgba,alphaextract[ckalpha];[ckalpha][3:v]blend=all_mode=and,format=rgba[maskedalpha];[2:v][maskedalpha]alphamerge[masked];[fade][masked]overlay[out]" -map "[out]" -r 5 -t 4 alphamerged.webm

Then I wanted to fade the masked video out by the end.

ffmpeg -loglevel 32 -y -loop 1 -i test.jpg -t 4 -loop 1 -i test2.jpg -t 4 -i balloons.mp4 -loop 1 -i mask.png -t 4 -filter_complex "[1:v][0:v]blend=all_expr='A*(if(gte(T,4),1,T/4))+B*(1-(if(gte(T,4),1,T/4)))'[fade];[2:v]chromakey=0x00ff00:0.1:0.2,format=rgba,alphaextract[ckalpha];[ckalpha][3:v]blend=all_mode=and,format=rgba[maskedalpha];[2:v][maskedalpha]alphamerge[masked];[masked]fade=type=out:st=2:d=1:alpha=1[maskedfade];[fade][maskedfade]overlay[out]" -map "[out]" -r 10 -t 4 alphamerged.webm

When A- finally went to bed, we arranged the presents, using the GoPro to take a picture at each step of the way. I cropped and resized the images, using Krita to figure out the cropping rectangle and offset.

for FILE in *.JPG; do convert $FILE -crop 1558x876+473+842 -resize 1280x720 cropped/$FILE; done

I used ImageMagick to calculate the masks automatically.

files=(*.JPG)
i=0
j=1
len="${#files[@]}"
while [ "$j" -lt $len ]; do
  compare -fuzz 15% cropped/${files[$i]} cropped/${files[$j]} -compose Src -highlight-color White -lowlight-color Black masks/${files[$j]}
  convert -morphology Open Disk -morphology Close Disk -blur 20x5 masks/${files[$j]} processed-masks/${files[$j]}
  i=$((i+1))
  j=$((j+1))
done

Then I faded the images together to make a video.

import ffmpeg
import glob
files = glob.glob("images/cropped/*.JPG")
files.sort()
fps = 15
crf = 32
out = ffmpeg.input(files[0], loop=1, r=fps)
duration = 3
for i in range(1, len(files)):
    out = ffmpeg.filter([out, ffmpeg.input(files[i], loop=1, r=fps).filter('fade', t='in', d=duration, st=i*duration, alpha=1)], 'overlay')
args = out.output('images.webm', t=len(files) * duration, r=fps, y=None, crf=crf).compile()
print(' '.join(f'"{item}"' for item in args))

"ffmpeg" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2317.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2318.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2319.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2320.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2321.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2322.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2323.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2324.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2325.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2326.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2327.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2328.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2329.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2330.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2331.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2332.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2333.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2334.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2335.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2336.JPG" "-loop" "1" "-r" "15" "-i" "images/cropped/GOPR2337.JPG" "-filter_complex" "[1]fade=alpha=1:d=3:st=3:t=in[s0];[0][s0]overlay[s1];[2]fade=alpha=1:d=3:st=6:t=in[s2];[s1][s2]overlay[s3];[3]fade=alpha=1:d=3:st=9:t=in[s4];[s3][s4]overlay[s5];[4]fade=alpha=1:d=3:st=12:t=in[s6];[s5][s6]overlay[s7];[5]fade=alpha=1:d=3:st=15:t=in[s8];[s7][s8]overlay[s9];[6]fade=alpha=1:d=3:st=18:t=in[s10];[s9][s10]overlay[s11];[7]fade=alpha=1:d=3:st=21:t=in[s12];[s11][s12]overlay[s13];[8]fade=alpha=1:d=3:st=24:t=in[s14];[s13][s14]overlay[s15];[9]fade=alpha=1:d=3:st=27:t=in[s16];[s15][s16]overlay[s17];[10]fade=alpha=1:d=3:st=30:t=in[s18];[s17][s18]overlay[s19];[11]fade=alpha=1:d=3:st=33:t=in[s20];[s19][s20]overlay[s21];[12]fade=alpha=1:d=3:st=36:t=in[s22];[s21][s22]overlay[s23];[13]fade=alpha=1:d=3:st=39:t=in[s24];[s23][s24]overlay[s25];[14]fade=alpha=1:d=3:st=42:t=in[s26];[s25][s26]overlay[s27];[15]fade=alpha=1:d=3:st=45:t=in[s28];[s27][s28]overlay[s29];[16]fade=alpha=1:d=3:st=48:t=in[s30];[s29][s30]overlay[s31];[17]fade=alpha=1:d=3:st=51:t=in[s32];[s31][s32]overlay[s33];[18]fade=alpha=1:d=3:st=54:t=in[s34];[s33][s34]overlay[s35];[19]fade=alpha=1:d=3:st=57:t=in[s36];[s35][s36]overlay[s37];[20]fade=alpha=1:d=3:st=60:t=in[s38];[s37][s38]overlay[s39]" "-map" "[s39]" "-crf" "32" "-r" "15" "-t" "63" "-y" "images.webm"

Next, I faded the masks together. These ones faded in and out so that only one mask was active at a time.

import ffmpeg
import glob
files = glob.glob("images/processed-masks/*.JPG")
files.sort()
files = files[:-2]  # Omit the last two, where I'm just turning off the lights
fps = 15
crf = 32
out = ffmpeg.input('color=black:s=1280x720', f='lavfi', r=fps)
duration = 3
for i in range(0, len(files)):
    out = ffmpeg.filter([out, ffmpeg.input(files[i], loop=1, r=fps).filter('fade', t='in', d=1, st=(i + 1)*duration, alpha=1).filter('fade', t='out', st=(i + 2)*duration - 1)], 'overlay')
args = out.output('processed-masks.webm', t=len(files) * duration, r=fps, y=None, crf=crf).compile()
print(' '.join(f'"{item}"' for item in args))

"ffmpeg" "-f" "lavfi" "-r" "15" "-i" "color=s=1280x720" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2318.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2319.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2320.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2321.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2322.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2323.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2324.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2325.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2326.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2327.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2328.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2329.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2330.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2331.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2332.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2333.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2334.JPG" "-loop" "1" "-r" "15" "-i" "images/processed-masks/GOPR2335.JPG" "-filter_complex" "[1]fade=alpha=1:d=1:st=3:t=in[s0];[s0]fade=st=5:t=out[s1];[0][s1]overlay[s2];[2]fade=alpha=1:d=1:st=6:t=in[s3];[s3]fade=st=8:t=out[s4];[s2][s4]overlay[s5];[3]fade=alpha=1:d=1:st=9:t=in[s6];[s6]fade=st=11:t=out[s7];[s5][s7]overlay[s8];[4]fade=alpha=1:d=1:st=12:t=in[s9];[s9]fade=st=14:t=out[s10];[s8][s10]overlay[s11];[5]fade=alpha=1:d=1:st=15:t=in[s12];[s12]fade=st=17:t=out[s13];[s11][s13]overlay[s14];[6]fade=alpha=1:d=1:st=18:t=in[s15];[s15]fade=st=20:t=out[s16];[s14][s16]overlay[s17];[7]fade=alpha=1:d=1:st=21:t=in[s18];[s18]fade=st=23:t=out[s19];[s17][s19]overlay[s20];[8]fade=alpha=1:d=1:st=24:t=in[s21];[s21]fade=st=26:t=out[s22];[s20][s22]overlay[s23];[9]fade=alpha=1:d=1:st=27:t=in[s24];[s24]fade=st=29:t=out[s25];[s23][s25]overlay[s26];[10]fade=alpha=1:d=1:st=30:t=in[s27];[s27]fade=st=32:t=out[s28];[s26][s28]overlay[s29];[11]fade=alpha=1:d=1:st=33:t=in[s30];[s30]fade=st=35:t=out[s31];[s29][s31]overlay[s32];[12]fade=alpha=1:d=1:st=36:t=in[s33];[s33]fade=st=38:t=out[s34];[s32][s34]overlay[s35];[13]fade=alpha=1:d=1:st=39:t=in[s36];[s36]fade=st=41:t=out[s37];[s35][s37]overlay[s38];[14]fade=alpha=1:d=1:st=42:t=in[s39];[s39]fade=st=44:t=out[s40];[s38][s40]overlay[s41];[15]fade=alpha=1:d=1:st=45:t=in[s42];[s42]fade=st=47:t=out[s43];[s41][s43]overlay[s44];[16]fade=alpha=1:d=1:st=48:t=in[s45];[s45]fade=st=50:t=out[s46];[s44][s46]overlay[s47];[17]fade=alpha=1:d=1:st=51:t=in[s48];[s48]fade=st=53:t=out[s49];[s47][s49]overlay[s50];[18]fade=alpha=1:d=1:st=54:t=in[s51];[s51]fade=st=56:t=out[s52];[s50][s52]overlay[s53]" "-map" "[s53]" "-crf" "32" "-r" "15" "-t" "54" "-y" "processed-masks.webm"

I ended up using this particle glitter video because the gifts were small, so I wanted a video that was dense with sparkly things. I also wanted the sparkles to be more concentrated on the area where the gifts were, so I resized it and positioned it.

ffmpeg -loglevel 32 -y -f lavfi -i color=black:s=1280x720 -i sparkles4.webm -ss 13 -filter_complex "[1:v]scale=700:392[sparkles];[0:v][sparkles]overlay=x=582:y=194,setpts=(PTS-STARTPTS)*1.05[out]" -map "[out]" -r 15 -t 53 -shortest sparkles-trimmed.webm
ffmpeg -y -stream_loop 2 -i sparkles-trimmed.webm -t 57 sparkles-looped.webm              

Lastly, I combined the videos with the sparkles.

ffmpeg -loglevel 32 -y -i images.webm -i sparkles-looped.webm -i processed-masks.webm -filter_complex "[1:v]chromakey=0x0a9d06:0.1:0.2,format=rgba,alphaextract[ckalpha];[ckalpha][2:v]blend=all_mode=and,format=rgba[maskedalpha];[1:v][maskedalpha]alphamerge[masked];[masked]fade=t=out:st=57:d=1:alpha=1[maskedfaded];[0:v][maskedfaded]overlay[combined];[combined]tpad=start_mode=clone:start_duration=4:stop_mode=clone:stop_duration=4[out]" -map "[out]" -r 15 -crf 32 output.webm

After many iterations and a very late night, I got (roughly) the video I wanted, which I'm not posting here because of reasons. But it worked, yay! Now I don't have to manually place stars frame-by-frame in Krita, and I can just have all that magic happen semi-automatically.