Looking for temporal upsampling / motion interpolation software

Question

I'm looking for something (prereably FOSS software) which can take an animation with N images as input, and which will output an animation with M frames, where M is in the range 2N to 5N or so.

I believe the general technique is called "temporal upsampling" or possibly "inbetweening" (or "'tweening" for short). Note that it does need to make some effort to do motion tracking of things in the scene ("optical flow"); just fading ("dissolve") between keyframes isn't going to cut it. Googling "temporal upsampling" turns up any number of papers on the subject, but I've yet to discover any code/software (a gstreamer plugin would be perfect) I can just use to try the technique out.

Any suggestions ?

Additional information: "Motion interpolation" seems to be the more widely used name in the TV world for what I'm looking for. See this and the video here too. Interestingly, some cinephiles seem to hate what these technologies do to 24FPS film: e.g a comment "makes Gone With The Wind look like a soap opera shot on video".

Answer 1

The MVTools plugin for AVISynth performs frame interpolation. AviSynth (sourceforge) is a script based video editing tool. You define operations to perform in a script text file.

The script i had to playback Charlie Wilson's War at 120fps:

cyia.avs:

source = DirectShowSource("Cyia Batten - Charlie Wilson's War.avi")

# assume progressive NTSC Film 23.976 source

# Fast Search
#============
# we use explicit idx for more fast processing
#backward_vec = source.MVAnalyse(blksize=16, isb = true, chroma=false, pel=1, searchparam=1, idx=1)
#forward_vec = source.MVAnalyse(blksize=16, isb = false, chroma=false, pel=1, searchparam=1, idx=1)


# Slow Search
# ===========
# Use block overlap, halfpixel accuracy and Exhaustive search
backward_vec = source.MVAnalyse(overlap=4, isb = true, pel=2, search=3, idx=1)
forward_vec = source.MVAnalyse(overlap=4, isb = false, pel=2, search=3, idx=1)


#double framerate
#source.MVFlowFps(backward_vec, forward_vec, num=2*FramerateNumerator(source), den=FramerateDenominator(source), idx=1)

#triple framerate
#source.MVFlowFps(backward_vec, forward_vec, num=3*FramerateNumerator(source), den=FramerateDenominator(source), mask=0, idx=1)

#120fps
source.MVFlowFps(backward_vec, forward_vec, num=120, den=1, mask=0, idx=1)

Yes, it does destroy the movie quality when converted to anything higher than 24fps.

Answer 2

Short answer:

It takes a lot of manual work with current technology to make it work.

Longer answer:

The usual method of getting a "in-between" image (to so-called optical flow) is by mapping points in both pictures (source, destination) and calculating the transform between the two pictures. Of course, we do have edge-finding algorithms or predictive algorithms that tracks different objects (Blob, kernel-based, contour, etc, etc.) But a full automation of a transform of "real life" motion picture is at best experimental.

Answer 3

slowmoVideo

slowmoVideo is an OpenSource program that creates slow-motion videos from your footage.

But it does not simply make your videos play at 0.01× speed. You can smoothly slow down and speed up your footage, optionally with motion blur.

How does slow motion work? slowmoVideo tries to find out where pixels move in the video (this information is called Optical Flow), and then uses this information to calculate the additional frames.

Gstreamer slowmo plugin

The plugin uses code from slowmovideo, porting it from QT and its own internal decoding "framework" based on ffmpeg to gstreamer.