Motion Flow on ImageMagick

[xyrph]

Can IM be used to interpolate the motion between two frames of a video that have been saved as images?

In other words, given images 1.0 and 2.0, I am looking for a way to have IM create image 1.5, placing objects that have moved exactly halfway in their movement.

I film basketball games at 60 frames per second at 1080p and am looking for a way to produce super slow motion at key moments in the footage (highlights, etc.). I can use FFmpeg to run the video at 30 fps, which slows down the action of the players exactly by half. But slowing it down any more than this causes strange motion artifacts, since FFmpeg simply copies frames to fill in the gaps rather than interpolating the motion and creating accurate intermediary virtual frames.

I had hoped that FFmpeg would offer a filter to do the Motion Flow, but alas I have found that it does not. I am not concerned with speed for this project. I am looking for quality and have no problem breaking the video segment into individual frames if IM can be used to create the accurate-motion intermediary frames.

I am using Windows and can program batch files. I am using ImageMagick-6.8.6-Q16, but am open to using any version that can help me achieve this goal.

[snibgo]

It is possible but not trivial.

Suppose we have a constant background (the camera is on a tripod) and a moving foreground object (a basketball player), and we have at least one frame with no object (a "clean" frame).

Then, from two frames that have the object at two positions, we can find the object position in each frame, and can interpolate any intermediate position, such as halfway. We can simply paint the object at that position on the clean frame.

If the camera is hand-held, the background also moves. We can use IM to stabilise the frames. (I do this by finding small areas that correspond between the frames, and deriving a distortion that maps one image to the other. This process is itself non-trivial.)

If you have no clean frame, you can build one provided the object has moved far enough against the background.

You have the extra complication that the object itself changes from one frame to the next: eg the player's arms move with respect to each other. This, again, can be accounted for in another stabilisation process.

EDIT: Of course, there are other complications such as object blur within a frame and non-linear motion.

[xyrph]

Thank you for the info. So, it is possible, but it is not easy.

Can you share some commands or scripts? Would it help if I posted a link to some frames? All of my footage is shot from tripod.

[snibgo]

I'm about to leave for the day. If you can put up a few frames, prefereably including a clean frame, I'll look at them later.

[xyrph]

OK. Here is a seven second clip from a highlight:

http://ehasamples.excelhero.com/video/sample/sample.mp4


Remember the camera records at 60 frames per second. I have uploaded each of the clip's frames as BMP files. The numbering starts at 0016.bmp and goes through 0472.bmp. They are all in the same folder as the sample file above, for example:

http://ehasamples.excelhero.com/video/sample/0016.bmp
http://ehasamples.excelhero.com/video/sample/0017.bmp
http://ehasamples.excelhero.com/video/sample/0018.bmp

etc.

The last frame is probably the cleanest:

http://ehasamples.excelhero.com/video/sample/0472.bmp

[snibgo]

Putting the camera on a tripod is good, but not moving the camera would be even better.

I've extracted the frames to fr_%06d.png. We can construct a false-clean frame, eg:

Code: Select all

convert ^
  fr_000015.png ^
  fr_000020.png fr_000025.png ^
  fr_000030.png fr_000035.png ^
  fr_000040.png fr_000045.png ^
  fr_000050.png fr_000055.png ^
  fr_000060.png fr_000065.png ^
  -evaluate-sequence median ^
  med.png

From this, we can make cutout figures for each frame, eg:

Code: Select all

convert ^
  med.png ^
  fr_000050.png ^
  ( -clone 0 -clone 1 ^
    -compose Difference -composite ^
    -level 5%%,10%% ^
  ) ^
  -delete 0 ^
  -alpha off ^
  -compose Copy_Opacity -composite ^
  fCutout.png

This shows where figures are against a transparent background in frame 50. Some of the players show up well. But where a player is wearing black, against a background spectator also in black, we have problems. When we rebuild motion, this doesn't matter.

We can use this as a mask to get the "moving pixels" for this frame, repeat for all frames, and these could be painted against a constant background: transparent or green or whatever. Then each frame can be broken into arbitrarily small areas to find the movements from each frame to the next. (This is similar to the stabilisation problem but with different heuristics. We need to ensure that every source area has a destination area, or limbs will disappear.)

Now we have the movement of each area, linear (or other) interpolation gives any intermediate position.