Problem with compare

[kinder]

Hi,

I'm comparing 2 set of screenshots got from 2 different files (it's the same movie encoded with 2 different programs, movie "A" has a better resolution then movie "B").
I would like to know if 2 screenshot are the same exact "frame" for each movie, but I have same problem with compare. I tried all the possible metric parameter but I can't have expected result.

These are 8 frames, 4 from source A (a-1, a-2, a-3, a-4) and 4 from source B (b-1, b-2, b-3, b-4).

(I want: ok value) (MAE value: 1886)

(I want: ko value) (MAE value: 1624)

(I want: ko value) (MAE value: 365)

(I want: ko value) (MAE value: 1397)

I would like whick parameters and value use with compare so that I get:
compare -something_here a-1.jpeg b-1.jpeg diff.png -> GOOD MATCH VALUE (same "frame")
compare -something_here a-2.jpeg b-2.jpeg diff.png -> BAD MATCH VALUE (same "frame")
compare -something_here a-3.jpeg b-3.jpeg diff.png -> BAD MATCH VALUE (same "frame")
compare -something_here a-4.jpeg b-4.jpeg diff.png -> BAD MATCH VALUE (same "frame")

I'm using -metric MAE but this don't help (1886 for the good case, 365, 1397 and 1624 for the bad cases). I would like to know with set of parameter can give me the best match for movie-exact-frame search.

Please help

[fmw42]

It is likely image dependent. But I like to use -metric rmse (and look at the fractional value in the parenthesis, not the first raw value). Perhaps you need to normalize by the image mean (rmse/mean). Don't really know for sure.

see http://www.imagemagick.org/Usage/compare/#statistics


What you really need is a proper normalized cross correlation value. But IM does not have that. see http://en.wikipedia.org/wiki/Cross-corr ... orrelation

[kinder]

Thank you: RMSE works better than MAE for my job.
Now during a test I got correct matches for RMSE values up to 0.169657 and wrong matches for values from 0.146048, so only values in a little range are uncertain (from 0.146048 up to 0.169657).

Thanks for links too, I'll try to study them.

To be honest, I hoped it was a simpler job. I'll try to add some heuristic on a subset of frames ...

[magick]

It would be quite simple to add a -normalize option to the compare program to subtract the mean and divide by the standard deviation before the compare. Would that be a useful addition to the compare program?

[fmw42]

This is a cut at a grayscale Normalized Cross Correlation Compare of two images of the same size. (From mathematics at http://en.wikipedia.org/wiki/Cross-corr ... orrelation)

Bad values are near 0 and good values near 1. A perfect match should be 1. Change infile1 and infile2 to your two images.

To be more accurate, you could do each channel separately and then get the rms combination of the 3 channels rms=sqrt(ncc-r^2 + ncc-g^2 + ncc-b^2)/sqrt(3)

For your thumbnail images, I got:

a-1 vs b-1: 0.880103
a-2 vs b-2: 0.309664
a-3 vs b-3: 0.239163
a-4 vs b-4: 0.8076

a-1 vs a-1: 1.00407 (perfect match but slight error as it should not be larger than 1)
a-1 vs the negated version of a-1 (white is black and black is white): 0 (no correlation)

infile1="a-1.jpg"
infile2="b-1.jpg"
inname1=`convert $infile1 -format "%t" info:`
inname2=`convert $infile2 -format "%t" info:`
convert $infile1 -type grayscale ${inname1}g.png
convert $infile2 -type grayscale ${inname2}g.png
n1a=`convert ${inname1}g.png -format "%[fx:w*h]" info:`
n1b=`convert ${inname2}g.png -format "%[fx:w*h]" info:`
if [ $n1a -ne $n1b ]; then
echo "--- IMAGES NOT SAME SIZE ---"
else
m1a=`convert ${inname1}g.png -format "%[fx:mean]" info:`
m1b=`convert ${inname2}g.png -format "%[fx:mean]" info:`
s1a=`convert ${inname1}g.png -format "%[fx:standard_deviation]" info:`
s1b=`convert ${inname2}g.png -format "%[fx:standard_deviation]" info:`
if [ "$s1a" = "0" -o "$s1b" = "0" ]; then
echo "--- ONE OR BOTH IMAGES ARE CONSTANT VALUES (NCC=0/0) ---"
ncc=0
else
num=`convert ${inname1}g.png ${inname2}g.png -fx "(u-$m1a)*(v-$m1b)" miff:- |\
convert - -format "%[fx:mean]" info:`
ncc=`convert xc: -format "%[fx:$num/($s1a*$s1b)]" info:`
fi
fi
echo "ncc=$ncc"

Indenting for clarity:

Note due to the use of FX it will be slower for full size images. The thumbnails were adequately fast.

However, here is another version that avoids fx and so should be faster for large images.

infile1="1a.jpg"
infile2="1b.jpg"
inname1=`convert $infile1 -format "%t" info:`
inname2=`convert $infile2 -format "%t" info:`
convert $infile1 -type grayscale ${inname1}g.png
convert $infile2 -type grayscale ${inname2}g.png
n1a=`convert ${inname1}g.png -format "%[fx:w*h]" info:`
n1b=`convert ${inname2}g.png -format "%[fx:w*h]" info:`
if [ $n1a -ne $n1b ]; then
echo "--- IMAGES NOT SAME SIZE ---"
else
m1a=`convert ${inname1}g.png -format "%[fx:mean]" info:`
m1b=`convert ${inname2}g.png -format "%[fx:mean]" info:`
m1ab=`convert xc: -format "%[fx:$m1a*$m1b]" info:`
s1a=`convert ${inname1}g.png -format "%[fx:standard_deviation]" info:`
s1b=`convert ${inname2}g.png -format "%[fx:standard_deviation]" info:`
if [ "$s1a" = "0" -o "$s1b" = "0" ]; then
ncc=0
else
num=`convert ${inname1}g.png ${inname2}g.png -compose mathematics \
-set option:compose:args "1,-$m1a,-$m1b,$m1ab" -composite miff:- |\
convert - -format "%[fx:mean]" info:`
ncc=`convert xc: -format "%[fx:$num/($s1a*$s1b)]" info:`
fi
fi
echo "ncc=$ncc"

[fmw42]

It would be quite simple to add a -normalize option to the compare program to subtract the mean and divide by the standard deviation before the compare. Would that be a useful addition to the compare program?

In my opinion a proper NCC equal size image compare would be useful. See my script above. Don't know if this is what you had in mind.

Even better would be to also do it for unequal size images, but that can be tricky as you need to get a different mean and std for each subsection of the large image. But if you want to go that far, I know of some tricks to speed it up that I can forward to you.

By the way, I have the latter coded in an (HDRI unfortunately) FFT script, but it does not seem to normalize correctly, even though it finds the best match.

Fred

[kinder]

Thanks to all, men.

Fred, your ncc function is really a kind of magic, it works like a charm in both version. That's awesome!!!

I tell you the project I'm working to. When you have 2 movies from 2 different countries or 2 different sources, often these movies have different length. The frame-per-second problem isn't the problem (example: 23.976024 in the USA, 25 in Europe), the problem is movies includes different frames or scenes.

So, if you want merge video from movieA with audio from movieB it's often a problem. You have to find all the differences, manually cut those scenes and then dub.

I'm developing a script that should do the work for you. The script works in this way:
* with ffmpeg I get all the frames for each movie (oh, my poor hard disk). If I need, I can cut and resize frames so frames from movieA and movieB have the same size, aspect-ratio, etc.
* I check every 100 frames (faster check) for a change of scene. I'm using MSE with a limit of 3000, so if I get a bigger value I have a change of scene. In a 90' movie I get about 255 "main" scenes.
* I compare all the "main" scenes from both movies (using Fred ncc compare procedure), that is I match main scenes from movieA with main scenes from movieB
* I check the offset between a match and the next one. Example: a-31.jpg (movie A frame: 1100) matches with b-31.jpg (movie B frame: 1150). a-32.jpg (movie A frame: 1230) matches with b-32.jpg (movie B frame: 1280). The offset is the same: +50 frames, so I can suppose all the frames from main frame-31 to main frame-32 are ok. If b-32.jpg was movie-B-frame 1283, then I have to match deeper all the frames from 1100 to 1230 (movieA) with frames from 1150 to 1283 (movie B) looking for the 3 frames more in the movie B
* using virtualdub, avidemux or avisynth I can cut the unwanted frames and finally dub the movie as I want

My work is in progress (I started 2 days ago), I'm not a super-coder and I'm a newbie, so ...

[fmw42]

My ncc compare seems to be working and useful, so I have now created and uploaded a script, similar, to my web site below.

[absolutelyrelative]

Hey Fred,
My project is a mobile visual search engine ( images only restricted to automobile logos ). I thought of using your compare script for the comparing the input image with the images in the database. But problem is i'm using Imagick in PHP. I dont know how to execute these commands in PHP. Any help would be highly appreciated. Thanks in advance.

[fmw42]

Imagick is somewhat limited in keeping up with changes with Imagemagick. Now the compare function in ImageMagick has a -metric ncc. But I don't know if the Imagick developer has kept up with those changes. Otherwise, the main issue would be the -compare Mathematics operation. The best I can recommend is use the PHP exec function to execute IM command lines or my script. You don't lose much in time if any and exec is much more flexible as it can run any IM operation and even my scripts. See Rubblewebs web site for PHP with Imagemagick. http://www.rubblewebs.co.uk/index.php