compare's exit code is always 0

[obfuscated]

Hi,
I want to use the compare tool to compare images in my regression test scripts.
But it turns out that the exit code from the tool is always zero even if there are differences in the images.
Is this the expected behaviour or is this a bug?
This problem happens only on linux, tested both 6.8.x on Centos 5.x and 6.7.x on Gentoo.
On windows it seems to work fine.

[magick]

ImageMagick utilities return 0 on success and non-zero if the command failed. For compare, you must check the DB level to determine if two images differ. If the DB is 0, the images are identical.

[obfuscated]

Is it possible to add a command line option to force a non-zero exit code if the images are different?

[magick]

Makes sense to have this feature. We'll add it to the next release of IM. Thanks.

[obfuscated]

Do you have an estimate date for a release?

[benze]

Makes sense to have this feature. We'll add it to the next release of IM. Thanks.

I'm using ImageMagick 6.8.6-6 2013-08-31 Q16 on a Mac OSX installed using MacPorts. It appears that this feature is already present. Is there any way to disable this and have it still return 0 since it actually completed successfully? It is breaking things with Im4Java, where Im4Java is now throwing an exception that Compare did not complete without an error code.

Thanks,

Eric

[dlemstra]

I think you should report the problem to im4java so they can change their code. There is no option available to force the exit code to zero and we should probably also not add such an obscure feature.

[benze]

I think you should report the problem to im4java so they can change their code. There is no option available to force the exit code to zero and we should probably also not add such an obscure feature.

Fair enough - will look at Im4Java to work my way around this. However, just to play devil's advocate for a moment, the idea of returning a non-zero exit code would imply that there was a non-standard termination of the application (ex: an error, a warning, etc). At least according to exit code conventions (of other software). In this particular case, Compare is actually working properly and exiting properly. The fact that the two images are not identical should not necessarily force a non-zero exit code as the software completed as expected. The output produced in the stderr stream gives the result of the compare. I would have expected the user need to validate/examine the output to know what the state of the compare command was, rather than depend on a non-standard way of producing and exit code.

From my understanding Compare originally did function this way - ie: produce a zero exit code unless there were errors or warnings produced, but was changed to accommodate regression test scripts that were "too lazy" (forgive the term - am not implying that the developer was too lazy to code it, just that it was easier to validate an exit code vs checking a DB level) to validate the DB level. I just question the decision to make this kind of a change global and not based on a command line option.

Thanks,

Eric

[magick]

The exit code is 0 if the ImageMagick command -line utility completes without any error exceptions. The compare utility returns an exception if two images are not similar. If you want to ignore this exception, set the -similarity-threshold high so that all dissimilar images are considered similar and therefore no exception is generated.

[benze]

The exit code is 0 if the ImageMagick command -line utility completes without any error exceptions. The compare utility returns an exception if two images are not similar.

Really? If I compare two completely different images, I get an exception and an exit code of 1:

Code: Select all

Eric-MacBook-Pro:temp eric$ compare -metric AE Disney-Princess-Kida-disney-princess-30168400-2560-1117.jpg DISNEY\ PRINCESS\ SLEEPING\ BEAUTY\ 3D\ LAMP.jpg /dev/nul
compare: image widths or heights differ `Disney-Princess-Kida-disney-princess-30168400-2560-1117.jpg' @ error/compare.c/CompareImageCommand/977.
Eric-MacBook-Pro:temp eric$ echo $?
1

That I expect, and makes sense. However, if I compare two images of the same size, I get no exception/errors thrown and yet it still exits with an error code of 1. In fact it even tells me exactly what is different between the two images:

Code: Select all

Eric-MacBook-Pro:temp eric$ compare -verbose -metric AE compare_reference.png compare_circle.png null: 2>&1
compare_reference.png PNG 35x35 35x35+0+0 8-bit sRGB 3.55KB 0.000u 0:00.000
compare_circle.png PNG 35x35 35x35+0+0 8-bit sRGB 4.47KB 0.000u 0:00.000
Image: compare_reference.png
  Channel distortion: AE
    red: 511
    green: 511
    blue: 511
    alpha: 511
    all: 511
compare_reference.png=> PNG 35x35 35x35+0+0 8-bit sRGB 0.000u 0:00.000
Eric-MacBook-Pro:temp eric$ echo $?
1

Am I missing something, or is this a bug?

Thanks,

Eric

[fmw42]

I believe he was likely talking about compare of two different size images with subimage search.

Nevertheless, I tried same size image compare (IM 6.8.6.9 Q16 Mac OSX) and as you say, it still produces return code of 1

see
http://www.imagemagick.org/script/comma ... -threshold
http://www.imagemagick.org/script/comma ... -threshold
http://www.imagemagick.org/script/compare.php


convert rose: -negate rose_neg.png

compare -metric rmse rose: rose_neg.png null:
35855.1 (0.547114)
echo $?
1

compare -metric rmse -similarity-threshold 1 rose: rose_neg.png null:
35855.1 (0.547114)
echo $?
1

In case he meant -dissimilarity-threshold:
compare -metric rmse -dissimilarity-threshold 1 rose: rose_neg.png null:
35855.1 (0.547114)
echo $?
1


compare -metric rmse -dissimilarity-threshold 1 -similarity-threshold 1 rose: rose_neg.png null:
35855.1 (0.547114)
echo $?
1

[dlemstra]

I have an update. For IMv7 we are going to change the exit code. When the images are equal 0 will be returned and when the images are different we will return 1. This is the same as IM6 but we will add something for when an exception has occurred. The executable will then return an exit code that has the same value as described here: http://www.imagemagick.org/script/exception.php. If the program returns an exit code of 300 or higher an exception was thrown. This will also make it easier to ignore warnings in a shell script but stop the execution when an error was thrown. We could maybe also add this to IMv6 but that might break a lot of script that check exit codes so we are a bit hesitant to do this.

[magick]

Error codes are limited from 0 to 255.

This is still problematic because the convention for error codes is 0 on success or an integer in the range 1 - 255 on error. However, POSIX only mandates, for programs such as grep, cmp, and diff, that the exit status in case of error be greater than 1.

[dlemstra]

My bad, windows uses 32-bit signed integers as exit code. Maybe we should just stick to 3 for a warning, 4 for an error and 7 for a fatal error?

[magick]

The Linux exit status convention is 0 denotes success and any non-zero exit status denotes failure. Grep-like utilities returns 0 if the pattern was found, 1 if it wasn't, and 2 (or more) if there was an error such as a missing file. We'll keep the existing exit status for all ImageMagick utilities except the compare utility. For it, we will return the same exit status as grep. That is if the images are similar and no errors, return 0. If the images are dissimilar and no errors, return 1. If an error occurs, return 2.