Issue with SubImage search

[appu]

Hi,

I am using ImageMagick-6.8.6-Q16 in windows platform. I used IM to does image comparison for my application UI. Currently I've one issue with sub image search in IM.

Here the ideal app behavior to be verified is as follows;


Here the arrow indicator is the area to be verified(sub image is enclosed in red rectangle for identification). There can be an upper arrow or not depending on the application behavior.

So the sub image to be verified is :


With following image, the sub image comparison passed.



But if there is any issue with painting app, such that there is a vertical 'x pixel' shift. So with drifted UI the image which is to be searched is :


In this case also the comparison passed, which is a false pass. I expect a failure in this case. Actually it interprets the upper corner of the ellipse as 'arrow'.

command used is

Code: Select all

compare -metric rmse -subimage-search img1.jpg img2.jpg result.jpg

On experiments observed that RMSE with error % less than 20 can be considered as pass. But in this case it returns error as 18%. So consider the comparison as passed which is incorrect. Result.jpg is :


how can I accurately test in this scenario ?

PS: Color comparison is required as the ' arrow' can be highlighted or not. I think grayscale comparison is not suited for this case

[snibgo]

So the sub image to be verified is : ...

Why have you included so much blue area? Won't it always match, so it doesn't help to distinguish? You could crop it down to just the triangle.

If the arrow can be a different colour, then I would isolate any non-blue pixels, and probably threshold so blue becomes black and non-blue becomes white. Do this with both the test image and sub-image.

With these two techniques you should get a much clearer test for pass or fail.

[appu]

Thanks Snibgo for the inputs.

I used the tightly cropped sub image and RMSE error percentage reduced. But I can't apply your second point as I don't have permission to change the current app rendering scheme.

Also I need a general solution for subImage search to avoid these kind of false passes. Any other generic rule along with tightly cropped sub image regions ?

[snibgo]

Sorry, I wasn't clear. I didn't mean you should change the rendering. I meant that you should take the image from the screen and process this before doing the search. For example, change blue to black and everything else to white. Do this also with the image you are using to search. That way, the colour of the arrow on the screen won't matter, it will still be found.

You may have a problem that the captured image catches only part of the arrow. You should check the RMSE values, and decide what you want to be "pass" or "fail".

[appu]

OK Snibgo.

One issue with that approach is that, in certain specific scenarios the ellipse can be highlighted with yellow (as below).



I would appreciate a generic approach which works in most of the cases. Please correct me, if I am not understanding your answer properly.

It would be great, if you could share the IM commands for the mentioned approaches.

[snibgo]

First point: don't use jpg files. Never ever use jpg at any stage of image processing unless you really have to. Jpg is bad for photographs but very very bad for graphics like this.

Second point: is the arrow always the same colour? And is the blue always the same colour? If so, then just create a sub-image which contains an arrow with a blue background, but is cropped down to the minimum rectangle that contains the arrow.

If the arrow of background can be different colours, please provide samples.

[appu]

OK, we will use PNG as the image format instead of JPG.

To answer your queries;
1. Yes, always the arrow is in same color. But it can be visible or not.
2. Background blue also of the same color.

I tried to cropped the arrow enclosed in a small rectangular region. The RMSE error percentage increased from (11 to 13%).[I'm testing a negative scenario.)
But our sampled error percentage is 17% error. So we expect more error on the image.

So we are trying to do something differently in IM as below.

1. Do a subimage search using metric RMSE

2. If error % is below our required similarity range(17% in our case), get the x and y co-ordinates of the found region. Then crop the region from the large image using (x, y, small image width, small image height)

3. If error % is greater than similarity range(17%), signal it as region not found

4. In pass case, again do a compare verification using more efficient compare algorithms. We have identified NCC, pHash etc. Please share if any one knows a better comparison algorithm for this scenario. If this comparison again passed, then signal it as pass. Otherwise, region not found.

In one of the thread, it was mentioned that they are planning to incorporate perceptual hash(pHash) in IM(viewtopic.php?f=1&t=23399#p100270)

Any update on when we can expect this feature in IM ? Also please share your thoughts on the approach and any other better method for sub image search to avoid false passes.

[fmw42]

1. Yes, always the arrow is in same color. But it can be visible or not.

What do you mean by the arrow can be visible or not? That does not make sense. Do you mean in some images there is no arrow, but you are still trying to match your arrow template to that image.

With respect to perceptual hash, there is no specific date on that. I am tasked to come up with a script prototype when I can get to it. My thoughts are to use an algorithm based upon image moments. see http://www.naturalspublishing.com/files ... g3omq1.pdf

[fmw42]

Hi,

I am using ImageMagick-6.8.6-Q16 in windows platform. I used IM to does image comparison for my application UI. Currently I've one issue with sub image search in IM.

Here the ideal app behavior to be verified is as follows;


Here the arrow indicator is the area to be verified(sub image is enclosed in red rectangle for identification). There can be an upper arrow or not depending on the application behavior.

So the sub image to be verified is :


With following image, the sub image comparison passed.



But if there is any issue with painting app, such that there is a vertical 'x pixel' shift. So with drifted UI the image which is to be searched is :


In this case also the comparison passed, which is a false pass. I expect a failure in this case. Actually it interprets the upper corner of the ellipse as 'arrow'.

command used is

Code: Select all

compare -metric rmse -subimage-search img1.jpg img2.jpg result.jpg

On experiments observed that RMSE with error % less than 20 can be considered as pass. But in this case it returns error as 18%. So consider the comparison as passed which is incorrect. Result.jpg is :


how can I accurately test in this scenario ?

PS: Color comparison is required as the ' arrow' can be highlighted or not. I think grayscale comparison is not suited for this case

The problem here is that you are really more matching the background blue in your small and large images, so even changing the shape of the white is making little difference.

What you need is a white triangle with background transparent and a new compare that ignores any transparency in the small image when matching to the large image. I do not believe this is currently possible with IM, but I have suggested it in the past. This would need to be suggested again to the Developers forum.

[appu]

1. Yes, always the arrow is in same color. But it can be visible or not.

What do you mean by the arrow can be visible or not? That does not make sense. Do you mean in some images there is no arrow, but you are still trying to match your arrow template to that image.

With respect to perceptual hash, there is no specific date on that. I am tasked to come up with a script prototype when I can get to it. My thoughts are to use an algorithm based upon image moments. see http://www.naturalspublishing.com/files ... g3omq1.pdf

Yes, the arrow is visible only if the App meets certain criterias. So in order to ensure that our Imagemagick comparison works fine and not signals a false pass for an arrow region, in a case where arrow is not actually present.

Please let me know, when you finished the image moment implementation

[appu]

The problem here is that you are really more matching the background blue in your small and large images, so even changing the shape of the white is making little difference.

What you need is a white triangle with background transparent and a new compare that ignores any transparency in the small image when matching to the large image. I do not believe this is currently possible with IM, but I have suggested it in the past. This would need to be suggested again to the Developers forum.

OK. Got your point.

IM developers it would be great if you could implement a compare routine in IM which ignores the transparency !!!

[magick]

IM developers it would be great if you could implement a compare routing in IM which ignores the transparency !!!

Turn off the alpha channel with -alpha deactivate (or +matte).

Magick, I believe you misunderstand. The idea is to skip pixels during the metric evaluation that are transparent in the smaller reference image, thus only matching the non-fully-transparent pixels in the reference to the equivalent pixels in each subsection of the larger image. Basically the alpha channel is used as mask to decide with pixels to use for matching.

This would need another argument, such as -skip-alpha or -skip-transparent. (or a -define)

Fred

[snibgo]

Perhaps I misunderstand what is being said. "compare" currently ignores pixels that are transparent, unless "-alpha off" is used. For example:

Code: Select all

D:\web\im>c:\im\ImageMagick-6.8.7-Q16\convert xc:rgba(0,0,0,1) xc:rgba(0,0,0,0) +append c1.png

D:\web\im>c:\im\ImageMagick-6.8.7-Q16\convert xc:rgba(0,0,0,1) xc:rgba(100%,100%,100,0) +append c2.png

D:\web\im>c:\im\ImageMagick-6.8.7-Q16\convert xc:rgba(100%,100%,100%,1) xc:rgba(100%,100%,100,0) +append c3.png

D:\web\im>c:\im\ImageMagick-6.8.7-Q16\compare -metric RMSE c1.png c2.png NULL:
0 (0)

D:\web\im>c:\im\ImageMagick-6.8.7-Q16\compare -metric RMSE c1.png c2.png -alpha off NULL:
46340.2 (0.707107)

D:\web\im>c:\im\ImageMagick-6.8.7-Q16\compare -metric RMSE c1.png c3.png NULL:
40131.8 (0.612372)

D:\web\im>c:\im\ImageMagick-6.8.7-Q16\compare -metric RMSE c1.png c3.png -alpha off NULL:
65535 (1)

EDIT: I retract my statement that "compare" currently ignores pixels that are transparent, unless "-alpha off" is used. Although my examples suggest that the statement is true, I think they are merely special cases.

[fmw42]

Perhaps I misunderstand what is being said. "compare" currently ignores pixels that are transparent, unless "-alpha off" is used.

snibgo,

I don't have time this morning to check into that, but if so, I never knew that. I alway thought alpha was totally ignored (as if the alpha channel was turned off) except for metrics AE and FUZZ.

If you have time, try creating a mask for his triangle reference image so that only the white triangle is opaque and the rest is transparent. Then run that against his two other images (one with triangle and the other with the white arch) and see what happens.

If you don't have time, I will check it later this afternoon.

Thanks.

[snibgo]

Working with those JPEGs, pending PNGs, would be a waste of effort. JPG messes up both the blue and white.