Subimage search: overlay a thermal with a visible image

[thomas12]

I use a thermal image camera that makes simultaneously a real and a thermal image
I get three images:
- real image (rgb 2048x1536)
- thermal image (grayscale 160x120 Pixel)
- a color lookup table

In the next step I want to overlay the pattern of the real image with the color information of the thermal image.
I know the fixed ratio between real and thermal image, but axis between real and thermal image a not parallel.
(camera has two sensors and optics, it’s not a SLR camera)

Depending on distance to object there is a offset from center between “geometry -2+7” (infinite) and „geometry -42+81” (near 20cm).
Both extreme points can be connected with a straight line.

Here is a sample:

the visible image (precise resized for overlay to 339x254px)



8 bit grayscale thermal image (sensor size is only 160x120px)


and the 256 color lookup table (ad libitum, this is iron)


With known offset I can simple overlay the images (here is geometry -7+12 ):

Code: Select all

convert real.jpg -shade  45x30 -auto-level \( thermal.png clut.png -clut \) -gravity Center -geometry -7+12 -compose Overlay -composite overlay.jpg

and I get this result (Overlay Image)


I tried to find the "geometry +x+y" with compare -subimage-search …
and tested variants of -metric and -dissimilarity-threshold
but with no success.

Code: Select all

 compare: images too dissimilar

Is there a possibility to convert the real image and thermal image (i.e. edge detect), so that subimage-search can find a position?
Since the offset run along at a straight line, not all positions must be test trough subimage-search. I can also imagine to test all points at the straight line with a shell script. But I dont find a good compare/metric...

Thanks for help.

[fmw42]

Your thermal image has some graylevel reversals compared to the real image (when converted to gray). Thus you will have a hard time matching. The best way to work is to convert the two images to grayscale edges and do the compare on them. The following works for me with IM 6.8.5.3 Q16 Mac OSX Snow Leopard. If you are not in a newish version of IM to support -grayscale, then try -set colorspace RGB -colorspace gray.


convert real.jpg -grayscale rec601luma -morphology edge diamond:2 -evaluate multiply 5 real_edge5.png



convert thermal.png -morphology edge diamond:2 -evaluate multiply 5 thermal_edge5.png




compare -metric rmse -subimage-search -dissimilarity-threshold 1 real_edge5.png thermal_edge5.png result.png
11362.9 (0.173386) @ 82,79

[thomas12]

I tested it with different thermal images and it works fine.

Thanks for your help!

[anthony]

I believe they convert the normal image to a hue based color space,
Colorise the thrmal image using the color map and also convert to a hue based colorspace.

Then replace color/saturation with information from the thermal image, so as to psuedo color the image using thermal data.

I believe this is done by simply using something like -compose colorize with the colored thremal image.


There is no 'shade' involved in the process at all.

Fred the grey reversals in the thermal image is because it is cooler.

[fmw42]

Fred the grey reversals in the thermal image is because it is cooler.

Yes, that was understood. I was just pointing that out to show that a good match cannot be made on grayscale or color alone. An edge extraction however works much better.

My first image processing publication from many years ago, which I sent you a few years ago, had thermal images that needed to matches with other images, as I recall, and I came up with several approaches that were edge processed correlations (some using the FFT frequency domain for efficiency). One of the techniques I developed in that paper was a "dot-product" correlation matching process which used the edge direction, which can be extracted with -morphology convolve in IM from the directional kernels such as Sobel. You actually have an example of extracting the edge directions at http://www.imagemagick.org/Usage/convol ... al_kernels

[thomas12]

The geometrical offset of the real and thermal images runs along a line (depending on focus distance of the flir e40).
I tried to compare the subimages along the line with a shell loop (compare the cutted x/y-subimage with thermal image)

Now I have problems to understand the subimage search result. I can't get the same rmse-values by using compare on the cropped image
------------------------

For better comparing I used in the following sample the metric mae

first let's find the subimage with your both edge-pictures from your post above

Code: Select all

compare -metric mae -subimage-search -dissimilarity-threshold 1 real_edge5.png thermal_edge5.png result.png

3813.57 (0.0581913) @ 81,80

crop the found subimage for direct comparing

Code: Select all

convert real_edge5.png -crop 160x120+81+80 +repage subimage.png

and make the same test

Code: Select all

compare -metric mae subimage.png thermal_edge5.png compare.gif

17102.7 (0.26097) // 16 Bit


this value 0.260926 we can validate with a simple difference image

Code: Select all

convert subimage.png thermal_edge5.png -compose Difference -composite -colorspace gray -verbose  info: | grep -i mean

mean: 66.5473 (0.26097) // 8 Bit

How we can explain the difference mae=0.058(subimage-search) versus mae=0.261(crop image) ?
(The difference is the same with metric rmse...)

---
Version: ImageMagick 6.7.5-5 2012-02-11 Q16

[fmw42]

I can confirm that with respect to my original rmse compare, which had a subimage compare result of

compare -metric rmse -subimage-search -dissimilarity-threshold 1 real_edge5.png thermal_edge5.png result.png
11362.9 (0.173386) @ 82,79

convert real_edge5.png[160x120+82+79] real_edge5_sub.png

compare -metric rmse thermal_edge5.png real_edge5_sub.png null:
24063.2 (0.367181)


I have no explanation.

This should be reported on the bugs forum unless someone else has any further input.

[thomas12]

ok, this was only a comprehension question
I'm only searching the minimum and this works fine

Thanks for your help!