Aerial Imagery normalization

[SFMapper44]

First timer here,
I am trying to stitch together aerial images to create an orthomosaic using imagery obtained from an airplane. The issue I am experiencing is the fact that some of the images are either under clouds or were taken at slightly different times of day, making them darker than others.
I was hoping to learn more about how to take a collection of images (~1000) and normalize them so when they are mosaiced together there wont be streaks or entire sections of dark imagery. I have done some snooping around and came across this discussion:
viewtopic.php?f=1&t=23727
But it seems this use case would be for normalizing within a given image and not a collection of images.
Any help would be greatly appreciated!

[snibgo]

Can you put up some example adjacent frames?

[SFMapper44]

You can see the first image is slightly darker than the one below it. These differences may look minor at this level, but when you look at all the images mosaiced together at a higher elevation, the differences are pronounced.

https://www.dropbox.com/s/w9oilercwucwdrx/01.jpg?dl=0
https://www.dropbox.com/s/954h4x13xl1v1m1/08.jpg?dl=0

[fmw42]

Many years ago I was in the same business of creating and mosaicing aerial images. There are a number of ways to approach normalization that I have used.

1) Optimize or pick the best image in terms of viewability (brighness and contrast). You can try to optimize that one image by either using something like -equalize or better something like my unix shell script, redist or retinex. User snibgo has an adaptive contrast equalization Magick Filter that would work well also. Then you can do histogram matching to match all the rest to this first one. I have a script histmatch. Also user snibgo has a Magick Filter for something similar.

2) Optimize all images to the same statistics. I did not that years ago with something similar to my space script. This script adjusts the image in an adaptive manner to a specified mean and standard deviation

User snibgo may also have some further ideas, such as matching the image to the same global mean and standard deviation or perhaps his adaptive contrast equalization Magick Filter.

Please always identify your imagemagick version and platform, since syntax may differ.

[snibgo]

We can consider this as a general blending problem. A number of my pages (not all published yet) are about blending, though on a smaller scale, up to a dozen images. The same principles apply to thousands.

There are two major issues: geometry and photometry.

Geometry: The two shapes on these images can't be made to match with a simple translation. They also need a rotation and perspective transformation, perhaps due to tilt of the aircraft. To see this, try:

Code: Select all

%IM%convert ^
  aerial01.jpg ^
  ( aerial08.jpg ^
    -repage +2100+1000 ^
    -alpha set ^
    -channel A -evaluate set 50%% +channel ^
  ) ^
  +write info: ^
  -layers merge ^
  overlap18.png

(All code is Windows BAT syntax. Adjust as required for your shell.)


Provided the same camera and lens is used for all photos, and the lens is close to rectilinear, my page "Simple alignment by matching areas", using Perspective Distortion, should solve the geometry problem.

Once overlaps have been determined, one image can be made to match another photometrically (brightness, contrast and colour cast). In this example pair, we can extract the two overlaps, approximately:

Code: Select all

%IM%convert ^
  aerial01.jpg ^
  -crop 1000x1000+2100+1000 +repage ^
  over1.png

%IM%convert ^
  aerial08.jpg ^
  -crop 1000x1000+0+0 +repage ^
  over8.png

One simple photometric adjustment is gain and bias, which adjusts the brightness and contrast of each channel independently. One image can be made to match another, or both adjusted to a common standard. From my "Gain and bias" page:

Code: Select all

call %PICTBAT%imgGainBias ^
  over1.png ^
  over8.png ^
  over1match8.png

The overlaps match photometrically, more or less. The same transformation can be applied to an entire image.

Instead of gain/bias, we can match histograms:

Code: Select all

call %PICTBAT%matchHisto ^
  over1.png ^
  over8.png ^
  over1matchhist8.png

This gives a slightly different result. It is more accurate, provided the overlaps match geometrically. But when used on two images that merely contain an overlap, it will give spurious results, so the gain/bias method is better.

So, we have solutions for matching the geometry and photometry of images. Then it is a hard slog of processing the thousands of images, working either to a common standard or a sliding standard, and blending the results. This is a good solution for preparing accurate maps.

For flight simulators, I think there is a better solution. In the words of one researcher, "Don't blend it, cut it!".

Instead of matching geometry and photometry then blending one image with another, we simply cut a ragged edge around one image so that when that cut image is placed over the other, the join is not obvious. Then we add another image, and another, reaching thousands. The ragged cut is made where the images match best, a "dark path" within the difference of the images.

This "cutting" method removes the complexity of standards (whether common or sliding), precise geometric alignment, mismatching colour gamuts, and ghosting caused by blending. The virtual landscape below the aircraft will look realistic. True, it won't be as accurate as the technical mapping method, but it will appear realistic, and it is far simpler to make.

I haven't published the general technique for doing this with a large number of photos. The simple case of two images is on "Rectangle boundaries with dark paths", and some extensions are shown on other pages.

Code: Select all

call %PICTBAT%rectDp ^
  over1.png ^
  over8.png ^
  over1cut8.png ^
  30c 30c 0 0

[snibgo]

In the post above, I found the rough offset of the two photos (+2100+1000) by manual trial-and-error. Finding it automatically by a brute force technique on corresponding layers of Gaussian pyramids gives the result +2144+987. It takes about 20 seconds.

Using that overlap, and the entire overlapping areas, we want a ragged cut between the top-right and bottom-left corners. Using Dijkstra's algorithm:

The upper-left is from aerial01.jpg; the lower right is from aerial08.jpg. The boundary is obvious where it crosses the road, but hard to see elsewhere.

[fmw42]

Nice work snibgo. I did not know you had all those parts to combine so nicely.

One point, a proper orthorectification (he mentions orthomosaic), would require registration with a digital terrain image and perspective correction using a camera model for the photograph taking into account the terrain height at each image pixel (ignoring the man-made structures). Perhaps he did not mean to go that far, but simple a flat perspective correction. See for example,
https://en.wikipedia.org/wiki/Orthophoto
https://trac.osgeo.org/ossim/wiki/orthorectification

[snibgo]

Thanks, Fred. My ramblings about ragged cuts as a substitute for corrections should be ignored if the goal is for science or engineering.

[SFMapper44]

Thank you so much snibgo and fmw42 for your help on this question!
Yes, orthorectification and BBA of images will take place inside a photogrammetric software using a well known workflow. Its just, the images have to be looking consistent on the way in or the resulting mosaic will never be consistent. The old "garbage in, garbage out" saying applying here.

[fmw42]

Thank you so much snibgo and fmw42 for your help on this question!
Yes, orthorectification and BBA of images will take place inside a photogrammetric software using a well known workflow. Its just, the images have to be looking consistent on the way in or the resulting mosaic will never be consistent. The old "garbage in, garbage out" saying applying here.

I had assumed that you had your orthorectification separately. So I never went into all the registration work that snibgo has provided. Though his work is quite good and ragged cuts are very interesting for hiding mosaic merge lines. However, one could also do some overlap blending, which is what I had done years ago. See for example,
http://www.fmwconcepts.com/imagemagick/ ... composite1
http://www.imagemagick.org/Usage/photos/#overlap

[snibgo]

Yes, orthorectification and BBA of images will take place inside a photogrammetric software using a well known workflow. Its just, the images have to be looking consistent on the way in or the resulting mosaic will never be consistent. The old "garbage in, garbage out" saying applying here.

I'm trying to understand what problem you need to solve.

If I understand correctly, you have a thousand or whatever aerial photographs. You have software that that does the geometric corrections so the shapes all match up. I suppose it then cuts the very large result into tiles for mapping or whatever.

Your problem is that the software corrects only the geometry, and doesn't change colours. Hence, the original boundaries between the images are visible.

Is that correct?

If so, have you tried talking to whoever wrote that software? Once the geometry is corrected, the overlapping areas are well-defined, so colour-correction is fairly easy. Or the job can be done with ragged cuts (aka "seams", see Mosaicking of Aerial Photographic Maps Via Seams Defined by Bottleneck Shortest Paths).

If colour-correction can only be done before the geometry is corrected, the problem is more difficult. You could make all photos the same mean and standard deviation (ie lightness and contrast), but if one photo has many light-roofed buildings and its neighbour has many dark roads, making them the same mean lightness will introduce a boundary. But you could combine that technique with ragged cuts.

[SFMapper44]

Your problem is that the software corrects only the geometry, and doesn't change colours. Hence, the original boundaries between the images are visible.

Is that correct?

That is mostly correct, yes. The only change would be that the software blends but only at a boundary. There is no color normalization taking place globally, to the entire collection of images. Thats why I am looking to "even" everything out before the software processes them.
Below are samples of the result. One showing the how the mosaic looks zoomed out and a zoom in of each the dark and light areas. This is the result of the photogrammetric software and NOT raw images. Hopefully this helps define the issue.
https://www.dropbox.com/s/ukztgzgp9w2u0 ... t.png?dl=0
https://www.dropbox.com/s/93u0eztj5u4em ... l.png?dl=0
https://www.dropbox.com/s/3jorlpwktlfwa ... l.png?dl=0

[snibgo]

Okay.

The best result would come from finding overlaps. Each photo probably has up to six neighbours. When the overlapping areas are known, you can calculate what change would make this photo match neighbour number (n). You want to apply half of that change, at the pixel in the centre of the overlap area. Now you know the change required at six (or so) points, you can triangulate the image from these points. Every pixel has barycentric coordinates within the triangle, which gives the required proportion of each of the three changes.

Yeah, that is messy and complex.

Various assumptions might be made to simplify processing:

1. The major change is to lightness, not to hue or saturation. Perhaps the change to the Lightness channel of each image needs only two parameters (gain and bias) or even just one parameter (gamma).

2. A good result can come from raising the lightness of photos so that all overlap pairs have matching lightness.

If that is still too complex, or finding overlaps is too time-consuming, options are severely limited. Perhaps you don't even know what photos overlap what other photos. Then you can only make rash assumptions, eg that the mean lightness of all photos should be the same. This assumption looks true of the examples you have posted so far.

Code: Select all

f:\web\im>%IM%convert Dark_Detail.png -colorspace Lab -format %[fx:mean.r] info:
0.246223
f:\web\im>%IM%convert Light_Detail.png -colorspace Lab -format %[fx:mean.r] info:
0.49808

EDIT: I forgot to put ".r" in the above. It is needed because I want the mean values of just the Lightness channels.

Adjust Lightness gamma of both, making both means 0.5.

Code: Select all

f:\web\im>%IM%convert Light_Detail.png -colorspace Lab -channel R -auto-gamma +channel -colorspace sRGB aerial_light.png

f:\web\im>%IM%convert Dark_Detail.png -colorspace Lab -channel R -auto-gamma +channel -colorspace sRGB aerial_dark.png

The two results aerial_*.png visually match.

[SFMapper44]

Great, that is a huge help. Im playing with it now.