Producing flat frames

[geoland]

I am working on a method of combining flat frames (3 at this stage) for flat division of a number of images.

Following subtraction of the offset/bias from the flat frames, they are averaged into a single master flat; as follows

Code: Select all

convert -average -auto-level 3xflats.tiff master_flat.tiff

this produces a well illuminated master flat. However, it seems to be too bright and tends to darken the image frames when divided.

Code: Select all

light_*.tiff master_flat.tiff -compose Divide_Src -composite

An alternative method of combining the flats to produce a darker flat

Code: Select all

convert -average 3xflats.tiff -evaluate-sequence median master_flat.tiff

Which tends to blow out brighter parts of the image frames following flat division; also passes a median filter over the stack from what I read. The median filter can be omitted with similar flat fielding results.

I would like to find a happy medium - that is; the master flat not too dark and not too light. I have tried many different methods of combining the flats, but invariably come back to the two methods above, as they produce the best flat fielding results correcting vignetting properly.

[fmw42]

I do not know what a flat is? Please clarify?

Also it might be a good idea for your post an example of what you are trying to do at say some place like dropbox.com and put the URLs here so we can see what you are trying to do.

You need to tell us also what version of Imagemagick and what platform. Some older release of IM produced dark images.

Also your syntax is not completely correct. The input images should come right after convert and before any processing commands. See
http://www.imagemagick.org/Usage/basics/#why

Also -average is deprecated and is currently replaced by -evaluate-sequence mean.

If your resulting image is grayscale, then there are many ways to combine RGB channels to make grayscale. Examples can be seen at http://www.fmwconcepts.com/imagemagick/ ... /index.php

[geoland]

In astronomical image processing, a flat is an image taken at infinity illuminated by an even/flat light source to reveal dust, vignetting and other artifacts in the optical train. Probably, you know it by another name.

To isolate the signal in an image from noise and other artifacts we can apply the following; where Light = Signal*Flat + Dark noise + Bias

Signal = Light - Dark/Flat - Bias

A single flat is not enough we need lots to produce a representative mean and reduce noise in the merged stack. This applies to the other frames as well. So lots of darks to be combined into single master dark and applied to the Light frames. Similarly, the bias frames applied to the Flat frames, as above.

Signal = Lights - Master Dark/Master Flat (IM does this very well, but flat illumination is very important and requires some normalization to account for variations between frames.

The Lights are then deBayered, aligned and mean combined (-average) in another program.

I want to combine all the flat frames into a single master flat with an appropriate level of illumination -auto-level is as close as I can get - what I really want is to apply multiplicative normalization when combining the flats. I can't find a way in IM.

IM version 6.7.7 xubuntu 14.04.

I will fix the syntax.

Many thanks.

[fmw42]

Thanks. I understand your use of -average now.

Can you define multiplicative normalization? If I understand what you want, perhaps I can tell you want IM function to use. Perhaps it is -evaluate multiply X. You can also try using -contrast-stretch to expand the contrast or -sigmoidal-contrast. Also -function polynomial, can be use to do Y=aX+b, where X is an input, a and b are constants and Y is the output. See http://www.imagemagick.org/Usage/transf ... polynomial

Aside: IM 6.7.7 is rather ancient (about 150 versions old). You should upgrade to the current version of IM if you can.

[fmw42]

P.S. If you want to have different weights for each flat, you can use -poly. see http://www.imagemagick.org/script/comma ... s.php#poly

[geoland]

All the files are 16 bit integer CFA monochrome tiff files

Source IRAF image processing
"If during flat field exposure, the light source flickered or faded, the individual flats would have slightly different average values throughout the image. This would throw off the median combine in that it would always take the pixel from the middle exposure image and not reduce the noise. We need to include a small multiplicative correction to each flat field. The flats will be multiplied by a factor (close to one).

"Normalizing the Flats"

"If the light is divided by the flat the resultant image will have very small values. Ideally, we want to modify the values in the light image just enough to correct the multiplicative errors without significantly changing the pixel values in the light image. To do this we need to divide the flat frame by a constant to make the flat close to 1 before dividing the light by the flat (light/flat)."

"A normalized flat - most pixels are near to a value of one."

I take away from this that I need to

Code: Select all

convert flat_frames -evaluate-sequence median master_flat

which is what I was doing previously with the flats. And also find a way of calculating the multiplicative error between the flats prior to combining them. A method is to find the mode of the master_flat and do;

Code: Select all

master_flat/mode

to correct only the multiplicative errors.
I am yet to find the mode of an image in IM. I have looked but it escapes me for now.

[fmw42]

I do not believe that IM has a mode of multiple images.

If you need the normalize each image to some common mean value, you can compute the mean of each image as

Code: Select all

convert image -format "%[fx:mean]" info:

That will result in values between 0 and 1. You can then use (desiredmean/mean) as weights to -poly. If you want to compute the weights directly, then assuming the desired mean is mid gray (0.5)

Code: Select all

wt=`convert image -format "%[fx:0.5/mean]" info:

That puts the wt into a variable. You do that for each image as wt1, wt2 ... or name the wts for each image and script it over a loop for each image.

If you want the median of an image, then see my script, median, at the link below. Sorry I do not have one for mode.

[snibgo]

Mode is the value for which half the pixels are below and half are above.

IM doesn't have a built-in function to find the mode value of an entire image. You can create a text histogram and read it in a script. Or you can create an image histogram (with one of Fred's script, or my mkhisto process module) find the middle value.

Code: Select all

convert rose: -colorspace Gray -process 'mkhisto capnumbuckets 201 cumul norm' -crop 1x1+100+0 +repage txt:

# ImageMagick pixel enumeration: 1,1,255,gray
0,0: (209,209,209)  #D1D1D1  gray(209)

The mode value is 209 (out of 256).

[fmw42]

Mode is the value for which half the pixels are below and half are above.

I believe that is the median. The mode is the most frequent value.

see
https://en.wikipedia.org/wiki/Median
https://en.wikipedia.org/wiki/Mode_(statistics)

You can find the mode by sorting the histogram and taking the most populous value.

In unix:

Code: Select all

convert image -format "%c" histogram:info:  | sort -r -k1 | head -n 1

[snibgo]

The mode is the most frequent value.

Bother, yes, of course you are correct. Thanks. Forget what I wrote above. (Statistics was never my strong subject.)

A text histogram can be read in a script to find the most common value, aka the mode.

An image histogram (not cumulated) can also be used, by finding the location of the largest count. My process module "onelightest" can do that.

[geoland]

Thanks again. I think this can be made to work.

Something like this in my bash script

Code: Select all

mode=$(convert image -format "%c" histogram:info:  | sort -r -k1 | head -n 1)

convert master_flat -format "%[fx:$mode]" info: | something here to pipe result

I suspect this is not quite correct. It would be necessary to pipe the final image result - the corrected master_flat?

[fmw42]

Your mode in the first line is a variable, though you need to extract the mode value using the bash cut function to get the value from the text line and reformat it to the desired range of values you want (e.g. 0 to 255 or 0 to 1) or as a color from the last entry on the line. No need to pipe it. You can use the variable in any IM command or use fx to compute some other factor from the mode, such as a normalization gain.

[geoland]

This is a work around, for the time being, which may satisfy the need. Pixel values and appearance indicates its not far off.

This median combines the flats for noise reduction and auto-levels the master_flat, which produces a nice clean division of the light frames. Problem is it also creates very low pixel values.

Code: Select all

convert *.tiff -evaluate-sequence median -auto-level master_flat.tiff

This normalizes the light frames following flat division, which raises the pixel values to a normal level, without blowing out bright areas such as stars and bright nebula.

Code: Select all

light -compose Divide_Src -composite -normalize

[fmw42]

The median you have used should not produce dark images. The auto-level should stretch them to full dynamic range. If you have one bright outlier pixel, then that could prevent auto-level from stretching to full dynamic range.

However, my guess is that your IM 6.7.7 may be at the time when grayscale images were treated as linear. You might try upgrading IM to the current version.

Alternately, try

Code: Select all

convert *.tiff -colorspace RGB -evaluate-sequence median -auto-level master_flat.tiff

or

Code: Select all

convert *.tiff -colorspace sRGB -evaluate-sequence median -auto-level master_flat.tiff

And see if that helps.

You could also use -contrast-stretch to clip the bright outlier, if there is one. Such as

Code: Select all

convert *.tiff -evaluate-sequence median -contrast-stretch 0x1% master_flat.tiff

Change the 1% as desired to higher or lower values

[geoland]

Apologies. The value of the light frame pixels is reduced when dividing the lights by the auto-leveled master flat. I am then normalizing the lights to increase the pixel values.

I will try the other operators - so far median and auto-level produce flat frames that divide correctly and don't over or under compensate.

There must be a repository with the latest deb package.