Filtering Astronomical Images

[geoland]

Does the average exclueing min/max routine clip min/max values or reassign new mean values to those pixels.

Similarly, would a winsorizing routine be possible with IM. Winsorizing is perhaps a better choice for dealing with outliers in astronomical images.

[snibgo]

On Winsorizing:

We can readily set all pixels below (or above) a given value to that value. Just use "-evaluate max" (or "-evaluate min").

Code: Select all

convert in.png -evaluate max 33% out.png

How do we find what value to use as the threshold? You can do this in a script that reads the text histogram. Or make the histogram as a Nx1 image, and invert that histogram, and find the value at x=10% of the width:

Code: Select all

%IMDEV%convert toes_bw.png -process "mkhisto cumul norm" -process "mkhisto cumul norm" -crop 10%x0+0+0 +repage -gravity East -crop 1x1+0+0 +repage txt:

# ImageMagick pixel enumeration: 1,1,4294967295,gray
0,0: (1.41944e+09,1.41944e+09,1.41944e+09)  #549B549B549B  gray(33.049%)

This tells us that 10% of the pixel values are below 33% gray.

[fmw42]

Does the average exclueing min/max routine clip min/max values or reassign new mean values to those pixels.

It forms the average after throwing out the min and max values. So it averages all the remaining values excluding the min and max.

[fmw42]

-contrast-stretch x%,x% will clip x% off each end stretch the data to full dynamic range and replace the clipped values with min and max. I do not know if you want the stretch.

-black-threshold x% will replace all values below x in the range of 0 to 100% graylevel (not percent pixels) with black. Similarly -white-threshold x% will repllace all value above x with white. They do similar to what snibgo has for -evaluate max and -evaluate min.

As far as I know, IM does not have a specific Windsoring function that replace percentage of pixels directly.

[geoland]

I have tried the methods suggested and this has helped clarify what I am trying to achieve.

Hot pixels cosmic ray hits and other unwanted signal is characterized by a sudden change in contrast and/or brightness, whereas the transition around stars is gradual.

I need to find the offending pixels and average its value based on the surrounding pixels. From what I have read the average value may be determined by a ring around the offending pixel to act only upon that pixel.

That is the best explanation I can come up with

Similarly, the same could be applied to very dark pixels - which may be underexposed.

Am I making sense?

Not sure how to do this...

[fmw42]

I am not sure what you want. Possibly see statsfilt option statistic=14: average (arithmetic mean) without the center pixel. Otherwise, perhaps upload an image to free hosting service and then put the URL here. Then explain what the issue is relative to your image.

[snibgo]

There seem to be two problems:
(1) Identify "hot" and "cold" pixels.
(2) Decide what to do about them.

Provided these can be answered, it can be done in IM. For example, a possible definition of "hot" is: "a pixel that is above 95% of maximum, where the mean of a window of 9 pixels (itself and the surrounding 8 neighbours) are below 75% of maximum."

Or, more simply, "a pixel that is above the mean of 9 pixels". (But this will include images of stars.)

Or, "a pixel that is lighter than any of its 8 neighbours".

Or, "a pixel that is more than 40% lighter than any of its 8 neighbours".

Obviously percentages and window sizes can be changed.

Next problem: what to do with each hot pixel? "Make it the maximum value of its 8 neighbours." Or: "Make it the mean of the 3x3 window", or ...

The "cold" pixels, likewise.

[geoland]

The scripts produce different effects and several produce very good noise reduction. I think the solution I am looking for is a pixel comparison and add or subtract from the pixel as required - defined by surrounding pixels of 'normal' exposure values. Normal meaning they could be any value but representative of the local area.

There seem to be two problems:
(1) Identify "hot" and "cold" pixels.
(2) Decide what to do about them.

Provided these can be answered, it can be done in IM. For example, a possible definition of "hot" is: "a pixel that is above 95% of maximum, where the mean of a window of 9 pixels (itself and the surrounding 8 neighbours) are below 75% of maximum."

Or, more simply, "a pixel that is above the mean of 9 pixels". (But this will include images of stars.)

Or, "a pixel that is lighter than any of its 8 neighbours".

Or, "a pixel that is more than 40% lighter than any of its 8 neighbours".

Obviously percentages and window sizes can be changed.

Next problem: what to do with each hot pixel? "Make it the maximum value of its 8 neighbours." Or: "Make it the mean of the 3x3 window", or ...

The "cold" pixels, likewise.

This is what I had in mind with stars to be avoided, which narrows the hot pixel definition down. Similarly, a value for under exposed pixels with a comparison to surrounding pixels.

Make the hot or under exposed pixel the mean or median of the 3x3 window and similarly the under exposed or cold pixel. This is probably a good place to start.

Test image is a crop of the R channel near the top left corner of an aps-c size frame. It features some hot and cold/underexposed pixels and a satellite trail. I double sized the image prior to cropping in display. The satellite trail should be a good test as it is not as bright as the stars. Neighbouring contrast is probably the test criteria for each pixel in the trail in this case.

Should I start a new thread for this as a question quite apart from my query about scripts. On the other hand, a script of this type would be very very useful. I use Linux BTW.

[fmw42]

try converting anything less than 50% brightness to full black. Adjust the 50% as desired

Code: Select all

convert RCrop.tiff -black-threshold 50% RCrop_bt50.tiff

Does that do what you want?

[fmw42]

I have moved this to the User's forum for you, since it is not directly related to my scripts, but is more general.

[geoland]

try converting anything less than 50% brightness to full black. Adjust the 50% as desired

Code: Select all

convert RCrop.tiff -black-threshold 50% RCrop_bt50.tiff

Does that do what you want?

To retain image features I need to do something like this, it's partly effective, but all over the image rather than to an isolated pixel

Code: Select all

convert RCrop.tiff -black-threshold 1% -white-threshold 95% RCrop_btwt.tiff

As snigbo suggested

Code: Select all

1. find all hot and cold pixels based on extreme contrast change to neighbouring pixels
2. get the mean/median of pixels neighbouring the hot or cold pixel
3. set the hot or cold pixel to the mean/median of the neighbouring pixels

I can't see another way of dealing with isolated pixels.

[fmw42]

You do not have isolated hot or cold pixels that I can see. All of them are larger than 1 pixel. So can you explain how to distinguish hot and cold "regions", since the do not have single pixels but extend in many case many pixels.

[geoland]

I have not gone to any length to correct the terminology used so far. So this is probably a good time to reiterate my second post.

Hot pixels cosmic ray hits and other unwanted signal is characterized by a sudden change in contrast and/or brightness, whereas the transition around stars is gradual.

I need to find the offending pixels and average its value based on the surrounding pixels. From what I have read the average value may be determined by a ring around the offending pixel to act only upon that pixel. (I could have worded this better)

The image crop in question has a number of issues; it is underexposed with a predominance of black pixels in groups. These are not necessarily cold pixels. The image from which the crop was taken contains lots of hot and cold single pixels - perhaps the cropped region does not contain any hot pixels.

The groups of bright pixels are likely cosmic ray hits. Unlike stars they have no obvious point spread function (PSF) and the change in pixel value compared to the surrounding signal is abrupt. As a rule, astronomical images while displaying large dynamic range, transition gradually between light and dark, even around stars.

A sudden change in brightness or darkness usually indicates an artefact of some sort. For example, the satellite trail is not as bright as the larger stars, but the transition from one value to another is abrupt as the trail cuts through the background. I'm not too concerned about the satellite trails at the moment.

Finding the places of abrupt change in pixel values is the key to defining artefacts. Getting the mean or median of the background in the region of the offending pixel to then replace the offending pixels value with the mean or median of the neighbouring pixels.

In a nutshell, I need to find abrupt transitions in pixel values - light or dark - and replace the offending pixel/s value with a value representative of the region in which the offending pixel/s is located.

Most of the hot pixels are removed with dark subtraction, but not all and this is just a clean up of all the sub images prior to aligning and stacking.

Perhaps I need to use another program to do this, but if it can be done in IM then that would be excellent.

[fmw42]

There is a technique for finding and "removing" isolated point pixels surrounded by dark. It is called isonoise, It computes the median those pixels that it deems to be single pixels and replaces that with the the median values. Other pixels it leaves alone. You might modify it for those special "hot" pixels. See my script isonoise at the link below or at http://www.fmwconcepts.com/imagemagick/ ... /index.php.

What characterizes "cold" pixels - just very dark isolated vasues?

[snibgo]

A sudden change in brightness or darkness usually indicates an artefact of some sort.

That makes sense. But looking at the magnitude of the slopes in your supplied image, I can't see any changes in brightness.