How to fix extreme distortion of grayscale range

[briligg]

This is a section that is cut from the middle of a file that is an elevation map of the moon's surface. The original is a 16-bit TIFF too large for me to open. I have also reduced the file to 25% its original size, and 50% original size with conversion to jpg. All three resulting images show the grayscale shift seen in the screenshot below.



It's like the histogram wraps around the ends of the level range. The program it's open in is the new GIMP 2.1.

The original is a GeoTIFF file formatted for opening on USGS software, such as Isis. I am processing it for use in Blender, to create an accurate and detailed model of the moon. Is it possible that it has an embedded grayscale profile that isn't compatible with standard graphics programs? And could that profile be changed?

I have also been looking at the evaluate operators - https://www.imagemagick.org/script/comm ... p#evaluate - searching for a way to correct the distortion here. There may be an operator specifically for this, and I just can't pick it out as I don't recognize the term. I've scanned through a lot of the convert arguments and don't see one.

If there isn't, then adding to the value of pixels at the bottom end, and subtracting from the value of pixels at the top end would achieve the desired result. The issue is I don't know how to assess how much to add, and how much to subtract. I have been looking for a way to determine the values of tones in the image with no luck.

So here's hoping there is a simple solution for this problem. My machine runs Ubuntu 18.04 and the Image Magick version is 6.9.7-4 Q16.

[fmw42]

It always helps to provide your platform and Imagemagick version. Also can you not crop a piece of our TiFF file and provide that or a smaller image assuming it has the same "distortions" in gray level.

The issue may be the format of the TIFF file and how it is being read. Without more information, it is hard to know what to suggest.

[briligg]

Sorry, forgot to do that.

The image in the first post shows a large section of the image, behind the histogram dialog box. The original image is also 16-bit grayscale - the file shown is just a section extracted from the middle of it, so the colors should not have been touched. I have tried to look for embedded information about color space, but haven't found anything. The identify command returns

Code: Select all

Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.tif TIFF64 92160x46080 92160x46080+0+0 16-bit Grayscale Gray 8.4942GB 0.000u 0:00.099

[snibgo]

I would expect that NASA or USGS or someone has written documentation that describes the data. The moon has no "sea level", but perhaps there is an equivalent, and terrain (umm, lunain?) above that has positive height but terrain below it has negative height, which comes out in unsigned numbers as values near the top of the range.

This would mean that adding 50% modulus QuantumRange would do the trick, so "sea level" becomes 50%: "-evaluate AddModulus 50%".

[briligg]

The page on USGS with the file didn't have anything that helped. It's at https://astrogeology.usgs.gov/search/ma ... 8m_Mar2014 if you'd like to take a look. The software in which these files are normally loaded, Isis, has a support center. I ended up there once before, and i can say it's a really technical set of people and it is best to be very prepared if you expect to get help from them. They don't have time for noobs experimenting with their data for their own amusement.

If i can't find a way to resolve this, i will try talking to them. I'm going to try the approach you suggest now.

[briligg]

That really looks right. It's just a small part in the middle though. Now i'll try it on the whole thing.

Edit: for a reference image that is proper 16 bit PNG, an image that is reduced to 4096 x 2048 can be downloaded at the link (12 MB):
https://www.moonwards.com/project.html#assets

[fmw42]

If you use IM 7, which has HDRI enabled, or recompile IM 6 with HDRI enabled, then do in IM 7 the following. If IM 6 with HDRI, remove the magick from magick identify and change magick to convert for the second command line.

Code: Select all

magick identify -verbose your.tiff

or perhaps

Code: Select all

magick your.tiff -define quantum:format=floating-point -verbose info:

You should be able to see if it has negative values.

[snibgo]

From the linked site:

Thus elevation values are the distance above or below the reference sphere.

As I suspected, but "reference sphere" is far more appropriate than "sea level".

That really looks right.

Yes, that looks good. We can see more detail with...

Code: Select all

convert jSMMidd.png -gravity Center -crop 90%x90%+0+0 +repage -auto-level x.png

... and there are no obvious discontinuities or other problems.

[fmw42]

I believe that -auto-level will stretch the most negative value to 0 and the max positive value to your quantum level (Q16). So if the values are important, your are biasing and stretching your data and the values will be misleading. If you only care about not seeing wrap around of gray levels, then that is fine. Also you might have taken a crop in which you have no negative values.

[snibgo]

Yes, certainly. I suggested "-auto-level" merely to provide a visual check.

The area shown, centrally cropped to 90%, has values from 40% to 57%, with a mean of 48%. Seems reasonable. It is just 8-bit, so precision has been lost somewhere.

[briligg]

@snibgo - it's a good point, about the reference sphere. I wasn't at all sure if that would translate into the grayscale in the same way, or if it would be a translation of that into how grayscales work.

But according to GIMP, the modulated image is still a 16-bit image. I"m not happy about the bands in the histogram showing unused tones, but that was in the original too. Unless cropping changed the color range, which i assume it wouldn't.

I have installed IM 7.0.7, but running "make check" returned two fails. I am going to ask about that separately, as that might come up for others and should be made easy to find. I'm also going to report it on GitHub as requested.

I also need to increase the size of my root partition. Asking IM to do the same evaluation on the whole original .tif failed twice yesterday with the error "Maximum TIFF file size exceeded" again. I am guessing it once again ran out of memory. So hopefully using 7.0.7 and giving it more space will allow it to complete the task. If not i guess i'll need to do it as tiles. I have to do that anyhow, but somehow i'd like to have the whole thing in one piece - if only to offer it in my repo to others. (That's one of the great things about NASA, you can do that.)

[fmw42]

"Maximum TIFF file size exceeded" may mean that you need to install the bigtiff delegate. See http://bigtiff.org

[snibgo]

Some tips for images that are too large for memory:

1. If you need to do multiple operations on the image, save it as .mpc and work on that. Mpc files are on disk but memory-mapped to memory, so IM doesn't try to read the entire image and store the pixel cache in memory only to run out of memory so it stores the pixel cache on disk.

2. Break the image into tiles, possible overlapping tiles for operations like blur and sharpen. When you are ready to glue the tiles together into a tiff file, first write each file as MPC, then glue these together.

3. When messing around, or developing scripts for the above, do this at reduced scale, eg 25% or 10%, whatever fits in memory. When it does what you want, run the scripts while you have lunch/dinner/sleep/holiday.

4. VIPS is an alternative to IM, specifically designed to handle large images without eating memory or needing massive time with disk access. However, it has a steep learning curve -- even steeper than IM.

[briligg]

"Maximum TIFF file size exceeded" may mean that you need to install the bigtiff delegate.

I started to do that before but left it, as it seemed the library I have, libtiff5, could handle large files too. I didn't know where to put it to install it, and I couldn't find any instructions on the web page.

I have downloaded it again and unzipped it to my home directory. Should I run 'make' there?

[fmw42]

I believe it needs to be installed in the same path as where you have ImageMagick. Then you would likely need to recompile ImageMagick. Sorry, I really do not know much Linux installs of ImageMagick. I usually install all my delegates from MacPorts and then install ImageMagick from source.