Legacy ImageMagick Discussions Archive

IM 6.7.1.6 Q32 HDRI Mac OSX Tiger.

I am getting some weird results from simple conversions of an HDR format image. See image lampickaHDR.hdr which can be downloaded at http://dcgi.felk.cvut.cz/home/cadikm/tmo/hdr_images/.

Note in the original, the max values are 10^13 very large values


identify -version lampickaHDR.hdr

Image: lampickaHDR.hdr
Format: HDR (Radiance RGBE image format)
Class: DirectClass
Geometry: 1600x1200+0+0
Resolution: 72x72
Print size: 22.2222x16.6667
Units: Undefined
Type: TrueColor
Endianess: Undefined
Colorspace: XYZ
Depth: 32/25-bit
Channel depth:
red: 25-bit
green: 25-bit
blue: 25-bit
Channel statistics:
Red:
min: 0 (0)
max: 7.00939e+13 (16320)
mean: 5.0654e+10 (11.7938)
standard deviation: 3.454e+11 (80.4197)
kurtosis: 4446.22
skewness: 36.2988
Green:
min: 0 (0)
max: 6.51461e+13 (15168)
mean: 4.39835e+10 (10.2407)
standard deviation: 3.30074e+11 (76.8513)
kurtosis: 4349.06
skewness: 38.34
Blue:
min: 0 (0)
max: 3.47721e+13 (8096)
mean: 3.37688e+10 (7.86241)
standard deviation: 2.44982e+11 (57.0393)
kurtosis: 2111.01
skewness: 27.6055

If I now simply resize, I get much smaller max values, 10^5 (very small values)

convert lampickaHDR.hdr -resize 500x500 lampickaHDR_small.hdr

identify -version lampickaHDR_small.hdr
Image: lampickaHDR_small.hdr
Format: HDR (Radiance RGBE image format)
Class: DirectClass
Geometry: 500x375+0+0
Resolution: 72x72
Print size: 6.94444x5.20833
Units: Undefined
Type: TrueColor
Endianess: Undefined
Colorspace: XYZ
Depth: 32-bit
Channel depth:
red: 29-bit
green: 32-bit
blue: 31-bit
Channel statistics:
Red:
min: 0 (0)
max: 211968 (4.93526e-05)
mean: 74176 (1.72704e-05)
standard deviation: 30926.6 (7.20067e-06)
kurtosis: 0.098061
skewness: -0.51046
Green:
min: 0 (0)
max: 125440 (2.92063e-05)
mean: 33552.2 (7.81198e-06)
standard deviation: 24600.4 (5.72773e-06)
kurtosis: -1.73878
skewness: 0.0915995
Blue:
min: 0 (0)
max: 72704 (1.69277e-05)
mean: 31418.6 (7.31521e-06)
standard deviation: 21467.4 (4.99827e-06)
kurtosis: -1.58378
skewness: 0.284789


Similarly, if I just use -auto-level, I would have expected the max to be much larger, full 32-bit range, but again very small values.

convert lampickaHDR.hdr -auto-level lampickaHDR_al.hdr

identify -version lampickaHDR_al.hdr
Image: lampickaHDR_al.hdr
Format: HDR (Radiance RGBE image format)
Class: DirectClass
Geometry: 1600x1200+0+0
Resolution: 72x72
Print size: 22.2222x16.6667
Units: Undefined
Type: TrueColor
Endianess: Undefined
Colorspace: XYZ
Depth: 32-bit
Channel depth:
red: 32-bit
green: 32-bit
blue: 32-bit
Channel statistics:
Red:
min: 0 (0)
max: 143360 (3.33786e-05)
mean: 74.7691 (1.74085e-08)
standard deviation: 507.317 (1.18119e-07)
kurtosis: 13216.5
skewness: 65.3044
Green:
min: 0 (0)
max: 115200 (2.68221e-05)
mean: 32.5216 (7.57203e-09)
standard deviation: 314.402 (7.32024e-08)
kurtosis: 32971.5
skewness: 117.476
Blue:
min: 0 (0)
max: 32768 (7.62939e-06)
mean: 27.4133 (6.38266e-09)
standard deviation: 198.483 (4.62129e-08)
kurtosis: 2918.72
skewness: 30.7014

Any one able to reproduce this problem? Or am I misunderstanding something here about IM in Q32 HDRI, especially about max (32-bit) values when using -auto-level?

Sure it is Q32? I thought all HDRI versions of IM are Q16!

Also as internally the values are floating point, I would thing that the coder could save 32bit integers (is that what /hdr fileformat needs?) using the appropriate math, using a float to int32 conversion. Actually it also depends on the byte order MSB or LSB! when saving integers.

So it comes does to what does the .hdr files save format need, and is the 'hdr' coder doing it right when in HDRI mode!

It was definitely Q32 at the time. But explain why -auto-level reduces the dynamic range rather than stretch it to full Q32 float values? Is this a problem with -verbose info: or a real problem with HDRI mode and this image format?


Here is the verbose info change in Q16 HDRI. See the max values under Q32 from before at about 10^13. But under Q16 HDRI, I get only about 10^9 from the same downloaded image. Why is that? It appears that the dynamic range has changed under Q16 HDR vs Q32 HDRI!

Image: lampickaHDR.hdr
Format: HDR (Radiance RGBE image format)
Class: DirectClass
Geometry: 1600x1200+0+0
Resolution: 72x72
Print size: 22.2222x16.6667
Units: Undefined
Type: TrueColor
Endianess: Undefined
Colorspace: XYZ
Depth: 16-bit
Channel depth:
red: 16-bit
green: 16-bit
blue: 16-bit
Channel statistics:
Red:
min: 0 (0)
max: 1.06953e+09 (16320)
mean: 772907 (11.7938)
standard deviation: 5.27031e+06 (80.4197)
kurtosis: 4446.22
skewness: 36.2988
Green:
min: 0 (0)
max: 9.94035e+08 (15168)
mean: 671125 (10.2407)
standard deviation: 5.03645e+06 (76.8513)
kurtosis: 4349.06
skewness: 38.34
Blue:
min: 0 (0)
max: 5.30571e+08 (8096)
mean: 515263 (7.86241)
standard deviation: 3.73807e+06 (57.0393)
kurtosis: 2111.01
skewness: 27.6055


Furthermore, why does -auto-level not change the values properly (as seen by -verbose info:). Why does it reduce the range and not increase the range to full Q32 range values. It does not even seem to work right under Q16 as below.



convert lampickaHDR.hdr -auto-level lampickaHDR_al16.hdr

verbose info:


Image: lampickaHDR_al16.hdr
Format: HDR (Radiance RGBE image format)
Class: DirectClass
Geometry: 1600x1200+0+0
Resolution: 72x72
Print size: 22.2222x16.6667
Units: Undefined
Type: TrueColor
Endianess: Undefined
Colorspace: XYZ
Depth: 16-bit
Channel depth:
red: 16-bit
green: 16-bit
blue: 16-bit
Channel statistics:
Red:
min: 0 (0)
max: 65535 (1)
mean: 74.5745 (0.00113793)
standard deviation: 469.414 (0.0071628)
kurtosis: 2469.85
skewness: 29.3084
Green:
min: 0 (0)
max: 65535 (1)
mean: 32.4409 (0.000495017)
standard deviation: 291.558 (0.00444889)
kurtosis: 14099.8
skewness: 73.5687
Blue:
min: 0 (0)
max: 32767.5 (0.5)
mean: 27.4138 (0.000418308)
standard deviation: 198.501 (0.00302893)
kurtosis: 2919.22
skewness: 30.7109

Note, that I have never needed to build a Q32 version of IM, though I understand how it works. I did not think Q32 HDRI was possible, except by changing header files before compiling.

The internal Q level should have no bearing on the accuracy of HDRI IM. Coders should be taking Q level into account, though it may use it for image file formats that allow you to write 32bit integer values (I don't know of any that do, except posibly MIFF).

HDRI image file formats should have must higher accuracy that normal image file formats, of course. Typically by allowing you to save floating point values (single or double floats). I don't know how HDR image file format saves its values, and you enlighten?

fmw42 wrote: Then why does the verbose info change when in Q16 HDRI vs Q32 HDRI. See the max values under Q32 from before at about 10^13. But under Q16 HDRI, I get only about 10^9 from the same downloaded image. Why is that? It appears that the dynamic range has changed under Q16 HDR vs Q32 HDRI!

Channel statistics:
Red:
min: 0 (0)
max: 1.06953e+09 (16320)
mean: 772907 (11.7938)
standard deviation: 5.27031e+06 (80.4197)
kurtosis: 4446.22
skewness: 36.2988

Of course the first value changes. That is the value in the 0 to QuantumRange range.
from Q16 to Q32 that value would have been multiplied by 65536. The Q level will influence the internal meaning of QuantumRange and thus black (0) to maximum for red (2^Q-1) As such 10^13 verse 10^9 is about 5 orders of magnitude or about 65536!

The internal representation is like that as some functions (such as -expression add, -level with raw values, not percentages etc.) are still dependant on the Q value.

The value that does not depend on the internal Q is the one in parenthesis! It is supposed to represent the color values 'normalized' value! That is 0.0=black 1.0=white

The fact that max (for red) is larger tha 1.0 indicates to me that the color values are really HDRI and you have values that are far in advance of normal. Something that is not possible to get in non-HDRI integer based Imagemagick.

However the actual value of 16320 (roughtly half of 2^16 or 65536) seems to suggest that the image read is not quite right. It seems to be a liitle to strong to be just a HDRI fault, but a coder read fault.

NOTE: all -auto-level does is multiple the values (all values equally unless channel sync flag is removed) so that largest value becomes equal to the QuantumRange, and thus the normalized value in parenthesis becomes 1...

And that is exactly what happened! the values were multiplied by 1/16320

Channel statistics:
Red:
min: 0 (0)
max: 65535 (1)
mean: 74.5745 (0.00113793)
standard deviation: 469.414 (0.0071628)
kurtosis: 2469.85
skewness: 29.3084

There is nothing magical going on. just the understanding of the internal representation.

The Q level does not effect overall accuracy in HDRI, It is just an internal multiplier of the stored floating point values (that you can see in the output), so algorithms designed for integers continue to work as expected. You could have a Q value of 1.0 for the differences it 'should' make, though some integer based algorithms could go screwy (which operators are effected I have no idea, though most should work as expected. Failures for Q level is probably more likely within coders more than core image operations.

Anthony,

Please relook at my original post. Compare the original min,max,mean with that at the end after doing an auto-level. Why are the values after the auto-level compressed and not expanded?


Furthermore, why do I get the following by simply copying an .hdr image under Q16 HDRI:

Original verbose info:
Image: doll.hdr
Format: HDR (Radiance RGBE image format)
Class: DirectClass
Geometry: 922x901+0+0
Resolution: 72x72
Print size: 12.8056x12.5139
Units: Undefined
Type: TrueColor
Endianess: Undefined
Colorspace: XYZ
Depth: 16-bit
Channel depth:
red: 16-bit
green: 16-bit
blue: 16-bit
Channel statistics:
Red:
min: 0 (0)
max: 60159.1 (0.917969)
mean: 1493.18 (0.0227845)
standard deviation: 4438.41 (0.0677258)
kurtosis: 31.8239
skewness: 4.90443
Green:
min: 3.96869 (6.05583e-05)
max: 60159.1 (0.917969)
mean: 1378.22 (0.0210303)
standard deviation: 4346.93 (0.06633)
kurtosis: 35.4432
skewness: 5.21291
Blue:
min: 0 (0)
max: 60159.1 (0.917969)
mean: 1229.22 (0.0187567)
standard deviation: 4016.41 (0.0612865)
kurtosis: 41.7148
skewness: 5.58878


convert doll.hdr doll2.hdr
verbose info:

Image: doll2.hdr
Format: HDR (Radiance RGBE image format)
Class: DirectClass
Geometry: 922x901+0+0
Resolution: 72x72
Print size: 12.8056x12.5139
Units: Undefined
Type: TrueColor
Endianess: Undefined
Colorspace: XYZ
Depth: 16-bit
Channel depth:
red: 16-bit
green: 16-bit
blue: 16-bit
Channel statistics:
Red:
min: 0 (0)
max: 65535 (1)
mean: 2115.3 (0.0322775)
standard deviation: 6157.04 (0.0939504)
kurtosis: 29.095
skewness: 4.81494
Green:
min: 0 (0)
max: 57599.1 (0.878906)
mean: 1199.79 (0.0183076)
standard deviation: 4173.58 (0.0636847)
kurtosis: 38.8376
skewness: 5.57164
Blue:
min: 0 (0)
max: 54271.2 (0.828125)
mean: 1094.82 (0.0167059)
standard deviation: 3591.64 (0.0548048)
kurtosis: 42.5136
skewness: 5.6316


Why the changes in the RGB min and max values? The original max values were all the same, but after the copy, the are not the same any more so a color shift

For anyone following this thread, here is another interesting bug/feature

In Q16 HDRI (see http://www.fmwconcepts.com/imagemagick/ ... 40x480.hdr)


identify -verbose IMG_3133And8more_640x480.hdr
Image: IMG_3133And8more_640x480.hdr
Format: HDR (Radiance RGBE image format)
Class: DirectClass
Geometry: 640x480+0+0
Resolution: 72x72
Print size: 8.88889x6.66667
Units: Undefined
Type: TrueColor
Endianess: Undefined
Colorspace: XYZ
Depth: 16-bit
Channel depth:
red: 16-bit
green: 16-bit
blue: 16-bit
Channel statistics:
Red:
min: 49.7492 (0.000759125)
max: 401402 (6.125)
mean: 12219.9 (0.186463)
standard deviation: 17125.5 (0.261319)
kurtosis: 3.12344
skewness: 1.46073
Green:
min: 37.4994 (0.000572205)
max: 188413 (2.875)
mean: 12784.1 (0.195073)
standard deviation: 17426.2 (0.265906)
kurtosis: 0.294483
skewness: 1.123
Blue:
min: 38.9994 (0.000595093)
max: 132094 (2.01562)
mean: 12541.7 (0.191374)
standard deviation: 16801.1 (0.256368)
kurtosis: -0.417009
skewness: 0.974638
Image statistics:
Overall:
min: 37.4994 (0.000572205)
max: 401402 (6.125)
mean: 12515.2 (0.19097)
standard deviation: 17119.5 (0.261227)
kurtosis: 1.0159
skewness: 1.18982
Rendering intent: Undefined
Interlace: None
Background color: xyz(255,255,255)
Border color: xyz(223,223,223)
Matte color: xyz(189,189,189)
Transparent color: xyz(0,0,0)
Compose: Over
Page geometry: 640x480+0+0
Dispose: Undefined
Iterations: 0
Compression: RLE
Orientation: Undefined
Properties:
comment: ?RADIANCE
date:create: 2011-08-15T13:02:52-07:00
date:modify: 2011-07-15T13:55:50-07:00
hdr:APERTURE: 2.78
hdr:CAMERA: Canon PowerShot SD1000
hdr:CAPDATE: 2011:05:08 17:00:41
hdr:FOCAL_LENGTH: 5.80
hdr:ISO: 80
hdr:SHUTTER: 0.016669
signature: 88c6d843d1a330aeca5bf6353a96ce47683004168e9ccd541f21687956c52ddc
Artifacts:
verbose: true
Tainted: False
Filesize: 992KBB
Number pixels: 307KB
Pixels per second: 1.463MB
User time: 0.130u
Elapsed time: 0:01.210
Version: ImageMagick 6.7.1-7 2011-08-17 Q16 http://www.imagemagick.org


Just a copy (no -auto-level):

convert IMG_3133And8more_640x480.hdr IMG_3133And8more_640x480_B.hdr
identify -verbose IMG_3133And8more_640x480_B.hdr
Image: IMG_3133And8more_640x480_B.hdr
Format: HDR (Radiance RGBE image format)
Class: DirectClass
Geometry: 640x480+0+0
Resolution: 72x72
Print size: 8.88889x6.66667
Units: Undefined
Type: TrueColor
Endianess: Undefined
Colorspace: XYZ
Depth: 16-bit
Channel depth:
red: 16-bit
green: 16-bit
blue: 16-bit
Channel statistics:
Red:
min: 0 (0)
max: 65535 (1)
mean: 12930.3 (0.197303)
standard deviation: 18749 (0.286092)
kurtosis: 0.970188
skewness: 1.46863
Green:
min: 0 (0)
max: 65535 (1)
mean: 12613.2 (0.192465)
standard deviation: 17024.6 (0.259779)
kurtosis: -0.513628
skewness: 0.961824
Blue:
min: 0 (0)
max: 60159.1 (0.917969)
mean: 11274 (0.172031)
standard deviation: 15055.6 (0.229734)
kurtosis: -0.599246
skewness: 0.94293
Image statistics:
Overall:
min: 0 (0)
max: 65535 (1)
mean: 12272.5 (0.187267)
standard deviation: 17010.2 (0.259558)
kurtosis: 0.361873
skewness: 1.21592
Rendering intent: Undefined
Interlace: None
Background color: xyz(255,255,255)
Border color: xyz(223,223,223)
Matte color: xyz(189,189,189)
Transparent color: xyz(0,0,0)
Compose: Over
Page geometry: 640x480+0+0
Dispose: Undefined
Iterations: 0
Compression: RLE
Orientation: Undefined
Properties:
comment: ?RADIANCE
date:create: 2011-08-18T18:30:42-07:00
date:modify: 2011-08-18T18:30:42-07:00
signature: 554f5a402378abc4476a50174d6d947d9c7fd3ab179d0ca7d9086590947edcd7
Artifacts:
verbose: true
Tainted: False
Filesize: 994KBB
Number pixels: 307KB
Pixels per second: 2.363MB
User time: 0.120u
Elapsed time: 0:01.130
Version: ImageMagick 6.7.1-7 2011-08-17 Q16 http://www.imagemagick.org
freds-mac-mini:~ fred$


So in Q16 HDRI, the convert is doing an automatic auto-level but the values are not correct (in proportion).


BUT if I do an IM -auto-level, I get different values, which appear more correct in proportion with the originals:

convert IMG_3133And8more_640x480.hdr -auto-level IMG_3133And8more_640x480_C.hdr
identify -verbose IMG_3133And8more_640x480_C.hdr
Image: IMG_3133And8more_640x480_C.hdr
Format: HDR (Radiance RGBE image format)
Class: DirectClass
Geometry: 640x480+0+0
Resolution: 72x72
Print size: 8.88889x6.66667
Units: Undefined
Type: TrueColor
Endianess: Undefined
Colorspace: XYZ
Depth: 16-bit
Channel depth:
red: 16-bit
green: 16-bit
blue: 16-bit
Channel statistics:
Red:
min: 0 (0)
max: 65535 (1)
mean: 2221.1 (0.0338918)
standard deviation: 3522.21 (0.0537456)
kurtosis: 10.4084
skewness: 2.45567
Green:
min: 0 (0)
max: 24831.6 (0.378906)
mean: 2055.66 (0.0313673)
standard deviation: 2792.49 (0.0426106)
kurtosis: -0.167808
skewness: 1.01393
Blue:
min: 0 (0)
max: 21503.7 (0.328125)
mean: 1837.47 (0.0280379)
standard deviation: 2465.72 (0.0376245)
kurtosis: -0.456459
skewness: 0.965185
Image statistics:
Overall:
min: 0 (0)
max: 65535 (1)
mean: 2038.07 (0.031099)
standard deviation: 2959.94 (0.0451658)
kurtosis: 7.43253
skewness: 1.89483
Rendering intent: Undefined
Interlace: None
Background color: xyz(255,255,255)
Border color: xyz(223,223,223)
Matte color: xyz(189,189,189)
Transparent color: xyz(0,0,0)
Compose: Over
Page geometry: 640x480+0+0
Dispose: Undefined
Iterations: 0
Compression: RLE
Orientation: Undefined
Properties:
comment: ?RADIANCE
date:create: 2011-08-18T18:37:53-07:00
date:modify: 2011-08-18T18:37:53-07:00
signature: 4c0d8aa160a617f7d1dd165d3a70db6f7ea7d3c2ae61e15e1fcb7559e87a575d
Artifacts:
verbose: true
Tainted: False
Filesize: 1.003MBB
Number pixels: 307KB
Pixels per second: 2.363MB
User time: 0.120u
Elapsed time: 0:01.129
Version: ImageMagick 6.7.1-7 2011-08-17 Q16 http://www.imagemagick.org

Perhaps part of the issue may be that IM supports .hdr in RGBE format. But the images that I have been testing in IM have been .hdr in XYZE format. Perhaps IM cannot properly handle the XYZE format?

looks like I can convert from .hdri to .exr in Q16 HDRI without any color shift.

convert doll.hdr doll.exr

the resulting doll.exr even after using -auto-level does not show the same color shifts that using -auto-level on doll.hdr

Of course for very high dynamic ranges that would require Q32 HDRI, this would not work as IM only handles "half" or 16-bit float EXR format.