14bit Camera raw to 16bit tiff Logluv HDR

[yellow]

Hi

Is it possible using a hdri build of IM, Q32 if necessary Q16 otherwise to 'develop' camera raw format supported by dcraw, libraw and rawspeed and write 16bit Logluv tiffs, reason is to retain, hate to use this tired and overused phrase but retain the 'dynamic range' of a raw file but bake debayer, WB into a 16bit non linear format to keep benefit of a 32bit depth hdr format but keep file sizes down and options open.

If its not directly possible can assuming I can use standard out from libraw which offers 32bit depth output can IM do the rest or alternatively a lin to log into 16bit tiff otherwise.

Cheers

[snibgo]

According to Wikipedia, LogLuv stores one channel in 16 bits and the other two channels in 8 bits each. A developer would need to comment on the feasibility of implementing LogLuv: adding the colorspace conversion functions and modification to file read/write functions.

It could be simulated, with constant bits per channel. Something like:
Reading:

Code: Select all

convert
  -channel R -evaluate Exp +channel
  -colorspace sRGB
  ...{other processing}

Writing:

Code: Select all

convert
  ...{other processing}
  -colorspace Luv
  -channel R -evaluate Log +channel
  out.tiff

[fmw42]

I don't know that IM can convert to TIFF logluv format. For that matter, IM does not list logluv for one of its colorspaces. See http://www.imagemagick.org/script/comma ... colorspace. I do not see any -define for that output tiff format either at http://www.imagemagick.org/script/formats.php

convert rose: -colorspace logluv rose_logluv.tif
convert: unrecognized image colorspace `logluv' @ error/convert.c/ConvertImageCommand/1015.


This would probably need a request on the Development forum for an enhancement. But the IM developers are very busy now with IM 7 development. So I do not know how much priority this would get.

[yellow]

Oh my bad, I thought IM did logluv, now discovered however it was pfstools.

So that leaves lin to log, I can export four 16bit greyscale tiffs with libraw:

4channnels - splits RAW-file into four separate 16-bit grayscale TIFFs (per RAW channel).
Command line switches:

-s N selects N-th image from RAW with multiple images
-g gamma correction (gamma 2.2)
-A values autoscale by auto-calculated integer factor
-B turn off black subtraction
-N no RAW curve

So preferably by piping from libraw into IM, can I get 16bit full range Log tiffs out. Aim is to avoid going raw into a linear 'gamma' file format due to suspected loss of precision in shadows hence Log but maximising data in 16bit format. Prefer not going to 2.2 or similar in 16bit either. Or would half float EXR from the four greyscale images out of IM be better?

[snibgo]

Aim is to avoid going raw into a linear 'gamma' file format due to suspected loss of precision in shadows hence Log but maximising data in 16bit format.

Why would you lose precision in the shadows? Any integer format will potentially lose precision, but not especially in the shadows, I think.

The gamma function will spread out the shadow values (by a factor of 12.92, see http://en.wikipedia.org/wiki/SRGB ), so you won't get a quantization issue of two raw integer values mapping to one integer sRGB value in the shadows.

That is potentially a problem in the highlights, where values are squished together. It squishes by roughly 2, so 2 raw integer values map to a single integer sRGB value. However, transforming a 14-bit (per channel) raw to 16-bit sRGB, we have a factor of 4 to play with, so it's not a problem in highlights either.

One possibility to retain precision: IM can handle 16-bit CIELab files, so you might use that format. Or you could simply store the raw data, non-gamma-corrected, in 16-bit RGB format.

[yellow]

Aim is to avoid going raw into a linear 'gamma' file format due to suspected loss of precision in shadows hence Log but maximising data in 16bit format.

Why would you lose precision in the shadows? Any integer format will potentially lose precision, but not especially in the shadows, I think.

My understanding or lack thereof is the distribution of levels, the number levels given to shadows is less in lin compared to Log.

http://www.guillermoluijk.com/article/s ... /index.htm

http://www.luminous-landscape.com/forum ... opic=25064 Specifically first post, then posts 7 & 8.

The gamma function will spread out the shadow values (by a factor of 12.92, see http://en.wikipedia.org/wiki/SRGB ), so you won't get a quantization issue of two raw integer values mapping to one integer sRGB value in the shadows.

That is potentially a problem in the highlights, where values are squished together. It squishes by roughly 2, so 2 raw integer values map to a single integer sRGB value. However, transforming a 14-bit (per channel) raw to 16-bit sRGB, we have a factor of 4 to play with, so it's not a problem in highlights either.

One possibility to retain precision: IM can handle 16-bit CIELab files, so you might use that format. Or you could simply store the raw data, non-gamma-corrected, in 16-bit RGB format.

Ok, so 16bit integer is sufficient precision, even non gamma corrected and Logarithmic distibution has no advantage? I'll have a look at CIELab but may struggle with software tools supporting it. Many thanks.

[fmw42]

If you compile IM in Q16 HDRI, you can import the 32-bit data from dcraw and save as 32-bit TIFF (or NetPBM PFM format) or use IM MPC format (or with a proper -define you can also use MIFF). The Tiff and PFM formats are of course more universally readable.

See
http://www.imagemagick.org/script/high- ... -range.php
posting.php?mode=reply&f=1&t=23778 (regarding defines for MIFF and TIFF)

I use 32-bit PFM for FFT processing often in HDRI mode IM Q16

[snibgo]

Can dcraw create 32 bit/channel files? I didn't know that.


Interesting article. Google translation: http://translate.google.com/translate?u ... N&ie=UTF-8

It points out the limitations of linear colorspaces (in IM terms: colorspace RGB), as used by raw camera files. They can represent the same number of stops (Exposure Values, EVs) as they have bits. A 16-bit file can represent up to 16 stops. And, if it does represent 16 stops, there will be no detail at all in the darkest shadow.

A 14-bit camera can capture 14 stops, but the bottom few stops are horrible. If we look at them in detail by increasing the contrast, we get posterization. (I reckon a Nikon D800 gives me 10 useful stops of data.) This posterization is a natural consequence of how cameras work.

If we use multiple exposures to capture a wider constrast range, say 20 stops, storing these as linear would need at least 20 bits per channel. However, transforming by a gamma function (in IM terms, converting from RGB to sRGB) provides a wider dynamic range: 20 stops of light can be stored in 16 bits of data. Compared to linear, we have more possible values in the shadows EVs at the expense of fewer in the hightlight EVs.

I have no practical experience of log colorspaces. (IM has one, but I haven't played with it.) It seems to be the obvious choice for the arithmetic of merging multiple exposures into HDR images.

[fmw42]

Can dcraw create 32 bit/channel files? I didn't know that.

I do not know that either. I may have read too much into the OP's request and discussion of 32-bit data.

I think the real issue is how many bits does that particular camera capture in its rgb raw data? If only 16-bits, then using HDRI is overkill unless you combine multiple exposures.

Two high dynamic range formats are:

EXR see
http://en.wikipedia.org/wiki/OpenEXR (note IM only supports the half-format)
http://www.openexr.com/index.html
http://www.openexr.com/TechnicalIntroduction.pdf

HDRI see http://en.wikipedia.org/wiki/HDRi_(data_format) (RGBE and XYZE formats)

see also
http://en.wikipedia.org/wiki/High_dynamic_range_imaging
http://www.hdrsoft.com/resources/dri.html
http://www.anyhere.com/gward/hdrenc/hdr_encodings.html

I have some scripts that work in HDRI IM mode for tone mapping (for linux/mac). See link below. The test images were EXR and HDRI.

[yellow]

Can dcraw create 32 bit/channel files? I didn't know that.

I do not know that either. I may have read too much into the OP's request and discussion of 32-bit data.

No dcraw is 16bit output only afaik, which is why I mention libraw and rawspeed and outputing 4x 16bit raw channels or 32bit where possible. Darktable for sure uses libraw and rawspeed as it's 32bit Lab processing internally. But I think 32bit image format for storage of non HDR is probably overkill. Which is why I was thinking of Logluv for making perhaps more efficient use of 16bit levels rather than going to 32bit.

I think the real issue is how many bits does that particular camera capture in its rgb raw data? If only 16-bits, then using HDRI is overkill unless you combine multiple exposures.

Two high dynamic range formats are:

EXR see
http://en.wikipedia.org/wiki/OpenEXR (note IM only supports the half-format)
http://www.openexr.com/index.html
http://www.openexr.com/TechnicalIntroduction.pdf

Camera captures 14bit raw RGGB video as a series of dng's but whether it uses full 14bit once taking away the noise floor at the low end and sensor saturation at the high end probably well within that.

Interesting article. Google translation: http://translate.google.com/translate?u ... N&ie=UTF-8

It points out the limitations of linear colorspaces (in IM terms: colorspace RGB), as used by raw camera files. They can represent the same number of stops (Exposure Values, EVs) as they have bits. A 16-bit file can represent up to 16 stops. And, if it does represent 16 stops, there will be no detail at all in the darkest shadow.

A 14-bit camera can capture 14 stops, but the bottom few stops are horrible. If we look at them in detail by increasing the contrast, we get posterization. (I reckon a Nikon D800 gives me 10 useful stops of data.) This posterization is a natural consequence of how cameras work.

That's what I'm driving at, the first few paragraphs in a nut shell, I'd read that article and noticed the same in the processing I've been doing which lead me to investigate the use of Log or Logluv if a wide number of usable stops is to be contained, to just put the resultant RGB from the raw development process in a form that will give 'good' 'best' distribution of levels shadows to highlights, linear suggests loss and posterization at low end from insufficent levels being used in the links above or is it rather the poor quality of the sensor at the low end producing the percieved loss of levels precison in the shadows?.

Not wishing to widen the discussion but also other matters such as preferably an 'open' gamut rather sRGB. I'd obviously be previewing in the gamut / color space for final delivery though ie: sRGB and in a 32bit float processing chain but not transform from raw to sRGB in case of gamut clipping. I have no real interest in trying to reproduce the 'exact' color in sRGB from a color checker swatch for example or applying a camera curve specific to the camera, just transferring the raw into RGB for color processing in a form that is distributing levels efficiently, then temporal motion compensated NR & luminance sharpening and encoding to digital video.

But if 16bit integer in linear domain is more than sufficent in terms of containing the DR of 'useable' 14bit over sufficient levels, then maybe posterization and loss of precision in shadows is a levels mapping or scaling of values going from raw to linear RGB, really why I thought of Log. As it's video I'm not looking to do multiple exposures for video although there is a (Dual Channel Analogue to Digital dual ISO recording method available in camera for other models).

HDRI see http://en.wikipedia.org/wiki/HDRi_(data_format) (RGBE and XYZE formats)

see also
http://en.wikipedia.org/wiki/High_dynamic_range_imaging
http://www.hdrsoft.com/resources/dri.html
http://www.anyhere.com/gward/hdrenc/hdr_encodings.html

I have some scripts that work in HDRI IM mode for tone mapping (for linux/mac). See link below. The test images were EXR and HDRI.

Thanks both for your continued discussion and links to further reading.

[snibgo]

These are interesting questions but I doubt there are definitive answers; only pragmatic ones.

If you want something like sRGB but with a wider gamut, then scRGB might be a good choice. Or AdobeRGB, but that isn't currently supported by IM.

For some purposes, CIELab is better. It has a lightness channel, and saturation and hue are fairly easy to process, and it does this while being (reasonably) perceptually uniform.