How to add image file header to .RAW file without altering RAW data?

[BrianP007]

I am creating single channel, grayscale extracts of RGB/HSV/Lab files and would like to be able to use a "standard" file extension so the OS could show a thumbnail. But without altering the raw data.

I was trying to use IM -> Convert to simply add the most trivial possible header to a .raw file without altering the data:

Code: Select all

convert  -size "7378x4924" -depth 16 gray:pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw  pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm

 lsr -s pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw  pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm
pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw      72658544
pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm  72658563

add 72658563 -72658544   =-> 19 

<< Looks like a tiny, 19 byte header; just what I want:

Code: Select all

head -n 3 pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm
P5
7378 4924
65535

All I need is the XY resolution and the number of channels.

identify pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm
pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm PGM 7378x4924 7378x4924+0+0 16-bit Gray 72.66MB ...

And, Photoshop likes it! Perfect?

NO! The data are altered. It looks like roundoff error from a float -> integer conversion. But, reading 16 bit gray data into a 32 bit float with 23 bits of significand should be an exact copy unless a gamma is being applied or it is being converted to L*ab or ???

If I dump the first 16 bytes of the .RAW file, then fseek past the header of the PGM file and dump the next 16 bytes, the bytes are altered slightly:

Code: Select all

dumpb pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw 0 16 8  [[ Start at byte 0;  dump 16 Bytes; uint8 format ]]
II [0] 8  (26) (93) (253) (92)
II [4] 8  (131) (94) (250) (95)
II [8] 8  (215) (92) (132) (89)
II [12] 8  (35) (90) (183) (90)

pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm 20 16 8
Seeked 20 bytes from file start
II [20] 8  (26) (92)<<93 (253) (94)<<92
II [24] 8  (131) (95)<<94 (250) (92)
II [28] 8  (215) (89) (132) (90)
II [32] 8  (35) (90) (183) (91)

If I tell it to use "-COMPRESS NONE", it triples the file size without changing the number of channels or XY resolution:

Code: Select all

convert  -size "7378x4924" -depth 16 gray:pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw -compress None   pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm

=>
pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw           72658544
pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm  212467460  << 3X file size

identify pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm
pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm PGM 7378x4924+0+0 16-bit Grayscale 212.5MB  ...

Still shows 16 bit quanta, single channel gray, same XY res AND 3x the size. Where are the missing data? It shows the file size of an RGB file but only registers gray.

Why would turning off compression make it create invisible data? Creating new data without changing quanta, resolution or file type looks like an accounting bug.

If I tell it to use no compression AND output a "GRAY:" file, it does just that:

Code: Select all

convert  -size "7378x4924" -depth 16 gray:pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw -compress None   gray:pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm

 lsr -sm pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw  pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm
pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw      5d6c89c14e31fbfaf9ed9d44ecbf18eb  72658544
pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm  5d6c89c14e31fbfaf9ed9d44ecbf18eb  72658544  << Same MD5

It copies the file verbatim, ignoring the "convert to PGM" directive. And, it can't read what it just wrote:

Code: Select all

identify pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm                       identify: improper image header `pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm' @ error/pnm.c/ReadPNMImage/284.

Is it possible to convert .RAW (DCRaw ?) files to tiff/png/jpg/PGM/??? without compressing, resampling or changing the raw data?

===============
convert -> Version: ImageMagick 6.9.3-8 Q16 x86_64 2016-04-20
Features: Cipher DPC HDRI Modules OpenMP
Delegates (built-in): bzlib fontconfig freetype jng jpeg lcms ltdl lzma openexr pangocairo png tiff x xml zlib
IM-> identify from same source

[glennrp]

It ignored the "convert to PGM" directive because of the "gray:" prefix you supplied in the output filename.

[BrianP007]

It ignored the "convert to PGM" directive because of the "gray:" prefix you supplied in the output filename.

Is there an incompatibility between "gray" and "Portable GRAY Map"? "RGB" would be contradictory.

I tried this to get it to stop compressing/altering/tripling the data.

IM identifies the file as GRAYSCALE. Prefixing a GRAY file with GRAY should be redundant (and ignored). Instead, it produces a corrupt PGM file, identical to the .RAW file, that it can't read. This can not be useful behavior.

We both agree that [[ the "gray:" prefix ]] does not work. Would you have any suggestions on what would work to:

• Accept a Photoshop extract .RAW file (pure uint16, single channel data with no header at all)
  Create a .PNG/PGM/TIF/??? standard image file
  NOT CHANGE/ALTER/compress/resample/etc... the RAW data

If I move the "gray directive" from a prefix to a colorspace, it again takes a single channel, known 16 bit quanta, file of '-size "7378x4924" ' and creates a corrupt, RGB size file:

Code: Select all

convert  -size "7378x4924" -depth 16 gray:pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw -compress None  -colorspace gray pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm

lsr -sm pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw  pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm
pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw      5d6c89c14e31fbfaf9ed9d44ecbf18eb   72658544
pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm  25a000d207c11877ab8f925e8019d9b6  212467460

Based on the given XY resolution -size of "7378x4924" and given 16 bit quanta, IM should know that it has to be single channel, I.E GRAY:
channels * quantum_bytes/channel * Xres * Yres = RAW_file_size
Channels = 72658544 / 2 / 7378 / 4924 = 1.000000 => GRAYSCALE. Isn't 1 channel the definition of GRAYSCALE?
File size/res/quanta math appears to be correct.

Let's look at the IM output:

Code: Select all

 identify  pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm
pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm PGM 7378x4924 7378x4924+0+0 16-bit Grayscale Gray 212.5MB 1.100u 0:01.099

FSize = 7378x4924 pixels * 1 gray_channel/pixel * 2 bytes/channel = 72658544
In the same sentence, it reports a size of "212.5MB". This size is contradicted by the other correct data. Photoshop confirms that the file is random bits.

Try to remove the "GRAY:" on the input file:

Code: Select all

 convert  -size "7378x4924" -depth 16 pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw -compress None  -colorspace gray pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm
convert: delegate failed `"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=%u.png" "%i"' @ error/delegate.c/InvokeDelegate/1333.
convert: unable to open image `/tmp/magick-7438pLlngZu55iI9.ppm': No such file or directory @ error/blob.c/OpenBlob/2702.
convert: no images defined `pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm' @ error/convert.c/ConvertImageCommand/3252.

Try removing the "-compress None":

Code: Select all

 convert  -size "7378x4924" -depth 16 pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw    pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm
convert: delegate failed `"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=%u.png" "%i"' @ error/delegate.c/InvokeDelegate/1333.
convert: unable to open image `/tmp/magick-79728opSKEoiDidN.ppm': No such file or directory @ error/blob.c/OpenBlob/2702.
convert: no images defined `pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm' @ error/convert.c/ConvertImageCommand/3252.

Change the file type:

Code: Select all

convert  -size "7378x4924" -depth 16 gray:pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw    pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.png
identify pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.png
pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.png PNG 7378x4924 7378x4924+0+0 16-bit sRGB 60.47MB 0.000u 0:00.000

Highly compressed sRGB. Shows in file viewer and Photoshop likes it. But NOT gray and not pristine...
Coax it into a gray colorspace:

Code: Select all

convert  -size "7378x4924" -depth 16 gray:pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw   -colorspace gray pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.png
identify pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.png                       pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.png PNG 7378x4924 7378x4924+0+0 16-bit sRGB 60.47MB 0.000u 0:00.000

Ignores "-colorspace gray" and produces a sRGB

Leave nothing to the imagination: specify the colorspace, type, compression and depth:

Code: Select all

convert -size "7378x4924" -depth 16 -type Grayscale  gray:pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw   -colorspace gray  -type Grayscale -compress none  -depth 16 pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm
identify  pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm
pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pgm PGM 7378x4924 7378x4924+0+0 16-bit Grayscale Gray 212.5MB 1.090u 0:01.089

Again, triples file size from thin air. Looks like random gray mess in photoshop

Try PBM file:

Code: Select all

convert -size "7378x4924" -depth 16 -type Grayscale  gray:pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw   -colorspace gray  -type Grayscale -compress none  -depth 16 pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pbm
brianp@raptor:~/exp$ identify pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pbm
pf-249465.dcr.T6Fww.xyz.7378x4924.blu.raw.pbm PBM 7378x4924 7378x4924+0+0 1-bit Bilevel Gray 72.7MB 0.340u 0:00.359

It does appear to be black&white. At 1 BIT/pixel, the size is 16x too large; 1 bit/pixel the same size as uncompressed 16 bit gray.

I am out of IM leavers to throw to create any standard image file from a raw gray file without destroying the raw data. I am going to writeup a bug. Back to raw!