How to convert raw image to compressed tif?

[srulop]

Hello,

I have a bunch of raw images like this one (about 8M each):
http://dl.dropbox.com/u/14434681/HM1_di ... LR0001.edf
They are 2048x2048 in size, "unsigned short datatype", and have a 1024 bytes header. The colors are grayscale.
I want to convert them to compressed tif images, so they will take less space, but retain all information (lossless).
The command I found that should do it was:

Code: Select all

convert -depth 16 -compress lzw -size 2048x2048+1024 gray:HM1_diffTomo_zone2_LR0001.edf image.tif

But all it gives me is a totally black image, and not so much reduced in size. I know for sure this image isn't empty - I can open it in ImageJ.
Any ideas of how to perform this conversion?

Thanks,
L.

[anthony]

Note -size does not take a thrid argument (+1024)


Also -compression should be put just before the final TIFF save.

Is your image black, more just very nearly black?
What 'endian' is your raw image data?
Is it just gray values, or rgb triples?

[srulop]

Thanks for the fast reply!

* The third argument in -size is from the tip here:
http://www2.fz-juelich.de/vislab/softwa ... nvert.html
And it actually works, when I indicate "-depth 8" - I get a non-black image, just not the correct form.

* changed the placing of -compress:

Code: Select all

convert -depth 16 -size 2048x2048+1024 gray:HM1_diffTomo_zone2_LR0001.edf -compress lzw image.tif

- Same result.

* The resulting image is really black. Changing brightness doesn't make details appear.

* Managed to open it in ImageJ using "little-endian byte order".

* The picture is only in gray, no colors.

[srulop]

OK, Tried to set endianness:
+endian, -endian msb cause a distorted image, each in it's own way.
-endian lsb - makes black image, as before.

I've got a feeling that it's because it sees it as signed 16 bit. How do I tell it to read it as unsigned 16 bit?

[Drarakel]

Which ImageMagick version do you use? (In some versions, there were problem with TIFF files.)
With the current version (v6.7.5-4), this should give you the right image:

Code: Select all

convert -depth 16 -size 2048x2048+1024 -endian msb gray:HM1_diffTomo_zone2_LR0001.edf -compress lzw image.tif

(And, you should be able to further decrease the file size by e.g. using a high tiff:rows-per-strip value or by using ZIP compression.)

I think, the "-endian" option behaves a bit strange - as your file is indeed "lsb" (byte order), not "msb". But it seems to work - at least for me.

[srulop]

I use version 6.6.0.
With your line, I got the following result:
http://img18.imageshack.us/img18/6074/imageit.png
Is that what you got?

The correct result should look like that:
http://img594.imageshack.us/img594/7701/correctu.jpg
I got this with ImageJ, little endian option. By the way, ImajeJ saves it much darker than it's displayed in the display window, for some reason.

And what did you mean by using a high tiff:rows-per-strip value?
(Zip compression - found out how)

Thanks!

[srulop]

OK, compiled and installed the latest version, 6.7.5-5.

Now when I do:

Code: Select all

convert -endian MSB -depth 16 -size 2048x2048+1024 gray:HM1_diffTomo_zone2_LR0001.edf -compress lzw image.tif

I get:
http://img841.imageshack.us/img841/127/imagegt.png
Which is the same as I get in ImajeJ with big endian, and it is not correct.

When using -endian LSB, I still get a black image.

[Drarakel]

At least the distortions from 6.6.0 should be gone.

I first thought that you wanted that brighter image (as you wrote that the "totally black" image is wrong). So I wrongly identified the byte order. Sorry. That also means that the "-endian" option behaves correctly here. And you have to use "-endian lsb".

The resulting TIFF has the same 16bit values as your raw input file then. It's not totally black - it just has very low values.
With "-auto-level" ("-normalize" is another possiblity), you should be able to get about the same result as in "correctu.jpg":

Code: Select all

convert -endian LSB -depth 16 -size 2048x2048+1024 gray:HM1_diffTomo_zone2_LR0001.edf -auto-level -compress lzw image.tif

But I'm not certain if "-auto-level" is the best option. Perhaps others can comment on that aspect.

And what did you mean by using a high tiff:rows-per-strip value?

If you use LZW compression, you could add "-define tiff:rows-per-strip=2048" to your command. That ensures that all 2048 lines/rows get stored into one strip/segment. The file size will be a bit lower. That's all. With ZIP compression, you don't need that.

[anthony]

Thanks for the fast reply!

* The third argument in -size is from the tip here:
http://www2.fz-juelich.de/vislab/softwa ... nvert.html
And it actually works, when I indicate "-depth 8" - I get a non-black image, just not the correct form.

Okay. Yes I found the code that extracts an offset from the "size" setting. Learn something new every-day!

Adding that info to IM Examples, Formats, Raw Image Data. (give it an hour or two to appear)
http://www.imagemagick.org/Usage/formats/#rgb

Hmmm.. It with a quick look at the source the offset appears to be bytes, so that is good.

Smaller test image (raw data)

Code: Select all

   convert -size 100x100 xc: -draw 'circle 25,25 20,5' -depth 16 -endian MSB gray:image_data.raw

File is 20000 bytes... 100x100x2 -> 20000 bytes

prepend some junk (23 bytes including a newline character)

Code: Select all

  echo "this is raw image data" > image_data_offset.raw
  cat image_data.raw >> image_data_offset.raw

now read back using offset..

Code: Select all

   convert -size 100x100+23 -depth 16 -endian MSB gray:image_data_offset.raw  show:

Perfect image return!

Without the offset I get junk for the first few pixels, and fuzzy circle off set to right by 12 pixels.
The fuzziness is because 23 bytes is not divisible by two, so the 2 byte gray values are in the wrong order. However as the test image is black and white so results are still vaguely visible.

With an offset that is to big... EG: 51 bytes (odd so as to re-align the 2-yte endian), the circle is clear but shifted left 12 pixels ( (51-23) / 2 -> 12 pixels) but the whole last row is black, and IM produced an error (exception)...

convert: unexpected end-of-file `image_data_offset.raw': No such file or directory @ error/gray.c/ReadGRAYImage/206.

The "No such file or directory" part of the error message is rather mis-leading, but the error "unexpected end-of-file" is exactly the reason for the failure. Basically it did not find enough data!

[srulop]

Drarakel, thank you very much - the -auto-level option gives exactly the output that I was looking for!
I have read about what this option does, but I am not sure that it conserves all the data in the original image. So can I consider my new file lossless after using this option?

Another strange thing: when I execute this line with -size 2048x2048+1024 on the original file, and the same line, but with -size 2048x2048 on the same file after removing the header, I get two images that look the same, but their MD5 sums are different. How is this possible?

And Antony, I learned a lot from your site in the past months, thank you.

[Drarakel]

So can I consider my new file lossless after using this option?

Depends on what you mean with 'lossless'. You won't 'lose' some values with "-auto-level", as it doesn't clip them (contrary to "-normalize"). But the values are changed/stretched, and you can't go back easily to the original values - not without special knowledge about the original file (and there will be small rounding errors, too, of course).

Another strange thing: when I execute this line with -size 2048x2048+1024 on the original file, and the same line, but with -size 2048x2048 on the same file after removing the header, I get two images that look the same, but their MD5 sums are different.

I get two identical output files.
Perhaps you changed the output file name? ImageMagick saves the file name as EXIF 'DocumentName' tag. Then the files are not identical anymore (just because of the metadata).

[srulop]

Drarakel, thanks, all problems solved!

Now the only problem that remains is to do this to some 400,000 images, some with different header sizes and different dimensions. But I guess it's an off-topic.