Invalid files DPX -> TIF

[friolator]

We need to convert some 2k film scans (DPX, Linear) to TIF, for processing in another command line tool that only accepts TIF. This tool came with a shell script that batch processes all the DPX files in a folder, using ImageMagick, and outputs the grayscale TIF files it wants. However, the resulting TIF files don't work in that application, nor can we open them in Photoshop or other image viewing tools.

The shell script does the following:

Code: Select all

for filename in *
do
 convert -set reference-black 0 -set reference-white 1023 -depth 16 -colorspace Gray -gamma 1.0 $filename ../convertedTIFs/${filename%.dpx}_gl.tif
done

This correctly results in the creation of files in the 'convertedTIFs' folder. But these files are no good, at least according to Photoshop and the target application for the converted files.

I have uploaded a source file (DPX) and the ouput file (TIF) here:

https://www.dropbox.com/sh/p4bg3ao7dorxf9u/sIhiU_216R

Even if I strip that convert command down to the bare minimum: "convert source.dpx target.tif", the result is no good.

I am able to make a grayscale TIF in Photoshop, save it out and run it through the target command line app, so I've ruled that application out as the problem.

The system this is on:

Mac OS X 10.6.8
ImageMagick 6.8.8-7 (compiled from source this afternoon, default settings)
LibTIFF 3.8.2 (which the target app requires, not sure if it's relevant here)

The source files were scanned today on our film scanner, and we've tried with 10 and 16 bit DPX files. Because it's coming from a positive, our scanner automatically creates Linear files, so no log->lin conversion should be necessary.

Any thoughts? Suggestions?

What's the best way to compare the TIF file created by ImageMagick (which doesn't work) with the TIF created by Photoshop from the same DPX (this one does work)? Maybe that will shed some light on what's happening.

Thanks!

[fmw42]

I am on Mac OSX Snow Leopard but temporarily running IM 6.8.0.5 and LIBTIFF, Version 3.9.5. I can process your image with the following commands and I can open the file in PS CS. But the result is very grainy. I am not an expert on DPX. Perhaps you need to ugrade your libtif

Code: Select all

convert BaseUp_DPX_COLOR_10Bit_016.dpx -set reference-black 0 -set reference-white 1023 -set gamma 1.0 -colorspace Gray -depth 16  tmp.tif

It looks fine with png output or -depth 8 tiff output. So perhaps there is a bug in libtiff or it cannot handle 16-bit grayscale properly or a newer version of libtiff is needed.

Can you try converting using libtiff standalone? If that works, then there is a bug in IM?

[friolator]

Would you mind posting the TIF file you were able to make? This film is a bit unusual - it's a scan of a home movie made using a short-lived Kodak lenticular color process, so it's not going to look super sharp. There should be lots of vertical lines, and because it's from 16mm it's pretty grainy.

I don't see any tools in LibTIFF to convert from DPX, but there is a Color->BW converter (tiff2bw), so I may be able to scan direct to RGB 16bit TIFF files and then convert those using tiff2bw instead of ImageMagick. Looks like it also has a tool for comparing TIFFs, so I'll run both the IM and the Photoshop TIFFs through that tomorrow to see if I can find differences.

Thanks!

[fmw42]

http://www.fmwconcepts.com/misc_tests/d ... _16bit.tif

http://www.fmwconcepts.com/misc_tests/d ... t_8bit.tif

http://www.fmwconcepts.com/misc_tests/d ... _16bit.png

I used libtiff 3.9.5, but the latest is 4.0.3. Perhaps it will work with that version.

Perhaps repost this to the bugs or developers forum.

I think the IM developers should look into this and verify if it is an IM or libtiff issue with generating 16-bit grayscale tiff from dpx. It looks so bad compared to the 8-bit tiff or 16-bit png.


P.S. I think it is a libtiff issue, since going directly to PNG works, but this does not:

Code: Select all

convert BaseUp_DPX_COLOR_10Bit_016.dpx -set reference-black 0 -set reference-white 1023 -set gamma 1.0 -colorspace Gray -depth 16  PNG:- | convert - tmp.tiff

[snibgo]

With the current version of IM, on Windows 8.1, the command:

Code: Select all

convert BaseUp_DPX_COLOR_10Bit_016.dpx t.tiff

... succeeds but the result seems bad. Windows Photo Viewer says t.tiff is corrupt. Gimp opens it without objection, but shows a blank (transparent black) image. I get the same result from older versions of IM.

The following is more successful:

Code: Select all

convert BaseUp_DPX_COLOR_10Bit_016.dpx t.png
convert t.png t.tiff

According to "identify -verbose", t.png has gamma=1.

(Note that the operation "-gamma 1" should do nothing at all to any image, as the parameter is the required change, not an absolute value. To set an absolute value, you can use "-set gamma 1".)

"convert t.png t.tiff" copies the pixels without changing their values. "identify -verbose t.tiff" thinks t.tiff is sRGB, but I think tiff has no flags that could tell it otherwise.

[fmw42]

snibgo,

It is odd that the following does not work for me. The png file is good, but the tif file is not very good. Note in my command and above I used -set gamma (though the OP used -gamma, which as you say is a no-op and leaves the tiff gamma undefined)

Code: Select all

convert BaseUp_DPX_COLOR_10Bit_016.dpx -set reference-black 0 -set reference-white 1023 -set gamma 1.0 -colorspace Gray -depth 16  tmp.png

convert tmp.png tmp.tif

What version libtiff are you using?

[snibgo]

The png file is good, but the tif file is not very good.

That massive graininess is characteristic of swapped bytes. We can test this theory:

Code: Select all

convert result_16bit.tif r2.png

convert -define png:swap-bytes r2.png r2swap.tiff

r2swap.tiff looks good (to me).

What version libtiff are you using?

Whatever the pre-compiled Windows binary uses. Is there a command I can use to tell me the libtiff version?

[fmw42]

on unix, tiffinfo, tells the version. don't know if that works on windows.

I tried earlier to swap the bytes using -set tiff:endian lsb, but that did not help. If that is the issue, then libtiff has a bug, since converting to png works fine.

P.S. Your swap bytes does not work for me, but I am on an older IM 6.8.0.5 and that may not have been available back then. But that may be a good workaround for now.

[dlemstra]

Whatever the pre-compiled Windows binary uses. Is there a command I can use to tell me the libtiff version?

There is no command to get this information. You can view the version on the properties of the file 'CORE_RL_tiff_.dll'. A new version of ImageMagick on Windows will be using 4.0.3.

[snibgo]

I've reported my problem as a bug: viewtopic.php?f=3&t=25029

I think the workaround for the OP is simply to have two convert commands. The first creates a PNG. The second reads that PNG and creates a TIFF.

P.S. Your swap bytes does not work for me, but I am on an older IM 6.8.0.5 and that may not have been available back then.

Yes, png:swap-bytes is a very recent addition.

You can view the version on the properties of the file 'CORE_RL_tiff_.dll'.

Thanks. I use the static version, which doesn't have dll files.

[friolator]

Thanks for the suggestions. I'll give the DPX->PNG->TIFF path a try today, though I'm not nuts about having to create another set of images. There will be well over 20,000 of them in the final film, once we're done with the tests, so that's a lot of extra 2k data to generate and a lot more processing time. Ideally I'd like to find a way that avoids having to create too many intermediate files, to speed things up, so the first test will be to scan direct to 16 bit TIFFs and see if we can just convert them to greyscale using the tools provided by LibTIFF. If that doesn't work, then it's back to IM.

Thanks for all the suggestions!

[snibgo]

You don't have to keep the png file.

Over on the bug thread, magick points out that IM creates a 10-bit tiff, and that's why other software can't read it. I have confirmed that putting "-depth 16" before the output tiff name cures that problem. I see you have "-depth 16", but not just before the output name. I suggest you try that.

[friolator]

Thanks, I'll give the '-depth 16' flag a try when I sit down to work on this after lunch.

The drive space really isn't an issue - this is all happening on a machine with 21TB of fast storage. But it's the processing time I'm worried about. As it is, once scanned it will take quite a bit of time to convert the files to TIFF, and then at least 12-14 hours to process the final images through the software that these are ultimately destined for. So, I'm looking more at speed than space here.

Thanks!

[fmw42]

This should work without saving the png.

convert BaseUp_DPX_COLOR_10Bit_016.dpx -set reference-black 0 -set reference-white 1023 -set gamma 1.0 -colorspace Gray -depth 16 PNG:- | convert -define png:swap-bytes - tmp.tiff

[friolator]

I've tried running this directly through LibTIFF's tiff2bw tool from a 16 bit TIFF file fresh off the scanner, but I get an error that it only handles 8 bit files. I'm able to get it to work with an 8bit scan, but of course it only outputs 8 bit files, even though I need 16bit files for the target application later on. So that's out. I'm not sure if that's a LibTIFF limitation or a limitation of the tiff2bw too. My guess is that it's LibTIFF.

convert BaseUp_DPX_COLOR_10Bit_016.dpx -set reference-black 0 -set reference-white 1023 -set gamma 1.0 -colorspace Gray -depth 16 PNG:- | convert -define png:swap-bytes - tmp.tiff

This doesn't work for me. I get the following when I run this. Maybe there's a syntax error somewhere?

Code: Select all

$ ./convertCommand.sh
/Volumes/RAID/Lenticular/DPX_COLOR_10Bit
BaseUp_DPX_COLOR_10Bit_016.dpx to be processed...
convert: no decode delegate for this image format `/var/tmp/magick-6323FLOd5w4MLLzA' @ error/constitute.c/ReadImage/555.
convert: no images defined `../convertedTIFs/BaseUp_DPX_COLOR_10Bit_016_gl.tif' @ error/convert.c/ConvertImageCommand/3147.

The line in convertCommand.sh that's calling IM is as follows (a slightly modified version of the above, to fit into the loop that's iterating through files in this folder:

Code: Select all

convert $filename -set reference-black 0 -set reference-white 1023 -set gamma 1.0 -colorspace Gray -depth 16 PNG:- | convert -define png:swap-bytes - ../convertedTIFs/${filename%.dpx}_gl.tif

I like the idea of writing to a temp PNG file, and I think that's probably a viable solution for now. Hopefully there's just a dumb syntax error on my part that I'm not seeing.

thanks!