Huge TIFF conversion - Interger overflow - IM:7.0.4.5

[mc_pl]

Cheers!

I have run into a problem with huge TIFF files compressed with LZW.

The example file is mainly white with black pixels. It's size is ~90MB (19046x49704). I do also have larger files and all of them compressed with LZW.

When trying to rotate the image:

Code: Select all

convert.exe  "input.tif" -virtual-pixel Background -background "#000000" +distort SRT "0,0 90.2291806734696"  +repage "output.tif"

or trying to decompress the image:

Code: Select all

convert.exe image.tif -compress none out_image.tif

An error accours:

TIFFStripSize Integer overflow

The file seems not to be corrupted - it is opened properly in Photoshop and IrfanView, but ImageMagick and Paint.Net do have problems with processing this file.

I have downloaded the sources of ImageMagick (https://www.imagemagick.org/script/install-source.php) and came into the following conclusion when working with VisualMagick:

In file: tif_strip.c there is a method in line: 220: TIFFStripSize :

Code: Select all

tmsize_t
TIFFStripSize(TIFF* tif)
{
	static const char module[] = "TIFFStripSize";
	uint64 m;
	tmsize_t n;
	m=TIFFStripSize64(tif);
	n=(tmsize_t)m;
	if ((uint64)n!=m)
	{
		TIFFErrorExt(tif->tif_clientdata,module,"Integer overflow");
		n=0;
	}
	return(n);
}

it throws the exception in line 230.

Code: Select all

TIFFErrorExt(tif->tif_clientdata,module,"Integer overflow");

The exception is being thrown because the cast in line 227 fails:

Code: Select all

n=(tmsize_t)m;

The value of m is the result of uint64 TIFFStripSize64 but is casted to tmsize_t which is signed int

The tmsize_t is defined in tiffconf.h in line 38:

Code: Select all

#define TIFF_SSIZE_T signed int

The strip is huge and is calculated correctly, but cannot be casted to signed int if is greater than the maximum int value. The cast is true if run on a 32bit operating system. But I am running Windows10 64bit and decided to test another aproach:

Code: Select all

#ifdef _WIN64
#define TIFF_SSIZE_T unsigned long
#else
#define TIFF_SSIZE_T signed int
#endif

I cannot define unsigned long permanently because the result of TIFFStripSize is used in file: tif_write.c:660 and is passed as a value of parameter to _TIFFmalloc (tif_unix.c:311) . If running on 32Bit system I can not request more than allowed memory for object so I've decided to keep the #ifdef _WIN64.

By this I have allowed the system to create files that have huge strips. Unfortunatelly Windows is unable to show or create a preview thumb of the output image, and also is unable to show its properties. I believe that this is because I have probably broken a TIFF RFC, haven't I? But still the output image is 120MB in size after rotation and is shown properly in IrfanView and PhotoShop, and can also be rotated and scaled by ImageMagick.

So the question is:
Is the original file corrupted or is there a bug in ImageMagick or what?

Attached you can find a dump from exiftool with information about the StipByteCounts and StripOffsets:
http://pastebin.com/PkSWMh2Y

Regards!

[magick]

ImageMagick uses the libTIFF delegate library to read and write TIFF images. You isolated this bug (or feature) to this delegate library. For a resolution, we recommend you post to the libTIFF developers user group. Only they can offer advice or a bug fix.

[mc_pl]

OK. So I will leave this subject here for later viewers.
I have found a thread in libTiff mailing list from 7 years ago:
http://www.asmail.be/msg0055078017.html
They do discuss this problem but as far as I understand they are not keen on allowing to compile their library with a 64bit addressing space mode because this would create a possibility of beeing able to read a file on a 64bit system and unable to read this file in 32bit mode. So this is exactly where I am right now.

[magick]

Have you seen http://bigtiff.org? ImageMagick uses libtiff 4.0.7. If you can show that the latest version, libtiff 4.1, can read your image, we'll go ahead and upgrade libtiff in the next point release of ImageMagick.

[mc_pl]

Hi,
The good news is that libTiff 4.1 does process the file without the "Integer overflow" error.

Probably it's my bad, but all the files created with IM.7.0.4.5 + libTIFF4.1 are blank. No matter if the file had the "Integer overflow" error or not.
I used the jpeg-6b and zlib from magicks sources.
This goes both for Q8 and Q16.

[mc_pl]

I just found an old installation of IM 6.8.9 Q8 in one of my computers. I have taken the "convert.exe" and the "vcomp100.dll" and placed them in a new directory. Then I've tested the problematic file with this version and all worked fine. So I took this idea and I have downloaded the zip package from: http://ftp.icm.edu.pl/packages/ImageMag ... indows.zip

It uses a hard disk to cache the image instead of RAM - there is 16GB free RAM. Because of this it's very slooow.
It's also 32bit not 64 as I would like it to be.
And the file name of the zip states that it''s a Q16 but it's Q8.

I've decided to check if there is a difference in handling my problematic file between Q16 vs Q8 from installation vs portable version, x64 vs x86.

The earliest portable version of 64bit is 6.9.1-6 - this version also uses vcomp120.dll in difference to 6.8.9:
Both versions (x64 and x86) of 6.9.1-6 do process this file. The only difference is that x86 does not take advantage of RAM and uses harddisk caching which is 20x slower that processing this file in RAM by x64 bit version.

I did not test Q16 vs Q8 because Q8 is only available after full installation which I did not want to do in this environment.

I did not add any more libraries to the testing directory only: convert.exe and vcomp120.dll were used in these test.


The file is NOT processed correctly with the newest 6.9.7 or 7.0.4.5. In those versions it throws the "Integer overflow" error.

[mc_pl]

I have decided to narrow down the location of the problem in the releases so maybe it can help you.

It turns out that there are three stages of this problem:

• The file is being processed correctly and the output of the file is a valid image until release: 6.9.5.9

• Since release 6.9.5.10 the file is processed without an error but the image is not valid any more. It is corrupted.

• Since version 6.9.6.7 the process throws an exception "Integer overflow"

[fmw42]

I do not know if this is relevant, but check the changelog at http://legacy.imagemagick.org/script/changelog.php There were changes to TIFF and to PSD, which might affect TIFF as well around the time of your listed IM versions.

[mc_pl]

Hi, Thanks. I did see that there were some changes in the TIF hangling area.
Working on this subject would require me to compile the sources of both 6.9.5.9 and 6.9.5.10 versions.

But in both the versions there is no "VisualMagick" folder. Is there any manual on compiling these older versions?

[fmw42]

How about just testing without VisualMagick (i.e. IM command line) and see if it is IM or VisualMagick that is the problem?