unable to convert large PSB files

[ikelso82]

Environments tested within:
OSX 10.14.3 --- Version: ImageMagick 7.0.8-37 Q16 x86_64 2019-04-03
Docker Container running "FROM jboss/wildfly:9.0.2.Final" that has imagemagick installed --- Version: ImageMagick 7.0.8-34 Q16 x86_64 2019-03-18

Commands used:

convert psb:/3layer.psb -limit memory $x -limit map $y -intent relative -sample "1024x768>" -quality 95 -colorspace sRGB -strip -auto-orient 3layer.jpg -debug all

convert psb:/3layer.psb 3layer.jpg

I've tried multiple iterations of the above commands, attempting to set limits to 0, 1 and 2, 2 and 4, 4 and 8. I've tried setting the -limit area to 0, -limit disk to 0, and tried multiple combinations of all of these limits. I've attempted to change the temporary file location using the environment variable MAGICK_TEMPORARY_DIRECTORY just in case we're running into a /tmp folder size limitation, but imagemagick does not seem to adhere to this. Nothing I've tried has resulted in large PSB files with multiple layers being converted successfully. This feels like a resource constraint issue, but I also found a previous thread with one of my exact errors using a PSD file, so perhaps this is a bug still.

I've attempted this with 7.0.8-36 in OSX as well as -37 and I get the following error returned:

2019-04-03T14:38:12-05:00 7:38.978 393.330u 7.0.8 Exception convert[47211]: blob.c/OpenBlob/3485/Exception
unable to open image 'testconvert.jpg': No such file or directory
2019-04-03T14:38:12-05:00 7:38.978 393.330u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy
2019-04-03T14:38:12-05:00 7:38.978 393.330u 7.0.8 Policy convert[47211]: policy.c/IsRightsAuthorized/609/Policy
Domain: Path; rights=Read; pattern="testconvert.jpg" ...
2019-04-03T14:38:12-05:00 7:38.978 393.330u 7.0.8 Exception convert[47211]: blob.c/OpenBlob/3485/Exception
unable to open image 'testconvert.jpg': No such file or directory
2019-04-03T14:38:12-05:00 7:38.978 393.330u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy
2019-04-03T14:38:12-05:00 7:38.981 393.330u 7.0.8 Exception convert[47211]: convert.c/ConvertImageCommand/3297/Exception
`all'
2019-04-03T14:38:12-05:00 7:38.981 393.330u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy TestConvert.psb[0]
2019-04-03T14:38:13-05:00 7:39.206 393.550u 7.0.8 Resource convert[47211]: resource.c/RelinquishMagickResource/1069/Resource
Memory: 2280960000B/11.6837GiB/16GiB
2019-04-03T14:38:13-05:00 7:39.206 393.550u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy TestConvert.psb[0]
2019-04-03T14:38:13-05:00 7:39.243 393.590u 7.0.8 Resource convert[47211]: resource.c/RelinquishMagickResource/1069/Resource
Memory: 1710720000B/10.0905GiB/16GiB
2019-04-03T14:38:13-05:00 7:39.244 393.590u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy TestConvert.psb[0]
2019-04-03T14:38:13-05:00 7:39.285 393.630u 7.0.8 Resource convert[47211]: resource.c/RelinquishMagickResource/1069/Resource
Memory: 2280960000B/8553600000B/16GiB
2019-04-03T14:38:13-05:00 7:39.285 393.630u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy TestConvert.psb[0]
2019-04-03T14:38:13-05:00 7:39.330 393.680u 7.0.8 Resource convert[47211]: resource.c/RelinquishMagickResource/1069/Resource
Memory: 2280960000B/6272640000B/16GiB
2019-04-03T14:38:13-05:00 7:39.331 393.680u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy TestConvert.psb[0]
2019-04-03T14:38:13-05:00 7:39.372 393.720u 7.0.8 Resource convert[47211]: resource.c/RelinquishMagickResource/1069/Resource
Memory: 2280960000B/3991680000B/16GiB
2019-04-03T14:38:13-05:00 7:39.372 393.720u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy TestConvert.psb[0]
2019-04-03T14:38:13-05:00 7:39.415 393.760u 7.0.8 Resource convert[47211]: resource.c/RelinquishMagickResource/1069/Resource
Memory: 2280960000B/1710720000B/16GiB
2019-04-03T14:38:13-05:00 7:39.415 393.760u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy TestConvert.psb[0]
Assertion failed: (resource_info.memory >= 0), function RelinquishMagickResource, file MagickCore/resource.c, line 1025.
Abort trap: 6

Within my container i get the following error:

2019-04-03T20:44:00+00:00 0:46.170 65.530u 7.0.8 Resource convert[1080]: resource.c/RelinquishUniqueFileResource/1107/Resource
/tmp/magick-1080FcS4OgIyXP4X
2019-04-03T20:44:00+00:00 0:46.170 65.530u 7.0.8 Resource convert[1080]: resource.c/RelinquishMagickResource/1069/Resource
Disk: 6.1875MiB/0B/8EiB
convert: unable to open image '3layer.jpg': No such file or directory @ error/blob.c/OpenBlob/3485.
convert: `all' @ error/convert.c/ConvertImageCommand/3297.

You can find an example file here https://drive.google.com/open?id=1VlMOm ... -mPV6Z6NFx

[fmw42]

How big is the TIFF? How much RAM do you have? What does

Code: Select all

identify psb:/3layer.psb

return?'

What are your versions of libtiff and libjpeg? What is your platform/OS?

[ikelso82]

Sorry if I sound confused, is TIFF the same as PSB?

identify psb:3layer.psb shows:

psb:3layer.psb=>3layer.psb[0] PSB 9900x14400 9900x14400+0+0 8-bit sRGB 975.791MiB 0.600u 0:00.601
psb:3layer.psb=>3layer.psb[1] PSB 9900x14400 9900x14400+0+0 8-bit sRGB 0.000u 0:00.001
psb:3layer.psb=>3layer.psb[2] PSB 9900x14400 9900x14400+0+0 8-bit sRGB 0.000u 0:00.000
psb:3layer.psb=>3layer.psb[3] PSB 9900x14400 9900x14400+0+0 8-bit sRGB 0.000u 0:00.000
psb:3layer.psb=>3layer.psb[4] PSB 9900x14400 9900x14400+0+0 8-bit sRGB 0.000u 0:00.000

I've tested this against somewhat larger psb files. The 3layer.psb I've linked as a google drive link in my original post is about 975Mb. I have another file that exhibits the same failures that's 1.03GB in size. I have not tested smaller ones.

I have tested this on a system with 16GB of available RAM, and hundreds of GB of available disk space.

libtiff is 4.0.10_1
libjpeg is 9c

[fmw42]

PSB files are a variant of PSD. My mistake, I was thinking Big Tiff. I will let the IM developers check this.

[snibgo]

convert psb:/3layer.psb -limit memory $x -limit map $y -intent relative -sample "1024x768>" -quality 95 -colorspace sRGB -strip -auto-orient 3layer.jpg -debug all

This is a bad command, as an error message tells you:

convert: `all' @ error/convert.c/ConvertImageCommand/3297.

The command should end with your output filename. So IM is thinks your "3layer.jpg" is an input filename.

Put the debug at the start, not the end.

[fmw42]

Put the debug at the start, not the end.

He means right after convert.

[snibgo]

Thanks, yes, and I suggest the limits are also put there before the input is read, so the command could be:

Code: Select all

convert -debug all -limit memory $x -limit map $y psb:/3layer.psb -intent relative -sample "1024x768>" -quality 95 -colorspace sRGB -strip -auto-orient 3layer.jpg

"-intent" is a setting for "-profile", but you don't have a "-profile" so the "-intent" does nothing.

[ikelso82]

Thanks for all the feedback on the syntax. It has certainly helped me to get to where I understood a couple of the previous errors I was getting. I'm still having issues with a large file being converted without causing resourcing issues. The command I'm using is

Code: Select all

convert -debug all -limit memory 4GiB -limit map 4GiB -define registry:temporary-path=/imagemagicktmp psb:test.psb test-out.JPG

This seems to result in a core dump error along with a line that seems like a resource allocation issue.

Code: Select all

convert: MagickCore/resource.c:1025: RelinquishMagickResource: Assertion `resource_info.memory >= 0' failed.
Aborted (core dumped)

Is there anything to look at to get more information on what was going on when the process dumped?

[fmw42]

With IM 7 use magick not convert and not magick convert. Does that work?

[ikelso82]

Using just magick does not work.

Code: Select all

magick -debug all -limit memory 2GiB -limit map 2GiB -define registry:temporary-path=/imagemagicktmp psb:test.psb test-S.JPG

2019-04-09T14:46:17+00:00 28:58.450 165.560u 7.0.8 Cache magick[1352]: cache.c/WritePixelCachePixels/5811/Cache
test.psb[0][10181x1+0+14680]
2019-04-09T14:46:17+00:00 28:58.450 165.560u 7.0.8 Cache magick[1352]: cache.c/ClonePixelCacheRepository/842/Cache
Memory => Disk
magick: MagickCore/resource.c:1025: RelinquishMagickResource: Assertion `resource_info.memory >= 0' failed.
2019-04-09T14:46:39+00:00 29:21.050 165.700u 7.0.8 Exception magick[1352]: utility.c/ShredFile/1843/Exception
Failed to remove: /imagemagicktmp/magick-1352oXqKdwgGs1zu
2019-04-09T14:46:39+00:00 29:21.050 165.700u 7.0.8 Exception magick[1352]: utility.c/ShredFile/1843/Exception
Failed to remove: /imagemagicktmp/magick-1352ZefhuHy8wJRi
Aborted (core dumped)

[fmw42]

I am not an expert on this, but

magick: MagickCore/resource.c:1025: RelinquishMagickResource: Assertion `resource_info.memory >= 0' failed.

looks to me like you ran out of memory.

I will leave this to be commented or tested by the IM developers who have better resources for such a large image.

[ikelso82]

Yeah, I don't doubt that I ran out of memory. If I remove the limits the OOM killer in linux will kill the convert/imagemagick PID. I would expect that IM would utilize the temp disk space location if it can't get a handle to more memory from the system. Perhaps that's an incorrect assumption though? Or perhaps there's a timeout on how long it will wait while trying to write out to the temp space location and I'm running into I/O issues?

[snibgo]

Code: Select all

magick: MagickCore/resource.c:1025: RelinquishMagickResource: Assertion `resource_info.memory >= 0' failed.

IM is trying to relinquish (release) memory, not allocate it. I suspect a bug somewhere in IM.

[magick]

Thanks for the problem report. We can reproduce it and will have a patch to fix it in GIT master branch @ https://github.com/ImageMagick/ImageMagick later this week. The patch will be available in the beta releases of ImageMagick @ https://www.imagemagick.org/download/beta/ likely by the weekend.