Legacy ImageMagick Discussions Archive

Use https://github.com/ImageMagick/ImageMagick/discussions instead.
https://imagemagick.com/discourse-server/

unable to convert large PSB files

https://imagemagick.com/discourse-server/viewtopic.php?t=35791

Page 1 of 1

unable to convert large PSB files

Posted: 2019-04-03T14:26:31-07:00
by ikelso82
Environments tested within:
OSX 10.14.3 --- Version: ImageMagick 7.0.8-37 Q16 x86_64 2019-04-03
Docker Container running "FROM jboss/wildfly:9.0.2.Final" that has imagemagick installed --- Version: ImageMagick 7.0.8-34 Q16 x86_64 2019-03-18

Commands used:

convert psb:/3layer.psb -limit memory $x -limit map $y -intent relative -sample "1024x768>" -quality 95 -colorspace sRGB -strip -auto-orient 3layer.jpg -debug all

convert psb:/3layer.psb 3layer.jpg

I've tried multiple iterations of the above commands, attempting to set limits to 0, 1 and 2, 2 and 4, 4 and 8. I've tried setting the -limit area to 0, -limit disk to 0, and tried multiple combinations of all of these limits. I've attempted to change the temporary file location using the environment variable MAGICK_TEMPORARY_DIRECTORY just in case we're running into a /tmp folder size limitation, but imagemagick does not seem to adhere to this. Nothing I've tried has resulted in large PSB files with multiple layers being converted successfully. This feels like a resource constraint issue, but I also found a previous thread with one of my exact errors using a PSD file, so perhaps this is a bug still.

I've attempted this with 7.0.8-36 in OSX as well as -37 and I get the following error returned:
2019-04-03T14:38:12-05:00 7:38.978 393.330u 7.0.8 Exception convert[47211]: blob.c/OpenBlob/3485/Exception
unable to open image 'testconvert.jpg': No such file or directory
2019-04-03T14:38:12-05:00 7:38.978 393.330u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy
2019-04-03T14:38:12-05:00 7:38.978 393.330u 7.0.8 Policy convert[47211]: policy.c/IsRightsAuthorized/609/Policy
Domain: Path; rights=Read; pattern="testconvert.jpg" ...
2019-04-03T14:38:12-05:00 7:38.978 393.330u 7.0.8 Exception convert[47211]: blob.c/OpenBlob/3485/Exception
unable to open image 'testconvert.jpg': No such file or directory
2019-04-03T14:38:12-05:00 7:38.978 393.330u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy
2019-04-03T14:38:12-05:00 7:38.981 393.330u 7.0.8 Exception convert[47211]: convert.c/ConvertImageCommand/3297/Exception
`all'
2019-04-03T14:38:12-05:00 7:38.981 393.330u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy TestConvert.psb[0]
2019-04-03T14:38:13-05:00 7:39.206 393.550u 7.0.8 Resource convert[47211]: resource.c/RelinquishMagickResource/1069/Resource
Memory: 2280960000B/11.6837GiB/16GiB
2019-04-03T14:38:13-05:00 7:39.206 393.550u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy TestConvert.psb[0]
2019-04-03T14:38:13-05:00 7:39.243 393.590u 7.0.8 Resource convert[47211]: resource.c/RelinquishMagickResource/1069/Resource
Memory: 1710720000B/10.0905GiB/16GiB
2019-04-03T14:38:13-05:00 7:39.244 393.590u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy TestConvert.psb[0]
2019-04-03T14:38:13-05:00 7:39.285 393.630u 7.0.8 Resource convert[47211]: resource.c/RelinquishMagickResource/1069/Resource
Memory: 2280960000B/8553600000B/16GiB
2019-04-03T14:38:13-05:00 7:39.285 393.630u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy TestConvert.psb[0]
2019-04-03T14:38:13-05:00 7:39.330 393.680u 7.0.8 Resource convert[47211]: resource.c/RelinquishMagickResource/1069/Resource
Memory: 2280960000B/6272640000B/16GiB
2019-04-03T14:38:13-05:00 7:39.331 393.680u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy TestConvert.psb[0]
2019-04-03T14:38:13-05:00 7:39.372 393.720u 7.0.8 Resource convert[47211]: resource.c/RelinquishMagickResource/1069/Resource
Memory: 2280960000B/3991680000B/16GiB
2019-04-03T14:38:13-05:00 7:39.372 393.720u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy TestConvert.psb[0]
2019-04-03T14:38:13-05:00 7:39.415 393.760u 7.0.8 Resource convert[47211]: resource.c/RelinquishMagickResource/1069/Resource
Memory: 2280960000B/1710720000B/16GiB
2019-04-03T14:38:13-05:00 7:39.415 393.760u 7.0.8 Cache convert[47211]: cache.c/DestroyPixelCache/1040/Cache
destroy TestConvert.psb[0]
Assertion failed: (resource_info.memory >= 0), function RelinquishMagickResource, file MagickCore/resource.c, line 1025.
Abort trap: 6
Within my container i get the following error:
2019-04-03T20:44:00+00:00 0:46.170 65.530u 7.0.8 Resource convert[1080]: resource.c/RelinquishUniqueFileResource/1107/Resource
/tmp/magick-1080FcS4OgIyXP4X
2019-04-03T20:44:00+00:00 0:46.170 65.530u 7.0.8 Resource convert[1080]: resource.c/RelinquishMagickResource/1069/Resource
Disk: 6.1875MiB/0B/8EiB
convert: unable to open image '3layer.jpg': No such file or directory @ error/blob.c/OpenBlob/3485.
convert: `all' @ error/convert.c/ConvertImageCommand/3297.
You can find an example file here https://drive.google.com/open?id=1VlMOm ... -mPV6Z6NFx

Re: unable to convert large PSB files

Posted: 2019-04-03T14:52:05-07:00
by fmw42
How big is the TIFF? How much RAM do you have? What does

Code: Select all

identify psb:/3layer.psb
return?'

What are your versions of libtiff and libjpeg? What is your platform/OS?

Re: unable to convert large PSB files

Posted: 2019-04-03T17:39:16-07:00
by ikelso82
Sorry if I sound confused, is TIFF the same as PSB?

identify psb:3layer.psb shows:

psb:3layer.psb=>3layer.psb[0] PSB 9900x14400 9900x14400+0+0 8-bit sRGB 975.791MiB 0.600u 0:00.601
psb:3layer.psb=>3layer.psb[1] PSB 9900x14400 9900x14400+0+0 8-bit sRGB 0.000u 0:00.001
psb:3layer.psb=>3layer.psb[2] PSB 9900x14400 9900x14400+0+0 8-bit sRGB 0.000u 0:00.000
psb:3layer.psb=>3layer.psb[3] PSB 9900x14400 9900x14400+0+0 8-bit sRGB 0.000u 0:00.000
psb:3layer.psb=>3layer.psb[4] PSB 9900x14400 9900x14400+0+0 8-bit sRGB 0.000u 0:00.000

I've tested this against somewhat larger psb files. The 3layer.psb I've linked as a google drive link in my original post is about 975Mb. I have another file that exhibits the same failures that's 1.03GB in size. I have not tested smaller ones.

I have tested this on a system with 16GB of available RAM, and hundreds of GB of available disk space.

libtiff is 4.0.10_1
libjpeg is 9c

Re: unable to convert large PSB files

Posted: 2019-04-03T19:36:13-07:00
by fmw42
PSB files are a variant of PSD. My mistake, I was thinking Big Tiff. I will let the IM developers check this.

Re: unable to convert large PSB files

Posted: 2019-04-03T20:15:34-07:00
by snibgo
ikelso82 wrote:convert psb:/3layer.psb -limit memory $x -limit map $y -intent relative -sample "1024x768>" -quality 95 -colorspace sRGB -strip -auto-orient 3layer.jpg -debug all
This is a bad command, as an error message tells you:
ikelso82 wrote:convert: `all' @ error/convert.c/ConvertImageCommand/3297.
The command should end with your output filename. So IM is thinks your "3layer.jpg" is an input filename.

Put the debug at the start, not the end.

Re: unable to convert large PSB files

Posted: 2019-04-03T21:09:44-07:00
by fmw42
Put the debug at the start, not the end.


He means right after convert.

Re: unable to convert large PSB files

Posted: 2019-04-03T21:39:41-07:00
by snibgo
Thanks, yes, and I suggest the limits are also put there before the input is read, so the command could be:

Code: Select all

convert -debug all -limit memory $x -limit map $y psb:/3layer.psb -intent relative -sample "1024x768>" -quality 95 -colorspace sRGB -strip -auto-orient 3layer.jpg
"-intent" is a setting for "-profile", but you don't have a "-profile" so the "-intent" does nothing.

Re: unable to convert large PSB files

Posted: 2019-04-08T14:04:11-07:00
by ikelso82
Thanks for all the feedback on the syntax. It has certainly helped me to get to where I understood a couple of the previous errors I was getting. I'm still having issues with a large file being converted without causing resourcing issues. The command I'm using is

Code: Select all

convert -debug all -limit memory 4GiB -limit map 4GiB -define registry:temporary-path=/imagemagicktmp psb:test.psb test-out.JPG
This seems to result in a core dump error along with a line that seems like a resource allocation issue.

Code: Select all

convert: MagickCore/resource.c:1025: RelinquishMagickResource: Assertion `resource_info.memory >= 0' failed.
Aborted (core dumped)
Is there anything to look at to get more information on what was going on when the process dumped?

Re: unable to convert large PSB files

Posted: 2019-04-08T14:30:53-07:00
by fmw42
With IM 7 use magick not convert and not magick convert. Does that work?

Re: unable to convert large PSB files

Posted: 2019-04-09T08:00:21-07:00
by ikelso82
Using just magick does not work.

Code: Select all

magick -debug all -limit memory 2GiB -limit map 2GiB -define registry:temporary-path=/imagemagicktmp psb:test.psb test-S.JPG
2019-04-09T14:46:17+00:00 28:58.450 165.560u 7.0.8 Cache magick[1352]: cache.c/WritePixelCachePixels/5811/Cache
test.psb[0][10181x1+0+14680]
2019-04-09T14:46:17+00:00 28:58.450 165.560u 7.0.8 Cache magick[1352]: cache.c/ClonePixelCacheRepository/842/Cache
Memory => Disk
magick: MagickCore/resource.c:1025: RelinquishMagickResource: Assertion `resource_info.memory >= 0' failed.
2019-04-09T14:46:39+00:00 29:21.050 165.700u 7.0.8 Exception magick[1352]: utility.c/ShredFile/1843/Exception
Failed to remove: /imagemagicktmp/magick-1352oXqKdwgGs1zu
2019-04-09T14:46:39+00:00 29:21.050 165.700u 7.0.8 Exception magick[1352]: utility.c/ShredFile/1843/Exception
Failed to remove: /imagemagicktmp/magick-1352ZefhuHy8wJRi
Aborted (core dumped)

Re: unable to convert large PSB files

Posted: 2019-04-09T09:03:36-07:00
by fmw42
I am not an expert on this, but
magick: MagickCore/resource.c:1025: RelinquishMagickResource: Assertion `resource_info.memory >= 0' failed.
looks to me like you ran out of memory.

I will leave this to be commented or tested by the IM developers who have better resources for such a large image.

Re: unable to convert large PSB files

Posted: 2019-04-09T09:44:32-07:00
by ikelso82
Yeah, I don't doubt that I ran out of memory. If I remove the limits the OOM killer in linux will kill the convert/imagemagick PID. I would expect that IM would utilize the temp disk space location if it can't get a handle to more memory from the system. Perhaps that's an incorrect assumption though? Or perhaps there's a timeout on how long it will wait while trying to write out to the temp space location and I'm running into I/O issues?

Re: unable to convert large PSB files

Posted: 2019-04-09T10:31:18-07:00
by snibgo

Code: Select all

magick: MagickCore/resource.c:1025: RelinquishMagickResource: Assertion `resource_info.memory >= 0' failed.
IM is trying to relinquish (release) memory, not allocate it. I suspect a bug somewhere in IM.

Re: unable to convert large PSB files

Posted: 2019-04-09T15:22:40-07:00
by magick
Thanks for the problem report. We can reproduce it and will have a patch to fix it in GIT master branch @ https://github.com/ImageMagick/ImageMagick later this week. The patch will be available in the beta releases of ImageMagick @ https://www.imagemagick.org/download/beta/ likely by the weekend.
All times are UTC-07:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited