convert or mogrify failing with resource exhaustion

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
Post Reply
getut
Posts: 4
Joined: 2018-02-02T09:41:37-07:00
Authentication code: 1152

convert or mogrify failing with resource exhaustion

Post by getut »

How can I get convert or mogrify to tell me WHICH resource is getting exhausted?

I'm trying to convert large multi page color PDF's (8Mb 20-30 page docs) to monochrome and nothing I have tried has gotten me past the resource exhaustion.

I'm using a pretty powerful machine running Ubuntu 17.10 with ~50GB free and 16GB of ram. I've gone in to policy.xml and increased limits on every enabled policy (I even turned the file policy on) and still get the failures.

Code: Select all

  <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
  <policy domain="resource" name="memory" value="3GiB"/>
  <policy domain="resource" name="map" value="3GiB"/>
  <policy domain="resource" name="width" value="24KP"/>
  <policy domain="resource" name="height" value="24KP"/>
  <policy domain="resource" name="area" value="1024MB"/>
  <policy domain="resource" name="disk" value="10GiB"/>
  <policy domain="resource" name="file" value="1536"/>
I've tried running the command with command line switches to increase limits and still get failures. Why doesn't the error message plainly spell out which one is causing it, or is it doing so in a way that I don't understand?
Here is the command I am running.

Code: Select all

find . -iname '*.pdf' -print0|xargs -0 -L 1 mogrify -density 1200 -threshold 70% -resample 100 -limit file 2000 -limit disk 4GiB -verbose
Here is sample output with verbose turned on:

Code: Select all

'gs' -sstdout=%stderr -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 '-sDEVICE=pngalpha' -dTextAlphaBits=4 -dGraphicsAlphaBits=4 '-r1200x1200'  '-sOutputFile=/tmp/magick-25241wrDLmeuXFXZu%d' '-f/tmp/magick-25241PcpVgIkhhAHw' '-f/tmp/magick-25241gZIZwEpcUipy'
/tmp/magick-25241wrDLmeuXFXZu1 PNG 10200x13200 10200x13200+0+0 8-bit sRGB 1.796MB 2.230u 0:02.230
/tmp/magick-25241wrDLmeuXFXZu2 PNG 10200x13200 10200x13200+0+0 8-bit sRGB 1.522MB 2.570u 0:02.560
/tmp/magick-25241wrDLmeuXFXZu3 PNG 10200x13200 10200x13200+0+0 8-bit sRGB 1.515MB 2.700u 0:02.709
./CP 2017/S/TEST_12-28-17-100153266.pdf[0] PDF 10200x13200 10200x13200+0+0 16-bit sRGB 1.796MB 5.280u 0:05.269
./CP 2017/S/TEST_12-28-17-100153266.pdf[1] PDF 10200x13200 10200x13200+0+0 16-bit sRGB 1.796MB 2.710u 0:02.709
./CP 2017/S/TEST_12-28-17-100153266.pdf[2] PDF 10200x13200 10200x13200+0+0 16-bit sRGB 1.796MB 0.010u 0:00.000
./CP 2017/S/TEST12-28-17-100153266.pdf[0] PDF 10200x13200=>850x1100 850x1100+0+0 16-bit GrayscaleAlpha Gray 354KB 26.190u 0:10.469
mogrify-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
mogrify-im6.q16: cache resources exhausted `/tmp/magick-25241wrDLmeuXFXZu4' @ error/cache.c/OpenPixelCache/3982.
Top
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: convert or mogrify failing with resource exhaustion

Post by snibgo »

wrote:... with command line switches to increase limits ...
But they won't increase limits. They can only decrease limits.

Try commenting out all the limits in policy.xml.
snibgo's IM pages: im.snibgo.com
Top
getut
Posts: 4
Joined: 2018-02-02T09:41:37-07:00
Authentication code: 1152

Re: convert or mogrify failing with resource exhaustion

Post by getut »

<head slap> 3 days of reading pages and pages and not once was there a mention that those could only be tuned DOWN.

It worked. Thank you!
Top
User avatar
fmw42
Posts: 25562
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: convert or mogrify failing with resource exhaustion

Post by fmw42 »

The system manager can change the master policy.xml to increase the values. But on a server, you can only decrease the values. If this is your own personal computer, then there should be a way to modify the master values.
Top
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: convert or mogrify failing with resource exhaustion

Post by snibgo »

@getut: http://www.imagemagick.org/script/resources.php says policy.xml is used to set resource limits, and the final sentence says:
In addition, resource policies are maximums for each instance of ImageMagick (e.g. policy memory limit 1GB, the "-limit 2GB" option exceeds policy maximum so memory limit is 1GB).
The intention is that individual users can't relax system-wide limits on resource usage.
snibgo's IM pages: im.snibgo.com
Top
getut
Posts: 4
Joined: 2018-02-02T09:41:37-07:00
Authentication code: 1152

Re: convert or mogrify failing with resource exhaustion

Post by getut »

Ok.. now that it is technically "working". I can see what the problem was. And it was definitely disk space. But that leads to the realization that something is still wrong even though it is succeeding.

I am testing on an 18.6MB PDF file with 70 pages. System monitor showed over 40 Gigabytes !!!!!! of disk write activity before it converted that file.

What is going wrong?
Top
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: convert or mogrify failing with resource exhaustion

Post by snibgo »

The MB in the PDF file isn't relevant. How many pixels are there? Each pixel takes about 8 bytes of memory.
snibgo's IM pages: im.snibgo.com
Top
User avatar
fmw42
Posts: 25562
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: convert or mogrify failing with resource exhaustion

Post by fmw42 »

Adding to snibgo's comment, your -density 1200 is going to create a huge rasterize PDF. It is the size of the rasterized PDF that is going to be huge and controls the amount of memory and/or disk space needed. Multiply the density by the dimensions in inches for your PDF to get the dimensions in pixels.
Top
getut
Posts: 4
Joined: 2018-02-02T09:41:37-07:00
Authentication code: 1152

Re: convert or mogrify failing with resource exhaustion

Post by getut »

ok, then is there a way to have convert simply extract the existing images (these are all scans from a copier), maintain the density and just convert to black and white? Everything else that I tried resulted in a dramatic loss of readability from the original scan.
They don't need to be very high quality in the end as long as they don't lose quality during the conversion.

Now that I intercepted our shadow IT starting this process on their own and have the source documents being scanned properly from the beginning the 100 - 150 density range in black and white is very adequate. Its just the 300 density in color using default conversion settings makes it all but unreadable.
Top
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: convert or mogrify failing with resource exhaustion

Post by snibgo »

IM (with Ghostscript) is great at rasterizing PDF pages. But I don't think that's what you want to do.

Your PDF contains raster images, and you want to extract them. The tool for that job is "pdfimages".
snibgo's IM pages: im.snibgo.com
Top
Post Reply

Return to “Users”