Details about resources management

[narcisgarcia]

I need to process 200 images (from a PDF) at once, but avoiding much disk work derived from RAM exhaustion.

Previous status:

Code: Select all

$ convert --version
Version: ImageMagick 6.6.9-7 2012-08-17 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2011 ImageMagick Studio LLC
Features: OpenMP

$ free -m
             total       used       free     shared    buffers     cached
Mem:         12000       4342       7658          0        310       2396
-/+ buffers/cache:       1635      10365
Swap:         4095          0       4095

$ identify -list resource
File         Area       Memory          Map         Disk    Thread         Time
-------------------------------------------------------------------------------
 768     3.1458GB    11.719GiB    23.438GiB    unlimited         4    unlimited

The following are the commands executed in a script:

Code: Select all

export MAGICK_MEMORY_LIMIT=6631080
export MAGICK_MAP_LIMIT=6631080
export MAGICK_DISK_LIMIT=1105180
export MAGICK_FILE_LIMIT=10
mkdir 00pages
gs -sDEVICE=png16m -r300 -dNOPAUSE -dQUIET -dBATCH "-sOutputFile=./00pages/pag-%04d.png" "book1.pdf"
mkdir 01contrast
convert "./00pages/*.png" -quality 40 -brightness-contrast -33x40 "./01contrast/pag-%04d.png"

I've tried a lot of combinations for memory, map and disk limits, but in all cases ImageMagick breaks with a "Segmentation fault". Monitorizing resources usage, I see that the process works during 30 seconds normally (using few memory), and in the next 5 seconds eats all memory available and breaks. This last test was deactivating swap, to avoid waiting minutes or hours before failure.

I've read the details described at:
http://www.imagemagick.org/script/architecture.php
http://www.imagemagick.org/script/resources.php

But I have some questions not cleared in the documentation:

1. When ImageMagick is called to process several files (with wildcards), does it process one by one, or all at once? If it's all at once, how can I order to process in smaller series?
2. Can I specify all limits in bytes every time (memory, map, disk)?
3. The order of resource usage is: memory -> map (areas) -> disk (files). Am I right? If is there enough memory, I suppose map and disk aren't used; right?
4. Is the "pixel cache memory" a kind of disk cache for "pixel cache files"?
5. If I get a page (DIN-A4, 300ppi, 24bpp) I calculate that its binary size is 25MiB. Is this the necessary area for this single image?
6. Does ImageMagick need the possibility to open one pixel cache file per image, or more than one?
7. I understand memory variables for main working area, and map variables for a subsidiary area. Do have them any relation with using or not using system's swap memory? How can I equilibrate the numbers to maximize performance and avoid disk?

[snibgo]

1. "convert" reads all the images, then processes each one. You could put it in a loop in a script to process one at a time. Or use "mogrify".

As your IM is 16-bit, it needs 2 bytes per channel per pixel. So each page needs 50 M byte. 200 pages would need 10 GB. If you don't need 16 bits, then using Q8 will half the memory requirements.

Your IM is rather old, but I doubt this makes any difference.

[narcisgarcia]

snibgo, do you mean that convert LOADS all the images to memory and maintain them there while processes one by one? If all the images need 10GiB and I have 11GiB of physical memory available, this should not be a problem without limiting imagik resources. Then, which is the step that passes to use other resources than RAM?

Do you mean also that mogrify only loads to memory the image that is processing, and unloads it before processing next one?
Next processes I need to do to the images with my script are: posterize, resample and remake a final PDF file.

I'm using imagemagick from the main repository in Ubuntu 12.04. Can I specify to work in Q8 mode?

[snibgo]

Yes, "convert" first loads all the images. I think it then processes them all, then finally saves them all. So 10 GB for loading, and maybe another 10 GB for processing. You might do tests on a small subset to verify this.

Yes, "mogrify" loads one file, processes it and saves, then moves on to the next one.

I don't use mogrify much, so I would write a script and call convert once per file. But mogrify is quicker, if it does what you need.

Q8 isn't a mode. You need to recompile if you want Q8. That would be reasonable for just changing brightness and contrast, but might be too low quality for posterize etc.