Very slow conversion of large PSD on RHEL5

[rzuris]

It just took 3 hours and 3 minutes to convert a 17460x5220 590Mb PSD to JPEG on a 32-bit RHEL5 server with 4 CPUs and 8Gb RAM. This server also has Apache and MySQL, and performance of those was suffering quite a bit during this convert process (which was triggered by a CLI PHP script). The resident memory was tapped out at 2Gb and it was using swap for part of the process, which no doubt was mainly responsible for the terrible performance, but I don't understand why it took so much memory as to need swap space. Can someone help me understand better how ImageMagick allocates memory for conversions?

The convert command used these options:

nice convert -limit map 1536mb -limit memory 1536mb -colorspace RGB -scale 10000x10000> -scale 1000x1000< -quality 85 -define registry:temporary-path=/opt/lightbox/magick theInputFile.psd theOutputFile.jpg

OS: RHEL5 32-bit
IM: 6.4.9-10 Q16
gs: 8.64
input format: PSD
WxH: 17460x5220
bytes: 590Mb


I'm curious why the "top" command showed 2.1Gb of resident memory when I set the memory limit to 1536mb. Does it add the 590Mb image size in bytes to the pixel cache size to arrive at 2.1Gb? I'm also curious why it is 2.1Gb instead of 2.0Gb when a 32-bit process can only address 2Gb of memory. But regardless, why does it need that much memory? Doesn't it take 4 bytes to store each pixel in the pixel cache (or does Q16 require 8 bytes per pixel?) For 17460x5220 pixels, that's either 348Mb at 4 bytes per pixel or 695Mb. And the output image is being constrained to a maximum of 10000 pixels in either direction, so it will take even less memory to store the output pixel cache. Why is this running out of memory and causing swapping, with the resulting horrible performance? Suggestions for improving this? I think we're going to switch to 64-bit RHEL and more RAM, but the numbers don't add up to me; the current environment ought to be able to handle this better.

[magick]

See http://www.imagemagick.org/script/architecture.php. The memory limit only affects the pixel cache. There are numerous algorithms that require heap memory allocation that adds to the limit you set. The conversion is most likely slow because the pixels were cached to disk which is 1000 times slower than memory. A 64-bit system may solve the problem. We've have had some conversions go from minutes to seconds on a 64-bit system (with sufficient memory). If caching to disk is required, fast disks (e.g. 15K SAS) can be a significant speed up. Finally, make sure your version of ImageMagick supports OpenMP (type convert -version). If so it can run most algorithms across all your cores simultaneously.

[rzuris]

We are moving to a 64-bit server with 36Gb of RAM and 8 CPU to solve this issue. We configure our app to be able to convert multiple files in parallel, so I need to set a memory limit for each instance of IM that will be large enough to process most high-resolution images like this 17460x5220 PSD without disk swapping, but not so large that we cannot process multiple files in parallel processes. So for a 17460x5220 image with the Q16 version, would that be 17460 * 5220 2 * 4 = 695Mb RAM for the pixel cache? I ran this conversion with -memory limit 1536mb, so I had plenty of room for the pixel cache, and the memory allocation topped out at 2Gb since this was a 32-bit server. That seems like a lot of memory besides the pixel cache. Is there a reasonable way to calcuate how much memory to reserve per process for an image of a given size, or is it just trial and error?

[magick]

Add -debug cache to your command line. For a particular command line conversion, it tells you whether the image pixels are in memory, memory-mapped, or on disk. Keep in mind that the memory limit is for pixels only. Individual algorithms may require temporary buffers that are not accounted for by the -limit option.