using disk for processing of huge images

[rogach]

Is there a way to tell ImageMagick to use in-memory processing for smaller images, but use disk when files are too big?

For example, I have a following command that requires a lot (~35Gb) of memory:

Code: Select all

$ time convert xc:white -extent 30000x30000 white.png
convert: unable to write pixel cache '/tmp/magick-30697WJTJFL2B7hAI': No space left on device @ error/cache.c/WritePixelCachePixels/5621.
convert: unable to extend cache 'white.png': No space left on device @ error/cache.c/OpenPixelCache/3684.
convert: memory allocation failed `white.png' @ error/png.c/WriteOnePNGImage/9262.

real	0m32.975s
user	0m22.634s
sys	0m10.289s

As you can see, it fails to finish. I tried using -limit options:

Code: Select all

$ time convert -limit memory 8gb -limit map 8gb -limit area 1000MP xc:white -extent 30000x30000 white.png
convert: unable to write pixel cache '/tmp/magick-27824B6CVaIqN2A0C': No space left on device @ error/cache.c/WritePixelCachePixels/5621.
convert: unable to write pixel cache '/tmp/magick-278242gCe9p110hp2': No space left on device @ error/cache.c/WritePixelCachePixels/5621.
convert: memory allocation failed `white.png' @ error/png.c/WriteOnePNGImage/9262.

real	0m33.451s
user	0m22.736s
sys	0m10.581s

The result stays exactly the same - I guess because /tmp is not actual disk, but tmpfs on most systems and is kept in memory.

Well, since we really want to use actual disk, we should instruct ImageMagick to use proper on-disk temp directory - so I tried setting registry:temporary-path:

Code: Select all

$ time convert -limit memory 8gb -limit map 8gb -limit area 1000MP -define registry:temporary-path=/var/tmp xc:white -extent 30000x30000 white.png

real	11m41.129s
user	1m37.469s
sys	1m41.773s

Now it works correctly. However, if I now use this options on small processing loads (that do fit into memory limit), performance is down 4 times (and it's on fast SSD, on slower disks it would be even worse):

Code: Select all

$ time convert xc:white -extent 10000x10000 white.png

real	0m7.166s
user	0m6.383s
sys	0m1.942s
$ time convert  -limit memory 8gb -limit map 8gb -limit area 1000MP -define registry:temporary-path=/var/tmp xc:white -extent 10000x10000 white.png

real	0m26.342s
user	0m6.815s
sys	0m6.935s

Thus my question - why ImageMagick still uses temporary directory, if processing fits into memory? Is there a way to make it skip disk if there is plenty of memory?

These test were done using ImageMagick 7.0.7-22, on Linux 4.14.13-1-ARCH.

[snibgo]

Weirdly, the case matters. Use "8GB" (capitals) not "8gb" (lower case).

[rogach]

Wow! Thanks!