using disk for processing of huge images
Posted: 2018-01-29T11:32:46-07:00
Is there a way to tell ImageMagick to use in-memory processing for smaller images, but use disk when files are too big?
For example, I have a following command that requires a lot (~35Gb) of memory:
As you can see, it fails to finish. I tried using -limit options:
The result stays exactly the same - I guess because /tmp is not actual disk, but tmpfs on most systems and is kept in memory.
Well, since we really want to use actual disk, we should instruct ImageMagick to use proper on-disk temp directory - so I tried setting registry:temporary-path:
Now it works correctly. However, if I now use this options on small processing loads (that do fit into memory limit), performance is down 4 times (and it's on fast SSD, on slower disks it would be even worse):
Thus my question - why ImageMagick still uses temporary directory, if processing fits into memory? Is there a way to make it skip disk if there is plenty of memory?
These test were done using ImageMagick 7.0.7-22, on Linux 4.14.13-1-ARCH.
For example, I have a following command that requires a lot (~35Gb) of memory:
Code: Select all
$ time convert xc:white -extent 30000x30000 white.png
convert: unable to write pixel cache '/tmp/magick-30697WJTJFL2B7hAI': No space left on device @ error/cache.c/WritePixelCachePixels/5621.
convert: unable to extend cache 'white.png': No space left on device @ error/cache.c/OpenPixelCache/3684.
convert: memory allocation failed `white.png' @ error/png.c/WriteOnePNGImage/9262.
real 0m32.975s
user 0m22.634s
sys 0m10.289s
Code: Select all
$ time convert -limit memory 8gb -limit map 8gb -limit area 1000MP xc:white -extent 30000x30000 white.png
convert: unable to write pixel cache '/tmp/magick-27824B6CVaIqN2A0C': No space left on device @ error/cache.c/WritePixelCachePixels/5621.
convert: unable to write pixel cache '/tmp/magick-278242gCe9p110hp2': No space left on device @ error/cache.c/WritePixelCachePixels/5621.
convert: memory allocation failed `white.png' @ error/png.c/WriteOnePNGImage/9262.
real 0m33.451s
user 0m22.736s
sys 0m10.581s
Well, since we really want to use actual disk, we should instruct ImageMagick to use proper on-disk temp directory - so I tried setting registry:temporary-path:
Code: Select all
$ time convert -limit memory 8gb -limit map 8gb -limit area 1000MP -define registry:temporary-path=/var/tmp xc:white -extent 30000x30000 white.png
real 11m41.129s
user 1m37.469s
sys 1m41.773s
Code: Select all
$ time convert xc:white -extent 10000x10000 white.png
real 0m7.166s
user 0m6.383s
sys 0m1.942s
$ time convert -limit memory 8gb -limit map 8gb -limit area 1000MP -define registry:temporary-path=/var/tmp xc:white -extent 10000x10000 white.png
real 0m26.342s
user 0m6.815s
sys 0m6.935s
These test were done using ImageMagick 7.0.7-22, on Linux 4.14.13-1-ARCH.