How to decrease memory usage of convert?
Posted: 2013-06-14T00:41:38-07:00
Hello.
I'm stitching together a map from many many screenshots of a video game.
My machine specs: Intel i7M Quad-Core @ 2.90GHz - 16 GB RAM - 500 GB SSD Harddisk - Ubuntu 12.04
My MAGICK environment: MAGICK_MEMORY_LIMIT=12G MAGICK_TEMPORARY_PATH=.
My imagemagick version:
At first I did it this way:
This worked quite fine and took about 20-30 minutes for 150 screenshots.
When the image size started to exceed 65000 pixels in width, JPG couldn't cope with it anymore so I tiled the map into 25600x25600 px sized .mpc as I read that this would be the most read-write efficient format.
I changed my stitching script to use multiple commands - one for each tile - like this:
This can now take upward of half an hour for a dozen-or-so screenshots, for each of the - by now - 8 tiles. This will also take copious amounts of HD space: Temp files of 20 GB.
I then use the command `deepzoom' from perls Graphics::DZI package to create the DZI files. For this I first montage the tiles:
This montage command uses over 50 GB of temporary HD storage to create a 650 MB PNG file.
And this now uses over 30 GB of temporary HD storage to create the DZI tiles.
Now, swapping stuff in and out of HD cache takes a lot of time and is probably not good for the longevity of my harddisk. Is there a way to decrease the amount of cache needed? Am I using the convert and montage commands in "less than good" ways?
I'm stitching together a map from many many screenshots of a video game.
My machine specs: Intel i7M Quad-Core @ 2.90GHz - 16 GB RAM - 500 GB SSD Harddisk - Ubuntu 12.04
My MAGICK environment: MAGICK_MEMORY_LIMIT=12G MAGICK_TEMPORARY_PATH=.
My imagemagick version:
Code: Select all
$ identify -version
Version: ImageMagick 6.6.9-7 2013-06-09 Q8 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2011 ImageMagick Studio LLC
Features:
Code: Select all
convert -page +0+0 "map.jpg" \
-black-threshold 1% -transparent black -page +100+0 "shot001.jpg" \
-black-threshold 1% -transparent black -page +200+0 "shot003.jpg" \
-black-threshold 1% -transparent black -page +500+1000 "shot003.jpg" \
...
-black-threshold 1% -transparent black -page +$x+$y "shotxxx.jpg" \
+page -alpha Set -virtual-pixel transparent -background None -layers merge +repage newmap.jpg
When the image size started to exceed 65000 pixels in width, JPG couldn't cope with it anymore so I tiled the map into 25600x25600 px sized .mpc as I read that this would be the most read-write efficient format.
I changed my stitching script to use multiple commands - one for each tile - like this:
Code: Select all
convert old-0-0.mpc -transparent-color black -background black \
-page 10000x10000 shot001.jpg -transparent black -black-threshold 1% \
-page 20000x20000 shot002.jpg -transparent black -black-threshold 1% \
-page 25000x10000 shot003.jpg -transparent black -black-threshold 1% \ # This overlaps on the right side but does not expand the size of the old-0-0.mpc file
-page -500x1000 shot004.jpg -transparent black -black-threshold 1% \ # same for the left
-flatten mpc:new-0-0.mpc
I then use the command `deepzoom' from perls Graphics::DZI package to create the DZI files. For this I first montage the tiles:
Code: Select all
montage map-0-0.mpc map-25600-0.mpc map-51200-0.mpc map-76800-0.mpc map-0-25600.mpc map-25600-25600.mpc map-51200-25600.mpc map-76800-25600.mpc \
-tile 4x2 -geometry 25600x25600+0+0 output.png
Code: Select all
deepzoom output.png
Now, swapping stuff in and out of HD cache takes a lot of time and is probably not good for the longevity of my harddisk. Is there a way to decrease the amount of cache needed? Am I using the convert and montage commands in "less than good" ways?