ImageMagick stalls when doing basic operations on large imag

[oneandonlydave]

I'm trying to perform some operations on an image that I would guess are relatively basic (rotate, resize and extent/gravity) but when I get to a certain size, the intermediate images are so large as to cause Imagemagick to stall. I think it gets into a mode where it starts thrashing the desk and the memory manager just can't successfully allocate enough memory for the process to finish. I have tried giving it a few minutes with no luck.

Here is the version and convert command I am running. I have tried it without the -limit commands but it just fails sooner in the processing.

The image that I'm running the command on is available at:

Any ideas or suggestions on how to improve the command?

Code: Select all

$ convert -version
Version: ImageMagick 6.3.3 05/28/07 Q16 file:///usr/local/share/doc/ImageMagick-6.3.3/index.html
Copyright: Copyright (C) 1999-2007 ImageMagick Studio LLC

$ convert -list resource
File       Area     Memory        Map       Disk
------------------------------------------------
22118      488mb      366mb  1.90625gb        4eb

$ convert -debug "All" -log "%u %m:%l %e" -limit memory 2048 -limit area 2048 /tmp/map_dubai.png -resize 5648x4861\! -background 'rgba(255,255,255,0)' -rotate 39.151047861254 -extent 7656x7379 -gravity SouthEast -extent 7680x7424 /tmp/pre_cut_map_dubai.png.gif


2007-08-20T03:08:26-07:00 0:01 0.000u 6.3.3 Configure convert[18278]: utility.c/unknown/812/Configure
  Command line: convert{-debug}{All}{-log}{%u %m:%l %e}{-limit}{memory}{2048}{-limit}{area}{2048}{/tmp/map_dubai.png}{-resize}{5648x4861!}{-background}{rgba(255,255,255,0)}{-rotate}{39.151047861254}{-extent}{7656x7379}{-gravity}{SouthEast}{-extent}{7680x7424}{/tmp/pre_cut_map_dubai.png.gif}
0.000u blob.c:2185   read 4096 magic header bytes
0.000u cache.c:1867 destroy 
0.000u configure.c:482 Searching for configure file: "/usr/local/lib/ImageMagick/config/magic.xml"
0.000u configure.c:482 Searching for configure file: "/usr/local/share/ImageMagick/config/magic.xml"
0.000u configure.c:482 Searching for configure file: "/usr/local/share/doc/ImageMagick-6.3.3/magic.xml"
0.000u configure.c:482 Searching for configure file: "/usr/local/share/ImageMagick-6.3.3/magic.xml"
0.000u configure.c:482 Searching for configure file: "/home/dgyang/.magick/magic.xml"
0.000u configure.c:482 Searching for configure file: "magic.xml"
0.000u magic.c:673 Loading magic map "/usr/local/share/ImageMagick/config/magic.xml" ...
0.000u png.c:2808 enter ReadPNGImage()
0.000u blob.c:2185   read 4096 magic header bytes
0.000u png.c:1677   enter ReadOnePNGImage()
0.000u png.c:1798     PNG width: 1800, height: 1522
0.000u png.c:1801     PNG color_type: 6, bit_depth: 8
0.000u png.c:1804     PNG compression_method: 0
0.000u png.c:1807     PNG interlace_method: 0, filter_method: 0
0.000u png.c:2207     Reading PNG IDAT chunk(s)
0.000u png.c:2219     Converting PNG pixels to pixel packets
0.000u resource.c:216 Area: 20.9015mb/20.9015mb/2gb
0.000u resource.c:216 Memory: 20.9015mb/20.9015mb/2gb
0.000u cache.c:3240 open /tmp/map_dubai.png[0] (anonymous memory, 1800x1522 20.9015mb)
0.406u png.c:2643     Reading PNG text chunk
0.406u png.c:2665       Keyword: Software
0.406u png.c:2769   exit ReadOnePNGImage()
0.406u png.c:2873 exit ReadPNGImage()
0.406u resource.c:216 Area: 209.464mb/209.464mb/2gb
0.406u resource.c:216 Memory: 209.464mb/230.366mb/2gb
0.406u cache.c:3240 open /tmp/map_dubai.png[0] (anonymous memory, 5648x4861 209.464mb)
0.406u cache.c:1637 memory => memory
0.492u resource.c:216 Area: 66.7557mb/66.7557mb/2gb
0.492u resource.c:216 Memory: 66.7557mb/297.122mb/2gb
0.492u cache.c:3240 open /tmp/map_dubai.png[0] (anonymous memory, 1800x4861 66.7557mb)
0.492u cache.c:1637 memory => memory
2.516u cache.c:3685 /tmp/map_dubai.png[0][2x4861+0+0]
2.516u cache.c:4597 /tmp/map_dubai.png[0][1x4861+0+0]
2.523u cache.c:3685 /tmp/map_dubai.png[0][2x4861+0+0]
2.523u cache.c:4597 /tmp/map_dubai.png[0][1x4861+1+0]
2.523u cache.c:3685 /tmp/map_dubai.png[0][3x4861+0+0]
2.539u cache.c:4597 /tmp/map_dubai.png[0][1x4861+2+0]
2.539u cache.c:3685 /tmp/map_dubai.png[0][3x4861+0+0]
2.539u cache.c:3685 /tmp/map_dubai.png[0][3x4861+0+0]
2.539u cache.c:4597 /tmp/map_dubai.png[0][1x4861+4+0]
2.539u cache.c:3685 /tmp/map_dubai.png[0][4x4861+0+0]
2.547u cache.c:3685 /tmp/map_dubai.png[0][4x4861+0+0]
2.547u cache.c:3685 /tmp/map_dubai.png[0][4x4861+0+0]
2.547u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1+0]
2.547u cache.c:4597 /tmp/map_dubai.png[0][1x4861+8+0]
2.547u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1+0]
2.555u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1+0]
2.555u cache.c:3685 /tmp/map_dubai.png[0][4x4861+2+0]
2.555u cache.c:3685 /tmp/map_dubai.png[0][4x4861+2+0]
2.563u cache.c:3685 /tmp/map_dubai.png[0][4x4861+2+0]
2.570u cache.c:4597 /tmp/map_dubai.png[0][1x4861+16+0]
2.570u cache.c:3685 /tmp/map_dubai.png[0][4x4861+4+0]
2.570u cache.c:3685 /tmp/map_dubai.png[0][4x4861+4+0]
2.570u cache.c:3685 /tmp/map_dubai.png[0][4x4861+4+0]
2.594u cache.c:3685 /tmp/map_dubai.png[0][4x4861+8+0]
2.594u cache.c:3685 /tmp/map_dubai.png[0][4x4861+8+0]
2.602u cache.c:3685 /tmp/map_dubai.png[0][4x4861+8+0]
2.602u cache.c:4597 /tmp/map_dubai.png[0][1x4861+32+0]
2.648u cache.c:3685 /tmp/map_dubai.png[0][4x4861+16+0]
2.648u cache.c:3685 /tmp/map_dubai.png[0][4x4861+16+0]
2.648u cache.c:3685 /tmp/map_dubai.png[0][4x4861+16+0]
2.664u cache.c:4597 /tmp/map_dubai.png[0][1x4861+64+0]
2.750u cache.c:3685 /tmp/map_dubai.png[0][4x4861+32+0]
2.750u cache.c:3685 /tmp/map_dubai.png[0][4x4861+32+0]
2.750u cache.c:3685 /tmp/map_dubai.png[0][4x4861+32+0]
2.797u cache.c:4597 /tmp/map_dubai.png[0][1x4861+128+0]
2.953u cache.c:3685 /tmp/map_dubai.png[0][4x4861+64+0]
2.953u cache.c:3685 /tmp/map_dubai.png[0][4x4861+64+0]
2.961u cache.c:3685 /tmp/map_dubai.png[0][4x4861+64+0]
3.055u cache.c:4597 /tmp/map_dubai.png[0][1x4861+256+0]
3.313u cache.c:4597 /tmp/map_dubai.png[0][1x4861+384+0]
3.352u cache.c:3685 /tmp/map_dubai.png[0][4x4861+128+0]
3.359u cache.c:3685 /tmp/map_dubai.png[0][4x4861+128+0]
3.359u cache.c:3685 /tmp/map_dubai.png[0][4x4861+128+0]
3.570u cache.c:4597 /tmp/map_dubai.png[0][1x4861+512+0]
3.828u cache.c:4597 /tmp/map_dubai.png[0][1x4861+640+0]
4.094u cache.c:4597 /tmp/map_dubai.png[0][1x4861+768+0]
4.172u cache.c:3685 /tmp/map_dubai.png[0][4x4861+256+0]
4.172u cache.c:3685 /tmp/map_dubai.png[0][4x4861+256+0]
4.180u cache.c:3685 /tmp/map_dubai.png[0][4x4861+256+0]
4.352u cache.c:4597 /tmp/map_dubai.png[0][1x4861+896+0]
4.617u cache.c:4597 /tmp/map_dubai.png[0][1x4861+1024+0]
4.875u cache.c:4597 /tmp/map_dubai.png[0][1x4861+1152+0]
4.992u cache.c:3685 /tmp/map_dubai.png[0][4x4861+384+0]
4.992u cache.c:3685 /tmp/map_dubai.png[0][4x4861+384+0]
5.000u cache.c:3685 /tmp/map_dubai.png[0][4x4861+384+0]
5.141u cache.c:4597 /tmp/map_dubai.png[0][1x4861+1280+0]
5.398u cache.c:4597 /tmp/map_dubai.png[0][1x4861+1408+0]
5.656u cache.c:4597 /tmp/map_dubai.png[0][1x4861+1536+0]
5.813u cache.c:3685 /tmp/map_dubai.png[0][4x4861+512+0]
5.813u cache.c:3685 /tmp/map_dubai.png[0][4x4861+512+0]
5.813u cache.c:3685 /tmp/map_dubai.png[0][4x4861+512+0]
5.922u cache.c:4597 /tmp/map_dubai.png[0][1x4861+1664+0]
6.180u cache.c:4597 /tmp/map_dubai.png[0][1x4861+1792+0]
6.445u cache.c:4597 /tmp/map_dubai.png[0][1x4861+1920+0]
6.633u cache.c:3685 /tmp/map_dubai.png[0][4x4861+640+0]
6.633u cache.c:3685 /tmp/map_dubai.png[0][4x4861+640+0]
6.633u cache.c:3685 /tmp/map_dubai.png[0][4x4861+640+0]
6.703u cache.c:4597 /tmp/map_dubai.png[0][1x4861+2048+0]
6.969u cache.c:4597 /tmp/map_dubai.png[0][1x4861+2176+0]
7.227u cache.c:4597 /tmp/map_dubai.png[0][1x4861+2304+0]
7.453u cache.c:3685 /tmp/map_dubai.png[0][4x4861+768+0]
7.453u cache.c:3685 /tmp/map_dubai.png[0][4x4861+768+0]
7.453u cache.c:3685 /tmp/map_dubai.png[0][4x4861+768+0]
7.484u cache.c:4597 /tmp/map_dubai.png[0][1x4861+2432+0]
7.750u cache.c:4597 /tmp/map_dubai.png[0][1x4861+2560+0]
8.008u cache.c:4597 /tmp/map_dubai.png[0][1x4861+2688+0]
8.273u cache.c:3685 /tmp/map_dubai.png[0][4x4861+896+0]
8.273u cache.c:4597 /tmp/map_dubai.png[0][1x4861+2816+0]
8.273u cache.c:3685 /tmp/map_dubai.png[0][4x4861+896+0]
8.273u cache.c:3685 /tmp/map_dubai.png[0][4x4861+896+0]
8.531u cache.c:4597 /tmp/map_dubai.png[0][1x4861+2944+0]
8.797u cache.c:4597 /tmp/map_dubai.png[0][1x4861+3072+0]
9.055u cache.c:4597 /tmp/map_dubai.png[0][1x4861+3200+0]
9.094u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1024+0]
9.094u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1024+0]
9.094u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1024+0]
9.313u cache.c:4597 /tmp/map_dubai.png[0][1x4861+3328+0]
9.578u cache.c:4597 /tmp/map_dubai.png[0][1x4861+3456+0]
9.836u cache.c:4597 /tmp/map_dubai.png[0][1x4861+3584+0]
9.906u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1152+0]
9.914u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1152+0]
9.914u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1152+0]
9.914u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1152+0]
10.102u cache.c:4597 /tmp/map_dubai.png[0][1x4861+3712+0]
10.359u cache.c:4597 /tmp/map_dubai.png[0][1x4861+3840+0]
10.625u cache.c:4597 /tmp/map_dubai.png[0][1x4861+3968+0]
10.727u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1280+0]
10.734u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1280+0]
10.734u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1280+0]
10.883u cache.c:4597 /tmp/map_dubai.png[0][1x4861+4096+0]
11.148u cache.c:4597 /tmp/map_dubai.png[0][1x4861+4224+0]
11.406u cache.c:4597 /tmp/map_dubai.png[0][1x4861+4352+0]
11.547u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1408+0]
11.555u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1408+0]
11.555u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1408+0]
11.664u cache.c:4597 /tmp/map_dubai.png[0][1x4861+4480+0]
11.930u cache.c:4597 /tmp/map_dubai.png[0][1x4861+4608+0]
12.188u cache.c:4597 /tmp/map_dubai.png[0][1x4861+4736+0]
12.367u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1536+0]
12.367u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1536+0]
12.375u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1536+0]
12.453u cache.c:4597 /tmp/map_dubai.png[0][1x4861+4864+0]
12.711u cache.c:4597 /tmp/map_dubai.png[0][1x4861+4992+0]
12.969u cache.c:4597 /tmp/map_dubai.png[0][1x4861+5120+0]
13.188u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1664+0]
13.188u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1664+0]
13.188u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1664+0]
13.234u cache.c:4597 /tmp/map_dubai.png[0][1x4861+5248+0]
13.484u cache.c:4597 /tmp/map_dubai.png[0][1x4861+5376+0]
13.742u cache.c:4597 /tmp/map_dubai.png[0][1x4861+5504+0]
13.992u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1792+0]
13.992u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1792+0]
14.000u cache.c:3685 /tmp/map_dubai.png[0][4x4861+1792+0]
14.000u cache.c:4597 /tmp/map_dubai.png[0][1x4861+5632+0]
14.031u cache.c:4597 /tmp/map_dubai.png[0][1x4861+5647+0]
14.039u resource.c:819 Memory: 66.7557mb/230.366mb/2gb
14.039u cache.c:1867 destroy /tmp/map_dubai.png[0]
14.047u resource.c:819 Memory: 20.9015mb/209.464mb/2gb
14.047u cache.c:1867 destroy /tmp/map_dubai.png[0]
14.047u resource.c:216 Area: 209.464mb/209.464mb/2gb
14.047u resource.c:216 Memory: 209.464mb/418.929mb/2gb
14.047u cache.c:3240 open /tmp/map_dubai.png[0] (anonymous memory, 5648x4861 209.464mb)
14.047u cache.c:1637 memory => memory
16.703u resource.c:216 Area: 209.464mb/209.464mb/2gb
16.703u resource.c:216 Memory: 209.464mb/628.393mb/2gb
16.703u cache.c:3240 open /tmp/map_dubai.png[0] (anonymous memory, 5648x4861 209.464mb)
16.703u cache.c:1637 memory => memory
17.063u resource.c:216 Area: 1.53927gb/1.53927gb/2gb
17.063u resource.c:216 Memory: 1.53927gb/2.15294gb/2gb
17.063u resource.c:819 Memory: 1.53927gb/628.393mb/2gb
17.063u resource.c:216 Disk: 1.53927gb/1.53927gb/4eb
17.063u resource.c:406 /var/tmp/magick-m0wJK77D
17.063u resource.c:216 File: 0b/0b/21.5996kb
17.063u cache.c:3153 extend /tmp/map_dubai.png[0] (/var/tmp/magick-m0wJK77D[3], disk, 1.53927gb)
17.063u resource.c:216 Area: 1.53927gb/1.53927gb/2gb
17.063u resource.c:216 Map: 1.53927gb/1.53927gb/1.90625gb
17.063u resource.c:819 Map: 1.53927gb/0b/1.90625gb
17.063u cache.c:3353 open /tmp/map_dubai.png[0] (/var/tmp/magick-m0wJK77D[3], disk, 14366x14381 1.53927gb)
17.063u cache.c:1485 memory => disk

[magick]

Try adding -limit memory 1 -limit map 1 to your command line. It will be slow but it will run without thrashing your system. Make sure you have plenty of free space in /tmp.

[oneandonlydave]

Try adding -limit memory 1 -limit map 1 to your command line. It will be slow but it will run without thrashing your system. Make sure you have plenty of free space in /tmp.

Hi - I tried setting those two limits and let it run for ~45 min and it just kept making disk write commands. I guess what I would like to know is if there is a way to avoid loading the entire image into the pixel cache and causing it to grow to such a large size.

Another question is: is it possible to set a limit so that if IM determines it needs to start swapping to disk it just cancels the entire operation chain and returns an error exit code?

[magick]

You can monitor progress with the -monitor or -debug cache option. The operation will finish as long as ImageMagick does not run out of temporary disk space. You can force ImageMagick to exit if it caches pixels to disk with -limit disk 0. To significantly speed up ImageMagick use modern hardware with a Linux OS and 4+ GB of memory and 15000RPM SAS disk. Alternatively, use some other package to perform your image processing operation. See http://www.imagemagick.org/script/architecture.php for a discussion.

[oneandonlydave]

You can monitor progress with the -monitor or -debug cache option. The operation will finish as long as ImageMagick does not run out of temporary disk space. You can force ImageMagick to exit if it caches pixels to disk with -limit disk 0. To significantly speed up ImageMagick use modern hardware with a Linux OS and 4+ GB of memory and 15000RPM SAS disk. Alternatively, use some other package to perform your image processing operation. See http://www.imagemagick.org/script/architecture.php for a discussion.

Hi magick - thanks for the tip. When you say use some other package, do you mean some other kind of method of storing images (like streaming versus pixel cache) or just another package altogether? The machine I'm running on is FreeBSD 4x with 4gb of memory and a SCSI disk so I don't think that is the issue. I'll try with monitor and debug to see what's going on.

david

[magick]

Can you upgrade your version of ImageMagick? You might get better results. Not sure why its failing for you. We can rotate your image in seconds. You could try increasing your limits: -limit area 8192 -limit memory 8192 and see if that makes a difference.

[oneandonlydave]

Can you upgrade your version of ImageMagick? You might get better results. Not sure why its failing for you. We can rotate your image in seconds. You could try increasing your limits: -limit area 8192 -limit memory 8192 and see if that makes a difference.

Hi magick - I think it's the extent commands because the rotate seems to complete when I use -monitor but the extent commands never start. I'm going to compile a new version (-extent is relatively new right) and try it out.

Thanks again for all the help.

David