Memory allocation failure on Windows when generating GIFs

[hunky-dory]

Main OS: Windows 7 with 8GB of memory
Virtual OS: Debian with 384MB of memory

Versions tried (on Windows):

- ImageMagick-6.8.7-5-Q16-x86-dll.exe
- ImageMagick-6.8.7-5-Q8-x86-dll.exe
- ImageMagick-6.8.7-5-Q16-x64-dll.exe

Output from identify -list resource (on Windows):

Code: Select all

  File       Area     Memory        Map       Disk   Thread  Throttle       Time
--------------------------------------------------------------------------------
  1536   8.4068GB  3.9147GiB  7.8294GiB  unlimited        4         0  unlimited

Versions tried (on virtual Debian):

- Version: ImageMagick 6.6.0-4 2012-05-02 Q16

Output from identify -list resource (on virtual Debian):

Code: Select all

File         Area       Memory          Map         Disk    Thread         Time
-------------------------------------------------------------------------------
 768      777.8MB     556.3MiB    1.4488GiB  18.446744EB         1    unlimited

I use the following script on Windows to generate GIFs from PNG files:

Code: Select all

@ECHO OFF
SETLOCAL
SET OutputName=%1
SET delay=%2
SET ImgMagConvert="C:\Program Files (x86)\ImageMagick-6.8.7-Q8"

IF "%1"=="" SET OutputName=.
IF "%2"=="" SET delay=4

IF NOT EXIST  "%OutputName%" (
MKDIR "%OutputName%"
)
@ECHO ON
%ImgMagConvert%\convert +repage -fuzz 0.0%% -delay %delay% -loop 0 frames/*.png -layers OptimizePlus -layers OptimizeTransparency "%OutputName%\Almost-00.gif"
%ImgMagConvert%\convert +repage -fuzz 0.5%% -delay %delay% -loop 0 frames/*.png -layers OptimizePlus -layers OptimizeTransparency "%OutputName%\Almost-05.gif"
%ImgMagConvert%\convert +repage -fuzz 1.0%% -delay %delay% -loop 0 frames/*.png -layers OptimizePlus -layers OptimizeTransparency "%OutputName%\Almost-10.gif"
%ImgMagConvert%\convert +repage -fuzz 1.5%% -delay %delay% -loop 0 frames/*.png -layers OptimizePlus -layers OptimizeTransparency "%OutputName%\Almost-15.gif"
%ImgMagConvert%\convert +repage -fuzz 2.0%% -delay %delay% -loop 0 frames/*.png -layers OptimizePlus -layers OptimizeTransparency "%OutputName%\Almost-20.gif"
%ImgMagConvert%\convert +repage -fuzz 2.5%% -delay %delay% -loop 0 frames/*.png -layers OptimizePlus -layers OptimizeTransparency "%OutputName%\Almost-25.gif"
%ImgMagConvert%\convert +repage -fuzz 3.0%% -delay %delay% -loop 0 frames/*.png -layers OptimizePlus -layers OptimizeTransparency "%OutputName%\Almost-30.gif"

gifsicle -O3 --colors 128 "%OutputName%\Almost-00.gif" >"%OutputName%\%OutputName%-00-128.gif"
gifsicle -O3 --colors 256 "%OutputName%\Almost-00.gif" >"%OutputName%\%OutputName%-00-256.gif"
gifsicle -O3 --colors 128 "%OutputName%\Almost-05.gif" >"%OutputName%\%OutputName%-05-128.gif"
gifsicle -O3 --colors 256 "%OutputName%\Almost-05.gif" >"%OutputName%\%OutputName%-05-256.gif"
gifsicle -O3 --colors 128 "%OutputName%\Almost-10.gif" >"%OutputName%\%OutputName%-10-128.gif"
gifsicle -O3 --colors 256 "%OutputName%\Almost-10.gif" >"%OutputName%\%OutputName%-10-256.gif"
gifsicle -O3 --colors 128 "%OutputName%\Almost-15.gif" >"%OutputName%\%OutputName%-15-128.gif"
gifsicle -O3 --colors 256 "%OutputName%\Almost-15.gif" >"%OutputName%\%OutputName%-15-256.gif"
gifsicle -O3 --colors 128 "%OutputName%\Almost-20.gif" >"%OutputName%\%OutputName%-20-128.gif"
gifsicle -O3 --colors 256 "%OutputName%\Almost-20.gif" >"%OutputName%\%OutputName%-20-256.gif"
gifsicle -O3 --colors 128 "%OutputName%\Almost-25.gif" >"%OutputName%\%OutputName%-25-128.gif"
gifsicle -O3 --colors 256 "%OutputName%\Almost-25.gif" >"%OutputName%\%OutputName%-25-256.gif"
gifsicle -O3 --colors 128 "%OutputName%\Almost-30.gif" >"%OutputName%\%OutputName%-30-128.gif"
gifsicle -O3 --colors 256 "%OutputName%\Almost-30.gif" >"%OutputName%\%OutputName%-30-256.gif"
@ENDLOCAL

Problem is, sometimes it works, sometimes it gives a lot of "memory allocation failed" errors or ImageMagick crashes resulting in incomplete GIFs. There seems to be no pattern to it; it may fail on very small gifs (small and few images) and succeed on large gifs (large and many images), but if it fails on certain images, it will keep failing for those even after a reboot, even though sometimes one of the commands that failed may succeed, while a command that completed will now fail. It's random is what I'm saying.

But! Those commands will still succeed on the virtual Debian OS with 384MB of memory for those images that fail on Windows.

So, how is this possible? How can I run out of memory on my main OS when it works on the virtual OS? What's causing it and how can I fix it? Thanks.

By the way, I have seen this: http://imagemagick.sourceforge.net/http/www/FAQ.html#C9, and there is zero possibility that I'm running out of memory when there's only about 30 images of size 250x184 which even with the highest Quantum depth should give about 920000 bytes required for one image which is what, 27.6MB for the whole GIF.

[magick]

Add -limit memory 128MiB to your convert command line.

[hunky-dory]

Add -limit memory 128MiB to your convert command line.

I think your post first said -limit memory 0, so I tried that as well as the 128MiB and neither worked. A few of the commands still complete, a few will crash convert.exe, and the rest will give a bunch of memory allocation failed errors:

convert.exe: Memory allocation failed `drylake-revolution-250px\Almost-20.gif' @ error/colormap.c/AcquireImageColormap/146.
convert.exe: Memory allocation failed `drylake-revolution-250px\Almost-20.gif' @ error/quantize.c/AssignImageColors/517.
convert.exe: Memory allocation failed `drylake-revolution-250px\Almost-20.gif' @ error/colormap.c/AcquireImageColormap/146.
convert.exe: Memory allocation failed `drylake-revolution-250px\Almost-20.gif' @ error/quantize.c/AssignImageColors/517.
convert.exe: Memory allocation failed `drylake-revolution-250px\Almost-20.gif' @ error/gif.c/WriteGIFImage/1643.

I did a complete reboot and the performance tab in task manager says there's still 5.3GB free and 6.4GB available memory. I also installed ImageMagick-6.8.7-5-Q16-x86-dll.exe.

Having the task manager open I can see the numbers for convert.exe starting with ~4MB memory, then if it doesn't crash or fail to allocate memory it goes up to ~70MB. This GIF I'm trying to generate now has 59 PNG images @ 250x108 ~20KB. CPU usage is around ~30-40%.

[snibgo]

Do you have enough free space in %TEMP%?

EDIT: You might also try adding "-synchronize" as the first option on the convert command to ensure "early reporting if there is not enough disk space for the image pixel cache."

[hunky-dory]

Do you have enough free space in %TEMP%?

EDIT: You might also try adding "-synchronize" as the first option on the convert command to ensure "early reporting if there is not enough disk space for the image pixel cache."

I cleared the %TEMP% directory (2GB) but that did not seem to have any effect.

After adding -synchronize every command seems to crash convert.exe.

Thanks for the suggestions so far.

[magick]

Add -debug cache to your command line and post the first 20-30 lines of output. Also post a URL to your image. We'll need to reproduce the problem before we can comment further.

[hunky-dory]

Download links to two different sets of images that fail in similar ways:

https://mega.co.nz/#!DVRghQzA!cL0c5UrDP ... ZQGjW570Ew
https://mega.co.nz/#!fVBzHbZL!N3X8agksF ... BLTc0v5RBY

Here's output from both sets of images:

C:\giffactory>"C:\Program Files (x86)\ImageMagick-6.8.7-Q16\convert.exe" -debug cache -limit memory 128MiB +repage -fuzz 0.0% -delay 16 -loop 0 frame
s/*.png -layers OptimizePlus -layers OptimizeTransparency test.gif
2013-11-14T22:22:26+02:00 0:00.015 0.016u 6.8.7 Cache convert.exe[3748]: cache.c/DestroyPixelCache/956/Cache
destroy
2013-11-14T22:22:26+02:00 0:00.024 0.016u 6.8.7 Cache convert.exe[3748]: cache.c/OpenPixelCache/3589/Cache
open frames\000.png[0] (Heap Memory, 250x108 211KiB)
2013-11-14T22:22:26+02:00 0:00.033 0.016u 6.8.7 Cache convert.exe[3748]: cache.c/DestroyPixelCache/956/Cache
destroy
2013-11-14T22:22:26+02:00 0:00.038 0.016u 6.8.7 Cache convert.exe[3748]: cache.c/OpenPixelCache/3589/Cache
open frames\001.png[0] (Heap Memory, 250x108 211KiB)
2013-11-14T22:22:26+02:00 0:00.047 0.016u 6.8.7 Cache convert.exe[3748]: cache.c/DestroyPixelCache/956/Cache
destroy
2013-11-14T22:22:26+02:00 0:00.052 0.016u 6.8.7 Cache convert.exe[3748]: cache.c/OpenPixelCache/3589/Cache
open frames\002.png[0] (Heap Memory, 250x108 211KiB)
2013-11-14T22:22:26+02:00 0:00.061 0.016u 6.8.7 Cache convert.exe[3748]: cache.c/DestroyPixelCache/956/Cache
destroy
2013-11-14T22:22:26+02:00 0:00.066 0.016u 6.8.7 Cache convert.exe[3748]: cache.c/OpenPixelCache/3589/Cache
open frames\003.png[0] (Heap Memory, 250x108 211KiB)
2013-11-14T22:22:26+02:00 0:00.077 0.016u 6.8.7 Cache convert.exe[3748]: cache.c/DestroyPixelCache/956/Cache
destroy
2013-11-14T22:22:26+02:00 0:00.082 0.016u 6.8.7 Cache convert.exe[3748]: cache.c/OpenPixelCache/3589/Cache
open frames\004.png[0] (Heap Memory, 250x108 211KiB)
2013-11-14T22:22:26+02:00 0:00.091 0.016u 6.8.7 Cache convert.exe[3748]: cache.c/DestroyPixelCache/956/Cache
destroy
2013-11-14T22:22:26+02:00 0:00.096 0.016u 6.8.7 Cache convert.exe[3748]: cache.c/OpenPixelCache/3589/Cache
open frames\005.png[0] (Heap Memory, 250x108 211KiB)
2013-11-14T22:22:26+02:00 0:00.105 0.016u 6.8.7 Cache convert.exe[3748]: cache.c/DestroyPixelCache/956/Cache
destroy
2013-11-14T22:22:26+02:00 0:00.109 0.016u 6.8.7 Cache convert.exe[3748]: cache.c/OpenPixelCache/3589/Cache
open frames\006.png[0] (Heap Memory, 250x108 211KiB)
2013-11-14T22:22:26+02:00 0:00.118 0.016u 6.8.7 Cache convert.exe[3748]: cache.c/DestroyPixelCache/956/Cache
destroy

C:\giffactory>"C:\Program Files (x86)\ImageMagick-6.8.7-Q16\convert.exe" -debug cache -limit memory 128MiB +repage -fuzz 0.0% -delay 16 -loop 0 frame
s/*.png -layers OptimizePlus -layers OptimizeTransparency test2.gif
2013-11-14T22:27:38+02:00 0:00.007 0.016u 6.8.7 Cache convert.exe[3996]: cache.c/DestroyPixelCache/956/Cache
destroy
2013-11-14T22:27:38+02:00 0:00.013 0.016u 6.8.7 Cache convert.exe[3996]: cache.c/OpenPixelCache/3589/Cache
open frames\000.png[0] (Heap Memory, 250x184 359KiB)
2013-11-14T22:27:38+02:00 0:00.026 0.016u 6.8.7 Cache convert.exe[3996]: cache.c/DestroyPixelCache/956/Cache
destroy
2013-11-14T22:27:38+02:00 0:00.031 0.016u 6.8.7 Cache convert.exe[3996]: cache.c/OpenPixelCache/3589/Cache
open frames\001.png[0] (Heap Memory, 250x184 359KiB)
2013-11-14T22:27:38+02:00 0:00.044 0.016u 6.8.7 Cache convert.exe[3996]: cache.c/DestroyPixelCache/956/Cache
destroy
2013-11-14T22:27:38+02:00 0:00.048 0.016u 6.8.7 Cache convert.exe[3996]: cache.c/OpenPixelCache/3589/Cache
open frames\002.png[0] (Heap Memory, 250x184 359KiB)
2013-11-14T22:27:38+02:00 0:00.061 0.016u 6.8.7 Cache convert.exe[3996]: cache.c/DestroyPixelCache/956/Cache
destroy
2013-11-14T22:27:38+02:00 0:00.066 0.016u 6.8.7 Cache convert.exe[3996]: cache.c/OpenPixelCache/3589/Cache
open frames\003.png[0] (Heap Memory, 250x184 359KiB)
2013-11-14T22:27:38+02:00 0:00.077 0.016u 6.8.7 Cache convert.exe[3996]: cache.c/DestroyPixelCache/956/Cache
destroy
2013-11-14T22:27:38+02:00 0:00.081 0.016u 6.8.7 Cache convert.exe[3996]: cache.c/OpenPixelCache/3589/Cache
open frames\004.png[0] (Heap Memory, 250x184 359KiB)
2013-11-14T22:27:38+02:00 0:00.092 0.016u 6.8.7 Cache convert.exe[3996]: cache.c/DestroyPixelCache/956/Cache
destroy
2013-11-14T22:27:38+02:00 0:00.096 0.016u 6.8.7 Cache convert.exe[3996]: cache.c/OpenPixelCache/3589/Cache
open frames\005.png[0] (Heap Memory, 250x184 359KiB)
2013-11-14T22:27:38+02:00 0:00.107 0.016u 6.8.7 Cache convert.exe[3996]: cache.c/DestroyPixelCache/956/Cache
destroy
2013-11-14T22:27:38+02:00 0:00.112 0.016u 6.8.7 Cache convert.exe[3996]: cache.c/OpenPixelCache/3589/Cache
open frames\006.png[0] (Heap Memory, 250x184 359KiB)
2013-11-14T22:27:38+02:00 0:00.122 0.016u 6.8.7 Cache convert.exe[3996]: cache.c/DestroyPixelCache/956/Cache
destroy

[hunky-dory]

In the past I've created much larger GIFs that have worked with no problems. For instance this 132 frame GIF @ 600x338 which I made to show how easy it is to make a high quality GIF following a tutorial I had written: http://i.imgur.com/L8NB6kl.gif

I also made this very high quality 31 frame GIF @ 600x326 from the movie The Fall: http://i.imgur.com/cyQAtat.gif

And this 63 frame GIF @ 500x212 from a Japanese film: http://fat.gfycat.com/TerribleDefensiveGannet.gif

Or this 73 frame GIF @ 500x210 from another Japanese film: https://gs1.wac.edgecastcdn.net/8019B6/ ... o3_500.gif

And many many more at similar sizes. So the program and script definitely works. So it's weird it doesn't when these two are much smaller.

[snibgo]

Windows 7, IM 6.8.7-0. I downloaded the first set of frames and ran this command:

Code: Select all

convert +repage -fuzz 2.0% -delay 16 -loop 0 drylake\frames\*.png -layers OptimizePlus -layers OptimizeTransparency test.gif

It completed with no problems, and output looks okay.

After adding -synchronize every command seems to crash convert.exe.

That shouldn't happen.

[hunky-dory]

And it isn't just me. Several people tried to generate a GIF using the batch script in the first post from PNG files extracted with Avisynth (I've also tried JPG by the way, with same result) and both of them said ImageMagick just crashes on every call to convert.exe creating empty GIFs/GIFs with missing frames/artifacts. So go figure. I think one of them succeeded in creating a very small GIF from just a few frames, but anyway, not that successful. Mine's at least successful some of the time.

Anyway, thanks for all the help so far. I don't know what info I could give that would help but if there's something just let me know. The linux installation seems to work for me so I'll probably have to just resort to that for those GIFs that don't work on Windows.

[snibgo]

I mentioned above that ...

Code: Select all

convert +repage -fuzz 2.0% -delay 16 -loop 0 drylake\frames\*.png -layers OptimizePlus -layers OptimizeTransparency test.gif

... completed with no problems, and output looks okay, for Windows 7 and IM 6.8.7-0.

The same is true of 6.8.7-1.

However, on IM v6.8.7-5 it consistently crashes with the general Windows "Your program has stopped working" box.

Removing "-fuzz 2.0%" it consistently crashes with:

Code: Select all

D:\web\im>%IMG6875%convert +repage -delay 16 -loop 0 drylake\frames\*.png -layer
s OptimizePlus -layers OptimizeTransparency test.gif
convert.exe: Memory allocation failed `test.gif' @ error/colormap.c/AcquireImage
Colormap/146.
convert.exe: Memory allocation failed `test.gif' @ error/quantize.c/AssignImageC
olors/517.
convert.exe: Memory allocation failed `test.gif' @ error/colormap.c/AcquireImage
Colormap/146.
convert.exe: Memory allocation failed `test.gif' @ error/quantize.c/AssignImageC
olors/517.
convert.exe: Memory allocation failed `test.gif' @ error/gif.c/WriteGIFImage/164
3.

I can't get it to work by removing any remaining options.

[snibgo]

Update: My command works for the first 8 files and fewer, but fails for 9 files and above.

[dlemstra]

I can confirm this is a bug. We are working on a fix.

[snibgo]

Just to add another data point:

Update 2: Using all 59 frames, the following command doesn't fail for c6.8.7-5:

Code: Select all

D:\web\im>%IMG6875%convert -debug cache +repage -fuzz 2.0% -delay 16 -loop 0 drylake\frames\*.png -layers OptimizePlus -layers OptimizeTransparency test.gif

This is the same as my previous command but with the addition of "-debug cache". It takes a long time and the result seems to have artefacts that are not present when the gif is created by v6.8.7-0.

[magick]

We can reproduce the problem you posted and will have a patch in ImageMagick 6.8.7-6 Beta by late tomorrow. Thanks.