GIF allocates 128MB of memory to dither

[donniedarko]

I am using ImageMagick in a Linux application as part of an image translation pipeline. I've been having problems with the server OOM killing processes. The command I am running can be replicated from the command line:

$ convert -resize x64 s1.svg s1.gif

s1.svg looks like this:

Code: Select all

<svg xmlns="http://www.w3.org/2000/svg" version="1.1" viewBox="0 0 124 30">
<defs>
  <style type="text/css"><![CDATA[
    #background {
      fill: black;
    }
  ]]></style>
</defs>
<path id="background" d="M0,0h124v30h-124z"/>
</svg>

The command when run consumes 128Mb (134221824 bytes) of memory (as seen through strace):

Code: Select all

...
fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap(NULL, 134221824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff4f3c58000
futex(0x738d24, FUTEX_WAKE_PRIVATE, 2147483647) = 0
...

I traced the memory utilisation to the following lines in "GetCubeInfo()" in MagickCore/quantize.c:

Code: Select all

 length=(size_t) (1UL << (4*(8-CacheShift)));
  cube_info->memory_info=AcquireVirtualMemory(length,sizeof(*cube_info->cache));

In the above code, which appears to be independent of the size of the file being processed, length evaluated to 2^24 and size is 8, so the call ends up requesting 2^27 bytes (128Mb) of memory.

The purpose of the application is to convert two differing SVGs into GIFs and then stitch them together as an animated GIF for web display. The problem is that the memory being used by this process is causing the OS to OOM kill processes due to the memory requirement when multiple converts run simultaneously.

Is there any reason why GIF needs such a huge amount of memory for what works out to be a 5k GIF file?

Surely the dither resources (the memory allocation in GetCubeInfo) should be related to the size of the source file?

Setting resource limitations either in the call or via shell environment doesn't help. They are ignored.

Environment: debian Jessie (3.16.39-1+deb8u2)
ImageMagick: v7.0.6 (compiled and run on host)

I've also tried this on RedHat using packages and got the same result.

[fmw42]

What SVG renderer are you using? Imagemagicks MSVG/XML, RSVG delegate or Inkscape? You can check via

Code: Select all

convert -list format

and see which is listed for the line with SVG. Note that XML will be the worst, RSVG next and Inkscape the best.

Your IM 6 command should be

Code: Select all

convert  s1.svg -resize x64 s1.gif

You may want to add -depth 8 or less and +dither -colors XX to reduce the file size. You may also want to use -density YY before s1.svg rather than -resize to reduce the resulting pixel dimension. That may be faster.

Imagemagick will rasterize your vector SVG file, so that is why the file size gets large.

On my IM 6.9.9.5 Q16 Mac OSX, I get the following:

Code: Select all

time convert test.svg -resize x64 test.png

real 0m0.036s
user 0m0.036s
sys 0m0.012s

Code: Select all

time convert test.svg -resize x64 test.gif

real 0m0.065s
user 0m0.122s
sys 0m0.016s

Using IM 7.0.6.5 Q16 HDRI, I get:

Code: Select all

time im7 magick test.svg -resize x64 test.gif

real 0m0.115s
user 0m0.180s
sys 0m0.031s


So this is not causing any problems. My SVG renderer is

SVG SVG rw+ Scalable Vector Graphics (RSVG 2.40.17)

Sorry, I do not know how to look at the memory usage.

[fmw42]

P.S. Do you have restricted memory allocation in your policy.xml file?

[donniedarko]

Thank you for your quick reply

My SVG renderer is

Code: Select all

SVG    rw+   Scalable Vector Graphics  (XML 2.9.1)

The SVG files created by the application are quite small, so although they process badly, they don't cripple the system.

When I re-ran with the arguments re-arranged:

Code: Select all

convert  s1.svg -resize x64 s1.gif

It has no effect on memory usage. Still using 128Mb of memory.

However, when I re-ran with the +dither option, the memory usage drops to 8Mb, which is more than acceptable

Code: Select all

convert  s1.svg -resize x64 +dither -colors 64 s1.gif

This looks like it solves my problem.

I also note that

Code: Select all

convert  s1.svg -resize x64 s1.gif

uses 128Mb of memory

Code: Select all

convert  s1.svg -resize x64 s1.png

uses only 8Mb

So PNG doesn't appear to use the same buffer by default.

Dithering appears to be what fixes the memory usage

>> P.S. Do you have restricted memory allocation in your policy.xml file?

No, I've not installed any special restrictions. The policy file is what was installed by default.

I'll see if I can rebuild the same command with the full pipeline using the +dither option

Thank you

[fmw42]

Try installing Inkscape. IM will use that if it is installed. It may render better that IM's MSVG/XML. But then it just could be as you suggest in the gif writer.

[snibgo]

I suspect the problem is independent of SVG. You can test this:

Code: Select all

convert x1.svg -resize x64 tmp.png
convert tmp.png s1.gif

I suspect the memory problem is entirely in the second convert.

The image, tmp.png, is 124x30 pixels, entirely black. Is this what you get? Being only one colour, I would hope a colour-reduction process would perform quickly, without much memory. Perhaps something is going wrong.

A developer might take a look.

Meanwhile, "-treedepth" might reduce memory usage. See http://www.imagemagick.org/script/comma ... #treedepth

[magick]

ImageMagick 7.0.6-6, when released, permits large memory allocations for certain algorithms on disk rather than heap. Simply set this policy in policy.xml:

Code: Select all

 <policy domain="system" name="max-memory-request" value="64MiB"/>

The trade off is reduced memory with a slight degradation in performance.

[donniedarko]

I suspect the problem is independent of SVG
I suspect the memory problem is entirely in the second convert.
Meanwhile, "-treedepth" might reduce memory usage

If a GIF dither occurs, it appears the code pointed out at the top will allocate a 128Mb buffer for this purpose. The code doesn't make an allowance for the size or content of the inbound graphic from what I can see, so it doesn't matter what the source graphic is.

I tried this with every value from 0 to 8 to see how it affects the performance with the following command:

Code: Select all

convert s1.svg -resize x64 -treedepth <0..8> s1.gif

In all cases it still required 128Mb to execute

The magic seems to come with disabling dither which prevents the dither buffer from being allocated (many thanks to fmw42 for pointing this out)

ImageMagick 7.0.6-6, when released, permits large memory allocations for certain algorithms on disk rather than heap

This sounds like it could help as it wouldn't OOM my system, but my point would still be why the 128Mb is required in the first place? Shouldn't the dither buffer be based on the source image?

[magick]

We allocate a cache to improve performance for color lookup. Its size is based on the RGB triple whose range is [0..255]. It is not based on image size. We could have an option to not use the cache and lookup every color by traversing the color cube, but that would slow down processing considerably. Performance improves considerably with our current solution of allocating the cache to disk and memory mapping it-- if you request it with the max-memory-request security policy. There are algorithms that perform color reduction with a smaller memory footprint, however, our original test images were computer generated spheres. The current algorithm was deemed visually optimal for these types of images.