Memory Leak

Post any defects you find in the released or beta versions of the ImageMagick software here. Include the ImageMagick version, OS, and any command-line required to reproduce the problem. Got a patch for a bug? Post it here.
Post Reply
Yay295
Posts: 3
Joined: 2015-04-01T12:11:16-07:00
Authentication code: 6789

Memory Leak

Post by Yay295 »

Using the Windows binary release ImageMagick-6.9.1-0-Q16-x64-dll.exe on a 64bit Windows 7 with 16GB of RAM.

ImageMagick Code Used

Code: Select all

convert big/frame%05d.png[1-3909] big/frame%05d.png[6067-34405] -resize 240 +dither -fuzz 1% -layers OptimizePlus -layers OptimizeTransparency big.gif
The folder 'big' is about 60GB and contains 37,332 1080p png images. Don't ask why I'm doing this; I'm not sure myself.

Windows Task Manager showed convert.exe using only about 637MB, but 90% of my RAM was in use. After closing the cmd window to stop the operation, the amount of RAM in use slowly decreased to appropriate levels (16%). convert.exe didn't close (and couldn't be force-closed) until RAM levels were back to normal.

I know there is a command to limit the amount of RAM used, but I let this run overnight so I thought it would be fine.
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Memory Leak

Post by snibgo »

Convert is reading about 32000 images of 1080x1920 pixels. At 8 bytes/pixel, this needs 531 GB of memory. Thus, disk will be used, and it will be slow.

I would do this by first looping through the 32000 images, resizing each one (to 240x427). I don't know what dither or fuzz settings do in your command. The layers operations need all the images, so this is merely 26 GB.
snibgo's IM pages: im.snibgo.com
Yay295
Posts: 3
Joined: 2015-04-01T12:11:16-07:00
Authentication code: 6789

Re: Memory Leak

Post by Yay295 »

That's what I did first. I just wanted to try doing it this way instead. Also, you have your dimensions backwards. 1080p is 1920x1080, which would make my resized gif 240x135. All 37,332 frames at that size only take up 1.5GB, and compress to a 118MB gif. This doesn't change the fact that ImageMagick is leaking memory somewhere.
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Memory Leak

Post by snibgo »

... you have your dimensions backwards ...
So I did. Sorry.

240*135*8*37000/1e9 = 9.6 GB. This is how much memory is needed to read the images.

You don't say if your commands are failing. If you kill a command, this might cause memory (and disk) leakage. Things will take time to get back to normal. How do you know there is memory leakage?
snibgo's IM pages: im.snibgo.com
Yay295
Posts: 3
Joined: 2015-04-01T12:11:16-07:00
Authentication code: 6789

Re: Memory Leak

Post by Yay295 »

After taking a closer look, I don't think there is one. It looks like my monitoring program froze when it ran out of RAM, so it looked like ImageMagick was still using all of it. Everything looked fine in the Resource Monitor.
User avatar
glennrp
Posts: 1147
Joined: 2006-04-01T08:16:32-07:00
Location: Maryland 39.26.30N 76.16.01W

Re: Memory Leak

Post by glennrp »

I think you should be using mogrify instead of convert for this project. convert reads in the whole batch of images, then writes them out. mogrify reads the images one at a time, read-write-read-write-etc., and only needs enough memory to process one image. So,

Code: Select all

mogrify -resize 240  -quality 20 -format PNG big/frame%05d.png[1-3909] big/frame%05d.png[6067-34405]
convert  big/frame*.PNG +dither -fuzz 1% -layers OptimizePlus -layers OptimizeTransparency big.gif
The intermediate files are frame*.PNG so you can distinguish them from the input frame*.png
I suggest using a low quality value for speed, since these are temporary files that don't need highly optimized compression.
Post Reply