Magick.NET slows down over time while processing a large batch of images?

[ZakO12]

Hi,

I'm using Magick.NET-Q16-AnyCPU (7.4.3) with .NET Core 2.0, need to batch process ~25,000 images, (make square, resize to large/thumb), just using a single thread at the moment:

Code: Select all

            using (MagickImage image = new MagickImage(inputFile))
            {
                image.ColorFuzz = new Percentage(1);
                image.Trim();

                var largestDimension = Math.Max(image.Width, image.Height);
                image.BackgroundColor = MagickColor.FromRgb(255, 255, 255);
                image.Extent(largestDimension, largestDimension, Gravity.Center);

                image.Format = MagickFormat.Jpg;
                image.Quality = 65;
                image.Resize(600, 600);
                image.Write(largeOutputFile);

                image.Resize(200, 200);
                image.Write(thumbOutputFile);
            }

It processes the first 300-400 images in 10-15 seconds, but then grinds to a halt, from there on it takes up to 10s per image. All the images are of a similar size / content, the memory usage remains low so there's no leak, I just can't figure out why all of a sudden it goes slow. Tested on three different computers all with the same outcome. Does anyone know why this might be happening?


Edit: it's the Resize command that starts slowing down, for the first images:

500x331px
Trim: 2.9678ms
Extent: 4.4123ms
Large: 24.8456ms
Thumb: 14.398ms
-
500x275px
Trim: 2.5952ms
Extent: 3.477ms
Large: 17.557ms
Thumb: 13.7364ms
-

After ~300 images:

500x357
Trim: 5.8368ms
Extent: 12.1412ms
Large: 2656.029ms
Thumb: 18.721ms
-
500x298
Trim: 5.7799ms
Extent: 11.5524ms
Large: 2647.8159ms
Thumb: 22.1286ms
-
500x261
Trim: 5.9073ms
Extent: 12.3215ms
Large: 2743.2966ms
Thumb: 19.2876ms
-

Edit2: Ok, so when the slowdown starts imagemagick starts writing to %temp%/magick-* instead of (presumably) doing the operations in memory. Why is it doing this? The process is only using 100mb memory, the machine has 32GB memory with ~26GB free.

[dlemstra]

The slow down is caused because by your process is running out of memory and ImageMagick will use your disk because it cannot allocate more memory. I suspect that a memory leak is happening somewhere or our internal memory booking is broken at some point. Would it be possible to create an example in a for loop that demonstrates the issue? It would help me a lot if you could pinpoint which method is causing this issue.

[ZakO12]

Hi, thanks for your reply.

If I use this image (https://upload.wikimedia.org/wikipedia/ ... lesia2.jpg) and loop over resizing it:

Code: Select all

        static void Main(string[] args)
        {
            var sw = new Stopwatch();
            for (int i = 1; i <= 500; i++)
            {
                sw.Restart();
                using (MagickImage image = new MagickImage(@"C:\test\in.jpg"))
                {
                    image.Resize(1000, 1000);
                }
                Console.WriteLine("Iteration {0}: {1}ms", i, sw.Elapsed.TotalMilliseconds);
            }
        }

Using Magick.NET-Q16-AnyCPU and .NET Core 2.0, at iteration 74 on my machine the %temp% usage starts and performance slows:

Iteration 1: 172.5328ms
Iteration 2: 79.418ms
...
Iteration 73: 73.8936ms
Iteration 74: 14202.4944ms
Iteration 75: 14363.0111ms

The actual process itself is only using 100MB at this point.

Edit: As a temporary workaround if I set ResourceLimits.Memory to a high amount, all 500 iterations without any slowdown and total memory usage still just hovers around ~100MB. With this set high it was able to process the 25,000 images I mentioned in my original post without slowdown or consuming much memory. Does the Magick.NET library itself try to keep track of how much memory it's allocating or is that within the ImageMagick core itself? I tried to look into it further but didn't get very far.

[dlemstra]

Thanks for reporting this and it looks like you found a bug in the internal memory resource counters. This will be resolve in the next version of Magick.NET that I will try to release this weekend.

[adhoc]

Zak012 - I am noticing this same issue. Everything works great in isolation, but repeated resizing seems to be causing ImageMagick to at some point start caching to disk which causes a tremendous slowdown.

What value did you use for

Code: Select all

ResourceLimits.Memory

Right now, I'm setting to:

Code: Select all

ResourceLimits.Memory = 5000000000;

but it seems to not be fixing the issue. I'm using Q8-AnyCPU FWIW.

Thank you! And thank you Dlemstra for such fast support on a great library!

[ZakO12]

Thanks for your replies dlemstra.

@adhoc - ResourceLimits.Memory is in bytes, I just set mine to 100GB (100_000_000_000) which was the lowest I could use to process all my images (although the process itself never went above 150MB usage). Obviously this isn't recommended to do with anything important though, I just changed it to run the task locally once and then set it back to the default to make sure my process can't accidentally use an insane amount of memory

[snibgo]

I don't know Magick.net (dlemstra knows it far better than I do), but I guess ResourceLimits.Memory is like the other interfaces: it doesn't give you extra resources. It defines an upper limit on memory use.

[blackangelnt]

Hi, are you fix this issue ? I see similar this issue on my app. Tks so much . I using Q16-AnyCPU version 7.501

When i put this code on loop
Image 50 kb only . after 60 times , memory lost more 500 mb and slowly

Code: Select all

for(int i=0;i < 100; i++)
{
 using (MagickImage image = new MagickImage("Y:\\Desktop\\test.png"))
                {
                    if (image.GetPixels().GetPixel(600, 1000).ToColor() == MagickColor.FromRgb(255,255,255))
                    {
                        // Empty  code
                    }
                }

}

[dlemstra]

The instance that is returned by `image.GetPixels()` is `IDisposable`. You are leaking memory.