Combining many images is incredibly slow on VM

[Excludos]

For a project I'm working on I need to combine a bunch of images (20-30) into one long one, and then resize them down a bit. This is the code I've put together so far:

Code: Select all

var imageFiles = Directory.GetFiles(imageFolder, "*.png");
var magickImages = new List<MagickImage>();
var imageHeight = new List<string>;

foreach (var imageFile in imageFiles)
{
    var image = new MagickImage(imageFile);
    imageHeight.Add(imageFile);
    var heightLengthInPixels = getHeightInPixels(imageFile).ToString();
    image.Resize(new MagickGeometry("154x" + heightLengthInPixels + "!"));
    magickImages.Add(image);
}

var settings = new MagickReadSettings
{
    Width = 154,
    Height = getFullHeightInPixels(imageHeight.First(), imageHeight.Last())
};

var canvas = new MagickImage("xc:white", settings) {Format = MagickFormat.Png32};

for (int i = 0; i < magickImages.Count; i++)
{
    canvas.Composite(magickImages[i], 0, getHeightInPixels(imageHeight[i]));
}

canvas.Resize(new MagickGeometry("154x10000!"));
canvas.Write(imageFolder + "/CombinedImage.png");

It uses a system where the name of the image dictates its height and placement.

On my personal computer, this literally takes less than a second. But I need this to run on a backend server, provided by a VM in Azure, and this takes several hours. I've tried different VMs, with and without GPU, and with varying CPU and Memory size, and they are all incredibly slow. (At the moment of writing, my VM is using 13% CPU power and 8% memory). I was expecting them to be a bit slower, sure, but not by a factor of 7000+.

Has anyone had similar problems before? And/or have I written the code stupidly?

[snibgo]

less than a second ... several hours

The difference is likely to be that on your PC the pixel caches are stored in memory, but your server doesn't have enough memory available so pixel caches are stored on disk which is far slower. System management tools will tell you if the process is using disk heavily.

The obvious solution is to get more memory in your server VM.

Assuming V6 Q16 integer, each pixel needs 8 bytes of memory. You could insert printf() statements to find the total number of pixels.

You might need less memory if you first convert each png input to mpc format, then read those instead of png. But the resized intermediate images will need memory.

[Excludos]

That does make sense, and I am seeing a large uptick in disk usage, tho both the pc and VM has 32gb of ram. Each image is roughly 154x1000 pixels, and there's around 30 of them, which should be around 37mb of ram + the finished image which is equally large before downsizing, so another 37mb. Not exactly super computer territory.

What makes Magick decide whether to use the memory or disk? Since the code is equal on both I'm assuming there's a check somewhere that mistakenly returns a false flag. Can I force it to always use memory?

[snibgo]

What makes Magick decide whether to use the memory or disk?

IM requests the memory from the operating system. If the OS says "no", IM will use disk.

Can I force it to always use memory?

Not exactly. You can tell IM to never use disk, eg "-limit disk 0" when using the command-line interface, so the command will fail when there isn't enough memory.

You can ask IM whether it is limiting memory or other usage. For example:

Code: Select all

f:\web\im>%IMG7%magick -list resource

Resource limits:
  Width: 107.374MP
  Height: 107.374MP
  List length: -1
  Area: 6.40681GP
  Memory: 2.9834GiB
  Map: 5.9668GiB
  Disk: unlimited
  File: 1536
  Thread: 8
  Throttle: 0
  Time: unlimited

On my computer, which has 12GB memory, IM won't use more that 2.98GiB.

[dlemstra]

You are disposing none of your images. Does this happen instantly or only after a couple times? And you might want to take a look at the MagickImageCollection class that has "Append" methods to combine images.

[Excludos]

You are disposing none of your images. Does this happen instantly or only after a couple times? And you might want to take a look at the MagickImageCollection class that has "Append" methods to combine images.

Hi.

A bit of both. It's slow already from the first image, but it does slow down dramatically more for every one (first takes a few seconds, the second a lot more, by the 10th it's a few minutes, and by the last it's nearly 15 minutes on its own).

Do you have a link or similar to this append function? I have not been able to find any kind of documentation for Magick.net (If it exists, I'm looking in the wrong places), so I'm sure my bit of code is completely stupid.

[Excludos]

Thanks for the dispose tip btw, I rewrote the code a little bit. It's much faster (image 1-29 takes about 5 minutes combined now, and the last canvas.resize takes around 15, so 20 minutes in total. It's still way way slower than on my personal computer, and not really within acceptable limits for the project, but it's heading in the right direction at least:

Code: Select all

var imageFiles = Directory.GetFiles(imageFolder, "*.png");

var settings = new MagickReadSettings
{
    Width = 154,
    Height = getFullHeightInPixels(imageFiles.First(), imageFiles.Last())
};

var canvas = new MagickImage("xc:white", settings) {Format = MagickFormat.Png32};

for (int i = 0; i < imageFiles.Count; i++)
{
    var image = new MagickImage(imageFiles[i]);
    var heightLengthInPixels = getHeightInPixels(imageFiles[i]).ToString();
    image.Resize(new MagickGeometry("154x" + heightLengthInPixels + "!"));
    canvas.Composite(image, 0, getHeightInPixels(imageFiles[i]));
    image.Dispose();
}

canvas.Resize(new MagickGeometry("154x10000!"));
canvas.Write(imageFolder + "/CombinedImage.png");
canvas.Dispose();

[Excludos]

What makes Magick decide whether to use the memory or disk?

IM requests the memory from the operating system. If the OS says "no", IM will use disk.

Can I force it to always use memory?

Not exactly. You can tell IM to never use disk, eg "-limit disk 0" when using the command-line interface, so the command will fail when there isn't enough memory.

You can ask IM whether it is limiting memory or other usage. For example:

Code: Select all

f:\web\im>%IMG7%magick -list resource

Resource limits:
  Width: 107.374MP
  Height: 107.374MP
  List length: -1
  Area: 6.40681GP
  Memory: 2.9834GiB
  Map: 5.9668GiB
  Disk: unlimited
  File: 1536
  Thread: 8
  Throttle: 0
  Time: unlimited

On my computer, which has 12GB memory, IM won't use more that 2.98GiB.

On my VM:

Code: Select all

C:\temp>magick -list resource
Resource limits:
  Width: 214.748MP
  Height: 214.748MP
  List length: -1
  Area: 17.1796GP
  Memory: 7.99989GiB
  Map: 15.9998GiB
  Disk: unlimited
  File: 1536
  Thread: 8
  Throttle: 0
  Time: unlimited

Should be plenty. That said there might be differences between MagickImage cmd line and Magick.net in how they behave in a VM.

[dlemstra]

Is your application running in 64 bit on your local machine and in 32 bit on your VM? But that would probably still not cause such a big difference. And you probably want to also follow my other tip to use AppendVertically or AppendHorizontally of the MagickImageCollection. This should also give you a huge speedup.

[Excludos]

Is your application running in 64 bit on your local machine and in 32 bit on your VM? But that would probably still not cause such a big difference. And you probably want to also follow my other tip to use AppendVertically or AppendHorizontally of the MagickImageCollection. This should also give you a huge speedup.

Vm is 64 bit as well, and running the same bit application.

Unless I am mistaken, it seems that AppendVertically appends continuously without any input on the position other than the order. The reason this doesn't work for me is because there might be a missing image in the sequence, which I want to show up as white space in the final concatenated one to clearly indicate to the user where there's a gap.