Different behaviour between stack and heap

[berathebrain]

Hi. I am having problem when ImageMagick caches files on disk. I have discovered that this happens because malloc returns NULL.
I have discovered why this is happening. For some strange reason this have something to do with allocating ImageMagick::Image class on the stack and on the heap.
For example take a look at the next code:

Code: Select all

#include "stdafx.h"
#include <Magick++.h>
#include <iostream>
#include <windows.h>

using namespace std;
using namespace Magick;

#define numberOfFiles 257

string dir;
string inputImageFileName;
string outputImageFileName;

string  fileNames[numberOfFiles] = {"00114.TIF",
"00116.TIF",
"00215.TIF",
"00216.TIF",
"00309.TIF",
"00310.TIF",
"00311.TIF",
"00313.TIF",
"00314.TIF",
"00315.TIF",
"00410.TIF",
"00412.TIF",
"00414.TIF",
"00415.TIF",
"00416.TIF",
"00509.TIF",
"00513.TIF",
"00514.TIF",
"00603.TIF",
"00604.TIF",
"00606.TIF",
"00607.TIF",
"00608.TIF",
"00609.TIF",
"00610.TIF",
"00611.TIF",
"00612.TIF",
"00613.TIF",
"00614.TIF",
"00615.TIF",
"00616.TIF",
"00701.TIF",
"00702.TIF",
"00703.TIF",
"00704.TIF",
"00705.TIF",
"00706.TIF",
"00707.TIF",
"00708.TIF",
"00709.TIF",
"00710.TIF",
"00711.TIF",
"00712.TIF",
"00713.TIF",
"00714.TIF",
"00715.TIF",
"00716.TIF",
"00801.TIF",
"00802.TIF",
"00803.TIF",
"00804.TIF",
"00805.TIF",
"00806.TIF",
"00807.TIF",
"00808.TIF",
"00809.TIF",
"00810.TIF",
"00811.TIF",
"00812.TIF",
"00813.TIF",
"00814.TIF",
"00815.TIF",
"00816.TIF",
"00901.TIF",
"00902.TIF",
"00905.TIF",
"00906.TIF",
"00907.TIF",
"00909.TIF",
"00910.TIF",
"00911.TIF",
"00912.TIF",
"00913.TIF",
"00914.TIF",
"00915.TIF",
"00916.TIF",
"01102.TIF",
"01103.TIF",
"01104.TIF",
"01105.TIF",
"01106.TIF",
"01107.TIF",
"01108.TIF",
"01109.TIF",
"01110.TIF",
"01111.TIF",
"01112.TIF",
"01113.TIF",
"01114.TIF",
"01115.TIF",
"01116.TIF",
"01201.TIF",
"01202.TIF",
"01203.TIF",
"01204.TIF",
"01205.TIF",
"01206.TIF",
"01207.TIF",
"01208.TIF",
"01209.TIF",
"01210.TIF",
"01211.TIF",
"01212.TIF",
"01213.TIF",
"01214.TIF",
"01215.TIF",
"01216.TIF",
"01301.TIF",
"01302.TIF",
"01303.TIF",
"01304.TIF",
"01305.TIF",
"01306.TIF",
"01307.TIF",
"01308.TIF",
"01309.TIF",
"01310.TIF",
"01311.TIF",
"01312.TIF",
"01313.TIF",
"01314.TIF",
"01315.TIF",
"01316.TIF",
"01401.TIF",
"01402.TIF",
"01403.TIF",
"01404.TIF",
"01405.TIF",
"01406.TIF",
"01407.TIF",
"01408.TIF",
"01409.TIF",
"01410.TIF",
"01411.TIF",
"01412.TIF",
"01413.TIF",
"01414.TIF",
"01415.TIF",
"01416.TIF",
"01501.TIF",
"01502.TIF",
"01503.TIF",
"01504.TIF",
"01505.TIF",
"01506.TIF",
"01507.TIF",
"01508.TIF",
"01509.TIF",
"01510.TIF",
"01511.TIF",
"01512.TIF",
"01513.TIF",
"01514.TIF",
"01515.TIF",
"01516.TIF",
"01601.TIF",
"01602.TIF",
"01603.TIF",
"01604.TIF",
"01605.TIF",
"01606.TIF",
"01607.TIF",
"01608.TIF",
"01609.TIF",
"01610.TIF",
"01611.TIF",
"01612.TIF",
"01613.TIF",
"01614.TIF",
"01615.TIF",
"01616.TIF",
"01701.TIF",
"01705.TIF",
"01709.TIF",
"01710.TIF",
"01713.TIF",
"02402.TIF",
"02403.TIF",
"02404.TIF",
"02406.TIF",
"02407.TIF",
"02408.TIF",
"02410.TIF",
"02411.TIF",
"02412.TIF",
"02416.TIF",
"02501.TIF",
"02502.TIF",
"02503.TIF",
"02504.TIF",
"02505.TIF",
"02506.TIF",
"02507.TIF",
"02508.TIF",
"02509.TIF",
"02510.TIF",
"02511.TIF",
"02512.TIF",
"02514.TIF",
"02515_16.TIF",
"02601.TIF",
"02602.TIF",
"02603.TIF",
"02604.TIF",
"02605.TIF",
"02606.TIF",
"02607.TIF",
"02608.TIF",
"02609.TIF",
"02610.TIF",
"02611.TIF",
"02612.TIF",
"02613.TIF",
"02615.TIF",
"02616.TIF",
"02701.TIF",
"02702.TIF",
"02703.TIF",
"02704.TIF",
"02705.TIF",
"02706.TIF",
"02707.TIF",
"02708.TIF",
"02709.TIF",
"02710.TIF",
"02711.TIF",
"02712.TIF",
"02713.TIF",
"02714.TIF",
"02715.TIF",
"02716.TIF",
"02801.TIF",
"02802.TIF",
"02803.TIF",
"02804.TIF",
"02805.TIF",
"02806.TIF",
"02807.TIF",
"02808.TIF",
"02901.TIF",
"02905.TIF",
"04303.TIF",
"04304.TIF",
"04307.TIF",
"04308.TIF",
"04310.TIF",
"04311.TIF",
"04312.TIF",
"04315.TIF",
"04316.TIF",
"04401.TIF",
"04402.TIF",
"04403.TIF",
"04404.TIF",
"04405.TIF",
"04406.TIF",
"04409.TIF"
};

int main(int argc, char** argv)
{
	dir = "d:/Temp/BolcanoTIFF/";

	Magick::InitializeMagick(*argv);
	try
	{
		long time = GetTickCount();
		Geometry geometry = Geometry(1024,768);
		Geometry cropGeometry = Geometry(2000,2000);
		Magick::Image inputImage;
		Magick::Image cropedImage;
		Magick::Image finalImage;
		for (int i = 0; i<numberOfFiles; i++)
		{
			printf("\b\b\b\b\b%d", i);
			inputImageFileName = fileNames[i];
			inputImage.read(dir+inputImageFileName);
			cropedImage = inputImage;
			cropedImage.crop(cropGeometry);
			finalImage = cropedImage;
			finalImage.resize(geometry);
		}

		cout << GetTickCount() - time << endl;

	}catch(Magick::Exception& e)
	{
		cout << e.what() << endl;
	}

	return 0;
}

Caching happens on a 169th file. I am using 257 images that have 11000*9000 pixels. The images are in TIFF format.
Now, on the next code the caching does not occur. This is because I am allocating images on the heap.

Code: Select all

#include "stdafx.h"
#include <Magick++.h>
#include <iostream>
#include <windows.h>

using namespace std;
using namespace Magick;

#define numberOfFiles 257

string dir;
string inputImageFileName;
string outputImageFileName;

string  fileNames[numberOfFiles] = {"00114.TIF",
"00116.TIF",
"00215.TIF",
"00216.TIF",
"00309.TIF",
"00310.TIF",
"00311.TIF",
"00313.TIF",
"00314.TIF",
"00315.TIF",
"00410.TIF",
"00412.TIF",
"00414.TIF",
"00415.TIF",
"00416.TIF",
"00509.TIF",
"00513.TIF",
"00514.TIF",
"00603.TIF",
"00604.TIF",
"00606.TIF",
"00607.TIF",
"00608.TIF",
"00609.TIF",
"00610.TIF",
"00611.TIF",
"00612.TIF",
"00613.TIF",
"00614.TIF",
"00615.TIF",
"00616.TIF",
"00701.TIF",
"00702.TIF",
"00703.TIF",
"00704.TIF",
"00705.TIF",
"00706.TIF",
"00707.TIF",
"00708.TIF",
"00709.TIF",
"00710.TIF",
"00711.TIF",
"00712.TIF",
"00713.TIF",
"00714.TIF",
"00715.TIF",
"00716.TIF",
"00801.TIF",
"00802.TIF",
"00803.TIF",
"00804.TIF",
"00805.TIF",
"00806.TIF",
"00807.TIF",
"00808.TIF",
"00809.TIF",
"00810.TIF",
"00811.TIF",
"00812.TIF",
"00813.TIF",
"00814.TIF",
"00815.TIF",
"00816.TIF",
"00901.TIF",
"00902.TIF",
"00905.TIF",
"00906.TIF",
"00907.TIF",
"00909.TIF",
"00910.TIF",
"00911.TIF",
"00912.TIF",
"00913.TIF",
"00914.TIF",
"00915.TIF",
"00916.TIF",
"01102.TIF",
"01103.TIF",
"01104.TIF",
"01105.TIF",
"01106.TIF",
"01107.TIF",
"01108.TIF",
"01109.TIF",
"01110.TIF",
"01111.TIF",
"01112.TIF",
"01113.TIF",
"01114.TIF",
"01115.TIF",
"01116.TIF",
"01201.TIF",
"01202.TIF",
"01203.TIF",
"01204.TIF",
"01205.TIF",
"01206.TIF",
"01207.TIF",
"01208.TIF",
"01209.TIF",
"01210.TIF",
"01211.TIF",
"01212.TIF",
"01213.TIF",
"01214.TIF",
"01215.TIF",
"01216.TIF",
"01301.TIF",
"01302.TIF",
"01303.TIF",
"01304.TIF",
"01305.TIF",
"01306.TIF",
"01307.TIF",
"01308.TIF",
"01309.TIF",
"01310.TIF",
"01311.TIF",
"01312.TIF",
"01313.TIF",
"01314.TIF",
"01315.TIF",
"01316.TIF",
"01401.TIF",
"01402.TIF",
"01403.TIF",
"01404.TIF",
"01405.TIF",
"01406.TIF",
"01407.TIF",
"01408.TIF",
"01409.TIF",
"01410.TIF",
"01411.TIF",
"01412.TIF",
"01413.TIF",
"01414.TIF",
"01415.TIF",
"01416.TIF",
"01501.TIF",
"01502.TIF",
"01503.TIF",
"01504.TIF",
"01505.TIF",
"01506.TIF",
"01507.TIF",
"01508.TIF",
"01509.TIF",
"01510.TIF",
"01511.TIF",
"01512.TIF",
"01513.TIF",
"01514.TIF",
"01515.TIF",
"01516.TIF",
"01601.TIF",
"01602.TIF",
"01603.TIF",
"01604.TIF",
"01605.TIF",
"01606.TIF",
"01607.TIF",
"01608.TIF",
"01609.TIF",
"01610.TIF",
"01611.TIF",
"01612.TIF",
"01613.TIF",
"01614.TIF",
"01615.TIF",
"01616.TIF",
"01701.TIF",
"01705.TIF",
"01709.TIF",
"01710.TIF",
"01713.TIF",
"02402.TIF",
"02403.TIF",
"02404.TIF",
"02406.TIF",
"02407.TIF",
"02408.TIF",
"02410.TIF",
"02411.TIF",
"02412.TIF",
"02416.TIF",
"02501.TIF",
"02502.TIF",
"02503.TIF",
"02504.TIF",
"02505.TIF",
"02506.TIF",
"02507.TIF",
"02508.TIF",
"02509.TIF",
"02510.TIF",
"02511.TIF",
"02512.TIF",
"02514.TIF",
"02515_16.TIF",
"02601.TIF",
"02602.TIF",
"02603.TIF",
"02604.TIF",
"02605.TIF",
"02606.TIF",
"02607.TIF",
"02608.TIF",
"02609.TIF",
"02610.TIF",
"02611.TIF",
"02612.TIF",
"02613.TIF",
"02615.TIF",
"02616.TIF",
"02701.TIF",
"02702.TIF",
"02703.TIF",
"02704.TIF",
"02705.TIF",
"02706.TIF",
"02707.TIF",
"02708.TIF",
"02709.TIF",
"02710.TIF",
"02711.TIF",
"02712.TIF",
"02713.TIF",
"02714.TIF",
"02715.TIF",
"02716.TIF",
"02801.TIF",
"02802.TIF",
"02803.TIF",
"02804.TIF",
"02805.TIF",
"02806.TIF",
"02807.TIF",
"02808.TIF",
"02901.TIF",
"02905.TIF",
"04303.TIF",
"04304.TIF",
"04307.TIF",
"04308.TIF",
"04310.TIF",
"04311.TIF",
"04312.TIF",
"04315.TIF",
"04316.TIF",
"04401.TIF",
"04402.TIF",
"04403.TIF",
"04404.TIF",
"04405.TIF",
"04406.TIF",
"04409.TIF"
};

int main(int argc, char** argv)
{
	dir = "d:/Temp/BolcanoTIFF/";

	Magick::InitializeMagick(*argv);
	try
	{
		long time = GetTickCount();
		Geometry geometry = Geometry(1024,768);
		Geometry cropGeometry = Geometry(2000,2000);
		Magick::Image *inputImage;
		Magick::Image *cropedImage;
		Magick::Image *finalImage;
		for (int i = 0; i<numberOfFiles; i++)
		{
			printf("\b\b\b\b\b%d", i);
			inputImageFileName = fileNames[i];

			inputImage = new Magick::Image();
			inputImage->read(dir+inputImageFileName);

			cropedImage = new Magick::Image(*inputImage);
			cropedImage->crop(cropGeometry);

			finalImage = new Magick::Image(*cropedImage);
			finalImage->resize(geometry);

			delete inputImage;
			delete cropedImage;
			delete finalImage;
		}

		cout << GetTickCount() - time << endl;

	}catch(Magick::Exception& e)
	{
		cout << e.what() << endl;
	}

	return 0;
}

I am using Windows XP SP3 32-Bit, Intel Core2Duo 2.8GHz, 4GB DDR2 RAM. Tried this using ImageMagick 6.6.1-3 beta.

Again, this is very subtle problem. You need big images and lots of them to reproduce this kind of bug.

[magick]

• Hi. I am having problem when ImageMagick caches files on disk.

What is the problem? At some point ImageMagick will cache images to disk when it runs out of memory or if one of the cache resource limits are reached (see identify -list resource). When memory is exhausted, bad things can happen. Some algorithms may need memory to complete and the system may refuse an allocation request. That is why ImageMagick has resource limits to prevent resource exhaustion if it can. Set your map and memory limits to 80mb, for example, and rerun your process. Does it still fail? The limits ensure there is still plenty of memory so ImageMagick algorithms can allocate heap as needed.

[berathebrain]

Have you looked at the two code examples above? I don't see why would image magick at some point cache files on disk in the first example and in the second example would not. The problem is that in both examples,
the running process doesn't consume more than 600MB of memory and there is 1.4GB of memory for the process to consume ( I think processes on Win 32-bit machines can allocate at most 2GB of memory), so there is plenty of memory to allocate an image.

[magick]

The two algorithms behave exactly the same. The difference is the response the OS gives when ImageMagick asks for memory. If the request is denied, ImageMagick automatically caches the image to disk. You can run the same program over and over and in some cases, the OS will permit a memory allocation and other times it may not.

[ivan.savu]

If you look closely at the provided example you will notice that memory should never get exhausted. Each image is loaded from disk, processed and then thrown away. Since each image is exactly the same size, memory requirements should be constant during entire life cycle of the program. When image is loaded into Magick::Image variable that resides on the stack, and is recycled, memory requirements of the program rise linearly till it runs out of memory and starts caching, when we use Magick::Image that is on the heap, which we manually delete, memory requirements are constant. This probably implies that there is memory leak in Image::read. We will test this a bit more and provide you with the resulting data. Thank you for your time.

[berathebrain]

I have tested the two examples above by running it 20 times. First example was caching each and every time and the second example did not. So I don't see how this got to do with OS responsiveness.
Furthermore here are another two examples that uses stack and in the first example the files are being cached and in the other example they are not.
First example:

Code: Select all

Magick::Image inputImage;
for (int i = 0; i<numberOfFiles; i++)
{
	inputImageFileName = fileNames[i];
	inputImage.read(dir+inputImageFileName);
}

Second example:

Code: Select all

for (int i = 0; i<numberOfFiles; i++)
{
        Magick::Image inputImage;
	inputImageFileName = fileNames[i];
	inputImage.read(dir+inputImageFileName);
}

In the first example the memory consumption constantly jumps up and down from 500MB to 900MB. In the second example the memory jumps from 100MB to 500MB. Clearly here destructor releases memory in the second example and in the first example the destructor is not being called(which is logical and obvious) but I don't see why would Image class when reading a new image retain the memory of an old image in the memory thus the total amount of memory consumption is twice the normal image and then at some point release that memory. Maybe it would be better if the old image would be deallocated before reading a new image.

Thank you for your patience.

[magick]

To force the C++ garbage collector on each loop, put the static allocation inside the loop:

Code: Select all

      for (int i = 0; i<numberOfFiles; i++)
      {  
         Magick::Image inputImage;
         Magick::Image cropedImage;
         Magick::Image finalImage;
         printf("\b\b\b\b\b%d", i);
         inputImageFileName = fileNames[i];
         inputImage.read(dir+inputImageFileName);
         cropedImage = inputImage;
         cropedImage.crop(cropGeometry);
         finalImage = cropedImage;
         finalImage.resize(geometry);
      }

[ivan.savu]

Unfortunately the code provided here is just a simple proof of concept demonstrating the issue we have with IM. In our real life example Image object is shared between many objects and the only way to ensure calling of destructor would be to allocate images on heap, and manually delete them. And as far as I understand this is not a desirable way to use ImageMagick.