Resource limits are not working

[signsrus]

output of: cat /proc/version
Linux version 2.6.9-42.0.10.ELsmp (mockbuild@builder7.centos.org) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-3)) #1 SMP Tue Feb 27 10:11:19 EST 2007

output of: identify -version
Version: ImageMagick 6.5.0-0 2009-03-17 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2009 ImageMagick Studio LLC

problem:
ImageMagick is filling up the /tmp folder that it is using once in a while. Sometimes it writes files over 1GB!!!

I tried setting limits on the command line:

Code: Select all

$limit = "-limit memory 16 -limit map 32 -limit area 64 -limit disk 128";
$pngpath = /somedir/somefile.png;

if($imageext == ".eps" || $imageext == ".ai") {
$call = "/usr/bin/convert $limit -colorspace RGB $file_path -trim -resize '960x960>' $png_path";
exec($call, $id_output, $id_status);
				
} else if($imageext == ".psd") {
$call = "/usr/bin/convert $limit -flatten $file_path -trim -resize '960x960>' $png_path";
exec($call, $id_output, $id_status);
				
} else {
$call = "/usr/bin/convert $limit $file_path -trim -resize '960x960>' $png_path";
exec($call, $id_output, $id_status);
}

I also set limits as environment variables in /etc/profile

...
export MAGICK_FILE_LIMIT="5"
export MAGICK_MEMORY_LIMIT="16mb"
export MAGICK_MAP_LIMIT="32mb"
export MAGICK_AREA_LIMIT="64mb"
export MAGICK_DISK_LIMIT="128mb"
...

Please help.

This is detrimental because PHP uses the /tmp folder to hold sessions and if that is full, our users cannot login.

Thank you,

~Shawn

[magick]

Not a bug. Use MAGICK_TMPDIR to direct temporary files to an alternative path.

At our image site, http://www.imagemagick.org/MagickStudio ... Studio.cgi, we perform a number of sanity checks to prevent a denial of service problem. The first thing we do is limit the number of incoming characters of any upload, next we use Ping() and check to see if the width / height of the image exceeds a maximum value (ping is lightweight and does not load the image pixels). Next we set the area, memory, map, and disk limits to ensure that large images are cached to disk so the processor is not taxed and very large images return an exception. The script is available at ftp://ftp.imagemagick.org/pub/ImageMagi ... 9.3.tar.gz. Its written in Perl but the principles can be translated to imagick / PHP. This script has been in use for over 10 years now without a single incidence of denial of service.

[signsrus]

I very much appreciate your response. I am still just not understanding the point of the limits. I am under the impression that setting limits on the disk cache (i.e. area, disk limit) SHOULD stop it from filling up past the given amount.

Thank you for the directory variable. I will definitely use that for now.

Here is the dealio, we allow 16MB file uploads. Once in the PHP tmp upload folder, I check the file extension, not the best solution, but then I use identify on it to make sure it is a valid image. All I do is check to see if it is one of those valid images and if identify is spitting out that first line explaining what it is. I am not so concerned with the image size. Then I use convert to change it to a size down and make a png. I am not sure whether it is during the identify or during the convert that is making it fill up. And it is only once in a while.

Will Ping check to see about the different file types. Like, will it check if it is a valid .eps, .ai, etc. file?

Thank you so much,

~Shawn

[signsrus]

I just opened that file. I don't see Ping. I am not quite sure what to do here.

Thank you,

~Shawn

[magick]

I just opened that file. I don't see Ping. I am not quite sure what to do here.

• -> grep Ping MagickStudio-1.9.3/scripts/MagickStudio.cgi
  @extents=$composite->Ping("MagickStudio.dat");
  ($width,$height,$size,$format)=$image->Ping("$path/MagickStudio.mpc");
  @extents=$image->Ping("$filename");
  @extents=$image->Ping("*");
  @extents=$image->Ping("$format$path/MagickStudio.mpc$scene");
  ($width,$height)=$image->Ping("$path/MagickStudio.mpc");
  ($width,$height)=$image->Ping("$filename" . '[0]');
  ($width,$height)=$image->Ping("$filename" . '[0]');
  @extents=$image->Ping("$path/MagickStudio.mpc");

[magick]

I am still just not understanding the point of the limits. I am under the impression that setting limits on the disk cache (i.e. area, disk limit) SHOULD stop it from filling up past the given amount.

Correct. It will return a fatal exception if the pixel cache on disk exceeds the MAGICK_DISK_LIMIT. This is specific to the pixel cache only and the limit is for a particular instance of an ImageMagick process for all images instantiated in that instance. Note that the disk limit is only checked when the pixel cache is cached to disk. If you have a 4GB memory limit it must be exhausted before the disk limit is checked.

[signsrus]

So, what you are saying is that the temp Magick file COULD get really large while the memory limit has NOT been reached yet?

[magick]

Have you read http://www.imagemagick.org/script/architecture.php?

[signsrus]

I just got finished reading "Cache Storage and Resource Requirements." That paragraph is still reading as though pixel caching adheres to the limits that are set.

ypically the pixel cache is stored in memory using heap memory. If heap memory is exhausted, pixels are stored in in an anonymous map; if the anonymous memory map is exhausted, we create the pixel cache on disk and attempt to memory-map it; and if memory-map memory is exhausted, we simply use standard disk I/O. Disk storage is cheap but it is also very slow, upwards of 1000 times slower than memory. We can get some speed improvements, up to 5 times, if we use memory mapping to the disk-based cache. These decisions about storage are made automagically by the pixel cache manager negotiating with the operating system. However, you can influence how the pixel cache manager allocates the pixel cache with cache resource limits. The limits include:

Are the limits simply not being adhered to?

Sorry, I am not trying to argue. I am just still not understanding the limits. I understand what it says it should do but I am not seeing it work as I think it should or how it reads like it should.

I also read the next two following paragraphs, "Magick Persistent Cache Format" and "Best Practices."

Although you can force the pixel cache to disk using appropriate resource limits, disk access...

That is telling me the same thing.

Thank you for the direction to the page. I have redirected the temp directory so at least we won't get DOS.

Interestingly, I read about the MPC. So, if I do a convert to mpc on it first, could I essentially determine at that point whether the image file is legit or not? Because, if it doesn't convert, then the image is bad???

Thank you so much,

~Shawn

[signsrus]

And, where to I download PerlMagick, I can't find it anywhere.

Thank you,

~Shawn

[magick]

PerlMagick is part of every source and binary distribution of ImageMagick. Look for ImageMagick-6.5.1-3/PerlMagick to find it in the source distribution.

The cache limits work fine. You may not have set them up properly or perhaps you do not understand how they work. To get more insight, type

• identify -list resource

and to see them in action add -debug cache to your command line:

• convert -debug cache,resource image.jpg image.png

The output tells you how ImageMagick is setting up its pixel cache and how much memory, memory mapping, and disk is utilized. Now force your image to disk and see the difference in pixel cache allocation strategy:

• convert -limit area 1 -debug cache,resource image.jpg image.png

[signsrus]

Code: Select all

root [~]# identify -list resource
File       Area     Memory        Map       Disk
------------------------------------------------
   5       64mb       16mb       32mb      128mb

I will do the debugging.

Thank you again,

~Shawn

[magick]

Don't set the file limit. Its default is optimal.

[signsrus]

Is that the problem? What is the default?

[signsrus]

What does this all mean???

Code: Select all

15:10:42 0:01 0.010u 6.0.7 convert[4524]: resource.c/RelinquishMagickResource/710/Resource
 area: 0/0/64.0mb
15:10:42 0:01 0.010u 6.0.7 convert[4524]: cache.c/DestroyCacheInfo/1193/Cache
 destroy
15:10:42 0:01 0.010u 6.0.7 convert[4524]: resource.c/RelinquishMagickResource/710/Resource
 area: 0/0/64.0mb
15:10:42 0:01 0.010u 6.0.7 convert[4524]: cache.c/DestroyCacheInfo/1193/Cache
 destroy
15:10:42 0:01 0.020u 6.0.7 convert[4524]: resource.c/RelinquishMagickResource/710/Resource
 area: 0/0/64.0mb
15:10:42 0:01 0.020u 6.0.7 convert[4524]: cache.c/DestroyCacheInfo/1193/Cache
 destroy
15:10:42 0:01 0.020u 6.0.7 convert[4524]: resource.c/AcquireUniqueFileResource/369/Resource
 /tmp/magick-XXEzYgqI
15:10:42 0:01 0.020u 6.0.7 convert[4524]: resource.c/AcquireMagickResource/209/Resource
 file: 1/1/12kb
15:10:42 0:01 0.020u 6.0.7 convert[4524]: resource.c/AcquireUniqueFileResource/369/Resource
 /tmp/magick-XXxwkYP0
15:10:42 0:01 0.020u 6.0.7 convert[4524]: resource.c/AcquireMagickResource/209/Resource
 file: 1/2/12kb
15:10:42 0:01 0.020u 6.0.7 convert[4524]: resource.c/RelinquishMagickResource/710/Resource
 area: 0/0/64.0mb
15:10:42 0:01 0.020u 6.0.7 convert[4524]: cache.c/DestroyCacheInfo/1193/Cache
 destroy
15:10:42 0:01 0.020u 6.0.7 convert[4524]: resource.c/AcquireMagickResource/209/Resource
 area: 3.7mb/3.7mb/64.0mb
15:10:42 0:01 0.020u 6.0.7 convert[4524]: resource.c/AcquireMagickResource/209/Resource
 memory: 3.7mb/3.7mb/16.0mb
15:10:42 0:01 0.020u 6.0.7 convert[4524]: cache.c/OpenCache/2372/Cache
 open /tmp/magick-XXxwkYP0[0] (memory, 3.7mb)
15:10:42 0:01 0.040u 6.0.7 convert[4524]: resource.c/RelinquishUniqueFileResource/743/Resource
 /tmp/magick-XXEzYgqI
15:10:42 0:01 0.040u 6.0.7 convert[4524]: resource.c/RelinquishMagickResource/710/Resource
 file: 1/1/12kb
15:10:42 0:01 0.040u 6.0.7 convert[4524]: resource.c/RelinquishUniqueFileResource/743/Resource
 /tmp/magick-XXxwkYP0
15:10:42 0:01 0.040u 6.0.7 convert[4524]: resource.c/RelinquishMagickResource/710/Resource
 file: 1/0/12kb
15:10:42 0:01 0.070u 6.0.7 convert[4524]: resource.c/AcquireMagickResource/209/Resource
 area: 3.7mb/7.4mb/64.0mb
15:10:42 0:01 0.070u 6.0.7 convert[4524]: resource.c/AcquireMagickResource/209/Resource
 memory: 3.7mb/7.4mb/16.0mb
15:10:42 0:01 0.070u 6.0.7 convert[4524]: cache.c/OpenCache/2372/Cache
 open users/21/29/2778034911cafa1f7cf/files/Talisma_Customer_Client.pdf[0] (memory, 3.7mb)
15:10:42 0:01 0.080u 6.0.7 convert[4524]: resource.c/RelinquishMagickResource/710/Resource
 memory: 3.7mb/3.7mb/16.0mb
15:10:42 0:01 0.080u 6.0.7 convert[4524]: resource.c/RelinquishMagickResource/710/Resource
 area: 3.7mb/3.7mb/64.0mb
15:10:42 0:01 0.080u 6.0.7 convert[4524]: cache.c/DestroyCacheInfo/1193/Cache
 destroy /tmp/magick-XXxwkYP0[0]
15:10:42 0:01 0.180u 6.0.7 convert[4524]: resource.c/RelinquishMagickResource/710/Resource
 memory: 3.7mb/0/16.0mb
15:10:42 0:01 0.180u 6.0.7 convert[4524]: resource.c/RelinquishMagickResource/710/Resource
 area: 3.7mb/0/64.0mb
15:10:42 0:01 0.180u 6.0.7 convert[4524]: cache.c/DestroyCacheInfo/1193/Cache
 destroy users/21/29/2778034911cafa1f7cf/files/Talisma_Customer_Client.pdf[0]

I did this command: export MAGICK_TMPDIR=/home/magick, but it looks like it is still writing to /tmp.

I put all the export stuff in /etc/profile. I believe my Apache/PHP is running as nobody. How can I tell if these env vars are being adhered to when it is running as nobody? Is it possible that this is the problem???

Thank you,

~Shawn