Memory allocation failed 'Cannot allocate memory'

[annoyed_n_aggravated]

The setup:
I have a custom Perl script that I use for taking a set of JPG images in a directory and creating an HTML gallery with thumbnails. The script was written years ago and was using "pnmscale" (from the old Netpbm utilities) for scaling the images to thumbnail size. Due to changes on my webserver this program has stopped functioning correctly and in looking for alternatives came across ImageMagick and PerlMagick. NOTE: These images are not huge (mostly in the range of 900x600 pixels).

The problem:
I originally rewrote my "thumnail" script from using something like: "djpeg <infile.jpg | pnmscale -width 150 | cjpeg >outfile.jpg" to use the ImageMagick "convert -thumbnail" functionality. Having tested and found my "thumbnail" script apparently working I started trying to generate galleries. I would get memory allocation errors at random points during the running of the script. I thought it might be due to some sort of permissioning issue on the webserver (with my main gallery generation Perl script calling a separate Perl thumbnail script) and decided to see if using PerlMagick to directly access the ImageMagick functionality through the Perl object oriented interface would work around these problems. No dice.

The code (well the important snippet anyway):
my $image = new Image::Magick;
$image->Read("${file}.jpg");
my ($height, $width) = $image->Get('height', 'width');
if ($height > $width)
{
$width *= 150 / $height;
$height = 150;
}
else
{
$height *= 150 / $width;
$width = 150;
}
$image->Thumbnail(width => $width, height => $height);
$image->write(filename => "${file}_thumb.jpg", compress => "JPEG");

The error:
The script works to a different extent every time I run it. It will process some of the JPG files just fine (resulting in perfectly adequate thumnail images) but bomb on some images in a seemely purely random order. When the script bombs it will throw an error message in the form:

PerlMagick: Memory allocation failed 'Cannot allocate memory @ <module>

Where <module> could be one of any of the following (randomly):

cache.c/AcquirePixelCacheInfo/213
cache.c/AcquirePixelCacheNexus/288
cache.c/AcquirePixelCacheNexus/293
semaphore.c/AllocateSemaphoreInfo/174
exception.c/AcquireExceptionInfo/108

etc....

The version:
ImageMagick 6.4.8 2009-02-30 Q16 OpenMP ....

The things I've tried:
1) I've seen (though web searches) suggestions of limiting the amount of memory allowed (either increasing to allow more memory or decreasing to force it to use memory mapped files instead). No change in the program's random failure point based on a variety of limit values set.
2) Wailing and gnashing of teeth. Not much help their either (and my teeth now hurt).
3) Joining this board and posting this message to see if cooler heads can prevail in finding a solution to this annoying (and random) memory allocation problem. I hate random problems. I hate them even more when they come from a piddling little script that I just want to work so I can get on and do something more useful.

The plea:
HELP!



KM

[fmw42]

The version:
ImageMagick 6.4.8 2009-02-30 Q16 OpenMP ....

Random errors have often been associated with a bad OpenMP. Try recompiling IM with OpenMP disabled and see if that fixes the issue.

see for example: viewtopic.php?f=3&t=17029

[annoyed_n_aggravated]

Random errors have often been associated with a bad OpenMP. Try recompiling IM with OpenMP disabled and see if that fixes the issue.

see for example: viewtopic.php?f=3&t=17029

Sounds like a worthy line of research in attempting to locate the source of the random memory allocation errors. Unfortunately, this is on a webhosting server that I have no control over (other than requests for unusual Perl modules to be loaded). Unless there is a programmatic solution (i.e. fixing something bone-headed that I've coded through the PerlMagick interface) or possibly an ImageMagick configutation fix that my web host should have made but hasn't, then I'm SOL with the ImageMagick solution. If need be I'll resurrect my fossilized NetPBM solution and try to fix that. At least that solution (which was working at one point) now fails consistently. It's hard to debug a program that fails at random places each time it is run (even harder to find a solution online).

[magick]

Put these lines in the beginning of your script:

• our $AreaLimit = "128mb";
  our $DiskLimit = "1.5GB";
  our $MapLimit = "512mb";
  our $TimeLimit = 120;
  Image::Magick->new('area-limit'=>$AreaLimit,'disk-limit'=>$DiskLimit,
  'map-limit'=>$MapLimit,'memory-limit'=>$MemoryLimit,'time-limit'=>$TimeLimit);

You may need to play with the limits a bit. For an older version of PerlMagick, it may not accept the units. In that case just use 128, 1, 512, and you may need to remove time-limit.

[annoyed_n_aggravated]

Put these lines in the beginning of your script:

• our $AreaLimit = "128mb";
  our $DiskLimit = "1.5GB";
  our $MapLimit = "512mb";
  our $TimeLimit = 120;
  Image::Magick->new('area-limit'=>$AreaLimit,'disk-limit'=>$DiskLimit,
  'map-limit'=>$MapLimit,'memory-limit'=>$MemoryLimit,'time-limit'=>$TimeLimit);

You may need to play with the limits a bit. For an older version of PerlMagick, it may not accept the units. In that case just use 128, 1, 512, and you may need to remove time-limit.

Nice idea. My version of PerlMagick didn't seem to choke on any of the units but using the values you provided (or any variety of other combinations higher or lower) had no effect on the random memory allocation errors of the script. One time it ran nearly to completion (some 20 images) but died near the end. When running the script again moments later it crashed and burned after the first two images.

Still randomly and inexplicably failing. Any other ideas to be tried?

[magick]

Try using low values for memory / map. Try 1, for example. If that fails your only other option is upgrading your release of ImageMagick or posting your script for us to download / try in our environment. Without a stack trace its a difficult problem to debug remotely.

[annoyed_n_aggravated]

Setting the values to 1MB still had no positive result. I'll work on a reduced version of the script stripped of unnecessary complication. When I come up with a simplified example that still crashes randomly on the webserver I'm trying to run it on I'll post that to this board. It should be able to be tested with any random assortment of JPG files of the same general size (600x900 pixels).

Because this error is so transient I can already guess the outcome though. I'll have a script that fails miserably on my host's webserver but it will probably work fine when tested elsewhere. Such is the lot of trying to track down random errors like this. It may still provide some clues as to what can be done on my webserver to make things work.

Stay tuned for a slimmed down script sometime tomorrow. Thanks for everybody's efforts so far.

[annoyed_n_aggravated]

Here is a reduced Perl script that exhibits the random memory allocation failure on my webserver. This script first nukes any thumbnails to prevent generating thumbs of thumbs of thumbs, etc. Then it opens a JPG file, determines its dimensions, and creates a thumbnail version of the larger file so that it is 150 pixel on its longest side. I named this script 'thumb' but you can call it what you like. Put it in a directory with a handfull of JPG images and see if it will make it to the end. My record is about 7 images before the script bombs - though many times it fails to make it past the first image. I suspect this will probably work without error for anybody who takes the time to try it on their machine. I guess if it works everywhere but my webserver then I can contact tech support at the web hosting company and see if they can reproduce the error and leave it to them to debug.

Code begins below:

Code: Select all

#!/usr/bin/perl -w

use Image::Magick;

my $AreaLimit = "128MB";
my $DiskLimit = "1.5GB";
my $MapLimit = "512MB";
my $MemoryLimit = "512MB";
my $TimeLimit = 120;
Image::Magick->new(
    'area-limit'=>$AreaLimit, 
    'disk-limit'=>$DiskLimit,
    'map-limit'=>$MapLimit,
    'memory-limit'=>$MemoryLimit,
    'time-limit'=>$TimeLimit);

#
# Cleanup
#
unlink <*_thumb.jpg>;

for my $file (<*.jpg>)
{
    $file =~ s/\.jpg$//;

    print "$file\n";

    #
    # Create thumbnail image
    #
    my $image = new Image::Magick;
    $image->Read("${file}.jpg");
    my ($height, $width) = $image->Get('height', 'width');

    if ($height > $width)
    {
	$width *= 150 / $height;
	$height = 150;
    }
    else
    {
	$height *= 150 / $width;
	$width = 150;
    }

    $image->Thumbnail(width => $width, height => $height);
    $image->write(filename => "${file}_thumb.jpg", compress => "JPEG");
}

exit(0);

[magick]

Check for exceptions:

• $x = $image->Read("${file}.jpg");
  warn $x if "$x";

Do the same for Write(). Are any obvious exceptions thrown. If not, change Thumbnail() to Scale(). If that fails you may need to upgrade your release of ImageMagick.

[annoyed_n_aggravated]

Ding ding ding! Give that man a gold star for the day....

I left off exception checking from the example script for clarity. Adding it back in did nothing as no exceptions are thrown. The turning point though was the switch from Thumbnail() to Scale(). I assumed that Thumbnail() would be the appropriate method to use since it claims to toss and associated profiles (and since I was, in fact, making thumbnails). Changing to Scale() made the script work as expected. I was able to remove all of the "limit" code which is now unnecessary.

So for some reason the Thumbnail() method is running into issues with either my images or my web host's installation of ImageMagick. Either way I don't really care as long as I have a solution that lets me get on with what I'm trying to do (post some image galleries). I'll monitor this board for a little while in case anybody has any desire to look into the Thumbnail() failure a little closer. Otherwise, thanks for the help in finding a workaround to a vexing problem. BTW: The thumbnails generated by Scale() are actually slightly (less than 100 bytes) smaller than those generated using the Thumbnail() method. No idea why this is but just mentioning it in passing.