Temp Files not being closed and deleted

[GeneralHQ]

We have an application that uses ImageMagick to manipulate many (possibly thousands) of images in a batch mode, and have come across a problem that ImageMagick temporary files are not being properly cleaned up, and eventually we will get an error for 'Too Many Open Files'.

It appears as if the old version (around Ver 6.2.6-Q16) worked fine, but this has been an issue for all version since then, including the current 6.4.2 release.

I have taken the liberty of trying to debug this issue, and believe I have found the problem.

The bug appears in our code when we execute the following line of code:
tiffimg.write(poblob);
where tiffimg is an Image and poblob is a pointer to a Blob.
The Image is a .TIFF image, that may cause a warning such as 'tags not sorted in ascending order', which as far as our program is concerned, is ok and not a problem.

Tracing into the write() function, the bug appears to be in the ‘CloseBlob’ function, around line 473 of the file blob.c (based on the 6.4.1-10 build for Windows), where it currently reads:
if (image->blob->exempt != MagickFalse)
{
image->blob->type=UndefinedStream;
return(MagickTrue);
}

If this if() condition is true, then it seems to return from the CloseBlob() function prematurely, without first closing the file associated with this blob, as it otherwise would do further down in the function, in that same file at line 516.
status=fclose(image->blob->file);

Later, after returning from the CloseBlob() function, an attempt is made to delete this temporary file, but that delete fails with an ‘Access’ error, because the operating system cannot delete the file while it is still open. That is why the temporary files are not deleted, and also why we end up having too many files open after processing many images.

By being sure to always call the fclose() function before returning from CloseBlob(), the problem seems to be fixed.

Is there any reason that if() clause is not located further down in the CloseBlob() function, after the file is closed, or a reason why the file is not closed if the if() condition is true? It does seem to me that the file should always be closed in CloseBlob() no matter what, and the fact that there is a code path that does not close it is a bug. No?

And Is there a way we can get a fix for this incorporated into the next release of ImageMagick?

Thank you,
Robert E. Lee
GeneralHQ

[magick]

We tried a quick program with ImageMagick 6.4.2-3 and put the 'tiffimg.write(blob)' in a loop and checked the number of file descriptors and it always returned the same number (3) which suggests there is no file leak. To investigate further we need to reproduce the problem. Post a small program that we can download and run to reproduce the problem. If we can reproduce the problem, we will have a fix for you within a day or two. Thanks.

[GeneralHQ]

ok. I shall try to produce a small program that demonstrates the issue, and will post it here once I have that working.

This problem doesn't seem to always happen with all images, but seems to happen when our source TIFF image generates innocent warnings from ImageMagick. Also, since the problem happens on the write side, not the read side, I probably should also have mentioned that we are trying to take these TIFF images to produce PCL output. So in our environment, what we are doing is using ImageMagick to read TIFF files and produce output PCL files. Therefore, I will try to put together a simple TIFF to PCL converter program and a sample TIFF data file that demonstrates the issue.

Stay tuned ...

[GeneralHQ]

ok. I have a demonstration program ready to post. How do I do that?

It is a C++ program (using Visual Studio 2003) that includes the source files, project files, and a sample image data file. When you build this program and run it against the data file, the problem will manifest itself.

I have this all ready to post as a ZIP file package with all you need to reproduce the problem ... but can' see how to upload the zip file to this board. Help!

GeneralHQ

[magick]

Post a URL to your Zip file so we can download it or e-mail it to generalhq@virginimage.org.

[GeneralHQ]

E-mail sent. Thank you.

[magick]

We can reproduce the problem you posted and will have a fix in ImageMagick 6.4.2-4 Beta within a day or two. Thanks.

[GeneralHQ]

Thank you.

[magick]

Increasing the memory limit does not guarantee that the pixels will be processed in memory. The operating system may reject our request for a large memory area.

We tried reproducing the problem with TIFF and could not. All temporary files are removed when the task completes. We are using ImageMagick 6.4.3-10.

Can you post the command you are using and a URL to one of your images. We need to reproduce the problem before we can be of any further help.

[goranga]

I am having a similar problem with the dumping of temporary data specifically in the form of magick-* files. The command I am using is

>convert *.jpg foo.mpg

*.jpg ends up being about 2000 40kb images, this is down from 4000 40kb images which would cause convert to choke and die horribly but thats besides the point.

each run uses ~ 6gigs of ram and dumps ~3 gigs of 2mb temporary files

This is using ImageMagick 6.4.3 Q16

Please let me know if there is any more information I can provide you the help sort out this problem.

[magick]

We can reproduce the problem you posted about and have a patch in the Subversion trunk available sometime tomorrow. Thanks.

To fix the problem yourself, change coalesce_image=DestroyImage() to coalesce_image=DestroyImageList() in WriteMPEGImage() in coders/mpeg.c.

[goranga]

Thanks for fixing that problem so quickly! Unfortunately I have another using the same set up I outlined before, except using 6.4.4. Occasionally an error will be sent to all Administrative users that "ffmpeg.exe encountered a problem and needed to close."

the event viewer also produced,

Date: 10/9/2008
Time: 10:58:54 PM
Description:
Reporting queued error: faulting application ffmpeg.exe, version 0.0.0.0, faulting module msvcrt.dll, version 7.0.3790.3959, fault address 0x00037e23.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74 Applicat
0008: 69 6f 6e 20 46 61 69 6c ion Fail
0010: 75 72 65 20 20 66 66 6d ure ffm
0018: 70 65 67 2e 65 78 65 20 peg.exe
0020: 30 2e 30 2e 30 2e 30 20 0.0.0.0
0028: 69 6e 20 6d 73 76 63 72 in msvcr
0030: 74 2e 64 6c 6c 20 37 2e t.dll 7.
0038: 30 2e 33 37 39 30 2e 33 0.3790.3
0040: 39 35 39 20 61 74 20 6f 959 at o
0048: 66 66 73 65 74 20 30 30 ffset 00
0050: 30 33 37 65 32 33 037e23

[magick]

We include ffmpeg.exe as a convenience. We did not write it, instead its a delegate program ImageMagick uses to process some video formats. The path forward would be to contact the ffmpeg developers or remove it from your system (of course you won't be able to read video formats if you do).

[goranga]

I see. So would the following usage of convert require ffmpeg?

>convert *.jpg foo.mpg

[magick]

Yes.