Page 1 of 2
Exception thrown No space for output buffer
Posted: 2018-02-15T08:58:29-07:00
by Keith.Vinson
Hello All,
When processing some fairly large TIFF images 400MB+ I randomly get this error (which doesn't seem to be related to any specific image from the block of images I'm working with), which seem to me to indicate some sort of an out of memory error. However at the time of the error the process's memory utilization is fairly low typically less than 200 MB on a machine showing about 8GB free & plenty of free disk space. Any ideas as to how I should fix or address this situation? Are there some config parameters I should be looking at?
Thanks,
Keith
Re: Exception thrown No space for output buffer
Posted: 2018-02-15T09:34:39-07:00
by snibgo
What version of ImageMagick? On what platform?
How many pixels are in memory at any one time?
What does "magick -list resource" say? On my computer, this is:
Code: Select all
Resource limits:
Width: 214.7MP
Height: 214.7MP
Area: 25.627GB
Memory: 11.934GiB
Map: 23.867GiB
Disk: unlimited
File: 1536
Thread: 8
Throttle: 0
Time: unlimited
If you have a small limit in there, that may be the problem. Fix it by editing delegates.xml.
Re: Exception thrown No space for output buffer
Posted: 2018-02-15T09:49:35-07:00
by Keith.Vinson
I'm using Magick.NET-Q8-AnyCPU" version="7.4.0"
Let me look for how to get the equivalent of magick -list from the DotNet library.
I looked at the delegates.xml file but I didn't see any tiff config params listed
Re: Exception thrown No space for output buffer
Posted: 2018-02-15T10:10:50-07:00
by snibgo
Can you paste the exact error message here? One possibility is that pixels have consumed all the available physical memory, leaving insufficient for the output processing. If so, then putting low limits on memory and map in delegates.xml will store pixels on disk, leaving room for output processing.
This will slow the program, so may not be the final solution, but is a good test.
Re: Exception thrown No space for output buffer
Posted: 2018-02-15T10:25:31-07:00
by magick
ImageMagick has resource limits to help push large images to disk to conserve memory, however, some TIFF images are decoded by the libTIFF delegate library rather than ImageMagick. If so, libTIFF may try to allocate enough memory for the entire image and fail. It all depends on the image type. Many TIFF images are processed by iterating over a single scanline at a time which typically does not consume much memory.
Re: Exception thrown No space for output buffer
Posted: 2018-02-15T10:42:02-07:00
by Keith.Vinson
Here's one of the errors as captured in the debugger...
ImageMagick.MagickCoderErrorException: 'No space for data buffer at scanline 4294967295. `TIFFFillStrip' @ error/tiff.c/TIFFErrors/650'
at ImageMagick.NativeInstance.CheckException(IntPtr exception, IntPtr result)\r\n
at ImageMagick.MagickImage.NativeMagickImage.ReadFile(MagickSettings settings)\r\n
at ImageMagick.MagickImage.Read(String fileName, MagickReadSettings readSettings, Boolean ping)\r\n
at ImageMagick.MagickImage..ctor(String fileName)\r\n
<the rest is just the stack trace of my crap...>
Re: Exception thrown No space for output buffer
Posted: 2018-02-15T10:48:51-07:00
by Keith.Vinson
The image in question is 14644 x 9346 as a 24 bit RGB uncompressed TIFF, its disk size is roughly 391 MB...
Re: Exception thrown No space for output buffer
Posted: 2018-02-15T13:03:32-07:00
by Keith.Vinson
IS thee a way I can control the delegate config params in code? If not I'm guessing I can by adding parameters into the delegates.xml file. Are the parameters documented somewhere?
Thanks,
Keith
Re: Exception thrown No space for output buffer
Posted: 2018-02-15T13:05:25-07:00
by magick
It looks like our speculation was correct. The libTIFF delegate library is attempting to allocate enough space to host the entire image in memory. We did not write nor do we support libTIFF so we can't help. If you post a URL to your image, we will download and inspect it. We might be able to read the image a scanline at a time which would resolve the problem and conversely the image might be in a format only libTIFF can handle. We can't tell without inspecting the image.
Re: Exception thrown No space for output buffer
Posted: 2018-02-15T13:08:20-07:00
by Keith.Vinson
Thanks, I'll post the image and send you a URL as soon as I can get it uploaded.
Re: Exception thrown No space for output buffer
Posted: 2018-02-15T14:37:12-07:00
by Keith.Vinson
Re: Exception thrown No space for output buffer
Posted: 2018-02-15T17:01:24-07:00
by Keith.Vinson
Just an FYI: while playing with it, I received a new class of error while importing image 4.tif.
"2018-02-15T18:00:17-06:00 4:24.578 70.188u 7.0.7 Exception OpusFreeFlow.exe[8204]: constitute.c/ReadImage/572/Exception\n Coder (TIFF) generated an image despite an error (450), notify the developers"
Re: Exception thrown No space for output buffer
Posted: 2018-02-15T18:38:29-07:00
by magick
The conversion worked for us with this command:
Code: Select all
convert -limit memory 0 -limit map 0 1.tif 1.png
We're using ImageMagick 7.0.7-22 under Linux. Try the command line as above. Does that work? If so, you need to set the memory and map resource limits in your Magick.NET script.
Re: Exception thrown No space for output buffer
Posted: 2018-02-16T11:40:20-07:00
by Keith.Vinson
Inside the Magick.NET library I tried setting the memory and map sizes to 0, it still throws the exception: "2018-02-16T12:29:22-06:00 0:16.702 1.703u 7.0.7 Exception OpusFreeFlow.exe[19276]: tiff.c/TIFFErrors/650/Exception\n No space for data buffer at scanline 4294967295. `TIFFFillStrip'"
I used the policy.xml file to set them like this:
<policy domain="resource" name="memory" value="0MiB"/>
<policy domain="resource" name="map" value="0MiB"/>
I haven't tried the command line version yet, I'll have to download a copy of it as the Dot.Net library doesn't expose the underlying DLL and EXEs for me to try...
Re: Exception thrown No space for output buffer
Posted: 2018-02-16T13:36:50-07:00
by Keith.Vinson
I'm guessing under windows this is the equivalent cmd line "magick -limit memory 0 -limit map 0 1.tif 1.png" which worked, but then again this cmd line also worked: "magick 1.tif 1.png". To me it seems that this issue seems to be more related to large object heap fragmentation issues. In that, sometimes the conversion works and sometimes it doesn't even when processing the same image in a loop. I've had the test fail on its first attempt, and then sometimes it may loop 3 or 4 times before the failure occurs. But sooner or later it will fail with the same exception "No space for data buffer "
Keith