C++ API graceful handling of MAGICK_TMPDIR becoming full

[whatdoido]

I have a C++ process on Linux that processes a large number of images - when TMPDIR / MAGICK_TMPDIR is full the process crashes with SIGBUS

I can make this situation appear more readily by creating a small 1MB tmpfs filesystem and pointing MAGICK_TMPDIR at it - the crash/output is produced with that setup

Code: Select all

Thread 1066 "imgcat" received signal SIGBUS, Bus error.
[Switching to Thread 0x7fff6ae31700 (LWP 9427)]
0x00007fffbddf561e in ReadJPEGImage () from /usr/lib64/ImageMagick-6.9.9/modules-Q16/coders/jpeg.so
(gdb) where
#0  0x00007fffbddf561e in ReadJPEGImage () from /usr/lib64/ImageMagick-6.9.9/modules-Q16/coders/jpeg.so
#1  0x00007ffff73f396e in ReadImage () from /lib64/libMagickCore-6.Q16.so.5
#2  0x00007ffff73be7a0 in BlobToImage () from /lib64/libMagickCore-6.Q16.so.5
#3  0x00007ffff7bb1107 in Magick::Image::read(Magick::Blob const&) () from /lib64/libMagick++-6.Q16.so.8
#4  0x00007ffff7bb1169 in Magick::Image::Image(Magick::Blob const&) () from /lib64/libMagick++-6.Q16.so.8
#5  0x000000000041db18 in ImgThumbGen::_genthumbnail (this=0xb726e0, path_=".thumbs/6091047-6450.jpg",

At frame 5

Code: Select all

Magick::Image  img(Magick::Blob( preview_.pData(), preview_.size() ));

the size of buffer provided is 4,296,964 bytes so not large/inline with the last several hundred images that were processed.

Is there a recommended way to trap and handle the condition whereby ImageMagick fails with generating its tmp files in TMPDIR/MAGICK_TMPDIR locations?

Using ImageMagick 6.9.9-13 Q16 on Fedora26 (from their official repos)

[magick]

To avoid SIGBUS, either point MAGICK_TMPDIR to a partition with plenty of free space or lose some performance by eliminating memory mapping. That is, set your MAP limit to 0. From the command-line its -limit MAP 0. Now when the resource limits are reached, you should get a graceful exit on I/O error.

[whatdoido]

Thanks for the response - I understand that to avoid this problem, its best to point at a large enough partition for internal magick operations.

However, by default on linux, it appears that magick uses /tmp. On many systems, /tmp is 1-2GB in size - with this default setup, my program crashes with SIGBUS which is not great. When you say:

lose some performance by eliminating memory mapping. That is, set your MAP limit to 0. From the command-line its -limit MAP 0

what is the equivalent when using the C++ API? And what happens to Magick API calls under this situation?

For instance, in the call to:

Code: Select all

Magick::Image  img(Magick::Blob( preview_.pData(), preview_.size() ));

would an exception (and if so, what?) is thrown?

Interestingly enough, when running my process a second time when MAGICK_TMPDIR was already full, I didn't get the SIGBUS, rather messages:

Code: Select all

Magick: unable to extend cache `': No space left on device @ error/cache.c/OpenPixelCache/4074

[magick]

Found a better solution-- set this policy

Code: Select all

<policy domain="cache" name="synchronize" value="True"/>

See https://www.imagemagick.org/script/security-policy.php. Set to True to ensure all image data is fully flushed and synchronized to disk. There is a performance penalty, however, the benefits include ensuring a valid image file in the event of a system crash and early reporting if there is not enough disk space for the image pixel cache.

[whatdoido]

Thanks - I set this value in /etc/ImageMagick-6/policy/xml and it does not bus error but still dies.

I still think there is a bug - the BUS error in original stack trace happens when, at the start of the program, the MAGICK_TMPDIR has capacity (ie not full). As the process runs, IM creates the tmp files of size

Code: Select all

11653 open("/tmp/magick-9076bm47ay5yKLRB", O_RDWR|O_CREAT|O_EXCL, 0600) = 3
11653 fchmod(3, 0600)                   = 0
11653 lseek(3, 0, SEEK_END)             = 0
11653 pwrite64(3, "\0", 1, 289218559)   = 1

The size of each tmp file is constant - in the BUS errror situation, it only happens when we can't do alloc this file. It seems there should be a better way to

If I run the program on the same input set with MAGICK_TMPDIR at 100% the BUS erorr does not happen.