magick++ copy-on-write forking

[jorgjorg]

I have a linux server with 32gb ram, a C++-application implementing imagemagick and a PDF-file.

If I malloc 16gb of ram when starting the application, imagemagick Magick::Image is unable to convert the pdf-file with ghostscript due to clone() and ENOMEM error.
A normal fork would share the memory (copy-on-write) when forking. It shouldn't actually allocate and copy 16gb of ram from the parent.

If I just call fork() 3 times after allocating 16gb of ram, the process would still only use 16gb ram.


[pid 17708] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f1284de49d0) = -1 ENOMEM (Cannot allocate memory)
[pid 17708] rt_sigaction(SIGINT, {SIG_IGN}, {SIG_DFL}, = 0
[pid 17708] rt_sigaction(SIGQUIT, {SIG_IGN}, {SIG_DFL}, = 0
[pid 17708] rt_sigprocmask(SIG_BLOCK, [CHLD], [], = 0
[pid 17708] clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7f1284dd495c) = -1 ENOMEM (Cannot allocate memory)

[jorgjorg]

What also happens when the process gets close to 16gb memory usage is that a pdf convert succeeds but the output gets incomplete. The output file becomes around half the size of what it should be, and when viewing the image you see it is incorrect. The output file (tif) validates ok with identify even though it is half the size.

[magick]

Set the MAGICK_MEMORY_LIMIT and MAGICK_MAP_LIMIT environment variables to 2GiB (see http://www.imagemagick.org/script/resources.php), then run your program again. Does that help? Also try ImageMagick 6.8.9-10, the latest release if you don't already have it. Does that help?

[jorgjorg]

I'm using the latest public release 6.8.8-10 on your website.
Exporting MAGICK_MEMORY_LIMIT and MAGICK_MAP_LIMIT environment variables to 2GiB did not help. I still get ENOMEM error (and also half size output a few times)
I verified the variables with "identify -list resource":

File Area Memory Map Disk Thread Throttle Time
--------------------------------------------------------------------------------
768 67.41GB 2GiB 2GiB unlimited 48 0 unlimited

[magick]

We need to reproduce the problem before we can comment further. Post a small workable Magick++ program that we can download, compile, and run to reproduce the problem.

[jorgjorg]

Should be easy to replicate (at least as long as you have 32gb ram). See code below.
Ulimit on tested machine is:
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
virtual memory (kbytes, -v) unlimited

Available memory when testing was 28gb
MAGICK_*LIMIT variables was set to both 2gib and unset. The same error happens.

Code: Select all

#include <stdlib.h>
#include <iostream>
#include <ImageMagick/Magick++.h>


int main(int argc, char **argv)
{
        if (argc < 2) return EXIT_FAILURE;

        // Allocate 16gb ram
        char *b = new char[16000000000];
        if (!b) return -2;
        // Fill it with some data so it really gets allocated
        for (unsigned long i(0); 15000000000 > i; i += 1000000)
                b[i] = '2';

        try
        {
                Magick::Image img(argv[1]);
        }
        catch (...)
        {
                std::cout << "Magick failed" << std::endl;
                return EXIT_FAILURE;
        }

        return EXIT_SUCCESS;
}

[magick]

Your program works fine for us. If its failing, its likely in your program where you allocate 16GB of memory. If your systems runs out of memory, ImageMagick should exit gracefully with an exception but any program can fail hard if all the memory and virtual memory is consumed. If you still think there is a problem within ImageMagick, you'll need to run a debugger such as gdb and post a stack trace showing where in ImageMagick it is failing and we will investigate further.

[jorgjorg]

Did you try with a PDF-file ?

The Magick::Image constructor will fail.
If you catch the error message you get:

root@n2:/root# ./test test.pdf
Magick failed - Magick: Postscript delegate failed `test.pdf': No such file or directory @ error/pdf.c/ReadPDFImage/713

The "no such file" comes from that it cant execute ghostscript due to ENOMEM, so the error message itself is also not entirely correct.



open("/usr/app/imagemagick/6.8.8-10/share/doc/ImageMagick-6/delegates.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
open("//.config/ImageMagick/delegates.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
open("//.magick/delegates.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
brk(0xcd4000) = 0xcd4000
stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=540672, ...}) = 0
open("/tmp/magick-25602tEJ6QzOnoVAd", O_RDWR|O_CREAT|O_EXCL, 0600) = 3
close(3) = 0
brk(0xcf5000) = 0xcf5000
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f93c8059a90) = -1 ENOMEM (Cannot allocate memory)
rt_sigaction(SIGINT, {SIG_IGN}, {SIG_DFL}, = 0
rt_sigaction(SIGQUIT, {SIG_IGN}, {SIG_DFL}, = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], = 0
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fff569e48bc) = -1 ENOMEM (Cannot allocate memory)
rt_sigaction(SIGINT, {SIG_DFL}, NULL, = 0
rt_sigaction(SIGQUIT, {SIG_DFL}, NULL, = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, = 0
open("/usr/app/imagemagick/6.8.8-10/share/ImageMagick-6/locale.xml", O_RDONLY) = 3
lseek(3, 0, SEEK_END) = 2403
mmap(NULL, 2403, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f93c806a000
munmap(0x7f93c806a000, 2403) = 0
close(3) = 0
open("/usr/app/imagemagick/6.8.8-10/lib/ImageMagick-6.8.8//config-Q16/locale.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/etc/app/imagemagick/ImageMagick-6/locale.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/app/imagemagick/6.8.8-10/share/doc/ImageMagick-6/locale.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
open("//.config/ImageMagick/locale.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
open("//.magick/locale.xml", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/app/imagemagick/6.8.8-10/share/ImageMagick-6/english.xml", O_RDONLY) = 3
lseek(3, 0, SEEK_END) = 48325
mmap(NULL, 48325, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f93c804c000
munmap(0x7f93c804c000, 48325) = 0
close(3) = 0
brk(0xd1f000) = 0xd1f000
unlink("/tmp/magick-25602xZhJhBt1ISkq") = 0
unlink("/tmp/magick-25602xZhJhBt1ISkq.cache") = -1 ENOENT (No such file or directory)
unlink("/tmp/magick-25602xZhJhBt1ISkq") = -1 ENOENT (No such file or directory)
unlink("/tmp/magick-25602ANZJT9hMXW4C.cache") = -1 ENOENT (No such file or directory)
unlink("/tmp/magick-25602ANZJT9hMXW4C") = 0
stat("/tmp/magick-25602tEJ6QzOnoVAd1", 0x7fff569e49f0) = -1 ENOENT (No such file or directory)
times({tms_utime=1, tms_stime=489, tms_cutime=0, tms_cstime=0}) = 4628589349
times({tms_utime=1, tms_stime=489, tms_cutime=0, tms_cstime=0}) = 4628589349
futex(0x7f93c6feeac4, FUTEX_WAKE_PRIVATE, 2147483647) = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f93c806a000
write(1, "Magick failed - Magick: Postscri"..., 120Magick failed - Magick: Postscript delegate failed `test.pdf': No such file or directory @ error/pdf.c/ReadPDFImage/713

[magick]

Yes, we tried a PDF. Our system has 12GB of memory and it completed without complaint. We're not sure why its failing for you.

[jorgjorg]

My mistake.
My first test used 15gb of ram, not 16.

I wasn't aware of that fork() requires the same amount of free memory as the entire process is using to clone itself, even though it never use that amount of ram due to copy-on-write.
So large processes waste 50% of the ram just in order to execute/fork.