Image slow in Multi-thread

[panzm]

1. Blob blob(data,len);
2. Image image(blob);
code above running in Multi-thread,Line 2 will run sequentially. is because image(blob) is a I/O operation and blocked？
i need deal with a lot of pictures，may i deal with them concurrently？

[magick]

Certain operations in Magick++ are locked, for example, to enable reference counting and image list maintenance (to support STL). These operations should be atomic and fast. Reading an image blob should not be locked and should be able to run in parallel (unless the underlying I/O layer is blocking). If you post a small program we can download, build, and execute to illustrate the problem, we will investigate further.

[panzm]

here is my code:

Code: Select all

#include <Magick++.h>
#include <iostream>
#include <unistd.h>
#include <time.h>
#include <pthread.h>
using namespace std;
using namespace Magick;
#define NUM_THREADS     2

void *testGM(void *ptr)
{
    long tid= pthread_self();
    cout << "enter Thread ID: " << tid << endl;

    Image watermark("/home/wans/0.jpg");

    time_t start=clock();

    Blob ret;
    watermark.write(&ret);

    time_t end=clock();
    cout<<"Thread ID: "<<tid<<"execute time:"<<double(end-start)*1000/CLOCKS_PER_SEC<<endl;

    pthread_exit(NULL);
}
int main(int argc, char **argv)
{
    InitializeMagick(NULL);
    pthread_t threads[NUM_THREADS];
    int rc;
    int i;
    for( i=0; i < NUM_THREADS; i++ ){
        //cout << "main() : creating thread, " << i << endl;
        rc = pthread_create(&threads[i], NULL, testGM,NULL);
        if (rc){
            cout << "Error:unable to create thread," << rc << endl;
            exit(-1);
        }
    }
    pthread_exit(NULL);

}

[panzm]

the output is:

Code: Select all

enter Thread ID: 139942202099456
enter Thread ID: 139942193706752
Thread ID: 139942202099456execute time:422.408
Thread ID: 139942193706752execute time:388.353

but if i set #define NUM_THREADS 20
the output is:

Code: Select all

enter Thread ID: 140502406002432
enter Thread ID: 140502397609728
enter Thread ID: 140502389217024
enter Thread ID: 140502380824320
enter Thread ID: 140502372431616
enter Thread ID: 140502364038912
enter Thread ID: 140502355646208
enter Thread ID: 140502271784704
enter Thread ID: 140502263392000
enter Thread ID: 140502254999296
enter Thread ID: 140502246606592
enter Thread ID: 140502238213888
enter Thread ID: 140502229821184
enter Thread ID: 140502221428480
enter Thread ID: 140502129174272
enter Thread ID: 140502120781568
Thread ID: 140502487537408execute time:4519.32
Thread ID: 140502495930112execute time:4497.89
Thread ID: 140502504322816execute time:4472.84
Thread ID: 140502406002432execute time:4440.6
Thread ID: 140502221428480execute time:4408.8
Thread ID: 140502254999296execute time:4379.11
Thread ID: 140502246606592execute time:4353.21
Thread ID: 140502364038912execute time:4324.89
Thread ID: 140502397609728execute time:4289.88
Thread ID: 140502129174272execute time:4261.5
Thread ID: 140502389217024execute time:4230.28
Thread ID: 140502238213888execute time:4205.29
Thread ID: 140502263392000execute time:4176.73
Thread ID: 140502137566976execute time:4142.68
Thread ID: 140502372431616execute time:4110.18
Thread ID: 140502120781568execute time:4082.82
Thread ID: 140502229821184execute time:4050.1
Thread ID: 140502271784704execute time:4020.05
Thread ID: 140502355646208execute time:3990.02
Thread ID: 140502380824320execute time:3954.88

[magick]

We're using ImageMagick 6.9.3-8 and built your script with this command:

• c++ `Magick++-config --cxxflags --cppflags` -O2 -o thread thread.cpp `Magick++-config --ldflags --libs`

Threading is obvious if we read a small image, 70x46 (rose:). We get 'enter Thread ID' intermixed with 'Thread ID' suggesting some of the threads are finishing as others are starting. If our image is larger, 4000x4000, the script returns all the 'enter Thread ID' comments before any of the initial threads complete. This is expected since a large image is time consuming whereas setting up the threads is relatively fast.

[panzm]

i read a small image,40X40,but didn`t get 'enter Thread ID' intermixed with 'Thread ID'.
i find this code in github:

Code: Select all

void Magick::Image::write(Blob *blob_)
{
  size_t
    length=0;

  void
    *data;

  modifyImage();
  GetPPException;
  data=[b]ImagesToBlob(constImageInfo(),image(),&length,exceptionInfo);[/b]
  if (length > 0)
    blob_->updateNoCopy(data,length,Blob::MallocAllocator);
  ThrowImageException;
}

is ImagesToBlob a I/O operation?

[panzm]

I want to deal a lot of image in parallel.
how can i do that efficiently?
thanks.

[magick]

Can you verify which version of ImageMagick you are using? Are you on Linux? Try 'ldd magick++` where magick++ is your compiled script. What does the ldd command report for the MagickCore and Magick++ libraries?

[panzm]

I'm on Linux.

Code: Select all

[fsdevops@localhost lib64]$ Magick++-config --ldflags --libs
-lMagick++-6.Q16 -lMagickWand-6.Q16 -lMagickCore-6.Q16 
-lMagick++-6.Q16 -lMagickWand-6.Q16 -lMagickCore-6.Q16 

[fsdevops@localhost lib64]$ ldd libMagick++-6.Q16.so
	linux-vdso.so.1 (0x00007fffccda6000)
	libMagickWand-6.Q16.so.2 => /lib64/libMagickWand-6.Q16.so.2 (0x00007f68f74c0000)
	libMagickCore-6.Q16.so.2 => /lib64/libMagickCore-6.Q16.so.2 (0x00007f68f6ff8000)
	liblcms2.so.2 => /lib64/liblcms2.so.2 (0x00007f68f6d9d000)
	libfftw3.so.3 => /lib64/libfftw3.so.3 (0x00007f68f699f000)
	libfontconfig.so.1 => /lib64/libfontconfig.so.1 (0x00007f68f675b000)
	libfreetype.so.6 => /lib64/libfreetype.so.6 (0x00007f68f64b0000)
	libXext.so.6 => /lib64/libXext.so.6 (0x00007f68f629e000)
	libXt.so.6 => /lib64/libXt.so.6 (0x00007f68f6035000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f68f5e0e000)
	libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f68f5bfe000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f68f59e8000)
	libltdl.so.7 => /lib64/libltdl.so.7 (0x00007f68f57dd000)
	libSM.so.6 => /lib64/libSM.so.6 (0x00007f68f55d4000)
	libICE.so.6 => /lib64/libICE.so.6 (0x00007f68f53b8000)
	libX11.so.6 => /lib64/libX11.so.6 (0x00007f68f5077000)
	libgomp.so.1 => /lib64/libgomp.so.1 (0x00007f68f4e55000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f68f4ad3000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f68f47d0000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f68f45b3000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f68f41f2000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f68f3fda000)
	libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f68f3db0000)
	libpng16.so.16 => /lib64/libpng16.so.16 (0x00007f68f3b7c000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f68f3978000)
	libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f68f3773000)
	libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f68f3550000)
	/lib64/ld-linux-x86-64.so.2 (0x00005579407cb000)
	libXau.so.6 => /lib64/libXau.so.6 (0x00007f68f334c000)

[fsdevops@localhost lib64]$ ldd libMagickCore-6.Q16.so
	linux-vdso.so.1 (0x00007ffe7138b000)
	liblcms2.so.2 => /lib64/liblcms2.so.2 (0x00007f793cdc0000)
	libfftw3.so.3 => /lib64/libfftw3.so.3 (0x00007f793c9c2000)
	libfontconfig.so.1 => /lib64/libfontconfig.so.1 (0x00007f793c77d000)
	libfreetype.so.6 => /lib64/libfreetype.so.6 (0x00007f793c4d3000)
	libXext.so.6 => /lib64/libXext.so.6 (0x00007f793c2c1000)
	libSM.so.6 => /lib64/libSM.so.6 (0x00007f793c0b7000)
	libICE.so.6 => /lib64/libICE.so.6 (0x00007f793be9b000)
	libX11.so.6 => /lib64/libX11.so.6 (0x00007f793bb5b000)
	libXt.so.6 => /lib64/libXt.so.6 (0x00007f793b8f1000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f793b6cb000)
	libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f793b4bb000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f793b2a4000)
	libltdl.so.7 => /lib64/libltdl.so.7 (0x00007f793b09a000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f793ad98000)
	libgomp.so.1 => /lib64/libgomp.so.1 (0x00007f793ab75000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f793a95e000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f793a741000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f793a37f000)
	libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f793a155000)
	libpng16.so.16 => /lib64/libpng16.so.16 (0x00007f7939f22000)
	libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f7939d1c000)
	libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f7939afa000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f79398f6000)
	/lib64/ld-linux-x86-64.so.2 (0x000055683209a000)
	libXau.so.6 => /lib64/libXau.so.6 (0x00007f79396f1000)

[magick]

We tried yet another Linux system with ImageMagick 6.9.3-8. It shows threading is working properly:

Code: Select all

-> convert rose: 0/jpg

-> ./thread 
enter Thread ID: -1208697968
Thread ID: -1208697968execute time:0
enter Thread ID: -1219187824
Thread ID: -1219187824execute time:0
enter Thread ID: -1229677680
Thread ID: -1229677680execute time:0
enter Thread ID: -1240167536
Thread ID: -1240167536execute time:0
enter Thread ID: -1250657392
enter Thread ID: -1261147248
enter Thread ID: -1271637104
enter Thread ID: -1282126960
enter Thread ID: -1292616816
enter Thread ID: -1303106672
Thread ID: -1250657392execute time:0
Thread ID: -1261147248execute time:0
Thread ID: -1271637104execute time:0
Thread ID: -1282126960execute time:0
Thread ID: -1292616816execute time:0
Thread ID: -1303106672execute time:0