MagickReadImage() very slow retrieving images by http

168gr · Post by **168gr** » 2013-03-26T02:39:27-07:00

Back with another question, though this time it's performance not function related.

MagickReadImage is really, really slow when retrieving network files.

I have an IP camera connected by ethernet directly to the computer - no switch, router, anything else. I can point a browser at http://192.168.10.30/image/jpeg.cgi and get a still image of whatever the camera is looking at.

With a browser or command line tool it's very quick - wget reports .03 seconds to retrieve the image. However, MagickReadImage() takes almost 30 seconds:

Code: Select all

pgg@g550 ~/test
$ cat test3.c
#include <stdio.h>
#include <sys/timeb.h>
#include <wand/magick_wand.h>

#define ThrowWandException(wand) \
{ \
        char *description; \
        ExceptionType severity; \
        description=MagickGetException(wand,&severity); \
        printf("\n\n-----\n%s %s %lu %s\n",GetMagickModule(),description); \
        description=(char *) MagickRelinquishMemory(description); \
        exit(-1); \
}

int main() {

  long milliseconds;
  struct timeb timer1, timer2;

  MagickWand *mw1 = NULL;


  MagickWandGenesis();
  mw1 = NewMagickWand();

  ftime( &timer1 );
  if( !MagickReadImage( mw1, "http://192.168.10.30/image/jpeg.cgi" ) )
    ThrowWandException( mw1 );
  ftime( &timer2 );
  milliseconds = timer2.millitm - timer1.millitm + 1000.0 * (timer2.time-timer1.time);
  printf( "Image retrieved in %ld milliseconds.\n", milliseconds );

  MagickWriteImage( mw1, "output.jpg" );

  mw1 = DestroyMagickWand(mw1);

  MagickWandTerminus();

  return 0;
}


pgg@g550 ~/test
$ gcc test3.c -o test3 -lMagickWand-6.Q16 -lgdi32 -I./include -L/usr/local/lib -DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16

pgg@g550 ~/test
$ ./test3
Image retrieved in 26942 milliseconds.

pgg@g550 ~/test
$ wget http://192.168.10.30/image/jpeg.cgi
--2013-03-26 14:00:41--  http://192.168.10.30/image/jpeg.cgi
Connecting to 192.168.10.30:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 179537 (175K) [image/jpeg]
Saving to: `jpeg.cgi'

100%[======================================================================================================================================================>] 179,537     --.-K/s   in 0.03s

2013-03-26 14:00:41 (6.12 MB/s) - `jpeg.cgi' saved [179537/179537]

It also takes about 30 seconds retrieve image files from an internet web address. Watching my network monitor, it appears MagickReadImage() sits for almost 30 seconds and then transfers the file rapidly. Is there anything I can do to speed this up? The difference between .03 seconds and 30 seconds is enormous.

Thank you.

168gr · Post by **168gr** » 2013-03-26T09:51:42-07:00

Update / more info ...

So I rewrote the test program to use curl for the network transfer of the file to memory (using this example code, and then MagickReadImageBlob to get the data into the ImageMagick structures. This is very fast, about 350 milliseconds to retrieve the file, do MagickReadImageBlob, and also write it to a file with MagickWriteImage:

Code: Select all

pgg@g550 ~/test
$ cat test4.c
#include <stdio.h>
#include <string.h>
#include <sys/timeb.h>
#include <wand/magick_wand.h>
#include <curl/curl.h>


#define ThrowWandException(wand) \
{ \
        char *description; \
        ExceptionType severity; \
        description=MagickGetException(wand,&severity); \
        printf("\n\n-----\n%s %s %lu %s\n",GetMagickModule(),description); \
        description=(char *) MagickRelinquishMemory(description); \
        exit(-1); \
}



struct MemoryStruct {
  char *memory;
  size_t size;
};

static size_t
WriteMemoryCallback(void *contents, size_t size, size_t nmemb, void *userp)
{
  size_t realsize = size * nmemb;
  struct MemoryStruct *mem = (struct MemoryStruct *)userp;

  mem->memory = realloc(mem->memory, mem->size + realsize + 1);
  if(mem->memory == NULL) {
    /* out of memory! */
    printf("not enough memory (realloc returned NULL)\n");
    return 0;
  }

  memcpy(&(mem->memory[mem->size]), contents, realsize);
  mem->size += realsize;
  mem->memory[mem->size] = 0;

  return realsize;
}



int main() {

  CURL *curl_handle;
  CURLcode res;
  struct MemoryStruct chunk;

  long milliseconds;
  struct timeb timer1, timer2;

  MagickWand *mw1 = NULL;
  MagickWandGenesis();
  mw1 = NewMagickWand();



  ftime( &timer1 );

  chunk.memory = malloc(1);  /* will be grown as needed by the realloc above */
  chunk.size = 0;    /* no data at this point */

  curl_global_init(CURL_GLOBAL_ALL);

  /* init the curl session */
  curl_handle = curl_easy_init();

  /* specify URL to get */
  curl_easy_setopt(curl_handle, CURLOPT_URL, "http://192.168.10.30/image/jpeg.cgi");

  /* send all data to this function  */
  curl_easy_setopt(curl_handle, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);

  /* we pass our 'chunk' struct to the callback function */
  curl_easy_setopt(curl_handle, CURLOPT_WRITEDATA, (void *)&chunk);

  /* some servers don't like requests that are made without a user-agent
     field, so we provide one */
  curl_easy_setopt(curl_handle, CURLOPT_USERAGENT, "libcurl-agent/1.0");

  /* get it! */
  res = curl_easy_perform(curl_handle);

  /* check for errors */
  if(res != CURLE_OK) {
    fprintf(stderr, "curl_easy_perform() failed: %s\n",
            curl_easy_strerror(res));
  }
  else {

   struct timeb timer1, timer2;

    if( !MagickReadImageBlob( mw1, chunk.memory, chunk.size ) )
      ThrowWandException( mw1 );

    MagickWriteImage( mw1, "output.jpg" );

    mw1 = DestroyMagickWand(mw1);

  }

  ftime( &timer2 );
  milliseconds = timer2.millitm - timer1.millitm + 1000.0 * (timer2.time-timer1.time);
  printf( "Image read and written to file in %ld milliseconds.\n", milliseconds );

  /* cleanup curl stuff */

  curl_easy_cleanup(curl_handle);

  if(chunk.memory)
    free(chunk.memory);

  curl_global_cleanup();


  MagickWandTerminus();

  return 0;
}

pgg@g550 ~/test
$ gcc test4.c -o test4 -lMagickWand-6.Q16 -lgdi32 -lcurl -I./include -L/usr/local/lib -DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16

pgg@g550 ~/test
$ ./test4
Image read and written to file in 343 milliseconds.

pgg@g550 ~/test
$ identify output.jpg
output.jpg JPEG 1280x800 1280x800+0+0 8-bit sRGB 184KB 0.000u 0:00.000

So my problem is worked around nicely, but not really solved. I can just use the curl library to do the network read as above, but I'm wondering if there was something I did improperly with ImageMagick that made MagickReadImage() take so long.

Post by **magick** » 2013-03-26T10:27:04-07:00

ImageMagick leverages the libxml2 delegate library to read images as URLs. We tried your original example to download http://www.imagemagick.org/MagickStudio ... wizard.jpg through a proxy. It ran in 647 milliseconds which seems reasonable. We're using ImageMagick 6.8.4-0 and libxml2 2.9.0.

168gr · Post by **168gr** » 2013-03-26T11:39:27-07:00

Well that is odd. As always I appreciate your help. I'm using ImageMagick 6.8.3-7 compiled from source under Cygwin on a Windows 7 laptop. Looks like the version of libxml2 I have is 2.9.0-1.

I did some more testing, first using http://www.imagemagick.org/MagickStudio ... wizard.jpg and alternating the test programs. Sometimes it's fast, sometimes it's very slow with really long delays.

MagickReadImage(): 16094, 132, 18062, 112, 110, 8571, 90, 101, 5641
curl + MagickReadImageBlob() + MagickWriteImage(): 2860, 110, 810, 340, 510, 2440, 70, 70, 2671

This is over a very congested wifi network then satellite internet connection in Afghanistan so there's definitely some other factors outside the code. But MagickReadImage() is consistently slower than curl + MagickReadImageBlob() + MagickWriteImage() over a bunch of iterations.

Code: Select all

pgg@g550 ~/test
$ ./test3
Image retrieved in 16094 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 2860 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 132 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 110 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 18062 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 810 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 112 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 340 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 110 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 510 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 8571 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 2440 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 90 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 70 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 101 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 70 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 5641 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 2761 milliseconds.

Then I did the same test with the local image source. Again this is an ethernet cable direct from the computer to the IP camera. No network congestion issues, caching, speed-of-light delay issues with geosynch satellite comms, etc. Nothing else running in the background, other than the usual Windows garbage.

MagickReadImage(): 26786, 540, 280, 288, 282, 612, 26825, 260, 230, 281, 290, 250
curl + MagickReadImageBlob() + MagickWriteImage(): 330, 320, 340, 310, 650, 660, 311, 510, 330, 320, 610, 321

Usually fast, but sometimes there's a huge delay.

Code: Select all

pgg@g550 ~/test
$ ./test3
Image retrieved in 26786 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 330 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 540 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 320 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 280 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 340 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 288 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 310 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 282 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 650 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 612 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 660 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 26825 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 311 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 260 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 510 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 230 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 330 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 281 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 320 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 290 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 610 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 250 milliseconds.

pgg@g550 ~/test
$ ./test4
Image read and written to file in 321 milliseconds.

The first retrieval with MagickReadImage() is always slow, even if I run the curl + MagickReadImageBlob() test program first:

Code: Select all

pgg@g550 ~/test
$ ./test4
Image read and written to file in 340 milliseconds.

pgg@g550 ~/test
$ ./test3
Image retrieved in 26823 milliseconds.

Moreover, if I space out the test runs by a minute, the MagickReadImage() is ALWAYS slow:

Code: Select all

$ ./test3; sleep 60; ./test3; sleep 60; ./test3; sleep 60; ./test3; sleep 60; ./test3; sleep 60; ./test3
Image retrieved in 26788 milliseconds.
Image retrieved in 26824 milliseconds.
Image retrieved in 26810 milliseconds.
Image retrieved in 26785 milliseconds.
Image retrieved in 26775 milliseconds.
Image retrieved in 26765 milliseconds.

And the curl + MagickReadImageBlob() + MagickWriteImage() is always fast:

Code: Select all

pgg@g550 ~/test
$ ./test4; sleep 60; ./test4; sleep 60; ./test4; sleep 60; ./test4; sleep 60; ./test4; sleep 60; ./test4
Image read and written to file in 680 milliseconds.
Image read and written to file in 330 milliseconds.
Image read and written to file in 350 milliseconds.
Image read and written to file in 340 milliseconds.
Image read and written to file in 330 milliseconds.
Image read and written to file in 320 milliseconds.

And just for grins:

Code: Select all

pgg@g550 ~/test
$ ./test3; ./test3; ./test3; ./test3; ./test3; ./test3; ./test3; ./test3; ./test3; ./test3; ./test3;
Image retrieved in 26804 milliseconds.
Image retrieved in 282 milliseconds.
Image retrieved in 260 milliseconds.
Image retrieved in 240 milliseconds.
Image retrieved in 300 milliseconds.
Image retrieved in 300 milliseconds.
Image retrieved in 300 milliseconds.
Image retrieved in 260 milliseconds.
Image retrieved in 300 milliseconds.
Image retrieved in 260 milliseconds.
Image retrieved in 310 milliseconds.

I don't have any idea why the first iteration is always so slow.

168gr · Post by **168gr** » 2013-03-27T10:09:30-07:00

Another issue -

MagickReadImage() doesn't handle HTTP authentication embedded in the URL. Is there a mechanism to make this work?

Code: Select all

pgg@g550 ~/test
$ ./test3
 

-----
 test3.c main 33 no data returned
 `http://admin:admin@192.168.10.30/image/jpeg.cgi' @ error/url.c/ReadURLImage/249
 
pgg@g550 ~/test
 $ ./test4
 Image read and written to file in 310 milliseconds.
 
pgg@g550 ~/test
 $ grep cgi *.c
 test3.c:  if( !MagickReadImage( mw1, "http://admin:admin@192.168.10.30/image/jpeg.cgi" ) )
 test4.c:  curl_easy_setopt(curl_handle, CURLOPT_URL, "http://admin:admin@192.168.10.30/image/jpeg.cgi");

Post by **magick** » 2013-03-28T05:43:40-07:00

That appears to be a problem with the libxm2 delegate library. ImageMagick calls their API to download http:// and ftp:// content and the failure is returned by their API. Contact the libxml2 developers and see if that is a bug or if there is a workaround. Let us know if you find a solution

168gr · Post by **168gr** » 2013-03-28T09:40:21-07:00

magick wrote:That appears to be a problem with the libxm2 delegate library. ImageMagick calls their API to download http:// and ftp:// content and the failure is returned by their API. Contact the libxml2 developers and see if that is a bug or if there is a workaround. Let us know if you find a solution

Thanks, will do.

Post by **anthony** » 2013-04-04T18:27:44-07:00

Note I believe https://... (secure http) may be a "wget" delegate rather than use the library.

Has this changed?

If authentication is involved than https is probably a better recommendation as it will nto pass the password across the net 'in the clear'.

Legacy ImageMagick Discussions Archive

MagickReadImage() very slow retrieving images by http

MagickReadImage() very slow retrieving images by http

Re: MagickReadImage() very slow retrieving images by http

Re: MagickReadImage() very slow retrieving images by http

Re: MagickReadImage() very slow retrieving images by http

Re: MagickReadImage() very slow retrieving images by http

Re: MagickReadImage() very slow retrieving images by http

Re: MagickReadImage() very slow retrieving images by http

Re: MagickReadImage() very slow retrieving images by http