IM Performance Issue

[rotem]

Hey all,

We're using a real time image processing server which currently runs ImageMagick commands such as "draw text" in command line.

Currently we're running one large command with a lots of convert commands with parallelism (using & operator in command line), in order to avoid spawning overhead.

We're wondering whether performance can be maximized by working with one of the bindings (go,ruby etc.) that work directly with libmagic.

Our current tests with RMagick have pointed (surprisingly to me) otherwise and are actually worse than the command line alternative.

Am I missing something? Is RMagick just not the correct choice or is command line really comparable or better than these bindings, performance-wise?

[fmw42]

Currently we're running one large command with a lots of convert commands with parallelism (using & operator in command line), in order to avoid spawning overhead.

This is not clear to me. Perhaps I misunderstand.

In my opinion, your best bet for performance would be to put multiple command lines into one big command line deleting temporary file as you do not need them. This can be done by managing either clones or in memory mpr: images. This avoids the time needed to save intermediate images to disk. If you are already doing that, then my apologies.

I do not know all the IM APIs and whether they run faster than command line. From my limited personal experience, PHP exec() runs just as fast as Imagick. I do not know RMagick or any of the other APIs. Perhaps using Magickcore or Magickwand from the C API might be faster. I have no real experience in that regard. So one of the developers or other users who might have this experience would need to answer.

I have run shell scripts on platforms that have multiple cores, e.g. 6 and we have done some timing tests. Seems like one gets the best performance by running one script copy per core/thread or at best 2 threads and running them simultaneously. That is spawning the same task for different sets of data to different copies of the same script.

[snibgo]

Am I missing something?

The important question is: what is taking the time? If 90% of the time is spent within IM, doing image processing, then changing APIs or whatever will save at most 10%. But if 50% of the time is spent in script activities like looping and calculating, you might save all of that by using a compiled language.

If most of the time is disk access, then that's what you should concentrate on.

[rotem]

fmw42: We don't use any temporary files.

snibgo: How can I know how much time is wasted on IM itself, the HD and the script commands?

This is example for how we run 2 commands in the same time:

Code: Select all

X=`gdate +%s%N`;            convert "-quality" "100" "/Users/rotemyakir/WebstormProjects/buggy-image-server/temp/test4/thumbs/mp4/thumb70.jpg" "-gravity" "SouthEast" "-draw" "image Over 2,2 0,0 assets/ic_gug_logo.png" "-font" "./assets/impact.ttf" -pointsize 40 -gravity West -stroke black -strokewidth 10 -draw "text +5+0 'H'" -draw "text +35+0 'E'" -draw "text +55+0 'L'" -draw "text +75+0 'L'" -draw "text +90+0 'O'" -draw "text +120+0 'M'" -draw "text +150+0 'Y'" -draw "text +180+0 'F'" -draw "text +200+0 'R'" -draw "text +220+0 'I'" -draw "text +240+0 'E'" -draw "text +260+0 'N'" -draw "text +280+0 'D'" -stroke none -fill white -draw "text +5+0 'H'" -draw "text +35+0 'E'" -draw "text +55+0 'L'" -draw "text +75+0 'L'" -draw "text +90+0 'O'" -draw "text +120+0 'M'" -draw "text +150+0 'Y'" -draw "text +180+0 'F'" -draw "text +200+0 'R'" -draw "text +220+0 'I'" -draw "text +240+0 'E'" -draw "text +260+0 'N'" -draw "text +280+0 'D'" "/Users/rotemyakir/WebstormProjects/buggy-image-server/temp/test4/result.jpg"          &       convert "-quality" "100" "/Users/rotemyakir/WebstormProjects/buggy-image-server/temp/test4/thumbs/mp4/thumb70.jpg" "-gravity" "SouthEast" "-draw" "image Over 2,2 0,0 assets/ic_gug_logo.png" "-font" "./assets/impact.ttf" -pointsize 40 -gravity West -stroke black -strokewidth 10 -draw "text +5+0 'H'" -draw "text +35+0 'E'" -draw "text +55+0 'L'" -draw "text +75+0 'L'" -draw "text +90+0 'O'" -draw "text +120+0 'M'" -draw "text +150+0 'Y'" -draw "text +180+0 'F'" -draw "text +200+0 'R'" -draw "text +220+0 'I'" -draw "text +240+0 'E'" -draw "text +260+0 'N'" -draw "text +280+0 'D'" -stroke none -fill white -draw "text +5+0 'H'" -draw "text +35+0 'E'" -draw "text +55+0 'L'" -draw "text +75+0 'L'" -draw "text +90+0 'O'" -draw "text +120+0 'M'" -draw "text +150+0 'Y'" -draw "text +180+0 'F'" -draw "text +200+0 'R'" -draw "text +220+0 'I'" -draw "text +240+0 'E'" -draw "text +260+0 'N'" -draw "text +280+0 'D'" "/Users/rotemyakir/WebstormProjects/buggy-image-server/temp/test4/result2.jpg"   &      wait      ;echo $(($(($X-`gdate +%s%N`)) / -1000000))  

[snibgo]

How can I know how much time is wasted on IM itself, the HD and the script commands?

Your operating system has tools that tell you how much time is spent in CPU, or waiting for disk, etc.

-draw "text +5+0 'H'" -draw "text +35+0 'E'" {...}

There's an opportunity, right there. You can draw many objects within a single "-draw", like this:

-draw "text +5+0 'H' text +35+0 'E' {...}"

I expect that will make a significant difference.

[fmw42]

You are also using two converts on the same input image and writing out two outputs. So you are reading the same image twice. You could make one command line read the image once and write the output twice.

[fmw42]

This is probably an IM issue being forgiving and allowing two alternate syntax, but in your -draw, you use +X+Y. The documentation at http://legacy.imagemagick.org/script/co ... s.php#draw specifies for -draw X,Y.

[rotem]

You are also using two converts on the same input image and writing out two outputs. So you are reading the same image twice. You could make one command line read the image once and write the output twice.

Changed it to read and write different files but it did not make any noticeable difference

you use +X+Y. The documentation at http://legacy.imagemagick.org/script/co ... s.php#draw specifies for -draw X,Y.

You're right, changed to X,Y

You can draw many objects within a single "-draw"

Reduce 270 milliseconds to 240!

Thank you guys!