Optimising this request!

[2Aces]

Hi guys,

Apologies if this is the wrong section.

I run a site which is processing hundreds of images per minute. We use redis and php-resque to process the ImageMagick requests in the background, but wait times for each image can be 2-3 minutes due to the amount of requests + the slow processing time. Each request probably takes around 2 seconds, so when 500 image requests get queued up, users can wait up to 2-3 minutes per image.

What I'd like to do is be able to optimise ImageMagick to process these images much more faster, whether a setting or optimising our code.

The code we use to process each image is something like this:

Code: Select all

convert -size 600x400 xc:none \( ".$path."assets/images/bases/base_image_69509021433289153_8_0.png -fill rgb\(255,15,127\) -colorize 100% \) -composite \( ".$path."assets/images/bases/eye_image_60444011438514404_8_0.png -fill rgb\(15,107,255\) -colorize 100% \) -composite \( ".$path."assets/images/markings/marking_clan_8_marking_10_1433289499.png -fill rgb\(255,79,79\) -colorize 100% \) -composite \( ".$path."assets/images/bases/shading_image_893252771433289153_8_0.png -fill rgb\(135,159,255\) -colorize 100% \) -compose Multiply -composite \( ".$path."assets/images/bases/highlight_image_629750231433289153_8_0.png -fill rgb\(27,35,36\) -colorize 100% \) -compose Overlay -composite \( ".$path."assets/images/bases/lineart_image_433715161433289153_8_0.png -fill rgb\(0,0,0\) -colorize 100% \) -compose Over -composite ".$path."assets/generated/queue/tempt_preview_27992_userid_0_".$filename."_file.png

I did some testing and found that reducing the number of images per request will speed it up quite a lot (3 images being used is a lot quicker). But that's not really a solution for us since we offer lots of customisation for our users.

Does anyone have any help or tips we could use to help optimise this process?

Thank you to anyone who replies, I really appreciate it

[fmw42]

If your hardware has multiple core processors, be sure to try enabling OpenMP upon compile. You can see if it is already enabled by

convert -version

Compiling IM as Q8 may help with memory usage.

You may be able to do more tuning using your policy.xml. see

viewtopic.php?f=4&t=26801
http://www.imagemagick.org/script/resou ... nvironment

I am not experience enough on the policy and environment variables to make any suggestions.

[snibgo]

You might also try disable multi-threading, on the basis that you are (I suppose) running many processes simultaneously, so ensuring each one uses only one processor may fully utilise capacity without the per-process overhead of multi-threading.

You command uses no pixel data from any of the images. For each one, pixel data is overwritten with "-colorize". (Incidentally, the "%" is not required.) It ma be quicker to get the image dimensions from "identify -ping", and use these to crate canvases of the required colours. The "convert" command would then read no images at all.

[fmw42]

You might also try disable multi-threading, on the basis that you are (I suppose) running many processes simultaneously,

User snibgo has a good point. Also certain Linux platforms have been known to work slower with OpenMP than without. So the best advice, I suppose, is try it both ways.

[2Aces]

Hi thanks for all the responses so far!

convert -version
Version: ImageMagick 6.7.2-7 2015-07-23 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2011 ImageMagick Studio LLC
Features: OpenMP

So it looks like we have it. How would I go about disabling it temporarily? Or do I need to recompile?

You command uses no pixel data from any of the images. For each one, pixel data is overwritten with "-colorize". (Incidentally, the "%" is not required.) It ma be quicker to get the image dimensions from "identify -ping", and use these to crate canvases of the required colours. The "convert" command would then read no images at all.

I'll definitely take a look at this to see if it helps us.

[fmw42]

You can recompile using --disable-openmp. Or export the environment variable MAGICK_THREAD_LIMIT=1 in your .profile file. Or put -limit thread 1 right after convert in your command line. Recompiling is global until you recompile again without it. Exporting the environament variable is global until you remove it from your .profile. The latter is just temporary to the one command. You can also just put MAGICK_THREAD_LIMIT=1 before your convert.

See viewtopic.php?f=2&t=20756&p=83407&hilit=thread#p83407.

[glennrp]

Version: ImageMagick 6.7.2-7 2015-07-23 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2011 ImageMagick Studio LLC

Note that the date is misleading. 6.7.2-7 was released on or about 2011-09-15. The
"Version" date is the date you installed it.

[fmw42]

IM 6.7.2.7 is rather old (193 versions old). You might consider upgrading.

[2Aces]

IM 6.7.2.7 is rather old (193 versions old). You might consider upgrading.

Our webhost people told us:

"It also has the latest available version for CentOS 6 obtained through the repositories as shown below:
Version: ImageMagick 6.7.2-7 2015-07-23 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2011 ImageMagick Studio LLC
Features: OpenMP "

They also said we have OpenMP installed and it's listed in the Features list. I think we must be multi-core since we have 16 cpu cores and i see them all 100% when I run a test of 500 images.

[fmw42]

You can ask your web people to install from source if you want to upgrade.

[2Aces]

Hi,

They wouldn't install from source for some reason. So I went ahead and did it myself- now it takes less than 30 seconds to process 500 images with 5 workers on my redis server. 10 workers = 19 seconds. Yay!