Decreased performance on sustained load for PNG images.

[ajatprabha]

Hi there,
I'm trying to build an HTTP based image processing service in Golang which will manipulate images on the fly based on query parameters in the URL of an image which is stored in an S3 bucket. ex: image.jpg?w=100&h=100

I'm using Imagick(https://github.com/gographics/imagick) which provides Go bindings to ImageMagick's MagickWand C API.

I load tested the service with a Golang load testing library with 200 requests/second for 120 seconds duration, the average response time is between 1.8-2.7s for JPG images. But when I tested this on PNG images, the response time starts increasing linearly and I start to face timeout errors in the load testing library. I tried to debug and found that reading the original image from disk and writing the processed image back to the disk is taking time and it is increasing with subsequent requests. I'm not sure why is this happening only with PNG images and not JPG images.
The only processes I'm applying right now are crop, resize, grayscale and compress.

PS: The disk is an SSD and I could've done the processing in memory but when such large requests are to be handled, memory is used up very quickly, so using the disk gave much better results here for sustained loads.

Version: ImageMagick 7.0.5-4 Q16 x86_64 2018-06-20

TIA

[snibgo]

Without seeing the test detail, commenting is difficult. Saving PNG needs more CPU and disk resources than saving JPG. If your test is intensive on either resourse, this may create queueing problems. If the load is light, but the hundredth iteration is significantly slower than the tenth, perhaps your testing process has a memory leak, or is not releasing some other resource.

[ajatprabha]

On light load, the images are getting served fine. Even if the size of resultant PNG is <= 100x100 the response time is good. But when dealing with bigger PNGs, the ReadImageBlob and WriteImage functions from the GoImagick library take more and more time on subsequent calls, they are bindings to MagickReadImageBlob and MagickWriteImage respectively. Is there a way I can optimize this? I think this is the only bottleneck now in the process. I've created object pool of MagickWant instances and I run around ~1800 of them concurrently. They all try to process the request simultaneously. Maybe the resources required by MagickReadImageBlob and MagickWriteImage functions are not available readily.
Can you shed some light here? Also, if you need to know anything specific about the testing process, please let me know.

[snibgo]

... and I run around ~1800 of them concurrently. They all try to process the request simultaneously.

I don't know IMagick or GoImagick, but trying to do too much concurrently can create problems, with too much effort being spent on task scheduling and swapping and locking of memory and files and IO interrupts, and these overheads can take more time than the useful work. Perhaps breaking the workload into consecutive portions instead of concurrent will improve performance.