identify -format '%k-%T' hangs on large files

[grigory]

Hey guys,

I have a web service using Imagemagick (Core2Duo + 8GB RAM).
When I run the following command on a 15MB GIF image (I tried different images):
/usr/local/bin/identify -format '%k-%T|' /home/webiste/image.gif
identify just hangs most of the time.

At first, it takes 100% CPU, then it takes 500MB of memory... 800MB... then it uses all the memory, all the swap, and then my server goes down in a minute.

Any ideas on how to solve this issue?

Thanks!

[snibgo]

%k calculates the number of unique colours. I don't know how it works. Perhaps it sorts all the pixels, and runs though these aggregating counts.

But a GIF can contain no more than 256 colours, so a faster method would be to simply walk through all the pixels, incrementing a counter for the appropriate palette entry.

How many pixels does your image have? What is the image -- a photograph, or a graphic with large areas of block colour?

[grigory]

Well, I use a GIF made from video. Its size is 2.23MB. Then I resize the file to bigger one so it's 15-20MB in size.
After that I try to use identify on that big GIF, and it hangs. I would say it has 2000x1000 or maybe even 2500x1500 in pixels.

[snibgo]

That isn't large. I can do %k on a 35 million pixel GIF in about 3 seconds.

What version of IM, on what platform? I use IM v6.9.1-0 on Windows 8.1.

EDIT: What is the value of $T? If that contains percent signs, that might explain the problem.

[grigory]

Sorry, I use '%k-%T', not '%k-$T'. Posted wrong string here.
I use CentOS and 6.9.1.-2 version of ImageMagick with OpenMP:

Version: ImageMagick 6.9.1-2 Q16 x86_64 2015-05-11 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2015 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Features: DPC OpenMP
Delegates (built-in): jng jpeg png zlib

[fmw42]

If this is a multi-frame gif, then how may frames do you have. IM may be running out of memory and swapping to disk, since it likely has to read every frame into memory rather than read each frame one at a time.

You could try converting to miff streaming format and that might then process each frame separately. Try

Code: Select all

convert yourlarger.gif miff:- | convert - -format "%k-%T" info:

or start with your smaller gif and do the resize in line.

Code: Select all

convert yoursmaller.gif -resize ... miff:- | convert - -format "%k-%T" info:

Does that help?

[grigory]

Hi fmw42,

Will your solution take longer to show the result?
I mean that miff streaming because of separately processing.

[fmw42]

Hi fmw42,

Will your solution take longer to show the result?
I mean that miff streaming because of separately processing.

I do not know, probably not much. But did you even try it to see if it works?

[grigory]

Yes, I've tried.

First solution takes more time (56 seconds for 10 repeats in a row vs. 47 seconds for my previous command). Here are results:

/usr/local/bin/convert /gifs/00005.gif miff:- | convert - -format "%k-%T|" info:
#real 0m55.736s
#user 0m45.543s
#sys 0m8.853s

and

/usr/local/bin/identify -format '%k-%T|' /gifs/00005.gif
#real 0m46.814s
#user 0m41.527s
#sys 0m3.420s

And I can't test you second solution because script doesn't know the path to previous smaller image to use it.

[fmw42]

And I can't test you second solution because script doesn't know the path to previous smaller image to use it.

Why do you not know where the smaller image resides, if you say above that you resize it in your old command sequence?

Nevertheless, if piping slows down one command it will slow down the other.

Sorry, I have no other ideas.