Histograms, MPRs, and Memory Management

[blue-j]

Hi all,

I've started a new topic in continuing my explorations of histograms introduced in a recent thread, as my concern now is is optimization and broader questions of memory management.

I am generating histograms with the code below, piping the image through RAM:

Code: Select all

convert input_file.jpg -separate -append -define histogram:unique-colors=false -write histogram:mpr:hgram +delete mpr:hgram -strip output_file.gif

I am using a Q16 build of ImageMagick 6.8.5-5. I have noted that when using this code, if I try to generate a histogram of an image of around 2.7MB, the histogram takes 200MB of RAM to generate. An image of 5.9MB eats up over 500MB of RAM (real memory, not virtual). Larger images of course take even more, and it gets rather unworkable quickly, despite having a machine with 32GB RAM. Some images that I'd like histograms for are over 100MB in size.

Is MPR not best for this? Am I piping inefficiently? Should i change the -depth at some point here? How to optimize this process further?

Thanks in advance for any effort you make toward addressing this with me. I sincerely appreciate it! I also just discovered Flattr for ImageMagick and will take advantage of it shortly to support this community.

Yours,
J

[magick]

Add '-limit memory 2mb -limit map 4mb' to your command line. Does that improve memory consumption?

[blue-j]

Hi, and thank you for your time! Yes, this appears to limit the resources used. In trying the values you suggested, it's taking a very long time and hasn't finished yet on a 5MB source file, but I assume your values were meant to be illustrative of what can be done and not literally to be used. Convert has held steady at 1 thread using 2.45MB real memory, with 595.39 virtual memory and hasn't budged in a while. I will experiment with larger values and test. (I hope it's just a matter of time before it finishes... it's been hung for quite a while now...)

to be exact about my command, and my positioning of the -limits:

Code: Select all

...-separate -append -define histogram:unique-colors=false -limit memory 2mb -limit map 4mb -write histogram:mpr:hgram +delete mpr:hgram...

In researching this, I did find something surprising. When I run:

Code: Select all

identify -list resource

I get

File Area Memory Map Disk Thread Throttle Time
--------------------------------------------------------------------------------
192 4.295GB 2GiB 4GiB unlimited 1 0 unlimited

And as I have 32GB of RAM, this seems suboptimal. I should think I could grant more access to RAM for IM than this generally speaking. I'll look into setting a policy of envar for that.

[GreenKoopa]

When you say 2.7MB or 5.9MB, you mean jpeg file size? Any image processing program will have to decompress the image before processing it. Just to hold an image:
memory = width X height X channels X depth
A program could decompress one jpeg block (8x8, or larger with chroma sub-sampling) at a time and process that. Some programs support this kind of stream processing, but this is the exception because it creates many special cases.

Why are you using mpr? I don't think it is necessary here. Can you use the 8-bit depth IM? It would cut memory usage in half. You could use it when generating histograms, and use the 16-bit version for making modifications. I don't think -depth cuts memory usage, just output file depth.

[fmw42]

convert input_file.jpg -separate -append -define histogram:unique-colors=false -write histogram:mpr:hgram +delete mpr:hgram -strip output_file.gif

Is there some reason you are using MPR other than to use -strip? You can write the output directly if you want without the -strip.

convert input_file.jpg -separate -append -define histogram:unique-colors=false -write histogram:output_file.gif

or perhaps


convert input_file.jpg -separate -append -strip -define histogram:unique-colors=false -write histogram:output_file.gif

[blue-j]

Oddly enough, not using MPR doesn't significantly change the memory used or the time the process takes! And it appears to take more CPU.

I think my best bet here is to try to get our of 16-bit. Is there a way to use 8-bit calculations within a 16-bit IM build?

Thanks!
J

[fmw42]

Not that I am aware. I think you must recompile as Q8

You can set the image depth to 8bit by using -depth 8.

[GreenKoopa]

-separate may triple memory use. If so, ImageMagick version 7 (still in alpha) may help this. Anyone know for sure?

[fmw42]

In IM 7, as I understand it, grayscale images will be truly 1 channel rather than 3 equal channels. So, yes, that should save memory.

See what is planned in IM 7 at http://magick.imagemagick.org/script/porting.php