IM not using enough memory

[mikmach]

Hello,

This is probably the first

In my case IM isn't using enough memory. I have 16GB of ram but identify -resource list returns:
[/c/obrazki800] % convert -list resource
Resource limits:
Width: 214.7MP
Height: 214.7MP
Area: 8.4301GP
Memory: 2B [sic!]
Map: 7.8512GiB
Disk: unlimited
File: 1536
Thread: 12
Throttle: 0
Time: unlimited
[/c/obrazki800] % convert -version
Version: ImageMagick 6.9.0-4 Q16 x86 2015-01-10 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2015 ImageMagick Studio LLC
Features: DPC OpenMP
Delegates (built-in): bzlib cairo freetype jbig jng jp2 jpeg lcms lqr openexr pangocairo png ps rsvg tiff webp xml zlib

This is static version of IM distributed as zip archive. System is Windows 7 Pro SP1 64-bit

Incresing memory usage by -limit option doesn't speed up processing. Also in system resources only ca. 4GB of memory are used, 12GB are available to use.

Example of command:
mogrify -depth 8 -verbose -resize ">2500x>2500" -density 72 -quality 80 -format jpg -path c:\\obrazki800 -sampling-factor 1x1
-limit memory 8192MB *.jpg

Example of -verbose output:
97439_1(1).jpg JPEG 12255x13440 12255x13440+0+0 8-bit sRGB 18.53MB 2.777u 0:02.786
97439_1(1).jpg=>c:\obrazki800\97439_1(1).jpg JPEG 12255x13440=>2280x2500 2280x2500+0+0 8-bit sRGB 951KB 312.626u 2:39.345

I am processing JPGs in sizes above 12000px x 12000px and it is really, really, slow.

[mikmach]

OK. I have found where problem is. It is really weird combination.

First I was processing original tiff files of the same size (up to 20000x15000 pixels). On operations like conversion to jpg it reached all limits and created 0 size JPG files. It was fixed by adding temp file in policy.xml . But when creating temporary-path I didn't create some 'overflow' resource but enforced IM to use disk as cache buffer effectively disabling processing in memory which significantly increased process times.

Please, is any solution to this?

The only think I can think now is to create two directories im690-big and im690-regular with separate configuration and use them accordingly - but it is not sane solution in my book

[mikmach]

Hello,

Bumping thread after a year. Still have those problems - ImageMagick isn't using enough of my resources. Checked CPU monitor and IM uses regularly only 2 thread and 1 process with only occasional spike which looks like pushing things a bit while Photoshop when performing practically the same operation (resize of 18000x18000 tif down to 2500x2500 jpg with jpg parameters as close as possible) constantly uses 56 threads and 8 processes and goes through files like hot knife through a butter.

[snibgo]

Do you still get:

Memory: 2B [sic!]

If so, that may be the problem. I'm no expert on resource limits, but my laptop (12 GB memory) shows "Memory: 11.934GiB", which seems reasonable.

[magick]

We assume you've read http://www.imagemagick.org/script/architecture.php?

ImageMagick is single threaded when it reads and writes images and in addition it stores the pixels in an intermediate cache, sometimes in memory, sometimes on disk. This allows for a robust method to support all image types (ImageMagick supports over 200 image formats). TIFF and JPEG are scanline oriented so the fastest performance when converting from TIF to JPEG would be to read scanlines on demand and write them directly to disk as required-- something ImageMagick does not do. Most, but not all the command-line options are threaded, for example, -resize.

Any slowdown is likely because your images are large and likely the pixels are cached to disk. Add -debug cache to your command line to confirm. Running the 64-bit version on ImageMagick on a x86_64 host may permit the pixels to cache to memory. This could speed up the process considerably. If it still caches to disk, you could try the Q8 version of ImageMagick (8-bit instead of 16-bit). This version uses less memory perhaps ensuring the image pixel cache can stay in memory.

[mikmach]

Ugh. I wrote long and detailed response to snibgo , to support it with additional tests and now checked which exactly version I have

And updated (from 6.9.0-x to 6.9.3.0)

And now test pack is processed in 129s (avg between 3 tests) instead of 658s (avg between 10 tests today)

I hope it wasn't result of some freak behaviour on my system but I look hopefully into the future

BTW - I've already read architecture.php and some of the tests run with limited number of threads to 6 (physical cores) which improve processing by about 10%, without that old average would be over 700s.

Thank you all!