convert image rotate excessive memory usage >2.5GByte

[ltaber@gmail.com]

The command:

Code: Select all

convert  -quality 95 -rotate 20 img_0095.jpg at2011-0095r.jpg

with the image avaliable here:
http://ltaber.blacksteel.com/at/at2011/ ... g_0095.jpg
was killed my my service provider for excessive memory usage. /usr/bin/time shows almost 3GBytes of memory usage:
Command terminated by signal 9

• Command being timed: "convert -quality 95 -rotate 20 img_0095.jpg at2011-0095r.jpg"
  User time (seconds): 8.70
  System time (seconds): 2.73
  Percent of CPU this job got: 99%
  Elapsed (wall clock) time (h:mm:ss or m:ss): 0:11.47
  Average shared text size (kbytes): 0
  Average unshared data size (kbytes): 0
  Average stack size (kbytes): 0
  Average total size (kbytes): 0
  Maximum resident set size (kbytes): 2892960

--truncated--
The version on the system:

• Version: ImageMagick 6.7.8-2 2012-07-07 Q16 http://www.imagemagick.org
  Copyright: Copyright (C) 1999-2012 ImageMagick Studio LLC
  Features: OpenMP

My system at home used Maximum resident set size (kbytes): 254136 running version

• Version: ImageMagick 6.7.7-10 2014-03-08 Q16 http://www.imagemagick.org
  Copyright: Copyright (C) 1999-2012 ImageMagick Studio LLC
  Features: OpenMP

[fmw42]

First your syntax is not proper for IM 6, where one should read the input before issuing processing commands. Nevertheless, it probably will work anyway.

On my Mac Mini OSX Snow Leopard running IM 6.8.9.8 Q16, the following command ran in about 14 s.

Code: Select all

time convert img_0095.jpg -rotate 20 -quality 95 tmp.jpg

real 0m13.831s
user 0m13.149s
sys 0m0.314s


I cannot comment on the disk space usage directly. But your RGB image, once read into memory would be 4000x3000 pixels at 24 bits per pixel = 288 Megabits = 288/8 = 36 Mbytes, if I have not made any mistakes. There could be 3 times that much for input, temp and output. So that would be 108 Mbytes. No where near your stated 3 Gbytes.

[snibgo]

Q16 needs 16 bits (2 bytes) per channel per pixel, and usually 4 channels, hence 8 bytes per pixel. 12 million pixels needs 96 million bytes. Multiply by 3 would be about 300 MB.

I agree that this is far less than 3 GB. Perhaps there was a bug in that old version (v6.7.8-2) or the jpeg library.

[fmw42]

Q16 needs 16 bits (2 bytes) per channel per pixel, and usually 4 channels, hence 8 bytes per pixel. 12 million pixels needs 96 million bytes.

I keep forgetting about Q16 and 16-bits per channel. Thanks for correcting me. But JPEG does not support alpha, so it in this case it is 3 channels (unless IM adds the alpha anyway in its internal format).

[snibgo]

(unless IM adds the alpha anyway in its internal format)

I believe it does. I'm no expert at IM internals (and would be happily corrected) but v6 seems to always allocate memory for 3 or 4 colour channels plus one alpha channel. CMYK needs 4 colour channels; everything else needs 3. Plus one alpha channel.

In this case, reading and writing jpeg, Q8 might be sufficient, and would need half the memory: 4 bytes per pixel.

[magick]

IMv6 uses 4 channels by default and adds an additional channel for CMYK and colormapped images. IMv7 only uses as many channels as required. RGB uses 3 channels, CMYK uses 4 channels, and grayscale uses one channel. Note, IMv7 is in Beta. The release date should be sometime next year.

[ltaber@gmail.com]

Hi,

I changed the order of the command line. This time is used 2.89 GBytes of RAM.

Code: Select all

/usr/bin/time -v convert img_0095.jpg -rotate 20 -quality 95 tmp.jpg

The process still got bounced by my service provider. Here is the output from /usr/bin/time -v

Code: Select all

Command being timed: "convert img_0095.jpg -rotate 20 -quality 95 tmp.jpg"
	User time (seconds): 6.71
	System time (seconds): 3.07
	Percent of CPU this job got: 94%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:10.39
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 2892832
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 2
	Minor (reclaiming a frame) page faults: 181209
	Voluntary context switches: 29
	Involuntary context switches: 51
	Swaps: 0
	File system inputs: 7416
	File system outputs: 2448
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Note: The built in bash time command provides less information than the /usr/bin/time command.

[magick]

Try this command:

• convert -limit memory 2MiB -limit map 4MiB img_0095.jpg -rotate 20 -quality 95 tmp.jpg

[ltaber@gmail.com]

The suggested command line:

Code: Select all

convert -limit memory 2MiB -limit map 4MiB img_0095.jpg -rotate 20 -quality 95 tmp.jpg

did worked. But took 13 minutes for the one image.

The calculations (2.9GBytes/12MPixel) show 240 Bytes per pixel being used (2.9GBytes/12MPixel). The comments about 3 or 4 bytes or even 6 or 8 aren't even close.

I still suspect there is just a bug in the code some place.

[snibgo]

I agree there must be a bug somewhere.

Is the command being executed from a terminal window, or something else?

For me (IM v6.8.9-5, Windows 8.1) the bare command ...

Code: Select all

convert img_0095.jpg -rotate 20 -quality 95 tmp.jpg

... takes 2 seconds.

With the limits, ...

Code: Select all

convert -limit memory 2MiB -limit map 4MiB img_0095.jpg -rotate 20 -quality 95 tmp.jpg

... it takes 107 seconds, and creates two temporary files: one of 96,000,000 bytes, the other of 160,426,720 bytes.

With these limits, what temporary files do you get?

[fmw42]

Older versions of IM used a three pass process to do rotation. The 3 intermediate images are larger than the original. This can add up in terms of memory requirements.

Newer versions of IM use -distort SRT type processing to do rotation in one 2D pass. So they use less memory. Unfortunately, I do not know when the transition occurred.

[ltaber@gmail.com]

I use a perl script to run convert, though it also uses excessive memory (200+ Bytes per pixel) when run from the command line.

How do I determine the size of the temporary files? I trust convert will delete them when it is finished.

Here is a snippet of the perl code (which I doubt makes any difference at all) some of which was added to help me locate the problem:

Code: Select all

        else
        {
        $rotate_file= "at".$year."-".$file_root."r.jpg";  # full size rotated image
        if( ! -e $rotate_file )
          {
          `convert  -quality 95 -rotate $angle $picture $rotate_file`;
          if( $? != 0 )
            {
            print STDERR "Return value: $?\n";
            print STDERR "convert  -quality 96 -rotate $angle $picture $rotate_file\n";     #TEST
            }
          $convert++;
          }

[ltaber@gmail.com]

I added the arguments:

Code: Select all

 -limit memory 500MiB -limit map 1GiB

to the convert command and it seems to work so far.

My service provider reports:

Code: Select all

 identify -list resource
File         Area       Memory          Map         Disk    Thread         Time
-------------------------------------------------------------------------------
 768     67.489GB    31.427GiB    62.854GiB    unlimited         8    unlimited

But then killed the process at around 3GByte usage.