Legacy ImageMagick Discussions Archive

Hi,

at the moment I have to rotate huge hires images (old town maps). One image has a resolution of 12000 x 9000 pixels. Rotation angle is 0.1
I use the convert command:
But this is awful slow! Loading and saving only (without rotate) needs 30 seconds!!! If I use rotate, ImageMagick needs over 15 Minutes (I killed the process) to rotate the image!! The CPU usage of convert is 1%...2% during process. Why? How I can speed this up?

For comparison:
Photoshop Elements: 8 seconds
GraphicConverter: 10 seconds

Thanks for hints

see http://www.imagemagick.org/Usage/files/#massive

also your syntax should be (though it won't help your problem here)

convert input.jpg -rotate 0.1 output.jpg

see http://www.imagemagick.org/Usage/basics/#why

I don't understand.... is this a memory problem?
The "massive" part of the manual is not very helpful. The only recommendation is "use the Q8 version", what?
About the -limit parameters. Which values I should check/increase?

Here you can test my problem with this single colored image:
http://www.lebostein.de/temp/flat.png (12000 x 9000, 550 kb)
No chance here....I can wait hours and nothing happens. I use the version ImageMagick 6.7.2-0 2011-09-10 Q16, installed with MacPorts on Mac OS X 10.6.8 (MacBook Pro).

You file is a compressed PNG of size 540 KBytes. But when read in and decompressed it goes to 12,000 x 9,000 x4(channels rgba) = 432 MBytes

It took minutes to even open in Mac PREVIEW and GraphicConverter for me.

This is a memory issue and depends upon your system and IM how much memory it allows you to use before paging to disk (as I understand it). However, I will defer to the IM developers to explain in more detail or correct me.

Also much of the time is spent reading and writing the output, especially the output as it is now writing 432 MBytes. You need to recompress the output before saving.

You can see how it progresses by adding -monitor to your command and time it by using time.


time convert -monitor flat.png -quality 25 flat_tst1.jpg

real 4m37.893s
user 0m35.959s
sys 0m15.866s

This is on my G4 Mac Mini PowerPC (one processor) OSX Tiger with only 512 MBytes of memory. So your whole image would barely fit and not allow any room for other code for processing. Thus it needs to page to disk.


If you watch, you will see the monitor show that it stops at times to write to disk as it read and decompresses the image. And then takes quite some time to write the image.

See http://www.imagemagick.org/script/archi ... .php#cache. Your image is large so its cached to disk rather than memory. Your task will complete-- it will just take along time. It converted for us on a large memory system in 11 seconds but took several minutes on our 4GB memory system.

fmw42 wrote:You file is a compressed PNG of size 540 KBytes. But when read in and decompressed it goes to 12,000 x 9,000 x4(channels rgba) = 432 MBytes It took minutes to even open in Mac PREVIEW and GraphicConverter for me.

I think this depends on the cleverness and the algorithms of the programm. Today I am here on a Window system (Dektop PC with 3 GHz). For example IrfanView needs 0.9 s (874 milliseconds) to load and show this image! (Rotating: 14 s, Saving 4 s)

Loading (decompressing) and Saving (compressing) ist NOT the problem. The time for reading and writing is OK. That works with convert. Problem ist the rotate parameter. If I include this parameter, I wait hours with Mac OS...

My question again: What is the problem with rotate and huge images? Can someone, who understands the workings (and the memory concept) of this program, give me a concrete hint?

Just now I have tested it on a 16 CPU unix cluster:



convert uses 16 CPUs and 4 GB memory! And the time is: 30 seconds. Twice as long as on a windows system with 1 cpu and IrfanView....

Is this my fault (wrong using of convert, wrong provision of resources) or is the rotating algorithm in comparison to other tools so slow?

The rotate operator is implemented using simple distorts followed by three image shears, a technique known as 'Rotate by Shear' (RBS) and first published in a research papers by Alan Paeth. You can also use the SRT distort (e.g. convert flat.png -distort SRT 0.5 flat.jpg).

Its possible some other tools use less sophisticated algorithms for image rotation. ImageMagick should give you the best results but it is slow due to making 3 passes over the image. And in your case the image pixels are cached to disk so its even slower. The trade-offs we made in designing ImageMagick are documented in the architecture pages we mentioned. We trade off some speed for quality of the results and the ability to support hundreds of image formats and the ability to compile and run on different architectures from a cluster of Xeon's to the iPhone.

Our opinion has always been-- if another package solves your problem better or faster than ImageMagick, use it. No one program can suitably solve all problems.

Try using the +distort SRT for the rotation. As cristy said, the -rotate needs at minimum 3 images in memory due to its use of 3 separate shears, while -distort only needs two.

If that still does not help then you can try to build a IM Q8 version (Bit quality which should half your memory use to hold the image data.

The other application probably uses a internal Q8 for its processing whcih is why it was quicker out of the box. IM defaults to Q16 because..

The higher the Quality Setting, used when compiling IM, the more precise the color values used to store the image in memory. That means that if in processing an image you generate a lot of very small, slight variations in color, then those variations will be preserved in the in-memory storage of ImageMagick, and not lost in later processing steps.

For a single operation (like rotate) a Q8 version of IM will work just fine, though if you add 'colorspace corrections' then you may be better off using Q16 anyway due to subtle value changes, that you want to preserve in Q16 (why else would you be doing colorspace corrections otherwise!)
http://www.imagemagick.org/Usage/resize ... colorspace

magick wrote:Our opinion has always been-- if another package solves your problem better or faster than ImageMagick, use it. No one program can suitably solve all problems.

I need a tool, that can be automated. Many people advised me ImageMagic. But this tool seems a little bit restricted (my problem with huge images).
Yesterday, I found a solution: Python Imaging Library (PIL). Fast enough and the best way for automatic processes...

We have a benchmark for ImageMagick vs. PIL and for the basics-- convert, rotate, transpose, resize, convolve, etc., the execution time is comparable. We reran the benchmark with a 12000x9000 pixel JPEG image and the execution time for both were within about 2%. Rotate was for 0, 90, 180, etc. True, odd sized rotations are dog slow for large images with ImageMagick.

magick wrote:We have a benchmark for ImageMagick vs. PIL and for the basics-- convert, rotate, transpose, resize, convolve, etc., the execution time is comparable. We reran the benchmark with a 12000x9000 pixel JPEG image and the execution time for both were within about 2%. Rotate was for 0, 90, 180, etc. True, odd sized rotations are dog slow for large images with ImageMagick.

Just curious, Magick, did you try timing your large test image for the odd size rotations with -distort SRT with and without -filter point and compare to -rotate?

Our benchmarks are not comprehensive. Just a few to see if ImageMagick was outside the performance norm as compared to PIL. Its not-- except for non-integral rotation which as mentioned uses the 3 pass Paeth shearing algorithm which has been proven to be slow when image pixels are cached to disk. Smaller images, such as the standard HDTV image format of 1920x1080 rotates in a reasonable time. The cause is that non-intergral rotation does not read from disk sequentially and random disk access is very slow as compared to memory. PIL holds as much of the image in memory (including swap) to take advantage of the fast swapping algorithms handled by the kernel. The problem with that is, as the images got large (24000x18000, 48000x32000, etc), PIL consumed all the available memory and swap, the system thrashed, became unresponsive, and eventually the PIL slowed to a crawl, the system halted and we had to reboot. ImageMagick handled the large images gracefully by caching the image pixels to disk rather than memory as expected. Other applications could run fine because it only needed 254MB of memory whereas for one case, PIL consumed 13GB of memory / swap. Each program has its advantages and disadvantages, each is useful depending on the requirements of the problem.

magick wrote:Our benchmarks are not comprehensive. Just a few to see if ImageMagick was outside the performance norm as compared to PIL. Its not-- except for non-integral rotation which as mentioned uses the 3 pass Paeth shearing algorithm which has been proven to be slow when image pixels are cached to disk. Smaller images, such as the standard HDTV image format of 1920x1080 rotates in a reasonable time. The cause is that non-intergral rotation does not read from disk sequentially and random disk access is very slow as compared to memory. PIL holds as much of the image in memory (including swap) to take advantage of the fast swapping algorithms handled by the kernel. The problem with that is, as the images got large (24000x18000, 48000x32000, etc), PIL consumed all the available memory and swap, the system thrashed, became unresponsive, and eventually the PIL slowed to a crawl and the system halted. ImageMagick handled the large images gracefully by caching the image pixels to disk rather than memory as expected. Other applications could run fine because it only needed 254MB of memory whereas for one case, PIL consumed 13GB of memory / swap. Each program has its advantages and disadvantages, each is useful depending on the requirements of the problem.

Thanks. I did understand all that. My question really pertained to whether using -distort SRT, which does not use the Paeth multipath approach, would be significantly faster so as to be helpful to the person who started the topic. Anthony had suggested earlier in this topic that -distort SRT as a one pass approach might be faster in this situation, and I would expect especially if EWA was turned off with -filter point.

Don't go to any trouble. It was just my own curiosity.

No it won't be faster because it still accesses pixels in a non-sequential manner and random access to disk is much slower than sequential access. Again if the image fits in memory, this random access is less problematic since memory access is 1000+ times faster than disk access.

magick wrote:No it won't be faster because it still accesses pixels in a non-sequential manner and random access to disk is much slower than sequential access. Again if the image fits in memory, this random access is less problematic since memory access is 1000+ times faster than disk access.

OK. Thanks for the clarification.