Processing large image

[sanderss]

Hi,

We're running into some trouble processing large images with convert.exe.

An example we can reproduce on several machines (WinXPPro, 2 or 4 GB RAM, 2 CPU's), is with a source image of 17244 by 12994 pixels and option "-resize x11811".

Convert.exe apparently reads the contents of the image, then writes about 750 MB, I suspect to a temp file(having a memory usage of about that much already), but then apparently falls idle. Nu CPU is used, no more or less memory is used. All I can see is 1 item on the disk queue (using PerfMon).

I've tried pushing the process priority to highest and even realtime. I've tried with "-limit memory 32" and "-limit memory 128", doesn't help.

I've tried loving care and tender and patience for two hours, but I can't let other systems depend on this type of effort and time-span.

Greetings
Stijn Sanders

[magick]

Not a bug. Your image is being cached to disk and your disk is dog slow. Add '-debug cache' to your command line to track progress. To speed up the process use the Q8 version of ImageMagick, add more memory to your system, purchase faster disks (e.g. 15K SCSI), run on a Linux-based system, or use some other program. We created a 17244x12994 image on our Redhat quad-core 3Ghz Xeon system with 16GB of memory and the resize took 46 seconds (version 6.4.2-2). With the -scale option it took 35 seconds and -sample took 15.

[sanderss]

Still, we've had this problem on our servers with brand new recent RAID systems, 4GB ram, multi-CPU's...

I've looked at what "-debug cache" gives, but it turns out the writing to disk is ok, then "cache.c/ReadCachePixels" and "cache.c/WriteCachePixels" gets called a number of times, I guess for a few megabytes. But then get called a few times every 10 seconds or so, with long times of no-CPU using inbetween...

This still looks like a dead-lock situation to me.

"-scale" may not handle upsampling the way we want, but doesn't seem to have the same issue. (-debug cache tells me -scale doesn't cache to disk, so -scale may have the same issue with even larger images)

[magick]

The debug cache has a back-off algorithm so as it progresses it reports less and less so the algorithm is running but reporting less. The resize option creates two internal image caches and one of them is accessed column-wise rather and row-wise and that is what is slowing down the process-- its reading / writing pixels in a random fashion from / to disk rather than sequentially. The scale option runs faster because it only creates one internal image cache and accesses the pixels sequentially. Another consideration is that ImageMagick is threaded on a modern Linux systems but runs on a single processor under Windows. Windows supports OpenMP but we have not successfully built a version with Visual C++ OpenMP support yet.

[johnsfine]

WinXPPro, 2 or 4 GB RAM,

If I understand correctly, WinXPPro is limited (by license, not technology) to not use ram which requires PAE, even when PAE is enabled. So with 4GB ram installed, you can only use a little more than 3GB.

add more memory to your system,

Won't help, because of that PAE problem and because of maybe an ImageMagick problem (see below).

purchase faster disks (e.g. 15K SCSI)

It would still be hopelessly slow.

run on a Linux-based system

Or use XP64 with the 64bit version of ImageMagick with lots of ram. I'm curious to see what this task looks like with a 32bit ImageMagick in 64bit Linux. The behavior of 32 bit ImageMagick in 64bit XP for this task seems to be seriously flawed, see below. Does only Windows 32 bit have this flaw?

quad-core 3Ghz Xeon system with 16GB of memory and the resize took 46 seconds

Was that Q8 or Q16?

My XP64, quad-core 2.3GHz Xeon with 16GB of (absurdly slow) ram took 99 seconds (Q16). I would expect only ram speed to matter in comparing your 46 seconds to my 99 seconds. This task is too ram intensive for openMP or CPU speed to matter much.

But something is very confusing about the cache behavior:

Code: Select all

\Tools\ImageMagick-6.4.2\32-Q8\convert -list resource
File       Area     Memory        Map       Disk
------------------------------------------------
1536   7.9948gb   5.9961gb    15.99gb        -1b

Code: Select all

\Tools\ImageMagick-6.4.2\64-Q16\convert -list resource
File       Area     Memory        Map       Disk
------------------------------------------------
1536   7.9948gb   5.9961gb    15.99gb        -1b

Why do I get exactly the same resource limits reported by the 32bit Q8 version and the 64bit Q16 version?

Code: Select all

2008-07-14T14:54:31-04:00 0:01 0.000u 6.4.2 Cache convert[2796]: cache.c/DestroyCacheInfo/2191/Cache
  destroy
2008-07-14T14:54:31-04:00 0:01 0.000u 6.4.2 Cache convert[2796]: cache.c/OpenCache/3526/Cache
  open bigger.jpg[0] (anonymous memory, 17244x12933 850.7mb)
2008-07-14T14:54:39-04:00 0:08 7.141u 6.4.2 Cache convert[2796]: cache.c/OpenCache/3526/Cache
  open bigger.jpg[0] (anonymous memory, 17244x11811 776.9mb)
2008-07-14T14:54:39-04:00 0:08 7.141u 6.4.2 Cache convert[2796]: cache.c/CloneMemoryToMemoryPixels/1957/Cache
  memory => memory
2008-07-14T14:55:04-04:00 0:33 32.344u 6.4.2 Cache convert[2796]: cache.c/ExtendCache/3439/Cache
  extend bigger.jpg[0] (c:/temp/magick-hgsGoomP[3], disk, 709.5mb)
2008-07-14T14:55:04-04:00 0:33 32.344u 6.4.2 Cache convert[2796]: cache.c/OpenCache/3639/Cache
  open bigger.jpg[0] (c:/temp/magick-hgsGoomP[3], disk, 15748x11811 709.5mb)

Why does the 32bit Q8 version go to disk so fast? (64bit Q16 doesn't go to disk at all).

In xp64 ordinary 32bit programs can use a full 4GB of ram. I'm not sure how you manage the anonymous mapping, but I expect that should let a 32bit program use far more than 4GB in xp64.

The -debug cache output has lots of detail on what is happening, but doesn't give any clue about why.

I tried a bunch of different -limit options and none made any difference at all.

Why can't the Windows 32-Q8 version use a larger amount of anonymous memory?

[magick]

Was that Q8 or Q16?

Q16.

Why do I get exactly the same resource limits reported by the 32bit Q8 version and the 64bit Q16 version?

Resource limits are set to default values except for the memory limit which is determined by the amount of memory available on your system.

Why does the 32bit Q8 version go to disk so fast? (64bit Q16 doesn't go to disk at all).

ImageMagick made a system request for anonymous memory mapping and then heap memory and apparently both failed. On your 64-bit OS, it apparently succeeded.

Why can't the Windows 32-Q8 version use a larger amount of anonymous memory?

It could be a Windows OS restriction. ImageMagick is asking for 700MB of anonymous memory map and then 700MB of heap memory and in both cases Windows refused to honor the request so ImageMagick caches the pixels to disk instead.

[johnsfine]

Resource limits are set to default values except for the memory limit which is determined by the amount of memory available on your system.

So why is Memory limit 5.9961gb ??

I'm not sure I understand what you mean by "memory limit". You seem to mean the heap within the process's address space. That heap cannot go above 4GB for a 32bit process. On my xp64 system, there is no limit near as low as 6GB on anything a 64bit process might want to do. I've run processes using far more than 6GB of heap. I have 16GB of physical ram (plus a big swap file).

So 5.9961gb is nonsense for anything determined from my system.

Why does the 32bit Q8 version go to disk so fast? (64bit Q16 doesn't go to disk at all).

ImageMagick made a system request for anonymous memory mapping and then heap memory and apparently both failed.

I don't have experience with anonymous memory mapping, so I can't be certain XP64 isn't doing something very wrong with a request from a 32bit program. But I have done a lot with heap memory in 32bit programs in xp64 and I'm sure that request would not have failed. In my own programs, I have had the heap grow to nearly the full 4GB. ImageMagick used only 1.6GB of anonymous memory and an insignificant amount of heap before failing the third allocation. There was plenty of virtual address space available and far more than plenty of physical ram. Even the OP of this thread, with 3GB of physical ram, probably had enough for the Q8 version to run this entire operation without going to disk.

The reasonable assumption is a bug in the way those requests are made or some other reason the cache code decides not to even try the map or heap requests.

I'm just not sure how to debug to find that problem (as mentioned in my other thread, I don't have the right Visual Studio to rebuild from source).

[magick]

See the Cache Storage and Resource Requirements subsection of http://www.imagemagick.org/script/archi ... .php#cache for a discussion of how resource limits are used. By default they are set to large values so images are processed in memory.

[sanderss]

Wait a minute, it's requesting 700MB in two ways and both fail? But monitoring reports the convert.exe process has allocated 800MB! (But still decides to use disk cache...)

[johnsfine]

it's requesting 700MB in two ways and both fail? But monitoring reports the convert.exe process has allocated 800MB!

Just 800MB ?? My copy allocated twice that much before going to disk.

Can you post the first few lines from running with -debug cache, as I did in this post:
viewtopic.php?p=37701#p37701

Notice in my post it has
open bigger.jpg[0] (anonymous memory, 17244x12933 850.7mb)
then
open bigger.jpg[0] (anonymous memory, 17244x11811 776.9mb)
then
extend bigger.jpg[0] (c:/temp/magick-hgsGoomP[3], disk, 709.5mb)
then
open bigger.jpg[0] (c:/temp/magick-hgsGoomP[3], disk, 15748x11811 709.5mb)

Those first two requests (850MB and 776MB) both succeeded in anonymous memory and I monitored the process to verify that ram remained allocated while the program became super slow with its third and/or fourth allocation on disk.

I have no clue what is going on with the third and fourth allocations in my output from -debug cache. Why does it need more than the first two? Where did the 15748 come from? What is "extend" (as opposed to "open"). Since my system has plenty of free virtual and physical ram (even after the first two allocations), why did the third allocation go to disk?

I'll try to look at the source code to understand some of the above. Maybe I'll find an answer for you. I find it hard to imagine any way in which the behavior I'm observing could be the designed behavior of the system, rather than a bug. I appreciate the difficulty of maintaining freeware programs and responding to problem reports from users (I do so myself for some MUCH simpler freeware), but the developer talking to us seems to be working from the assumption that there is no bug and therefor rejecting all information that indicates a bug. Maybe he's right. Hopefully, I'll get some clue from the source code. But I didn't get any clue, looking through the build process to find some way to recompile ImageMagick on my Windows system, so if I get any clue from the source code, I probably have no way to test changes.

Meanwhile, I ran the same test both 64bit Q16 and 32bit Q16 on my Linux system. 64bit took 89 seconds. 32bit took 48 Minutes. The -debug cache output from that 32bit run was surprisingly different (in format as well as content) from the Windows one and even less informative. So I have no clue whether anonymous memory allocation should be able to do more for a 32bit application (on a system with 8GB physical ram) or whether the resize task you selected is just too big for a 32bit Q16 copy of ImageMagick.

I'm also a bit confused about the 89 seconds. It took 99 seconds on a 2.3GHz Windows system with very slow ram. It should have been much faster on a 3.2GHz Linux system with fast ram.

[sanderss]

I also thought of having a look in the source-code myself, but don't have any ActivePerl handy for the moment, could you pass me a copy of a .sln suitable to debug-run convert.exe with?

[johnsfine]

could you

I hope you didn't mean me!

I haven't a clue how the build process works. I tried the configure program, thinking it could create a dsp and dsw file for my 1998 copy of Visual Studio, but I didn't get very far.

[magick]

I have no clue what is going on with the third and fourth allocations in my output from -debug cache. Why does it need more than the first two? Where did the 15748 come from? What is "extend" (as opposed to "open"). Since my system has plenty of free virtual and physical ram (even after the first two allocations), why did the third allocation go to disk?

As mentioned, resize first scales horizontally and then vertically so it requires two internal pixel caches for these scales. The 15748 comes from your resize argument of x11811. By default ImageMagick respects the aspect ratio of the image. You can force a fixed size by adding ! to your argument.

We would like to take future discussions to the users forum since so far no bugs have been revealed and this is the bugs forum. The pixel cache appears to be behaving as it was designed and the issue seems to be simply trying to understand how it works. Currently the only documentation on the pixel cache is here: http://www.imagemagick.org/script/architecture.php.

[sanderss]

All right then.
There is a bug with the pixel cache and "-resize".
On Windows with images large enough to force two pixel caches to be created on disk, they interlock, progress is unacceptably slow, and 1 item appears stuck on the PhysicalDisk queue.
As soon as I can find the time (and other resources) to dive into the code and pinpiont a location of the malbehaviour, I'll let you know.

[johnsfine]

"It's not a bug it's a feature."

OK, that definition can apply to anything. So I posted the work around at:
viewtopic.php?f=1&t=11683

Obviously, I still think several aspects of this perform enough worse than the documentation would lead you to expect that an ordinary user would call it a "bug".

The bottom line seems to be that you need a 64 bit version of Imagemagick to reasonably attack big problems. The work around I posted in that new thread increases by 50% the size of problem that is practical on a win32 system with 3+ GB of physical ram. The problem in this thread happens to be in that range (not practical to run without my workaround but quite practical with my work around).

But I don't agree with the "interlock" characterization. The pixel cache is simply inefficient enough when using disk files to be impractical to use for the operation discussed in this thread.