JPEG-2000 convert use all available memory

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
Post Reply
rnagy90
Posts: 3
Joined: 2016-06-23T00:59:36-07:00
Authentication code: 1151

JPEG-2000 convert use all available memory

Post by rnagy90 »

Hello!
I have a big problem, and i couldn't find solution. I triied about 10-20 options for this, but i still can't limit the memory usage for the jp2 conversion.
About 300-400 MB is free all the time, but the conversion use all the ramaining memory in my PC, and i am afraid of the failure in my server, because there is a web application.
I use the im4java lib to process the conversions in batch mode (10-12 at one time), and i thought the memory allocation error is lib's failure, but then i tested the normal cmd line conversion.

Example:
25/85 MB tif image with 4207x6522 8-bit sRGB.
Referring to the formula [(Quantum * height * width * channel)/8] => 16 * 4207 * 6522 * 3 = 157MB * 2 /*because of the pixel clone*/ ~ 320MB
And it's fine the total memory usage is about 340MB (i think because of the windows' task manager working with 1000 not with 1024) when i convert the file into a jpeg. But when i try the jp2 it's 1,2 GB.
Same is for the bigger, 8889x15120 image, but in this case, the memory usage always the available memory - 400MB, which is in my case ~5GB, intead of the ~1,5GB. It's like the memory allocation is growing exponential, and i don't know a multiplier "magic number".
I tried:
  • the image magick -limit options but it's not working.
    the JasPer limitations with the -define option. Example: ftp://ftp.tuwien.ac.at/graphics/ImageMa ... w/jp2.html
    also tried the quality with the normal -quality (but this isn't working at all, i think because of the delegating) and the -delegate jp2:quality=50. Result: the file is smaller, but the memory usage is still the maximum.
Anyone has an idea how can i calculate/limit the memory usage?
User avatar
fmw42
Posts: 25562
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: JPEG-2000 convert use all available memory

Post by fmw42 »

What is your IM version and platform? Is this on your computer or a server? What was your exact command line?
rnagy90
Posts: 3
Joined: 2016-06-23T00:59:36-07:00
Authentication code: 1151

Re: JPEG-2000 convert use all available memory

Post by rnagy90 »

fmw42 wrote:What is your IM version and platform? Is this on your computer or a server? What was your exact command line?
Testing on my computer
- OS: win 8.1 x64
- ImageMagick 6.9.3-8 Q16 x64
Resources:
8G RAM - about 4-5 GB available, depends on the number of webapps running.
8G swap (pagefile)
CPU: Intel(R) Core(TM) i5-3210M @ 2.50GHz

The default command after the im4java parse (http://im4java.sourceforge.net/):

Code: Select all

convert tiff:C:\workspace\test\image1 -compress JPEG2000 -format JP2 -quality 100.0 -type ColorSeparation jp2:C:\workspace\test\7
ColorSeparation is necessary, because the 2bit tiff's after the JP2 convert are black. Only visible after IrfanView's color correction.

I'm testing the conversions with this command and expanding with the limitations (see above). The server has more resources, so i can use parallel conversions. The implementation in the java lib is a simple loop, and when the big images are coming the server runs out of the memory, because of the high memory usage by the JasPer library.

Note: As i said it's not the java lib's fault, because the memory usage for 1 file in the loop is the same as the normal command line start with one file.
rnagy90
Posts: 3
Joined: 2016-06-23T00:59:36-07:00
Authentication code: 1151

Re: JPEG-2000 convert use all available memory

Post by rnagy90 »

I have just found that the jasper lib changed to openjpeg2 from the 6.8.8-2 version. But the memory usage by this lib is still ridiculous
User avatar
magick
Site Admin
Posts: 11064
Joined: 2003-05-31T11:32:55-07:00

Re: JPEG-2000 convert use all available memory

Post by magick »

We leverage the OpenJPEG2 delegate library to read and write JPEG-2000 images. The OpenJPEG2 API allocates the entire image in memory. If you spot opportunities in the API to only allocate a few rows of pixels at a time, let us know and we will investigate rewriting the JP2 coder to lower its memory requirements.
Post Reply