Converting VERY large images

[matthewk]

Hi all,
Have just about exceeded my technical limitations getting IM installed via DarwinPorts onto Mac OS X 10.5.6.
I wanted to get into command line conversion because I have a need to convert a large number of very big JPEG2000 images to TIFF. I succeeded in getting Jasper installed and getting IM to recognise it. I converted a smaller JP2 file to TIFF, worked superbly - thank you!
However, when I went for one of the bigger ones, the following happened:

Code: Select all

$ mogrify -format tiff /Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.jp2
mogrify(56710) malloc: *** mmap(size=1107787776) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
error: cannot decode code stream
mogrify: unable to decode image file `/Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.jp2' @ jp2.c/ReadJP2Image/389.

and I realise I have run into some kind of major limit.
The file is really, really large - 200 megapixels - and won't convert at full size using the shareware utility GraphicConverter (which I bought in order to do this job!).

Is there any useful workaround for this limitation? Is it a limit of the OS, as I suspect? The allocation size seems to be about a gigabyte and I have 4GB of RAM installed - and the OS X VM as well, of course.
I don't mind if it's slow, as long as it works.
Thanks for any help or pointers. As I say I am stretching the limits of my knowledge here so please be gentle!
Cheers,
Matthew.

[magick]

This command should work for you:

• convert -limit memory 16mb -limit map 32mb /Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.jp2 /Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.tif

This command caches the pixels to disk so it will be slow but will work assuming you have plenty of temporary disk space in /tmp. If you have another partition with plenty of free space, use this command:

• convert -limit memory 16mb -limit map 32mb -define registry:temporary-path=/data/tmp /Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.jp2 /Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.tif

Use your own local path instead of /data/tmp.

[matthewk]

Thank you so much. Superb. I hope I can find a way to contribute to the community in return.

If my machine is only baulking at the 1GB range, could I use limits like memory 320mb, map 640mb?

And one last question - I had been using mogrify so I could batch large numbers and leave it overnight (e.g. mogrify -format tiff *.jp2) but I gather the limit options are specific to convert. Would I need to use convert inside a shell script for the equivalent functionality?

I realise the question is probably extremely basic - my previous unix experience was minimal and happened 20 years ago ... !

[magick]

You can set the limits to whatever makes sense in your environment.

The mogrify program is probably best for your stated requirements. It accepts the limits option as well.

[matthewk]

Thank you again. Much appreciated. Hope others can benefit from your advice as well.

[matthewk]

Failed, sadly.

Code: Select all

matthewk$ mogrify -limit memory 32mb -limit map 64mb -format tiff [path]SectionA.jp2 
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
jpc_dec_decodepkts failed
error: cannot decode code stream
mogrify: unable to decode image file `[path]SectionA.jp2' @ jp2.c/ReadJP2Image/389.

I am thinking perhaps the failed allocation is happening in the JasPer decode, so that mogrify can't limit the memory it asks for.

[magick]

You could be right. We have little control over the Jasper delegate library and it may be asking for more memory than you computer will honor. In the mean-time we can limit how much memory ImageMagick requests, leaving more for Jasper. Try this command:

• convert -limit area 1mb /Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.jp2 /Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.tif

[matthewk]

Actually now that I read the error reports more carefully, it seems like mogrify is failing to allocate 16MB of space, which is nuts. Everything else is running fine; Activity Monitor reports 2.77GB free memory, 41.94GB VM size, 98GB free on the startup volume.

I'll try the limit area thing you mention. Sorry to take up your time.

Update: that failed as well:

Code: Select all

*** mmap(size=1107787776) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
error: cannot decode code stream
convert: unable to decode image file `/Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.jp2' @ jp2.c/ReadJP2Image/389.
convert: missing an image filename `/Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.tif' @ convert.c/ConvertImageCommand/2775.

[matthewk]

Just to downgrade the urgency - I have managed to used a plugin from fnord to open the images in Photoshop CS3, which is happy to save TIFFs at that size (albeit with a large graphical front end time penalty!). So I guess I will be automating and batching instead of flexing my CLI skills.