Page 1 of 1
Converting VERY large images
Posted: 2009-04-06T23:08:00-07:00
by matthewk
Hi all,
Have just about exceeded my technical limitations getting IM installed via DarwinPorts onto Mac OS X 10.5.6.
I wanted to get into command line conversion because I have a need to convert a large number of very big JPEG2000 images to TIFF. I succeeded in getting Jasper installed and getting IM to recognise it. I converted a smaller JP2 file to TIFF, worked superbly - thank you!
However, when I went for one of the bigger ones, the following happened:
Code: Select all
$ mogrify -format tiff /Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.jp2
mogrify(56710) malloc: *** mmap(size=1107787776) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
error: cannot decode code stream
mogrify: unable to decode image file `/Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.jp2' @ jp2.c/ReadJP2Image/389.
and I realise I have run into some kind of major limit.
The file is really, really large - 200 megapixels - and won't convert at full size using the shareware utility GraphicConverter (which I bought in order to do this job!).
Is there any useful workaround for this limitation? Is it a limit of the OS, as I suspect? The allocation size seems to be about a gigabyte and I have 4GB of RAM installed - and the OS X VM as well, of course.
I don't mind if it's slow, as long as it works.
Thanks for any help or pointers. As I say I am stretching the limits of my knowledge here so please be gentle!
Cheers,
Matthew.
Re: Converting VERY large images
Posted: 2009-04-07T05:53:20-07:00
by magick
This command should work for you:
- convert -limit memory 16mb -limit map 32mb /Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.jp2 /Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.tif
This command caches the pixels to disk so it will be slow but will work assuming you have plenty of temporary disk space in /tmp. If you have another partition with plenty of free space, use this command:
- convert -limit memory 16mb -limit map 32mb -define registry:temporary-path=/data/tmp /Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.jp2 /Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.tif
Use your own local path instead of /data/tmp.
Re: Converting VERY large images
Posted: 2009-04-07T16:49:22-07:00
by matthewk
Thank you so much. Superb. I hope I can find a way to contribute to the community in return.
If my machine is only baulking at the 1GB range, could I use limits like memory 320mb, map 640mb?
And one last question - I had been using mogrify so I could batch large numbers and leave it overnight (e.g. mogrify -format tiff *.jp2) but I gather the limit options are specific to convert. Would I need to use convert inside a shell script for the equivalent functionality?
I realise the question is probably extremely basic - my previous unix experience was minimal and happened 20 years ago ... !
Re: Converting VERY large images
Posted: 2009-04-07T17:28:04-07:00
by magick
You can set the limits to whatever makes sense in your environment.
The mogrify program is probably best for your stated requirements. It accepts the limits option as well.
Re: Converting VERY large images
Posted: 2009-04-07T18:17:12-07:00
by matthewk
Thank you again. Much appreciated. Hope others can benefit from your advice as well.
Re: Converting VERY large images
Posted: 2009-04-07T19:11:39-07:00
by matthewk
Failed, sadly.
Code: Select all
matthewk$ mogrify -limit memory 32mb -limit map 64mb -format tiff [path]SectionA.jp2
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mogrify(88579) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
jpc_dec_decodepkts failed
error: cannot decode code stream
mogrify: unable to decode image file `[path]SectionA.jp2' @ jp2.c/ReadJP2Image/389.
I am thinking perhaps the failed allocation is happening in the JasPer decode, so that mogrify can't limit the memory it asks for.
Re: Converting VERY large images
Posted: 2009-04-07T19:20:35-07:00
by magick
You could be right. We have little control over the Jasper delegate library and it may be asking for more memory than you computer will honor. In the mean-time we can limit how much memory ImageMagick requests, leaving more for Jasper. Try this command:
- convert -limit area 1mb /Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.jp2 /Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.tif
Re: Converting VERY large images
Posted: 2009-04-07T19:22:01-07:00
by matthewk
Actually now that I read the error reports more carefully, it seems like mogrify is failing to allocate 16MB of space, which is nuts. Everything else is running fine; Activity Monitor reports 2.77GB free memory, 41.94GB VM size, 98GB free on the startup volume.
I'll try the limit area thing you mention. Sorry to take up your time.
Update: that failed as well:
Code: Select all
*** mmap(size=1107787776) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
error: cannot decode code stream
convert: unable to decode image file `/Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.jp2' @ jp2.c/ReadJP2Image/389.
convert: missing an image filename `/Users/matthewk/Desktop/Nissl_Sagittal_91_SectionA.tif' @ convert.c/ConvertImageCommand/2775.
Re: Converting VERY large images
Posted: 2009-04-07T21:13:54-07:00
by matthewk
Just to downgrade the urgency - I have managed to used a plugin from fnord to open the images in Photoshop CS3, which is happy to save TIFFs at that size (albeit with a large graphical front end time penalty!). So I guess I will be automating and batching instead of flexing my CLI skills.