resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

[inMotion]

I lost tons of work due a hard drive crash - among them several gigapixel images created for a client that I need back. They are stored on gigapan.com but there is no way to download them. I found this script, which lead me to imagemagick. I have downloaded the 256x256 images that make up the gigapixel, but I am having problems merging them with IM. This is what is going on:

D:\gigapan\Jerusalem_Gigapan\Python27\149616>montage -depth 8 -geometry 256x256+ 0+0 -mode concatenate -tile 104x *.jpg stitchX.tiff
montage.exe: UnableToOpenPixelCache `stitchX.tiff': Too many open files @ error/
cache.c/OpenPixelCache/4030.

The total image is 6.33 Gigapixels that is currently in 96,729 256x256px .jpgs. I have 32gb RAM and 120gb HDD space. I'm using ImageMagick-6.9.8-4-Q16-x64-static. Am I doing anything wrong?

[fmw42]

You may be running out of resources or your JPG file may have limits on the width.

What do you get from

convert -list resource

If it does not show large enough resources to match your RAM and disk, then you should increase your limits by editing the policy.xml file.

I am not sure if JPG has a width limit.

Will your total file size for both the input images and the final output image fit into your RAM? If not, then IM will try to use disk space. So do you have enough free disk space?

[magick]

Your command works without complaint for us. We're using ImageMagick 6.8.10-8, the current release. We saved all the image file names to a file "filelist" to reduce the command line arguments size, then ran this command:

Code: Select all

$ montage -define registry:temporary-path=/data/tmp -limit memory 16mb -verbose -depth 8 \
  -geometry 256x256+0+0 -mode concatenate -tile 104x @filelist tiff64:stitchX.tiff

[inMotion]

Hmm. would limiting it to just 16mb exponentially increase the time it takes to process? it already takes hours. why not like 20GB?

[magick]

Sure, use 20GB. We wanted to post a command that is known to work as a baseline.

[inMotion]

D:\gigapan\Jerusalem_Gigapan\Python27\143766>montage -define registry:temporary-
path=data/tmp limit memory 8000mb -verbose -depth 8 -geometry 256x256+0+0 -mode
concatenate -tile 107x @filelist tiff64:stitchX.tiff
montage.exe: unable to open image `limit': No such file or directory @ error/blo
b.c/OpenBlob/2701.
montage.exe: no decode delegate for this image format `' @ error/constitute.c/Re
adImage/504.
montage.exe: unable to open image `memory': No such file or directory @ error/bl
ob.c/OpenBlob/2701.
montage.exe: no decode delegate for this image format `' @ error/constitute.c/Re
adImage/504.
montage.exe: unable to open image `8000mb': No such file or directory @ error/bl
ob.c/OpenBlob/2701.
montage.exe: no decode delegate for this image format `' @ error/constitute.c/Re
adImage/504.
montage.exe: unable to open image `@filelist': No such file or directory @ error
/blob.c/OpenBlob/2701.
montage.exe: no decode delegate for this image format `' @ error/constitute.c/Re
adImage/504.
montage.exe: missing an image filename `tiff64:stitchX.tiff' @ error/montage.c/M
ontageImageCommand/1793.

Am feel like I'm making an obvious and simple mistake....

[inMotion]

Your command works without complaint for us. We're using ImageMagick 6.8.10-9, the current release. We saved all the image file names to a file "filelist" to reduce the command line arguments size, then ran this command:

Code: Select all

$ montage -define registry:temporary-path=/data/tmp -limit memory 16mb -verbose -depth 8 \
  -geometry 256x256+0+0 -mode concatenate -tile 104x @filelist tiff64:stitchX.tiff

Isn't 7.0.8-8-Q16-x64 the current/latest release?

[fmw42]

6.8.10.9 is the current beta. 6.8.10.8 and 7.0.8.8 are the current releases.

[snibgo]

...montage -define registry:temporary-path=data/tmp limit memory ...

"-limit" is the option you want, with a "-" at the front. As you have no "-", IM thinks you want to read an image called "limit", and it can't find it.

[inMotion]

Well, I installed ImageMagick-6.9.9-37-Q16-HDRI-x64-dll

Tried this:
D:\gigapan\Jerusalem_Gigapan\Python27\143766>montage -define registry:temporary-path=data/tmp -limit memory 8000mb -verbose -depth 8 -geometry 256x256+0+0 -mode concatenate -tile 127x @filelist tiff64:stitchX.tiff

got this:
montage: unable to open image `@filelist': No such file or directory @ error/blob.c/OpenBlob/2761.montage: no decode delegate for this image format `' @ error/constitute.c/ReadImage/504.montage: missing an image filename `tiff64:stitchX.tiff' @ error/montage.c/MontageImageCommand/1795.

[fmw42]

Try enclosing @filelist in double quotes as "@filelist.txt" with a suffix on the actual file list.

[inMotion]

That gives me this:

montage: unable to open image `@filelist.txt': No such file or directory @ error/blob.c/OpenBlob/2761.montage: missing an image filename `tiff64:stitchX.tiff' @ error/montage.c/MontageImageCommand/1795.

Thank you so much for your help!

[fmw42]

Your actual file needs to be named filelist.txt. Perhaps you should post a link to your txt file and we can test with it.

Also try putting the @filelist.txt right after montage.

If on IM 7, then use magick montage and not montage.

Another issue might be that your resources have been restricted to prevent the use of @. See your policy.xml file. https://www.imagemagick.org/script/resources.php

Is this line commented out or not?

<!-- <policy domain="path" rights="none" pattern="@*" /> -->

If uncommented, then you are restricted from using it.

If that is the case or even not, then try

Code: Select all

cat path\to\filelist.txt | montage - -define registry:temporary-path=data/tmp -limit memory 8000mb -verbose -depth 8 -geometry 256x256+0+0 -mode concatenate -tile 127x tiff64:stitchX.tiff

[inMotion]

Well...i think we're getting somewhere. I don't know what you mean by filelist.txt, so i used *.jpg and it seemed to start to work. I had 150GB free when I started this. Do I just need to free up space or is there another issue with the cache I should know about?

This is what I used:

Code: Select all

montage -define registry:temporary-path=/data/tmp -limit memory 8000mb -verbose -depth 8 -geometry 256x256+0+0 -mode concatenate -tile 127x *.jpg tiff64:stitchX.tiff

And after many hours, I got this:

tiff64:stitchX.tiff=>stitchX.tiff JPEG 32512x204032 8-bit sRGB 76.453u 1:18.385
montage: UnableToExtendCache `tiff64:stitchX.tiff': No space left on device @ er
ror/cache.c/OpenPixelCache/4101.

[inMotion]

After 8 hours of processing, now i'm getting this error: https://www.dropbox.com/s/ix69yh5qq3aoh ... 1.jpg?dl=0

A ton of these:
montage: unable to open file `magick-16188shKu7TTUf-rE': Too many open files @ error/cache.c/WritePixelCachePixels/5942.
then this:
montage: too many exceptions (exception processing is suspended) @ error/exception.c/ThrowException/969.