resurrecting gigapixel image from 256px jpgs - "unable to open pixel cache" and "too many files open"

[magick]

You are not running the latest release of ImageMagick. Type

Code: Select all

magick --version

It should report version 7.0.8-8. If not, upgrade your release of ImageMagick and try your command again.

[inMotion]

You are not running the latest release of ImageMagick. Type

Code: Select all

magick --version

It should report version 7.0.8-8. If not, upgrade your release of ImageMagick and try your command again.

What is the difference between ImageMagick-7.0.8-8-Q16-x64-static.exe and ImageMagick-7.0.8-8-Q16-x64-dll.exe? Which one should I use?

[inMotion]

After uninstalling 6.9.x and installing mageMagick-7.0.8-8-Q16-x64-dll.exe" I get this error now:

D:\gigapan\Jerusalem_Gigapan\Python27>montage -define registry:temporary-path=/data/tmp -limit memory 8000mb -verbose -depth 8 -geometry 256x256+0+0 -mode concatenate -tile 127x *.jpg tiff64:stitchX.tiff
montage: unable to open image '*.jpg': Invalid argument @ error/blob.c/OpenBlob/3487.
montage: `tiff64:stitchX.tiff' @ error/montage.c/MontageImageCommand/1777.

[fmw42]

In Imagemagick 7, you must preface montage with magick montage. (convert is simply replaced with magick)

Though that is not likely the issue here.

Did you install all the delegates for jpg and tiff?

What do you get from

Code: Select all

magick -version

Does it list JPG and TIF under Delegates?

Have you checked and reset your policy.xml file?

Are your jpg files in your working directory

Have you tried your command with just a few specific input jpg files? Do you get the same error about not finding jpg files?

[inMotion]

I just realized I was in the wrong working directory! Thanks for pointing that out!

I didn't install jpg and tiff delegates. How do I do that?

It's processing now. Hopefully it won't error out.

What is the file size/pixel dimension limits for tiff? Could i be exceeding that?

This is magick -version:

C:\Windows\System32>magick -version
Version: ImageMagick 7.0.8-8 Q16 x64 2018-07-23 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2018 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Visual C++: 180040629
Features: Cipher DPC Modules OpenMP
Delegates (built-in): bzlib cairo flif freetype gslib heic jng jp2 jpeg lcms lqr
lzma openexr pangocairo png ps raw rsvg tiff webp xml zlib

[fmw42]

You do have jpeg and tiff listed for delegates. So that aspect is a non-issue. Sorry, I do not know the width limits of TIFF. But there may be width limits in your policy.xml file.

[inMotion]

this is the policy xml

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE policymap [
<!ELEMENT policymap (policy)+>
<!ELEMENT policy (#PCDATA)>
<!ATTLIST policy domain (delegate|coder|filter|path|resource) #IMPLIED>
<!ATTLIST policy name CDATA #IMPLIED>
<!ATTLIST policy rights CDATA #IMPLIED>
<!ATTLIST policy pattern CDATA #IMPLIED>
<!ATTLIST policy value CDATA #IMPLIED>
]>
<!--
  Configure ImageMagick policies.

  Domains include system, delegate, coder, filter, path, or resource.

  Rights include none, read, write, execute, and all.  Use | to combine them,
  for example: "read | write" to permit read from, or write to, a path.

  Use a glob expression as a pattern.

  Suppose we do not want users to process MPEG video images:

    <policy domain="delegate" rights="none" pattern="mpeg:decode" />

  Here we do not want users reading images from HTTP:

    <policy domain="coder" rights="none" pattern="HTTP" />

  Lets prevent users from executing any image filters:

    <policy domain="filter" rights="none" pattern="*" />

  The /repository file system is restricted to read only.  We use a glob
  expression to match all paths that start with /repository:

    <policy domain="path" rights="read" pattern="/repository/*" />

  Lets prevent users from executing any image filters:

    <policy domain="filter" rights="none" pattern="*" />

  Any large image is cached to disk rather than memory:

    <policy domain="resource" name="area" value="1GB"/>

  Define arguments for the memory, map, area, width, height and disk resources
  with SI prefixes (.e.g 100MB).  In addition, resource policies are maximums
  for each instance of ImageMagick (e.g. policy memory limit 1GB, -limit 2GB
  exceeds policy maximum so memory limit is 1GB).

  Rules are processed in order.  Here we want to restrict ImageMagick to only
  read or write a small subset of proven web-safe image types:

    <policy domain="delegate" rights="none" pattern="*" />
    <policy domain="coder" rights="none" pattern="*" />
    <policy domain="coder" rights="read|write" pattern="{GIF,JPEG,PNG,WEBP}" />
-->
<policymap>
  <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
  <!-- <policy domain="resource" name="memory" value="2GiB"/> -->
  <!-- <policy domain="resource" name="map" value="4GiB"/> -->
  <!-- <policy domain="resource" name="width" value="10MP"/> -->
  <!-- <policy domain="resource" name="height" value="10MP"/> -->
  <!-- <policy domain="resource" name="area" value="1GB"/> -->
  <!-- <policy domain="resource" name="disk" value="16EB"/> -->
  <!-- <policy domain="resource" name="file" value="768"/> -->
  <!-- <policy domain="resource" name="thread" value="4"/> -->
  <!-- <policy domain="resource" name="throttle" value="0"/> -->
  <!-- <policy domain="resource" name="time" value="3600"/> -->
  <!-- <policy domain="system" name="precision" value="6"/> -->
  <!-- <policy domain="coder" rights="none" pattern="MVG" /> -->
  <!-- <policy domain="delegate" rights="none" pattern="HTTPS" /> -->
  <!-- <policy domain="path" rights="none" pattern="@*" /> -->
  <policy domain="cache" name="shared-secret" value="passphrase" stealth="true"/>
</policymap>

10MP? 768? Those seem crazy low. this is going to be a several GIGAPIXEL image that is 204,000 pixels wide. Do I need to change these or else it will error out?

[fmw42]

All but these first 3 are commented out

<policy domain="delegate" rights="none" pattern="*" />
<policy domain="coder" rights="none" pattern="*" />
<policy domain="coder" rights="read|write" pattern="{GIF,JPEG,PNG,WEBP}" />


The default values show up for some of the others by

Code: Select all

convert -list resource

[magick]

It looks like you have everything you need. The Windows release of ImageMagick supports JPEG and TIFF. The tiff64: prefix to your output filename permits TIFF images that exceed 4GB. Replace -limit memory 8000mb with -limit area 0. We want to force the pixel cache to disk. This is very slow but we need to allow any memory to be available to the TIFF delegate library.

Now run your command. If it fails, post a description of the failure here so we can investigate further.

[inMotion]

So, right now after about 17 hours of processing, there is zero disk activity, 25gb of 32gb of ram is being used without fluctuations, and the CPU utilization for montage.exe is stuck at 17.5%. Could it have crashed? Or do I need to just wait longer?

[magick]

We run on a Linux system with modern SAS disks. The conversion took just under 2 hours. For Windows, and utilizing the pixel cache on disk, we are not surprised for it to be running this slow. Let it run to completion. If you need to do lots of these, consider switching to Linux and SD drives to increase performance.

[inMotion]

You did the conversion of my images?

[magick]

We don't have access to your images. We created 96000+ 640x480 images in the JPEG format and as mentioned, the conversion completed without complaint.

[inMotion]

Well, it's been running for about 3 days now, still no signs of progress

https://www.dropbox.com/sh/qwz43i5fy78n ... ELjPa?dl=0

There's access to the images and the script i've been using if anyone wants to take a crack at it.

[inMotion]

It's been going for 5 days now, the memory utilization is at 12gb now, down from 21gb. I'm not sure what's going on, as there is still zero disk activity.