Convert fails without error when cropping very big images even when it has plenty of memory

[malegro]

Hi everyone,

I'm using IM's convert to crop really big images (~13Gb). It works most of the time. However, it fails on some images and exits without any error. It only saves the first row of tiles and quits and I can't figure out what's wrong.
The image in question is TIFF64 85459x74581 85459x74581+0+0 8-bit TrueColor sRGB 13.408GB.
I'm using the following command line: convert -debug all -crop 17x20@ +repage +adjoin 000000_-85480_000000.tif tile_%04d.tif.
IM is running on Linux in a cluster. I'm setting the memory limit to 64Gb. When I check the debug logs, I don't see any error and it really only tries to save only the first row of images. The full log file is here: https://www.dropbox.com/s/udeedfoas36lh ... g.txt?dl=0

Would anybody have a advice on how to fix this?

Thanks!

[snibgo]

Please don't multi-post. I have removed your other posts.

Please always state the version of IM you are using. From the log, it seems to be v6.8.9.

Your command is in the wrong order. It should read the input, perform operations, then write the outputs.

I'm setting the memory limit to 64Gb.

How do you do this? With policy.xml? Please show the contents of that file.

[malegro]

Sorry about that! I'm using IM 6.8.9-9 Q16 x86_64 2017-07-31. It's running on Ubuntu 16.04.
Should my command be: convert -debug all 000000_-85480_000000.tif -crop 17x20@ +repage +adjoin tile_%04d.tif?
For memory, I'm setting the environment variables MAGICK_MEMORY_LIMIT to 64Gb and MAGICK_TMPDIR to a directory I know I have a lot of free space. After checking this site http://www.imagemagick.org/script/resou ... nvironment I had the impression it would be enough.

My policy.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE policymap [
<!ELEMENT policymap (policy)+>
<!ELEMENT policy (#PCDATA)>
<!ATTLIST policy domain (delegate|coder|filter|path|resource) #IMPLIED>
<!ATTLIST policy name CDATA #IMPLIED>
<!ATTLIST policy rights CDATA #IMPLIED>
<!ATTLIST policy pattern CDATA #IMPLIED>
<!ATTLIST policy value CDATA #IMPLIED>
]>
<!--
Configure ImageMagick policies.

Domains include system, delegate, coder, filter, path, or resource.

Rights include none, read, write, and execute. Use | to combine them,
for example: "read | write" to permit read from, or write to, a path.

Use a glob expression as a pattern.

Suppose we do not want users to process MPEG video images:

<policy domain="delegate" rights="none" pattern="mpeg:decode" />

Here we do not want users reading images from HTTP:

<policy domain="coder" rights="none" pattern="HTTP" />

Lets prevent users from executing any image filters:

<policy domain="filter" rights="none" pattern="*" />

The /repository file system is restricted to read only. We use a glob
expression to match all paths that start with /repository:

<policy domain="path" rights="read" pattern="/repository/*" />

Any large image is cached to disk rather than memory:

<policy domain="resource" name="area" value="1GB"/>

Define arguments for the memory, map, area, and disk resources with
SI prefixes (.e.g 100MB). In addition, resource policies are maximums for
each instance of ImageMagick (e.g. policy memory limit 1GB, -limit 2GB
exceeds policy maximum so memory limit is 1GB).
-->
<policymap>
<!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
<!-- <policy domain="resource" name="memory" value="2GiB"/> -->
<!-- <policy domain="resource" name="map" value="4GiB"/> -->
<!-- <policy domain="resource" name="area" value="1GB"/> -->
<!-- <policy domain="resource" name="disk" value="16EB"/> -->
<!-- <policy domain="resource" name="file" value="768"/> -->
<!-- <policy domain="resource" name="thread" value="4"/> -->
<!-- <policy domain="resource" name="throttle" value="0"/> -->
<!-- <policy domain="resource" name="time" value="3600"/> -->
<!-- <policy domain="system" name="precision" value="6"/> -->
<policy domain="cache" name="shared-secret" value="passphrase"/>
<policy domain="coder" rights="none" pattern="EPHEMERAL" />
<policy domain="coder" rights="none" pattern="URL" />
<policy domain="coder" rights="none" pattern="HTTPS" />
<policy domain="coder" rights="none" pattern="MVG" />
<policy domain="coder" rights="none" pattern="MSL" />
<policy domain="coder" rights="none" pattern="TEXT" />
<policy domain="coder" rights="none" pattern="SHOW" />
<policy domain="coder" rights="none" pattern="WIN" />
<policy domain="coder" rights="none" pattern="PLT" />
<policy domain="path" rights="none" pattern="@*" />
</policymap>

Thanks!

[snibgo]

Should my command be: convert -debug all 000000_-85480_000000.tif -crop 17x20@ +repage +adjoin tile_%04d.tif?

Yes.

I don't know why IM is failing with no error message. It shouldn't do that, unless it is killed externally of course. Does your sysadmin have something in place that kills "rogue" processes?

If you want to limit the memory usage, I think you should also set MAGICK_MAP_LIMIT, probably to the same value as MAGICK_MEMORY_LIMIT.

These define an upper limit on how much memory will be used. If you don't define limits, IM may use all your memory for pixel caches which leaves no room for other structures, and this may be the problem you are hitting.

If you have less than 64 GB of real memory, setting the upper limit to 64 GB will do nothing. How much memory do you have?

At 8 bytes per pixel, your image needs about 51 GB, plus about the same again for the crops. I suggest you set MAGICK_MAP_LIMIT and MAGICK_MEMORY_LIMIT to low values, eg 1 GB. This will force the pixel caches to disk, which is much slower but less likely to swamp your memory.

Another additional possibility is to first convert the input to MPC, then use that as the input. This stores input data on disk (in a *.cache file), so doesn't eat memory for the input. But you still need memory for the outputs.

Code: Select all

convert 000000_-85480_000000.tif tmp.mpc
convert -debug all tmp.mpc -crop 17x20@ +repage +adjoin tile_%04d.tif
rm tmp.mpc
rm tmp.cache