Persistent Cache doesn't work

[tmpfs]

Apology for no reproducer since I'm in a hurry.

I'm trying to use this script http://www.fmwconcepts.com/imagemagick/notch/ working with a fairly large image using mpc files (3304 x 4083).

Version: ImageMagick 6.9.7-4 Q16 x86_64 20170114

The operation failed with "unable to extend cache". The following is a debug trace.

Code: Select all

2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Configure convert-im6.q16[26344]: utility.c/ExpandFilenames/940/Configure
  Command line: convert {-debug} {All} {-verbose} {-monitor} {./notch_2_26311.mpc[0]} {-ping} {-format} {%w} {info:}
[...]
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Module convert-im6.q16[26344]: module.c/OpenModule/1288/Module
  Searching for module "MPC" using filename "mpc.la"
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Module convert-im6.q16[26344]: module.c/GetMagickModulePath/558/Module
  Searching for coder module file "mpc.la" ...
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Module convert-im6.q16[26344]: module.c/OpenModule/1297/Module
  Opening module at path "/usr/lib/x86_64-linux-gnu/ImageMagick-6.9.7//modules-Q16/coders/mpc.la"
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Module convert-im6.q16[26344]: module.c/OpenModule/1324/Module
  Method "RegisterMPCImage" in module "MPC" at address 0x7f980b6287a0
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Module convert-im6.q16[26344]: module.c/OpenModule/1338/Module
  Method "UnregisterMPCImage" in module "MPC" at address 0x7f980b628840
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Policy convert-im6.q16[26344]: policy.c/IsRightsAuthorized/580/Policy
  Domain: Path; rights=Read; pattern="./notch_2_26311.mpc" ...
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Blob convert-im6.q16[26344]: blob.c/OpenBlob/2593/Blob
    read 3 magic header bytes
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Cache convert-im6.q16[26344]: cache.c/DestroyPixelCache/1123/Cache
  destroy 
[...]
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Policy convert-im6.q16[26344]: policy.c/IsRightsAuthorized/580/Policy
  Domain: Coder; rights=Read; pattern="MPC" ...
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Policy convert-im6.q16[26344]: policy.c/IsRightsAuthorized/580/Policy
  Domain: Path; rights=Read; pattern="./notch_2_26311.mpc" ...
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Blob convert-im6.q16[26344]: blob.c/OpenBlob/2593/Blob
    read 3 magic header bytes
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Resource convert-im6.q16[26344]: resource.c/AcquireMagickResource/318/Resource
  Width: 4.08KB/4.08KB/16KB
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Resource convert-im6.q16[26344]: resource.c/AcquireMagickResource/318/Resource
  Height: 4.08KB/4.08KB/16KB
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Cache convert-im6.q16[26344]: cache.c/PersistPixelCache/4122/Cache
  attach persistent cache
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Resource convert-im6.q16[26344]: resource.c/AcquireMagickResource/318/Resource
  Width: 4.08KB/4.08KB/16KB
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Resource convert-im6.q16[26344]: resource.c/AcquireMagickResource/318/Resource
  Height: 4.08KB/4.08KB/16KB
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Resource convert-im6.q16[26344]: resource.c/AcquireMagickResource/318/Resource
  Area: 133.4MB/133.4MB/128MB
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Resource convert-im6.q16[26344]: resource.c/AcquireMagickResource/318/Resource
  Disk: 133.4MB/127.3MiB/1GiB
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Resource convert-im6.q16[26344]: resource.c/AcquireMagickResource/318/Resource
  File: 1B/1B/768B
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Cache convert-im6.q16[26344]: cache.c/SetPixelCacheExtent/3737/Cache
  extend ./notch_2_26311.mpc[0] (./notch_2_26311.cache[4], disk, 133.4MB)
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Resource convert-im6.q16[26344]: resource.c/AcquireMagickResource/318/Resource
  Map: 133.4MB/127.3MiB/512MiB
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Resource convert-im6.q16[26344]: resource.c/RelinquishMagickResource/1013/Resource
  File: 1B/0B/768B
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Cache convert-im6.q16[26344]: cache.c/OpenPixelCache/4022/Cache
  open ./notch_2_26311.mpc[0] (./notch_2_26311.cache[-1], Map, 4084x4084 127.3MiB)
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Resource convert-im6.q16[26344]: resource.c/AcquireMagickResource/318/Resource
  Width: 4.08KB/4.08KB/16KB
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Resource convert-im6.q16[26344]: resource.c/AcquireMagickResource/318/Resource
  Height: 4.08KB/4.08KB/16KB
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Cache convert-im6.q16[26344]: cache.c/PersistPixelCache/4122/Cache
  attach persistent cache
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Resource convert-im6.q16[26344]: resource.c/AcquireMagickResource/318/Resource
  Width: 4.08KB/4.08KB/16KB
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Resource convert-im6.q16[26344]: resource.c/AcquireMagickResource/318/Resource
  Height: 4.08KB/4.08KB/16KB
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Resource convert-im6.q16[26344]: resource.c/AcquireMagickResource/318/Resource
  Area: 133.4MB/133.4MB/128MB
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Resource convert-im6.q16[26344]: resource.c/AcquireMagickResource/318/Resource
  Disk: 133.4MB/254.5MiB/1GiB
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Resource convert-im6.q16[26344]: resource.c/AcquireMagickResource/318/Resource
  File: 1B/1B/768B
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Cache convert-im6.q16[26344]: cache.c/SetPixelCacheExtent/3737/Cache
  extend ./notch_2_26311.mpc[1] (./notch_2_26311.cache[4], disk, 266.9MB)
[...]
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Exception convert-im6.q16[26344]: cache.c/OpenPixelCache/3966/Exception
  unable to extend cache `./notch_2_26311.mpc': Bad file descriptor
2017-03-11T23:51:22-05:00 0:00.000 0.000u 6.9.7 Exception convert-im6.q16[26344]: mpc.c/ReadMPCImage/944/Exception
  unable to persist pixel cache `./notch_2_26311.mpc'

The problem here seems to be some logic error in the code. Here is the corresponding strace:

Code: Select all

25778 open("./notch_2_25746.cache", O_RDONLY) = 4
[...]
25778 lseek(4, 0, SEEK_END)             = 133432448
25778 pwrite64(4, "\0", 1, 266867839)   = -1 EBADF (Bad file descriptor)

The cache file is opened as read-only and called with pwrite(), which is obviously not ok and this isn't even a problem with memory.

[fmw42]

What is your IM version and platform? What was your exact command line? What is your input image format? How much RAM do you have?

Are you trying to provide MPC as input? Was the MPC file created with the same version of IM that you are using to run the script?

If your file is not MPC, can you provide that image for me to use to test. Unfortunately, your debug notes are over my head and I would have to be able to reproduce the error to get help on this.

Searching for module "MPC" using filename "mpc.la"

I do not understand a filename such as mpc.la! I have no idea where that is coming from. Internal to my script, I convert input images to tmp.mpc (and tmp.cache).

Try commenting out line 481

Code: Select all

trap "rm -ff $tmp1A $tmp1B $tmp2A $tmp2B $tmp3A $tmp3B; exit 1" ERR

and see if that works.

[snibgo]

In cache.c, SetPixelCacheExtent() will return MagickFalse if (among other conditions) count != 1.

But count may be set to the value returned by WritePixelCacheRegion(), which can call pwrite(), and it seems to me that other values >=1 should also indicate success.

[magick]

WritePixelCacheRegion(), in SetPixelCacheExtent(), writes one character, a NULL. As such, a return value for anything other than a successful write of one character is considered a fail. The purpose is to pre-allocate the pixel cache on disk and have the OS return an error if its unlikely the pixel cache can properly populate on the disk partition. You can force full allocation with the MAGICK_SYNCHRONIZE environment variable to ensure an early exception if the disk partition exceeds the allocation of free space on a disk partition.

[snibgo]

Ah, yes, for this case the length of the buffer written by WritePixelCacheRegion is one, so that is the only value it can return that means success. Thanks.

[tmpfs]

Here is a shortened reproducer

Code: Select all

$ convert -size 4000x4000 xc:white xc:white -adjoin 2pages.mpc; convert 2pages.mpc info:
convert-im6.q16: unable to extend cache `2pages.mpc': Bad file descriptor @ error/cache.c/OpenPixelCache/3966.
convert-im6.q16: unable to persist pixel cache `2pages.mpc' @ error/mpc.c/ReadMPCImage/944.
convert-im6.q16: no images defined `info:' @ error/convert.c/ConvertImageCommand/3258.

The platform is Debian. I don't think this is specific to Fred's script. Also, the system's resource limits

Code: Select all

  Width: 16KP
  Height: 16KP
  Area: 128MP
  Memory: 256MiB
  Map: 512MiB
  Disk: 1GiB
  File: 768
  Thread: 8
  Throttle: 0
  Time: unlimited

The coding error I referred to was in PersistPixelCache(), pwrite() was called within OpenPixelCache(image,ReadMode,exception). I don't know much about the design but If ReadMode indicates any intention here pwrite() shouldn't ever happen.

[fmw42]

I have moved this topic to the Bugs forum, since it did not seem to be specific to my scripts.

[fmw42]

Your command works fine for me on IM 6.9.8.0 Q16 Mac OSX.

Code: Select all

convert -size 4000x4000 xc:white xc:white -adjoin 2pages.mpc; convert 2pages.mpc info:
2pages.mpc[0] MPC 4000x4000 4000x4000+0+0 16-bit sRGB 833B 0.010u 0:00.099
2pages.mpc[1] MPC 4000x4000 4000x4000+0+0 16-bit sRGB 833B 0.010u 0:00.059

Code: Select all

Resource limits:
  Width: 214.7MP
  Height: 214.7MP
  Area: 4.295GP
  Memory: 2GiB
  Map: 4GiB
  Disk: unlimited
  File: 192
  Thread: 2
  Throttle: 0
  Time: unlimited

[snibgo]

Memory: 256MiB
Map: 512MiB

These are very low limits. They are set in policy.xml. What happens if you give more sensible limits?

[tmpfs]

These limits were added after last year's CVEs as security hardening. https://anonscm.debian.org/git/collab-m ... a6d7bf645b

The result is the same after doubling the two limits you mentioned. But when I doubled the area limit it no longer reproduces the issue. (With area limit at 256MP instead of 128MP)

Code: Select all

$ convert -size 4000x4000 xc:white xc:white -adjoin 2pages.mpc; convert 2pages.mpc info:
2pages.mpc[0] MPC 4000x4000 4000x4000+0+0 16-bit sRGB 835B 0.000u 0:00.000
2pages.mpc[1] MPC 4000x4000 4000x4000+0+0 16-bit sRGB 835B 0.000u 0:00.000

Also, I don't think this limit is the deciding factor. Several other ways can work (area limit 128MP)

Code: Select all

$ convert -size 4000x4000 xc:white xc:white -adjoin 2pages.miff; convert 2pages.miff info:
2pages.miff[0] MIFF 4000x4000 4000x4000+0+0 16-bit TrueColor sRGB 192MB 0.160u 0:00.180
2pages.miff[1] MIFF 4000x4000 4000x4000+0+0 16-bit TrueColor sRGB 192MB 0.080u 0:00.089
$ convert -size 4000x4000 xc:white 1page.mpc; convert 1page.mpc 1page.mpc -adjoin 2pages.mpc; convert 2pages.mpc info:
2pages.mpc[0] MPC 4000x4000 4000x4000+0+0 16-bit sRGB 891B 0.000u 0:00.000
2pages.mpc[1] MPC 4000x4000 4000x4000+0+0 16-bit sRGB 891B 0.000u 0:00.000
$ convert -size 4000x4000 xc:white xc:white -adjoin 2pages.mpc; convert 2pages.mpc info:
convert-im6.q16: unable to extend cache `2pages.mpc': Bad file descriptor @ error/cache.c/OpenPixelCache/3966.
convert-im6.q16: unable to persist pixel cache `2pages.mpc' @ error/mpc.c/ReadMPCImage/944.
convert-im6.q16: no images defined `info:' @ error/convert.c/ConvertImageCommand/3258.

[snibgo]

Yes, sorry, I hadn't noticed your very small area limit.

I call these limits "very small" because my images are typically 5000x7500 pixels, so need about 300 MB memory each (or 600 MB with Q32), and many commands need three images in memory concurrently. Your images aren't much smaller than mine.

For many purposes, eg web pages, those limits will be fine. The default security policy used to be weak, but we could increase the strength if we wanted. Now the default is strong, so we need to weaken it if needed.

If I understand correctly, reading an MPC may not take any memory, because the file format is exactly the same as the memory format, so IM treats this as a memory-mapped file.

[magick]

Perhaps your /tmp area is filling up. Assuming your work area is on a partition with plenty of free space, try this command:

Code: Select all

convert -define registry:temporary-path=./ -size 4000x4000 xc:white xc:white -adjoin 2pages.mpc
convert -define registry:temporary-path=./ 2pages.mpc info:

Does that work? It does for us.

[tmpfs]

All the previous commands were run under /tmp. And my /tmp is not separately mounted, just part of /.

[magick]

Can you post you entire security policy here? We're trying to reproduce the problem and so far have been unsuccessful. Your commands run on our system without complaint.

[tmpfs]

It's in the above debian git repo link. But well here it is too (policy.xml)

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE policymap [
<!ELEMENT policymap (policy)+>
<!ELEMENT policy (#PCDATA)>
<!ATTLIST policy domain (delegate|coder|filter|path|resource) #IMPLIED>
<!ATTLIST policy name CDATA #IMPLIED>
<!ATTLIST policy rights CDATA #IMPLIED>
<!ATTLIST policy pattern CDATA #IMPLIED>
<!ATTLIST policy value CDATA #IMPLIED>
]>
<!--
  Configure ImageMagick policies.

  Domains include system, delegate, coder, filter, path, or resource.

  Rights include none, read, write, and execute.  Use | to combine them,
  for example: "read | write" to permit read from, or write to, a path.

  Use a glob expression as a pattern.

  Suppose we do not want users to process MPEG video images:

    <policy domain="delegate" rights="none" pattern="mpeg:decode" />

  Here we do not want users reading images from HTTP:

    <policy domain="coder" rights="none" pattern="HTTP" />

  Lets prevent users from executing any image filters:

    <policy domain="filter" rights="none" pattern="*" />

  The /repository file system is restricted to read only.  We use a glob
  expression to match all paths that start with /repository:
  
    <policy domain="path" rights="read" pattern="/repository/*" />

  Let's prevent possible exploits by removing the right to use indirect reads.

    <policy domain="path" rights="none" pattern="@*" />

  Any large image is cached to disk rather than memory:

    <policy domain="resource" name="area" value="1GB"/>

  Define arguments for the memory, map, area, width, height, and disk resources
  with SI prefixes (.e.g 100MB).  In addition, resource policies are maximums
  for each instance of ImageMagick (e.g. policy memory limit 1GB, -limit 2GB
  exceeds policy maximum so memory limit is 1GB).
-->
<policymap>
  <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
  <policy domain="resource" name="memory" value="256MiB"/>
  <policy domain="resource" name="map" value="512MiB"/>
  <policy domain="resource" name="width" value="16KP"/>
  <policy domain="resource" name="height" value="16KP"/>
  <policy domain="resource" name="area" value="128MB"/>
  <policy domain="resource" name="disk" value="1GiB"/>
  <!-- <policy domain="resource" name="file" value="768"/> -->
  <!-- <policy domain="resource" name="thread" value="4"/> -->
  <!-- <policy domain="resource" name="throttle" value="0"/> -->
  <!-- <policy domain="resource" name="time" value="3600"/> -->
  <!-- <policy domain="system" name="precision" value="6"/> -->
  <!-- not needed due to the need to use explicitly by mvg: -->
  <!-- <policy domain="delegate" rights="none" pattern="MVG" /> -->
  <!-- use curl -->
  <policy domain="delegate" rights="none" pattern="URL" />
  <policy domain="delegate" rights="none" pattern="HTTPS" />
  <policy domain="delegate" rights="none" pattern="HTTP" />
  <!-- in order to avoid to get image with password text -->
  <policy domain="path" rights="none" pattern="@*"/>
  <policy domain="cache" name="shared-secret" value="passphrase" stealth="true"/>
</policymap>