Convert overwrites input when acting on raw (CR2, NEF, etc.)

[teuobk]

In version 6.7.5-10 on Windows (which includes a copy of convert that reports as 6.7.5-9), Q16, binary distribution, 32-bit, statically linked, executing the following command:

convert some_raw_file.cr2 output.jpg

...results in the raw file being overwritten with itself in addition to the jpeg file being generated. This is most visible as a change in the modification date for the file. The actual contents of the raw file appear to be identical pre- and post-convert. I believe that the raw (e.g., .cr2) file should not be written to during this command. Indeed, the raw file is not overwritten with earlier versions of convert.

For example, this does not occur with a 64-bit, Q16, Windows binary copy of convert that reports as version 6.6.7-1 (the only old copy of IM I had around). In that old version, only reading takes place; the raw file is never opened for writing.

This is not just a stat (i.e., file system) artifact. Inspection with Process Monitor reveals that the file is indeed being written, as shown in this snippet from the log:

(...snip...)
4:24:06.0571321 PM convert.exe 6596 ReadFile C:\Users\Jeff\Pictures\aaaa\IMG_9543.CR2 SUCCESS Offset: 24,903,680, Length: 262,144, Priority: Normal
4:24:06.0577580 PM convert.exe 6596 ReadFile C:\Users\Jeff\Pictures\aaaa\IMG_9543.CR2 SUCCESS Offset: 25,165,824, Length: 262,144, Priority: Normal
4:24:06.0582826 PM convert.exe 6596 ReadFile C:\Users\Jeff\Pictures\aaaa\IMG_9543.CR2 SUCCESS Offset: 25,427,968, Length: 34,963, Priority: Normal
4:24:06.0583923 PM convert.exe 6596 ReadFile C:\Users\Jeff\Pictures\aaaa\IMG_9543.CR2 END OF FILE Offset: 25,462,931, Length: 262,144, Priority: Normal
4:24:06.0630140 PM convert.exe 6596 CloseFile C:\Users\Jeff\Pictures\aaaa\IMG_9543.CR2 SUCCESS
4:24:20.0027346 PM convert.exe 6596 CreateFile C:\Users\Jeff\Pictures\aaaa\IMG_9543.CR2 SUCCESS Desired Access: Generic Write, Read Attributes, Disposition: OpenIf, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: 0, OpenResult: Opened
4:24:20.0038856 PM convert.exe 6596 WriteFile C:\Users\Jeff\Pictures\aaaa\IMG_9543.CR2 SUCCESS Offset: 0, Length: 262,144, Priority: Normal
4:24:20.0043010 PM convert.exe 6596 WriteFile C:\Users\Jeff\Pictures\aaaa\IMG_9543.CR2 SUCCESS Offset: 262,144, Length: 262,144, Priority: Normal
4:24:20.0046506 PM convert.exe 6596 WriteFile C:\Users\Jeff\Pictures\aaaa\IMG_9543.CR2 SUCCESS Offset: 524,288, Length: 262,144, Priority: Normal
4:24:20.0050095 PM convert.exe 6596 WriteFile C:\Users\Jeff\Pictures\aaaa\IMG_9543.CR2 SUCCESS Offset: 786,432, Length: 262,144, Priority: Normal
4:24:20.0055029 PM convert.exe 6596 WriteFile C:\Users\Jeff\Pictures\aaaa\IMG_9543.CR2 SUCCESS Offset: 1,048,576, Length: 262,144, Priority: Normal
4:24:20.0059948 PM convert.exe 6596 WriteFile C:\Users\Jeff\Pictures\aaaa\IMG_9543.CR2 SUCCESS Offset: 1,310,720, Length: 262,144, Priority: Normal
(...snip...)

This problem occurs with all Canon and Nikon raw files that I have tried. To aid in diagnosis, I've placed an example raw file for download on my site.

I'm assuming this is a bug?

Thanks,
Jeff

p.s. - ImageMagick is awesome


*Edit: Pasted the wrong data from Process Monitor. Corrected.

[anthony]

Yes over-writing input I would consider a bug, But it may be caused by the "delegate" program that IM uses to read the camera format files.

[magick]

Converting CR2 works fine for us-- that is the original file is not overwritten. ImageMagick utilizes the dcraw.exe delegate program to assist in the conversion of camera formats such as CR2. Perhaps you installed dcraw from another source that behaves differently that the version that is included with ImageMagick. If so, let us know and we will rename the utility to avoid conflicts.

[teuobk]

Thanks for the replies. I suspected a dcraw problem too, but I eliminated that possibility by swapping in dcraw from a known-good (i.e., older) ImageMagick installation.

So I dug into it a bit more, by which I mean I built from source so I could get more visibility into the problem. Turns out that the problem was introduced in changeset 6964, committed on Feburary 25, 2012 by "christy". The change was in MagickCore/delegate.c (copied and pasted from the browser, sorry):

r6226r6964
785 785 /*
786 Return if destination file already exists.
786 Copy source file to destination.
787 787 */
788 788 assert(source != (const char *) NULL);
789 789 assert(destination != (char *) NULL);
790 status=GetPathAttributes(destination,&attributes);
791 if (status != MagickFalse)
792 return(MagickTrue);
793 /*
794 Copy source file to destination.
795 */
796 790 destination_file=open_utf8(destination,O_WRONLY | O_BINARY | O_CREAT,S_MODE);
797 791 if (destination_file == -1)

If you put a breakpoint at delegate.c:813, you can watch it overwrite the input file.

As for what to do about it, I'm not sure what the standard procedure is for ImageMagick. What should my next step be?

Thanks,
Jeff

**Update: Just built 6.7.4 from source (which was released prior to the aforementioned change) and verified that it does not overwrite the input file when working on files delegated to dcraw. Definitely something introduced in 6.7.5.

[teuobk]

Sure. To be clear, it's overwritten with the same data, so if you're not watching the OS calls, the only difference you'll see before and after is a change in the modification timestamp of the input file. Here's a sequence I just ran.

First, let's check the version of convert:

Code: Select all

E:\Install\ImageMagick-6.7.5-10-Q16-windows\ImageMagick-6.7.5-10>convert --version
Version: ImageMagick 6.7.5-9 2012-03-02 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2012 ImageMagick Studio LLC
Features: OpenMP

Now, let's look at the modification timestamp on our input file:

Code: Select all

E:\Install\ImageMagick-6.7.5-10-Q16-windows\ImageMagick-6.7.5-10>dir img_9548*
 Volume in drive E has no label.
 Volume Serial Number is BCE3-9C82

 Directory of E:\Install\ImageMagick-6.7.5-10-Q16-windows\ImageMagick-6.7.5-10

03/10/2012  02:09 PM        24,056,510 IMG_9548.CR2
               1 File(s)     24,056,510 bytes
               0 Dir(s)   4,897,185,792 bytes free

Next, let's run convert with the -verbose flag:

Code: Select all

E:\Install\ImageMagick-6.7.5-10-Q16-windows\ImageMagick-6.7.5-10>convert -verbose IMG_9548.CR2 out.png
dcraw.exe -4 -w -O "C:/Users/Jeff/AppData/Local/Temp/magick-_wqXiJ6y.ppm" "C:/Users/Jeff/AppData/Local/Temp/magick-jyhc2nHY"
C:/Users/Jeff/AppData/Local/Temp/magick-_wqXiJ6y.ppm PPM 5202x3465 5202x3465+0+0 16-bit DirectClass 108.1MB 0.234u 0:00.236
IMG_9548.CR2=>C:/Users/Jeff/AppData/Local/Temp/magick-_wqXiJ6y.ppm CR2 5202x3465 5202x3465+0+0 16-bit DirectClass 108.1MB 0.000u 0:00.027
IMG_9548.CR2=>out.png CR2 5202x3465 5202x3465+0+0 16-bit DirectClass 85.92MB 17.800u 0:17.975

Now let's check the modification timestamp of the input file again. Note that it has changed:

Code: Select all

E:\Install\ImageMagick-6.7.5-10-Q16-windows\ImageMagick-6.7.5-10>dir img_9548*
 Volume in drive E has no label.
 Volume Serial Number is BCE3-9C82

 Directory of E:\Install\ImageMagick-6.7.5-10-Q16-windows\ImageMagick-6.7.5-10

03/10/2012  06:59 PM        24,056,510 IMG_9548.CR2
               1 File(s)     24,056,510 bytes
               0 Dir(s)   4,811,264,000 bytes free

In IM 6.7.4 and earlier, the input file is never written to, and the modification timestamp of the input file is not changed after a convert op. Again, the data in the file is the same; the (over)write of the input file is superfluous, messes with the filesystem timestamps, and creates a window for data loss in the input file (e.g., if convert dies halfway through writing back to the input file).

Specifically, the code change referenced above is causing an unconditional write from the temporary file C:/Users/Jeff/AppData/Local/Temp/magick-jyhc2nHY back into the input file IMG_9548.CR2

The call that does the copy to the input file is at delegate.c:1036:

Code: Select all

   if (LocaleCompare(decode,"SCAN") != 0)
      {
        if (CopyDelegateFile(image->filename,input_filename) == MagickFalse)
          (void) RelinquishUniqueFileResource(input_filename);
      }

I'm not sure what the purpose of this code is, but it's called after the delegate (dcraw in this case) has completed its work. Inspection with the debugger shows that with a raw file, decode == "dng:decode", image->filename == "C:/Users/Jeff/AppData/Local/Temp/magick-BiyIVy-8" (which when run in this instance contains the same data, confirmed with SHA1, as the input CR2 file), and input_filename == "IMG_9543.CR2". Since "dng:decode" != "SCAN", the call to CopyDelegateFile() is made. With those arguments, using the values in this example, the destination of the copy is "IMG_9543.CR2". Before the code change in changeset 6964, CopyDelegateFile() would return early when the destination file already existed, but after that change, the overwrite is unconditional.

I hope I'm being clear. I apologize if that is not the case. Does the problem make sense now?

Thanks,
Jeff

[magick]

We'll have a patch for the problem you reported by sometime tomorrow. Thanks.

[teuobk]

Great. Thanks!

Jeff