XMP being lost/corrupted on conversion

[djeyewater]

I'm using ImageMagick-6.5.4-5 on Ubuntu 9.04. Build is from source, without-perl.

When I convert a JPEG image with XMP metadata embedded, the XMP seems to get lost or corrupted. Viewing the exif of the converted image using the Exif Viewer v1.45 plugin for Firefox, it gives the following output:

Handling APPx (0xe0) block.
Handling APPx (0xed) block.
Handling APPx (0xe1) block.
Endian = Intel
* * * Parsing the IFD0 data * * *
18 directory entries (tags) found.
* * * Parsing the Exif SubIFD data * * *
38 directory entries (tags) found.
* * * Parsing the IFD1 data * * *
6 directory entries (tags) found.
* * * Parsing the GPS Information IFD data * * *
8 directory entries (tags) found.
Handling APPx (0xe2) block.
Handling APPx (0xe1) block.
Exif string not detected: 0x68 0x74 0x74 0x70 0x3a 0x2f
End of APPx data blocks reached.
Status = -3

Unable to extract some or all of the Exif data.

and then prints the exif, but doesn't have any XMP.

Also, if I view the converted image in Adobe Bridge CS4, the majority of the XMP is missing, though it still has some XMP like the exif and tiff XMP tags. It seems that exiftool can still read all the XMP.

Here's an example you can download and test:
Original image:
Converted image:

I'm converting just using

Code: Select all

convert "XMPTest.jpg" "-test.jpg"

, and also get the same results when converting through the Imagick PHP extension.

Dave

[magick]

ImageMagick transfers most profiles as blobs and writes them unaltered to the output format. The exception is EXIF which has a few fields updated such as the resolution. We've been doing it this way for over a decade without any complaints. Trying identify -verbose on the input and output images. It shows identical profiles (except for the EXIF resolution field). In addition exiftool can grok the output JPEG image suggesting the profiles are valid. We feel you have not proven ImageMagick is producing incorrect output profiles. If you provide additional evidence that the output JPEG image is corrupt, let us know and we will investigate further.

[djeyewater]

Thanks for the quick reply. I have done some more testing with exiftool, and it seems that while it can read the exif on the converted file, it can't write to it:

Code: Select all

exiftool -v3 -a -n -EXIF:ExposureProgram=3 -o "-testRewriteExif.jpg" "-test.jpg" > "-testRewriteExif.jpgExiftool"

Outputs

Writing ExifIFD:ExposureProgram
======== /home/brighto/public_html/photosite/CSI/Img/-test.jpg
'/home/brighto/public_html/photosite/CSI/Img/-test.jpg' --> '/home/brighto/public_html/photosite/CSI/Img/-testRewriteExif.jpg'
Rewriting /home/brighto/public_html/photosite/CSI/Img/-test.jpg...
Editing tags in: APP0 APP1 ExifIFD IFD0 JFIF
Creating tags in: APP1 ExifIFD IFD0
JPEG APP0 (14 bytes):
0000: 4a 46 49 46 00 01 01 01 01 2c 01 2c 00 00 [JFIF.....,.,..]
Rewriting JFIF
JPEG APP13 (2674 bytes):
0000: 50 68 6f 74 6f 73 68 6f 70 20 33 2e 30 00 38 42 [Photoshop 3.0.8B]
0010: 49 4d 04 04 00 00 00 00 01 73 1c 02 00 00 02 00 [IM.......s......]
0020: 02 1c 02 78 00 6b 53 74 20 4d 61 72 79 27 73 20 [...x.kSt Mary's ]
0030: 50 61 72 69 73 68 20 43 68 75 72 63 68 20 69 6e [Parish Church in]
0040: 20 74 68 65 20 76 69 6c 6c 61 67 65 20 6f 66 20 [ the village of ]
0050: 42 69 74 74 65 73 77 65 6c 6c 2c 20 4c 65 69 63 [Bitteswell, Leic]
0060: 65 73 74 65 72 73 68 69 72 65 2c 20 55 4b 20 6f [estershire, UK o]
[snip 2562 bytes]
JPEG APP1 (2280 bytes):
0000: 45 78 69 66 00 00 49 49 2a 00 08 00 00 00 12 00 [Exif..II*.......]
0010: 0e 01 02 00 6c 00 00 00 e6 00 00 00 0f 01 02 00 [....l...........]
0020: 12 00 00 00 52 01 00 00 10 01 02 00 0b 00 00 00 [....R...........]
0030: 64 01 00 00 12 01 03 00 01 00 00 00 01 00 00 00 [d...............]
0040: 01 00 00 00 01 00 00 00 6f 01 00 00 1b 01 05 00 [........o.......]
0050: 01 00 00 00 77 01 00 00 28 01 03 00 01 00 00 00 [....w...(.......]
0060: 02 00 00 00 31 01 02 00 1c 00 00 00 7f 01 00 00 [....1...........]
[snip 2168 bytes]
Rewriting IFD0
ExifByteOrder = II
Warning = [minor] Entries in IFD0 were out of sequence. Fixed.
Error = Bad format (0) for IFD0 entry 0
0 image files updated
1 files weren't updated due to errors

Whereas doing the same command on the original file updates it okay without errors.

Exiftool verbose dumps: -test.jpgExiftoolDump (converted file) XMPTest.jpgExiftoolDump

This may not be important, but I also found that if when doing a verbose level 3 dump with exiftool

Code: Select all

exiftool -v3 "-test.jpg" > "-test.jpgExiftoolDump"
exiftool -v3 "XMPTest.jpg" > "XMPTest.jpgExiftoolDump"

What I guess is the start of the XMP Packet looks like this in XMPTest (the original file):

JPEG APP1 (9886 bytes):
137a: 68 74 74 70 3a 2f 2f 6e 73 2e 61 64 6f 62 65 2e [http://ns.adobe.]
138a: 63 6f 6d 2f 78 61 70 2f 31 2e 30 2f 00 3c 3f 78 [com/xap/1.0/.<?x]
139a: 70 61 63 6b 65 74 20 62 65 67 69 6e 3d 22 ef bb [packet begin="..]
13aa: bf 22 20 69 64 3d 22 57 35 4d 30 4d 70 43 65 68 [." id="W5M0MpCeh]
13ba: 69 48 7a 72 65 53 7a 4e 54 63 7a 6b 63 39 64 22 [iHzreSzNTczkc9d"]
13ca: 3f 3e 20 3c 78 3a 78 6d 70 6d 65 74 61 20 78 6d [?> <x:xmpmeta xm]
13da: 6c 6e 73 3a 78 3d 22 61 64 6f 62 65 3a 6e 73 3a [lns:x="adobe:ns:]

but in the converted file (-test), it looks like this (note the space before the .<?xpacket):

JPEG APP1 (9887 bytes):
1578: 68 74 74 70 3a 2f 2f 6e 73 2e 61 64 6f 62 65 2e [http://ns.adobe.]
1588: 63 6f 6d 2f 78 61 70 2f 31 2e 30 2f 20 00 3c 3f [com/xap/1.0/ .<?]
1598: 78 70 61 63 6b 65 74 20 62 65 67 69 6e 3d 22 ef [xpacket begin=".]
15a8: bb bf 22 20 69 64 3d 22 57 35 4d 30 4d 70 43 65 [.." id="W5M0MpCe]
15b8: 68 69 48 7a 72 65 53 7a 4e 54 63 7a 6b 63 39 64 [hiHzreSzNTczkc9d]
15c8: 22 3f 3e 20 3c 78 3a 78 6d 70 6d 65 74 61 20 78 ["?> <x:xmpmeta x]
15d8: 6d 6c 6e 73 3a 78 3d 22 61 64 6f 62 65 3a 6e 73 [mlns:x="adobe:ns]

If I write an XMP tag using exiftool on the converted file, then do a verbose level 3 dump on the file using exiftool, I see it has 2 xmp sections, it looks like exiftool has written a new xmp block to the file, and the xmp tag written can be read by the Firefox exif plugin. The new xmp block looks like this (note no space before the .<xpacket):

JPEG APP1 (2844 bytes):
137a: 68 74 74 70 3a 2f 2f 6e 73 2e 61 64 6f 62 65 2e [http://ns.adobe.]
138a: 63 6f 6d 2f 78 61 70 2f 31 2e 30 2f 00 3c 3f 78 [com/xap/1.0/.<?x]
139a: 70 61 63 6b 65 74 20 62 65 67 69 6e 3d 27 ef bb [packet begin='..]
13aa: bf 27 20 69 64 3d 27 57 35 4d 30 4d 70 43 65 68 [.' id='W5M0MpCeh]
13ba: 69 48 7a 72 65 53 7a 4e 54 63 7a 6b 63 39 64 27 [iHzreSzNTczkc9d']
13ca: 3f 3e 0a 3c 78 3a 78 6d 70 6d 65 74 61 20 78 6d [?>.<x:xmpmeta xm]
13da: 6c 6e 73 3a 78 3d 27 61 64 6f 62 65 3a 6e 73 3a [lns:x='adobe:ns:]
[snip 2732 bytes]
+ [XMP directory, 2815 bytes]
| XMPToolkit = Image::ExifTool 7.79
| Source = Dave Kennard
| - Tag 'x:xmpmeta/rdf:RDF/rdf:Description/photoshop:Source'

The xmp section that Image Magick wrote (with the space before the .<xpacket) when creating the file is still there, but the Firefox exif plugin still can't read it, only the new XMP tag written by exiftool.

Here's the dump for the file converted by Image Magick then an XMP tag written by exiftool: -testRewriteXMP.jpgExiftoolDump

Dave

[magick]

Thanks for the detailed analysis of this problem. We will have patch to fix the problem in the next point release of ImageMagick within a day or two.

[djeyewater]

That's quick!

Just wanted to make sure you realise that my analysis that the problem was due to a space between 'http://ns.adobe.com/xap/1.0/' and '.<?xpacket', was only a guess (and probably wrong I expect).

Thanks for the work you put into this program.

Regards

Dave

[vulpes]

I am also seeing a similar issue.
After inserting xmp profile to an image,
Running identify -verbose 123.jpg does show

Code: Select all

  Profiles:
    Profile-APP12: 15 bytes
    Profile-xmp: 2920 bytes

But When I try to read the data using ff exifviewer plugin or adobe photoshop, nothing shows up.
The data does get read by exiftool so something does get added, just not in a proper format it seems?

Any updates on this would be greatly appreciated.

Thank you for the superb work on the wonderful application btw.

[djeyewater]

Any updates on this would be greatly appreciated.

I just tried my problem file in 6.5.4-10, and the XMP (and exif) could be read in FF exif viewer, Opanda Iexif FF plugin (in windows), the File Info Panel in Adobe Photoshop, and also the metadata panel in Adobe Bridge. However, when I tried writing a tag with exiftool (v7.89) to the exif of the converted file, the same as I did before, I get the same error as I did before and exiftool can't write the tag.

Regards

Dave