Metadata in PSD and TIFF files produced by Lightroom 2 bug

[dalex_257]

The Adobe Lightroom 2 and Photoshop CS3 produce PSD and TIFF files (when converting from RAW in PS CS3) that ImageMagick has trouble to get metadata from. By metadata I refer to EXIF, IPTC and XMP information. The identify command in verbose mode shows that both those formats have 8BIM profile which is a binary image resource (as defined by Photoshop format document) and ImageMagick does seem to have some logic parsing those but it is somehow flawed. In files produced by Lightroom and Photoshop CS3 ImageMagick extracts the whole image resource table at once if trying to extract it as 8BIM. Even if the resource table contains other 8BIM resources with thumbnails, embedded ICC profiles and EXIF/XMP metadata - those are not recognised at all.

It would be nice to fix this to allow extraction of the PSD tags as well as embedded metadata from those 8BIM resources. The XMP in PSD files are documented and implemented in a code in Adobe XMP SDK which is released under BSD license (the PSD format reader is quite well commented) so I guess its a matter of implementing it in IM (XMP resource ID is 0x424, EXIF resource ID is 0x422 and IPTC - 0x404 all taken from XMP SDK).

Also by some reason when trying to extract XMP resource/profile from PSD/TIF files with convert command, ImageMagick looks up the APP1 resource which I believe is wrong (outdated?).

Edit: forgot to add that the ExifTool by Phil Harvey does display all the metadata correctly from both of those formats (so I guess the code could be analysed and ported to ImageMagick).

Edit2: this is happening in the most recent released version of ImageMagick 6.4.2-6.

[magick]

Can you post a URL to a couple of PSD and TIFF images that illustrate the problem. We will use them as out test images to verify and fix the bug you reported. Thanks.

[dalex_257]

Can you post a URL to a couple of PSD and TIFF images that illustrate the problem. We will use them as out test images to verify and fix the bug you reported. Thanks.

Here you go - files exported from LR 2 with all defaults but reduced to 8bit per channel and reduced in size to 800x600 (to save the space).

Archive:
http://www.dalex.clara.net/LR2Test.zip

This contains the following files:
LR2Test-1.tif
LR2Test-2.tif
LR2Test-1.psd
LR2Test-2.psd

All of these don't show any EXIF/IPTC/XMP information with identify -verbose. For all of these, extraction of only 8BIM profile can be extracted with convert, no EXIF or XMP. All of those have EXIF (8BIM resource ID 0x422) and XMP (8BIM resource ID 0x424) embedded with the lots of camera/lens metadata and some other Lightroom settings. All commands used from the latest IM version (downloaded yesterday).

Please keep me informed by adding to this thread.

Edit: URL updated - now shared on my personal site without limitations

[dalex_257]

Can you post a URL to a couple of PSD and TIFF images that illustrate the problem. We will use them as out test images to verify and fix the bug you reported. Thanks.

Archive:
http://www.dalex.clara.net/LR2Test.zip

Did the file helped? Do you know the reason of the problem and is it going to be fixed?

Thanks

[magick]

We are just starting on this problem, we may need a week or two depending on the demands on our time.

[dalex_257]

We are just starting on this problem, we may need a week or two depending on the demands on our time.

Thanks.

[dalex_257]

We are just starting on this problem, we may need a week or two depending on the demands on our time.

Any progress with this problem?

[magick]

Download and try ImageMagick 6.4.3. We extract the embedded profiles from the Photoshop resource block. If you continue to have problems, let us know.

[dalex_257]

Download and try ImageMagick 6.4.3. We extract the embedded profiles from the Photoshop resource block. If you continue to have problems, let us know.

The 6.4.3 seems to be unavailable. The source links on your FTP does not exist for Windows and for UNIX the tar archive is 6.4.2-10.

I've tried a few mirrors but thay all refer to 6.4.2 versions.

[magick]

Grab the latest ImageMagick source from ftp://ftp.imagemagick.org/pub/ImageMagick. Some mirrors take a week or two before they catch up with the authoritative site.

[dalex_257]

Grab the latest ImageMagick source from ftp://ftp.imagemagick.org/pub/ImageMagick. Some mirrors take a week or two before they catch up with the authoritative site.

Thanks, it works now for PSD and TIFs. However, there is a nagging issue with JPEGs profuced by Lightroom (or Photoshop ACR). When trying to extract the XMP profile from them (by 'convert aaa.jpg bbb.xmp' command) the generated XMP file is invalid and includes extra bytes in header that do not belong there. Doing so for TIF and PSD produces correct XMP. The example of the beginning of such XMP extracted from JPEG:

Code: Select all

http://ns.adobe.com/xap/1.0/NULL<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Adobe XMP Core 4.2-c020 1.124078, Tue Sep 11 2007 23:21:40        ">
 <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <rdf:Description rdf:about=""

The first URL and following NULL character do not belong in XMP really. Again for how XMPs are stored in JPEGs, I wil refer you to Adobe XMP SDK sources. I did the sample of such JPEG which can be found in here http://www.aletan.com/img/v1/p98083919.jpg

EDIT: It seems also that writing XMPs in JPEG does not work well either. I have tried to convert the PSD file exported from Lightroom to JPEG (that should preserve the profiles) and also by manually extracting the XMP from PSD and adding it to JPEG. When both of those JPEGs opened in Lightroom or Photoshop, none has any metadata apart from a few EXIF fields (not all of them).

[magick]

The JPEG XMP profile includes the XMP namespace followed by a NULL followed by the XMP packet. We need to strip the namespace from the profile. Will have a fix in the ImageMagick subversion trunk within a day or two. Thanks.

[dalex_257]

The JPEG XMP profile includes the XMP namespace followed by a NULL followed by the XMP packet. We need to strip the namespace from the profile. Will have a fix in the ImageMagick subversion trunk within a day or two. Thanks.

Great, thanks. I'll monitor the releases for the change.

[ThomasH]

i've got the same problem with 6.4.x, and what troubles me is that it used to work just fine with older imagemagick versions. also, the problem seems to be with TIF files in generall, not necessarily created by lightroom. i'm both seeing profile problems with adobe photoshop CS2 and imagemagick 6.3.6 created files!

when doing an identify, the profiles seem to be read just fine:

Code: Select all

[b]identify -verbose test.tif[/b]
...
  Compression: None
  Orientation: TopLeft
  Properties:
    comment: Test
    create-date: 2008-12-10T01:20:26+01:00
    modify-date: 2008-10-01T13:02:30+02:00
    signature: edited
    tiff:artist: edited
    tiff:make: Canon
    tiff:model: Canon EOS 30D
    tiff:rows-per-strip: 1
    tiff:software: Adobe Photoshop CS2 Windows
    tiff:timestamp: 2008:09:22 05:05:45
  Profiles:
    Profile-8bim: 7046 bytes
    Profile-icc: 3144 bytes
      IEC 61966-2.1 Default RGB colour space - sRGB
    Profile-iptc: 2048 bytes
      unknown[2,0]: 
      Created Date[2,55]: 20080813
      Created Time[2,60]: 201607+0100
      Country[2,101]: Britain
      Country Code[2,100]: GBR
      Image Name[2,5]: Test
      Keyword[2,25]: Test
...

accessing the profiles fails:

Code: Select all

[b]convert test.tif xmp:test.xmp[/b]
convert: no APP1 data is available `test.xmp' @ meta.c/WriteMETAImage/2318.

[b]convert test.tif xmp:test.iptc[/b]
convert: no IPTC profile available `test.iptc' @ meta.c/WriteMETAImage/2249.

this also fails for files that were previously edited by imagemagick, and where imagemagick was previously able to manipulate the profiles:

Code: Select all

[b]identify -verbose 001.tif[/b]
...
 Compression: None
 Orientation: TopLeft
 Properties:
   comment: edited
   create-date: 2008-12-10T01:49:15+01:00
   modify-date: 2008-05-16T16:09:09+02:00
   signature: e4377aa1ac2dbdcee2270ac2c62a1bbed87908d2d1677190e46e34e00e1f85b6
   tiff:artist: unknown
   tiff:copyright: edited
   tiff:document: edited
   tiff:make: Canon
   tiff:model: Canon EOS 30D
   tiff:rows-per-strip: 1
   tiff:software: ImageMagick 6.3.6 10/19/07 Q16 http://www.imagemagick.org
   tiff:timestamp: 2008:05:13 22:31:54
 Profiles:
   Profile-8bim: 15168 bytes
   Profile-icc: 560 bytes
     Adobe RGB (1998)
   Profile-iptc: 152 bytes
    Profile-iptc: 152 bytes
      unknown[2,0]:
      Keyword[2,25]: edited1
      Keyword[2,25]: edited2
      Keyword[2,25]: edited3
      Keyword[2,25]: edited4
      Keyword[2,25]: edited5
      Byline[2,80]: edited
      unknown[2,92]: edited
      Copyright String[2,116]: edited
      Caption[2,120]: edited
  Artifacts:
    verbose: true
  Tainted: False
...


[b]convert 001.tif iptc:test.iptc[/b]
convert: no IPTC profile available `test.iptc' @ meta.c/WriteMETAImage/2249.

[b]convert 001.tif xmp:test.xmp[/b]
convert: no APP1 data is available `test.xmp' @ meta.c/WriteMETAImage/2318.

any help very much appreciated - i need a way to be able to programmatically extract and modify the XMP data quite urgently...

update: using an old 6.3.4 imagemagick installation, i can read, extract and modify the xmp profiles just fine...

[magick]

We're using ImageMagick 6.4.7-5 and cannot reproduce the problem you posted. We extracted the XMP profile from a number of TIFF images. Post a URL to a couple of your TIFF images that fail and the command you are using. We need to reproduce the problem before we can help.