Help with DCRaw Delegate Configuration

[kevinludlow]

Hi. I've spent about 5 days working on this without being able to get over the final hump and so I'm finally turning to the community.

Basically I have decided to change the ufraw-batch process to use dcraw instead. I've installed dcraw and cjpeg so that I can convert raw files (specifically .CR2 files in my case) to JPGs. I am running on a CentOS box using ImageMagick 6.7.8-9.

If I run the following from the command line, my system will successfully convert a CR2 file into a JPG:

Code: Select all

dcraw -c -q 3 -H 5 -w IMG_1830.CR2 | /opt/libjpeg-turbo/bin/cjpeg -quality 100 > IMG_1830.jpg

So basically dcraw processes the CR2 file to STDOUT. STDOUT then pipes that into the cjpeg program which converts it into a jpeg. Unfortunately I cannot for the life of me configure the delegate in the delegates.xml file and was hoping someone could shed some light on it. Here's what I've tried:

Code: Select all

<delegate decode="dng:decode" command=""dcraw" -c -q 3 -H 5 -w "%i" | "/opt/libjpeg-turbo/bin/cjpeg" -quality 100 > "%o""/>

Which as far as I can tell should run the dcraw command on the input file, process it the same way and then pipe the output to cjpeg which should in turn write it to the output file name. Unfortunately that does not happen.

Code: Select all

dev01::~/dcraw/#>convert -verbose IMG_1830.CR2 IMG.JPG
"dcraw" -c -q 3 -H 5 -w "/tmp/magick-VteWODb6" | "/opt/libjpeg-turbo/bin/cjpeg" -quality 100 > "/tmp/magick-7gqEr1sV"
Cannot decode file /tmp/magick-VteWODb6
Empty input file
convert: delegate failed `"dcraw" -c -q 3 -H 5 -w "%u" | "/opt/libjpeg-turbo/bin/cjpeg" -quality 100 > "%o"' @ error/delegate.c/InvokeDelegate/1065.
convert: unable to open image `/tmp/magick-VteWODb6.ppm': No such file or directory @ error/blob.c/OpenBlob/2642.
convert: no images defined `IMG.JPG' @ error/convert.c/ConvertImageCommand/3046.

The weirdest part is that the IMG_1830.CR2 file that was in the directory when I started will no longer be in the directory after I run this command and it fails. It also doesn't seem that those temp magick files are ever created in the /tmp/ directory. So I really haven't a clue what is going on at this point, but would love to solve this.

I've also taken the time to type up a step-by-step on how to get dcraw going on one's system, but I'd love to solve this final step!

[snibgo]

It shouldn't remove IMG_1830.CR2. Something is badly wrong.

I don't understand the proper use of delegates.xml, but I note the following:

Perhaps delegates can't cope with piplining within the delegate command. Try removing the secondary processing by cjpeg.

The line in the Windows delegates.xml looks like this:

<delegate decode="dng:decode" stealth="True" command="dcraw.exe -6 -w -O "%u.ppm" "%i""/>

This runs a customised version of dcraw with an option "-O outfilename", which you won't have.

The filenames are bracketed with " rather than the quote character. The output file is %u.ppm, rather than %o.

[kevinludlow]

That's a good start. I tried simplifying the delegate line. Initially I tried using the -O, but of course (and consistent with what you said) my DCRAW version doesn't contain that custom option. Still, I thought I'd see what happens when it was stripped down like this:

Code: Select all

  <delegate decode="dng:decode" stealth="True" command="dcraw -6 -w "%i""/>

And while I'm not sure exactly what it did, it definitely did something as it took some time to process it instead of just dying.[/code]

This was the initial output I got and I should note in this case the CR2 file did NOT disappear. So something to do with the piping is evidently moving the file. It's possible I'm using the %o and %i incorrectly and so it's getting overwritten, but I'm not sure. Initial output below:

Code: Select all

dev01::~/dcraw/#> convert IMG_1830.CR2 abc3.jpg
convert: unable to open image `/tmp/magick-PlL9WblE.ppm': No such file or directory @ error/blob.c/OpenBlob/2642.
convert: no images defined `abc3.jpg' @ error/convert.c/ConvertImageCommand/3046.

So then I ran the same thing using the same delegate formatting only with verbose mode:

Code: Select all

dev01::~/dcraw/#> convert -verbose IMG_1830.CR2 abc3.jpg
dcraw -6 -w "/tmp/magick-VW1XalCm"
convert: unable to open image `/tmp/magick-wYn24Bx8.ppm': No such file or directory @ error/blob.c/OpenBlob/2642.
convert: no images defined `abc3.jpg' @ error/convert.c/ConvertImageCommand/3046.

Perhaps more importantly about this part is that the tmp file ID that is is looking for there HAS been created as a PPM file in my /tmp/ directory.

Code: Select all

-rw-rw-r-- 1 ludlow ludlow 62539795 Nov  7 19:38 magick-VW1XalCm.ppm

The second file, `/tmp/magick-wYn24Bx8.ppm', however is NOT in the /tmp/ directory. So it seems that DCRAW is definitely taking the CR2 file as one would expect. Moreover it's processing it. But then it seems that ImageMagick doesn't know how to return it to the same directory I'm working in. The abc3.jpg file does not exist within the directory.

As one more tiny experiment I ran DCRAW from the command line by itself with the identical parameters from the delegate file.

Code: Select all

dev01::~/dcraw/#> dcraw -6 -w IMG_1830.CR2

And that created this file in the same working directory (as I think we might expect at this point):

Code: Select all

-rw-rw-r--  1 ludlow ludlow 62539795 Nov  7 19:43 IMG_1830.ppm

So anyway, the DCRAW portion is definitely doing it's thing. The trick is that I then have to convert that to JPG using cjpeg albeit haven't figured out exactly how to convert the PPM file using cjpeg. I can do it when DCRAW writes to STDOUT and then it gets piped into cjpeg, but not from a file itself. Weirdly still is that I can just use ImageMagick to convert the PPM file over to JPG!

Point being that all of the individual pieces are working, I just want to figure out how to make them work within the delegate handler so that I can rely on ImageMagick for all of my files (rather than fork my code so that any CR2 file uses some custom method I've come up with). I'm even cool writing a simple BASH script that I could then have the delegate invoke. Anyone know if that might be possible?

And if I didn't say so already, thank you very much for the thought on this.

[snibgo]

I can't recall why the IM developers customised dcraw with the "-O" option. Something wouldn't work without it but I forget what.

I'm even cool writing a simple BASH script that I could then have the delegate invoke. Anyone know if that might be possible?

Yes it is.

Well, I should qualify that with an example. My user-specific delegates.xml includes:

Code: Select all

  <delegate decode="png" encode="msph" command="msph.bat %i"/>

msph.bat is a shell script on my system path. It contains this:

Code: Select all

rundll32.exe "%ProgramFiles%\Windows Photo Viewer\PhotoViewer.dll", ImageView_Fullscreen %~dpnx1

This runs a standard Windows program that displays the image(s) that are in the file named in the script's parameter, so ...

Code: Select all

convert rose: msph:

... runs the viewer and shows the rose, but that's not important. I just wrote it as a test to see if IM could delegate-out to a script.

This is Windows. I would expect that bash would work just as easily.

[kevinludlow]

This runs a standard Windows program that displays the image(s) that are in the file named in the script's parameter, so ...

Sweet. Thank you very much for the suggestion and for the proof of concept at least in Windows. I am out and about right now, but will give that a go on my Linux box as soon as I get back and see if it fires off

[kevinludlow]

This is Windows. I would expect that bash would work just as easily.

So while it does seem that I can get ImageMagick to run custom bash scripts, it seems the problem may be as simple as I cannot understand what the delegates.xml input fields are. It would seem to me that the %i is the INPUT (as the documentation suggests), but experimenting with it shows that's not entirely the case. Instead %i is actually a reference to a /tmp/magick-xxxxxxx file of some kind. While this may technically still be the input file, it's largely deceptive from simply the argument passed in at the command line.

So I expanded the delegate to run a script of mine and then had that script output every single parameter possible.

The delegate line:

Code: Select all

<delegate decode="dng:decode" command="sh /home/ludlow/dcraw/dcraw.sh %i %o %u %Z %b %c %g %h %k %l %m %p %q %s %w %x %y"/>

My dcraw.sh bash script:

Code: Select all

#!/bin/bash
echo "The program outputs: $1 - $2 - $3 - $4 - $5 - $6 - $7 - $8 - $9 - ${10} - ${11} - ${12} - ${13} - ${14} - ${15} - ${16} - ${17} - ${18} - ${19} - ${20}";

And finally invoking the script through ImageMagick:

Code: Select all

dev01::~/dcraw/#> convert -verbose IMG_1830.CR2 sample.jpg
sh /home/ludlow/dcraw/dcraw.sh /tmp/magick-h_2IqaNu /tmp/magick-RljLgB6g /tmp/magick-B3dKcsFl /tmp/magick-ValDijeq 0B  0x0+0+0 0 0  CR2 0 16 0 0 72 Undefined 72 Undefined
The program outputs: /tmp/magick-h_2IqaNu - /tmp/magick-RljLgB6g - /tmp/magick-B3dKcsFl - /tmp/magick-ValDijeq - 0B - 0x0+0+0 - 0 - 0 - CR2 - 0 - 16 - 0 - 0 - 72 - Undefined - 72 - Undefined -  -  - 
convert: unable to open image `/tmp/magick-B3dKcsFl.ppm': No such file or directory @ error/blob.c/OpenBlob/2642.
convert: no images defined `sample.jpg' @ error/convert.c/ConvertImageCommand/3046.

So essentially the problem for me is just that I cannot for the life of me figure out where the IMG_1830.CR2 parameter comes into play. Or for that matter, I'm telling it that parameter #2 SHOULD be sample.jpg (as in that's what I want to name the output file), but again, it doesn't seem that my sample.jpg name is regarded in ANY of the parameters available to the delegates.

Perhaps if anyone could help me to understand why none of the parameters match up I could fill in the remaining gaps. It's kind of surprising to me how difficult it has been to hunt this stuff down given there is so much documentation for ImageMagick. It seems that the documentation will show every little technical ability, but fails to show the simple stuff.

Incidentally I did find a reference to creating custom delegates in this Google Book, but unfortunately page I need to read (page 293) has been removed from the preview. Well played, Michael Still. Well played. https://books.google.com/books?id=6KVZ8 ... de&f=false

Thanks again for any additional help!

[snibgo]

Your investigations are already deeper than mine, but a couple of points:

For explanations of %i, %o, %u etc, see http://www.imagemagick.org/script/escape.php . I've not seen an explanation of when we should use %u or %Z.

Don't forget that delegates can be used either for reading, or writing, or both.

When invoking a delegate to decode a file, I think IM always creates a temporary copy of that file, named "magick-something", and passes that invented name to the delegate. If IM is expecting a file back, it will invent a name for it, and pass that invented name to the delegate, and IM will look for that file with that name.

As far as I know, the delegate never sees the filenames the user gave.

[kevinludlow]

As far as I know, the delegate never sees the filenames the user gave.

Thanks very much for posting the link that you did. That helped me to understand the process a little bit further. I'm definitely not doing it the way that it's supposed to be done as ImageMagick will report an error, but I can kindof hack it with a bash script using the %f (filename) from that documentation. The obvious problem for me is that I still can't figure out how the OUTPUT filename works.

So in general if you were to do this:

Code: Select all

convert x.png y.jpg

...then the program would expect x.png to be in the local directory and for it to create a file called y.jpg. It will automatically interpret that it should load as a PNG and then use delegates to make a JPG.

But in my case of forcing the program to use a bash script I still cannot figure out how to use the output filename desired. So I would expect to be able to do this:

Code: Select all

convert x.cr2 y.jpg

And then the CR2 delegate will fire appropriately which in this case will fire off my bash script. So I can pass the %f through, but I still can't seem to find any variable that will let me dictate the output name.

Anyway, if you happen to know of any way to get around that I would say that I can write a full work-around for this, but at this point I'd be interested to see if I can fix the codebase myself and release a patch because it seems to be a pretty big shortcoming. I really love and appreciate how they've constructed the delegates.xml file as it should make things super easy, but in this case the idea behind it is definitely not working, unfortunately.

Thanks again!

[snibgo]

Ah, %f, yes, I'd forgotten that. There seems to be no %-escape for output filename. This could be useful to an "encode" delegate.

You can use other %-escapes. For example, my user-specific delegates.xml now includes:

Code: Select all

  <delegate decode="png" encode="testimdel" command="testimdel.bat %f %i %o %u %Z %[fx:1/7] %[artifact:mystuff]"/>

testimdel.bat is:

Code: Select all

echo 1=%1
echo 2=%2
echo 3=%3
echo 4=%4
echo 5=%5
echo 6=%6
echo 7=%7

So the text output from ...

Code: Select all

convert r.png -define mystuff=Hello testimdel:

... is:

Code: Select all

f:\web\im>
1=r.png
2=C:/Users/Alan/AppData/Local/Temp/magick-4616qz8t8QyjiXWL
3=C:/Users/Alan/AppData/Local/Temp/magick-46160_5ro4izOiE8
4=C:/Users/Alan/AppData/Local/Temp/magick-4616N1HuKnoSopr_
5=C:/Users/Alan/AppData/Local/Temp/magick-4616INuy7mnUDtK3
6=0.142857
7=Hello

So you could use an artifact or similar.

[henryj]

Very helpful discussion. I was trying to use dcraw delegate myself today and found this thread.

It's actually simple to implement -O option for dcraw according to viewtopic.php?t=22606#p103409.

So I've put something together today, if anyone the having same issue later can probably follow that instruction.

Available on github https://github.com/henryj/dcraw