Converting RAW images...

[cdannenberg]

I'm setting a server to batch convert raw digital camera images into tif files. Mostly right now I'm working with .CR2 files (the raw format used by newer Canon camera models). The result of: convert file.cr2 file.tif yields a tif file that is much darker than the the raw file that went in. I've found that a free windows-based converter app also darkens the output as well, but not as significantly as Imagemagick- Any suggestions or thoughts are greatly appreciated.

[jhfry]

Forgive me if I am way off base here... but aren't raw files neither light nor dark... they are simply what the CCD saw with absolutely no filtering by the camera.

If I am right in thinking this, then you can expect them to be way off in brightness and perhaps even color... even the best CCD on the market requires post capture processing to obtain a decent image.

It might be possible to script some of this into your workflow, as your input is all from the same CCD and thus it will require the same basic set of adjustments to compensate for its strengths and weaknesses.

DO NOT expect a direct RAW-to-TIFF conversion to yeild an accurate image... it's not supposed to. That's the whole reason behind the RAW format... it's so you can manipulate the image how you want to before the camera has gone and did what it 'THINKS' will look good.

[cdannenberg]

There is a histogram saved to them I think that dictates the correct light levels of the image, but since they are all proprietary formats I think it's just difficult for many programs to open them up.. and open them correctly. I have UFraw installed on the same server, and it will allow me to open the .CR2 image, it opens without darkening, and I can save it and the resulting file is the same as the raw that I opened... but I'm not sure how to automate this process.

They expect for numerous people to be sending in RAW files... most digital cameras create either jpg, or raw (only a couple will directly save as tif) and since they do not wish to accept .jpgs, and fear if they ask for tifs, people will convert their .jpgs to .tifs if they don't understand how to do so with their raw files... so though currently I'm working on getting good conversions of .CR2's, it will need to work with other formats as well.

[el_supremo]

IM uses the dcraw program as a delegate to handle the conversion of the CR2 file.
Look at the file config/delegates.xml and find the line containing "dcraw". Remove the -4 argument in that line, write the file and then try a conversion again.
-4 tells dcraw to produce a 16-bit linear output which, as you've seen, is very dark. Without the -4, dcraw will produce an 8-bit gamma corrected image.

Pete

[anthony]

You can also add a new delegate, exactly like the old but without the -4 and posibly called 'cr8' to mean cr at 8 bit. That way you can use cr8:filename to read that type of image.

The new delegate does NOT need to be added to the system delegate file, but can be in your own delegate.xml file in the ".magick" sub-directory of your home.

[cdannenberg]

Thanks, that at least lets me know there could be an easy solution; I do have a couple problems though. I grepped the delegates.xml file for both dcraw and instances of 4 and it does not appear to be there. I also do not have a .magick sub-directory in my home directory, I used ls -a to look since it would be hidden. Should I post the contents of delegates.xml or further information I can provide?

You will need to create your own ".magick" sub-directory, and create a delegates.xml by cutting down the system one. It has lots of comments as a built-in help guide. The convert -list option can be used to check on the delegates IM is seeing and from what file.

<= read
=> write
<=> both

As for it not being present, it may be id cdraw was NOT foun during the build/installation then it may not be present in the delegates file. Someone may be able to send you a copy of the right delegate to add, either personally or to the system (as you like). I don't have that delegate myself, but then I build my own Im and don't have cdraw installed.

PS: you nay notice that 'https:' is actually a delegate as the builtin http: can not handle the https secure reading.

[jhfry]

If it's too difficult to get a script that works in all cases, simply run a bunch of scripts and present the results to the user to choose the one that looks best.

I imagine a 9x9 grid of thumbnails, each with different RAW conversion settings and filters applied... the user selects the one that appears most accurate and the rest of the images he uploads are converted using those settings.

I am not sure of the nature of your site... but you could even make it possible for the users to create a custom script for their camera's... eventually you could build a DB of scripts for each camera that yeilds good results.

Sometimes trying to do all the work for the user actually makes your site LESS user friendly... give them choices and let them feel empowered!

[cdannenberg]

This is being carried out on a storage server which staff members can place their images on directly; I've set up a folder which a script checks frequently for RAW files, and when they show up it converts them into TIFs. Your idea sounds like it would be the most ideal, but might be too complex for me to follow (I'm kinda a noob) or require a greater explanation that you are willing to give. (Though if you are I'll certainly listen, try to follow, and be appreciative!). I believe there is a web interface where users will submit files (this concerns images of houses and other structures that are considered historic properties), but the website is controlled by a group of organizations, and I don't have control over the site's administration. If a web front end could be set up for our users in-house that might be an option, but the .CR2 will be primarily what they will be using so that great of complexity might not be needed.

[jhfry]

Sorry, I am certainly no expert, nor am I a developer... but I do understand technology.

Just because they are all using the same RAW format does not mean that all of the images will look good after being adjusted the same way. Cannon creates multiple image sensors, each of which have different characteristics and will result in different output. Most non-photographer types will expect a JPEG image that has already been adjusted within the camera's software to compensate for the CCD, the lighting levels, white balance, focal length, shutter speed, and any other settings that the camera senses or are set by the user.

RAW pictures simply do not work that way... they are just a dump of the "raw" CCD data. A CCD maybe very sensative to red or blue frequencies of light, another may be capable of night shots (very light sensative, or even infrared), and yet another might be best suited for highspeed shots. Each of these CCDs could be used to output a .CR2 raw file and each shot would appear different in the raw file. The JPEG output of the camera's using the different CCD's may be nearly indistinguishable as the manufacturer processes the raw data in the camera to achieve a fairly standard 'quality' of output... but ultimately that is a function of the camera's software that was written specifically for that particular CCD.

To assume that there is any chance of a one-size-fits-all approach to processing .CR2 or any other RAW format files is the result of not understanding why RAW files exist or what they are. RAW files exist because photographers want the output of their camera to be unfooled around with, allowing them the ultimate in fidelity, the best possible source to tweak to their liking. This means they want a file that has had ZERO post processing, ZERO compression, ZERO adjustments for white balance, ZERO adjustments for shutter speed, ZERO adjustments for light levels, ZERO, ZERO, ZERO. Therefore most RAW images look like CRAP until you apply these adjustments manually, and the adjustments must be tailored for the image.

If I take two pictures, one indoors and one outdoors, and apply the same adjustments to the indoor shot as I do the outdoor shot, the indoor shot will be WAY too dark as my adjustments must compensate for the sunlight that doesn't exist in the indoor shot.

One thing going for you however is that most/all raw formats include post processing information in the file... if your RAW filter knows how to use this data it can come up with an image that is nearly identical to the JPEG that your camera would have output.

The best tool I know for this is "dcraw" (http://www.cybercom.net/~dcoffin/dcraw/) which should allow you to convert to tiff from most raw formats... it will apply any adjustments that were configured on the camera at the time the photo was taken and result in an image similar to that of the JPEG that the camera would have produced (less the compression artifacts).

I would look at passing the files through dcraw to get a TIFF and then doing any required resizing and processing with IM.

DISCLAIMER: I could be completely wrong on everything I just said. IM may actually implement dcraw within it's code and yield the same result for all I know, however judging by your dark images, it doesn't.

[cdannenberg]

I completely agree- raw files are meant to be used as digital negatives, it's not that they look like crap, it's just that most programs can't open them or understand the post processing information correctly because they are proprietary formats. It's been a while, but I believe when I started in on this I could not use the convert command on a .CR2, and then after installing dcraw it would work. It just doesn't do it completely accurately. ...but it is possible since UFraw doesn't seem to have any problem in opening/converting them... that's a GUI app though that I'm not certain how to automate.... I'll read more on dcraw. Thanks.

[anthony]

This topic is extremely interesting, and I have extracted and refered to it from IM examples, Common Image Formats.

If anyone has more things to add or other findings, then I recommend they return to this topic and add it here.

[el_supremo]

Is it possible to create a jpg file containing all camera settings (white balance, temperature, sharpness, colours, ...) starting from a RAW file (.cr2)?

If by "contain" you mean just produce a JPG file having the same EXIF header as the raw file I think the answer is yes, although I haven't specifically tested that and you probably wouldn't use ImageMagick to do it.
But if you mean use ImageMagick to produce a jpg image from the raw file using all the original camera settings, then the answer is a definite no.
Although the dcraw program can use the original white balance information, it does not use the sharpness, contrast, saturation and other information that was used by the camera and nor does IM.
In my camera (Canon 10D) I can set the JPG compression to one of 6 values. In Imagemagick, JPG compression is controlled by specifying a "quality" which is a number from zero to 100, so there's no way to know how the six compression levels in the 10D correspond to a quality value in ImageMagick. The same type of problem applies to the contrast, saturation, sharpness and colour tone levels which can be set in the 10D. Each of these can be set to one of 5 levels which don't correspond to what IM uses.
After a lot of experimentation you might be able to get good approximations for these values for a specific camera but you would still have to write a script or program which reads the camera settings from the EXIF (or MakerNote?) header, convert these and then pass them on to an ImageMagick command.

Pete