NEF to JPG conversion yielding dark output

[cassiopeborealis]

I am using ImageMagick version 6.9.3-0_2.
I have .NEF files that I want to convert to .jpg. I want to preserve the image quality 100%. I was able to convert the image however the jpg is much darker than the original image. I have tried most of the suggestions in the forum but none of them fixed my issue (they either make it darker or lighter but never the exact conversion).
I use the following command for conversion:

Code: Select all

convert DSC_0684.NEF -quality 100% DSC_0684.jpg

when i do :

Code: Select all

identify -verbose DSC_0684.NEF

this is what i get

Code: Select all

Base filename: DSC_0684.NEF
  Format: NEF (Nikon Digital SLR Camera Raw Image File)
  Class: DirectClass
  Geometry: 6036x4020+0+0
  Units: Undefined
  Type: TrueColor
  Endianess: Undefined
  Colorspace: sRGB
  Depth: 16-bit
  Channel depth:
    red: 16-bit
    green: 16-bit
    blue: 16-bit
  Channel statistics:
    Pixels: 24264720
    Red:
      min: 0 (0)
      max: 65535 (1)
      mean: 7863.33 (0.119987)
      standard deviation: 5914.13 (0.0902439)
      kurtosis: 17.6988
      skewness: 2.95783
      entropy: 0.8162
    Green:
      min: 0 (0)
      max: 65535 (1)
      mean: 9157.82 (0.139739)
      standard deviation: 4752.79 (0.0725229)
      kurtosis: 14.6816
      skewness: 2.36324
      entropy: 0.834942
    Blue:
      min: 0 (0)
      max: 65535 (1)
      mean: 8997.15 (0.137288)
      standard deviation: 4588.12 (0.0700102)
      kurtosis: 6.22786
      skewness: 1.09649
      entropy: 0.872226
  Image statistics:
    Overall:
      min: 0 (0)
      max: 65535 (1)
      mean: 8672.77 (0.132338)
      standard deviation: 5119.14 (0.0781131)
      kurtosis: 15.0886
      skewness: 2.33448
      entropy: 0.841123
  Rendering intent: Perceptual
  Gamma: 0.454545
  Chromaticity:
    red primary: (0.64,0.33)
    green primary: (0.3,0.6)
    blue primary: (0.15,0.06)
    white point: (0.3127,0.329)
  Background color: white
  Border color: srgb(223,223,223)
  Matte color: grey74
  Transparent color: black
  Interlace: None
  Intensity: Undefined
  Compose: Over
  Page geometry: 6036x4020+0+0
  Dispose: Undefined
  Iterations: 0
  Compression: Zip
  Orientation: TopLeft
  Properties:
    date:create: 2016-03-24T16:12:31-04:00
    date:modify: 2016-03-24T16:12:31-04:00
    dng:Aperture: F3.5
    dng:AspectRatio: 1.501493
    dng:ChannelMultipliers: 2.386719 1.000000 1.597656 1.000000
    dng:CreateID: 1
    dng:Crop: 0 0 6036 4020
    dng:EXIFSource: exiv2 0.25
    dng:FocalLength: 18.0 mm
    dng:FocalLength35: 27.0 mm
    dng:Green: 1.072
    dng:ISOSpeed: 6400
    dng:Lens: Nikon AF-S DX Nikkor 18-140mm f/3.5-5.6G ED VR
    dng:Make: Nikon
    dng:MatrixInputProfile: Color matrix
    dng:Model: D7100
    dng:Orientation: 0
    dng:Rotation: 0.000000
    dng:Shutter: 2 s
    dng:sRGBOutputProfile: sRGB
    dng:Temperature: 5073
    dng:Timestamp: Sun Dec 20 22:54:32 2015
    dng:WB: Camera WB
    dng:WBFineTuning: 0
    exif:Artist:                                     
    exif:CFAPattern: 2, 0, 2, 0, 0, 1, 1, 2
    exif:ColorSpace: 1
    exif:Contrast: 0
    exif:Copyright:                                                       
    exif:CustomRendered: 0
    exif:DateTime: 2015:12:20 22:54:32
    exif:DateTimeDigitized: 2015:12:20 22:54:32
    exif:DateTimeOriginal: 2015:12:20 22:54:32
    exif:DigitalZoomRatio: 1/1
    exif:ExifOffset: 416
    exif:ExposureBiasValue: 0/6
    exif:ExposureMode: 1
    exif:ExposureProgram: 1
    exif:ExposureTime: 20/10
    exif:FileSource: 3
    exif:Flash: 16
    exif:FNumber: 35/10
    exif:FocalLength: 180/10
    exif:FocalLengthIn35mmFilm: 27
    exif:GainControl: 2
    exif:GPSInfo: 4768
    exif:GPSVersionID: 2, 3, 0, 0
    exif:ISOSpeedRatings: 6400
    exif:LightSource: 9
    exif:Make: NIKON CORPORATION

[snibgo]

I was able to convert the image however the jpg is much darker than the original image.

No. You haven't seen the original image. You will have seen the original image that has been demosaiced, converted to sRGB and so on, possibly with many enhancements. You are comparing the IM conversion with some other conversion.

As your means are very low, around 13%, I suspect you are not getting a conversion to sRGB. When you convert with "-verbose", what does it say? Is the delegate dcraw or something else? Please paste the text here.

[cassiopeborealis]

I guess before preceding with any further discussion, I need to mention that I am not an expert in ImageMagick.

No. You haven't seen the original image.

I have bunch of raw NEF images that I can visualize with preview in Mac OS. I guess you are referring to the preview image not being the real version of the image. Correct?

When I use the following command

Code: Select all

convert -verbose DSC_0684.NEF -quality 100% DSC_0684.jpg

I get

Code: Select all

"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=/var/tmp/magick-478035MBlLUJn1rf6.png" "/var/tmp/magick-47803UedPmHfthzpK"
 Found a generic profile, type exif
/var/tmp/magick-478035MBlLUJn1rf6.png PNG 6036x4020 6036x4020+0+0 16-bit sRGB 130.7MB 0.000u 0:00.000
DSC_0684.NEF=>/var/tmp/magick-478035MBlLUJn1rf6.png NEF 6036x4020 6036x4020+0+0 16-bit sRGB 130.7MB 0.000u 0:00.000
DSC_0684.NEF=>DSC_0684.jpg NEF 6036x4020 6036x4020+0+0 16-bit sRGB 39.74MB 0.000u 0:00.000

Let me know if you need any more output.
Thanks

[snibgo]

I guess you are referring to the preview image not being the real version of the image.

The preview version is processed from the raw image. It might have lightness adjustment, contrast adjustment, increased saturation, increased sharpness, and anything else.

You problem sounds like this thread: viewtopic.php?f=3&t=6936&p=129802

Try:

Code: Select all

convert DSC_0684.NEF -set colorspace RGB -colorspace sRGB -quality 100% DSC_0684.jpg

Does the lightness look correct? It may not be identical to any other version that is processed from the raw image, but does it look reasonable?

[cassiopeborealis]

The converted image looks reasonable now. Few observations though:

Image is of the night sky so I do see a background noise in the regions that are supposed to be black (as if there are lot of tiny red-green-blue dots)
However the other conversion was showing pitch black for the night sky but hiding the features i am interested in.
The converted image is larger (47.4 MB) while the raw NEF was (33.7 MB)

So it is safe (no information is lost) so I can digitize the image for further analysis?

Is there also a method to add timestamps on the images (the info extracted by exiftool)?

Thanks

[snibgo]

...as if there are lot of tiny red-green-blue dots

That will be noise, which shows mostly in "black" areas. One of the common steps when processing raw images is noise reduction. It is very difficult to automatically distinguish noise from, say, faint stars.

NEF files are 12 or 14 bits/channel/pixel, but the tiff may be 16. And they use different compression algorithms, so file sizes will be different.

So it is safe (no information is lost) so I can digitize the image for further analysis?

Yes. For image processing, it is best to start with the purest (least processed) image available.

Is there also a method to add timestamps on the images (the info extracted by exiftool)?

I mostly use "-annotate". See http://www.imagemagick.org/script/comma ... p#annotate

EDIT: You mention "further analysis", but also JPG format. JPG compression shouldn't be used until you have a final result to show for the internet or whatever, because JPEG compression is lossy, which means it mangles the image.

[cassiopeborealis]

Thanks a lot for your response.
Do you recommend doing my analysis on tiff images?
I am more interested in analyzing moving features in the image. I have images of an event taken with 5s exposure for 2-3 minutes.
I want to trace the motion of a certain image pixel throughout this 2-3 min period. Do you also know a certain method to do that?

Also the quality of converted image from raw isn't too good and if I were to make a video out of these images I need to probably process them with some noise reduction and smoothing techniques. What would be your recommendation for that?

[snibgo]

Do you recommend doing my analysis on tiff images?

I recommend that you don't use JPEG compression at any stage in the image processing change. If you compress, always use lossless compression. If you use tiff, just make sure you are not compressing these with jpeg.

I want to trace the motion of a certain image pixel throughout this 2-3 min period. Do you also know a certain method to do that?

Just a single pixel? That sound difficult. What distinguishes this pixel from others? Is it a different colour or something? Perhaps it is the only thing that changes between exposures. If so, then it's easy: find the difference between any pair of images.

If you provide sample images, perhaps advice can be more specific.

... some noise reduction and smoothing techniques. What would be your recommendation for that?

For noise reduction (removal of spurious low-light detail), I use dcraw. But I don't do astrophotography.

For smoothing, ImageMagick has a raft of possibilities. "-statistic median" is commonly used.

[cassiopeborealis]

Thanks for your comments and suggestions.
I have forwarded you an example image through PM. I was wondering if you got them?

[snibgo]

You provided one image, and asked about "labelling few of these rays and observe their relative motion as the image frames advance."

I think you want to identify features on one image that will have moved (and possibly changed in other ways) between images. Without seeing other images, I don't know what to suggest.

You can put examples here in this this thread, if you want. Just paste the URLs.

[cassiopeborealis]

The image I shared with you was taken from twitter but what i have are very similar to that one.
The images are not mine and unfortunately can not share them here for copyright reasons.

I think you want to identify features on one image that will have moved (and possibly changed in other ways) between images

Yes that's correct. They move horizontally but the motion is not always in one direction. That's the reason why I need to trace one of them between the rest of the images to see the motion.

[snibgo]

A basic technique is to identify some feature in one frame, and search for it in the next frame. For example, you might find the lightest pixel in the frame. Crop that image to perhaps 20x20 pixels, centred on that pixel. Then do a "-subimage-search" of that 20x20 image in the next frame.

If it won't move by much between frames, you don't need to search the entire next frame. You might crop it 100x100, centred on the same coordinates, and search that small image.

[cassiopeborealis]

Thanks for the suggestion!