Remove too dark photos from batch

[salmen]

Putting together a video of a construction site. I have a batch of photos all taken from the same exact point during a few years. Now I want to remove those that make the video too flickery. First that would be all taken at night. I guess

Code: Select all

identify -format "%[fx:mean]\n" filename

is useful. So I would need to move the files that are darker than a threshold.
Just started with IM and using commandline on Ubuntu. Any help appreciated!
Thanks

[fmw42]

Please always provide your IM version also. If you want to remove the dark ones, then you can write a shell script loop over each image, use your test but store the result as a variable. Then test that variable against a threshold and if it is too dark, then use the OS rm command to delete it from your system or mv to move it elsewhere. If you remove it, make a backup of your image folder.

[salmen]

Ok, thanks. "Version: ImageMagick 6.8.9-9 Q16 x86_64 2015-08-06 ". Now trying to write a shellscript

[salmen]

Obviously new to all of this.. Think I have a problem with the mean returning a float value which bash can not handle?

Code: Select all

#!/bin/bash
toodark=29000
avgluz=$(identify -format "%[mean]" file.dng)
echo $avgluz
if ((avgluz < toodark)); then
    echo "too damn dark!"
else
    echo "reasonable.."
fi

Output:

Code: Select all

29489.9
./test.sh: line 5: ((: 29489.9: syntax error: invalid arithmetic operator (error token is ".9")
reasonable..

[snibgo]

Shells are often bad at handling floating-point. Either use a calculator like bc, or do more calculation within IM, eg:

Code: Select all

identify -format "%[fx:mean>0.345?1:0]\n" filename

This returns 1 when the mean is greater than 0.345; otherwise it returns 0.

You may need to escape the ">".

[fmw42]

if ((avgluz < toodark)); then

try

Code: Select all

if [[$avgluz < $toodark]]; then

[salmen]

Thank you both! I belive I have got this part working. So I am able to sort out those that are too dark.
Any other tip to find images that cause a video to seem flickery? Quite alot of changes in weather.

[snibgo]

Can you put up some example frames?

How long were the intervals between frames? 5 minutes, one hour, or whatever?

Weather conditions may change the average brightness of photos, but will certainly change the amount of contrast: sunshine creates shadows. This can be smoothed out with automatic processing, but not entirely removed.

For example, compare two photos of the same subject, one in sunshine the other under overcast sky. The colour temperatures will be different, and they will vary across the photo. Why? Because areas in shade from the sun are illuminated by a blue sky, so take a blue cast. The same subject illuminated by an overcast sky will have a more neutral cast. In addition, local contrast of an object in shade from the sun, or under overcast sky, will be roughly the same but the contrast of the overall photos will be different.

You might find the technique on my "Gain and bias" page useful.

[salmen]

Thanks, will put up some frames shortly. The interval is 1 hour. I think I should do a heavy selection skipping lots of photos. Then see if its possible to make the remaining more similar by editing. Your "Gain and bias" looks interesting.

[salmen]

Two examples in the dropbox-links below (cant seemt to include images?). Since the photos are taken with such long interval there are alot of changes like this or more.
image1
image2

[snibgo]

Half the image is the sky, and half is the ground. When a cloud obscures the sun, the ground becomes darker but the sky won't change. So overall image statistics would be a compromise and we would still have heavy flickering.

Instead, we can use the statistics from just the bottom half of the images. For example, we can apply gain and bias to each image to give the bottom half of each a mean of 0.5 and standard deviation of 0.1667. Hence the brightness and contrast will match, in the lower halves.

I have named your images cons1.jpg and cons2.jpg.

Code: Select all

call %PICTBAT%imgMnSdLBot cons1.jpg 0.5 0.1667 cons_c1.png
call %PICTBAT%imgMnSdLBot cons2.jpg 0.5 0.1667 cons_c2.png

%IM%convert cons_c1.png -resize 600 -quality 40 cons_c1.jpg
%IM%convert cons_c2.png -resize 600 -quality 40 cons_c2.jpg

(I resize to low-quality jpegs just for speed of Web uploads.)



The ground of the two images is now similar, but the sky has become washed out in the second.

While the construction remains below the horizon, if the camera doesn't move, then we can use a mask to process the sky separately, or simply not process the sky at all.

I make a quick mask with Gimp, cons_mask.png:

Then I composite the ground over the original skies:

Code: Select all

%IM%convert ^
  cons1.jpg ^
  cons_c1.png ^
  cons_mask.png ^
  -composite ^
  cons_d1.png

%IM%convert ^
  cons2.jpg ^
  cons_c2.png ^
  cons_mask.png ^
  -composite ^
  cons_d2.png

%IM%convert cons_d1.png -resize 600 -quality 40 cons_d1.jpg
%IM%convert cons_d2.png -resize 600 -quality 40 cons_d2.jpg

The results are good.

The script %PICTBAT%imgMnSdLBot.bat is:

Code: Select all

rem From image %1,
rem applies gain and bias
rem so bottom 50% of image matches mean %2 and SD %3.
rem Output to %4.

set TMP_IMG=imsl_temp.miff
set TMP_BOT=imsl_bot.miff

%IM%convert ^
  %1 ^
  +write %TMP_IMG% ^
  -gravity South ^
  -crop 0x50%%+0+0 +repage ^
  %TMP_BOT%

call %PICTBAT%meanSdTr %TMP_BOT% igb1_

set igb2_mn_R=%2
set igb2_mn_G=%2
set igb2_mn_B=%2

set igb2_sd_R=%3
set igb2_sd_G=%3
set igb2_sd_B=%3

call %PICTBAT%calcGainBias igb1_ igb2_ igb_gb_

%IM%convert ^
  %TMP_IMG% ^
  -channel R -function Polynomial !igb_gb_gain_R!,!igb_gb_bias_R! ^
  -channel G -function Polynomial !igb_gb_gain_G!,!igb_gb_bias_G! ^
  -channel B -function Polynomial !igb_gb_gain_B!,!igb_gb_bias_B! ^
  +channel ^
  %4

This calls other scripts you will find on my Gain and Bias page.

[snibgo]

Incidentally, there is nothing magical about the number 0.1667 for the standard deviation. Some images look better with a different value. Your sample images look good at 0.2, or perhaps even 0.25 though that is too garish for my taste.

You might find, after assembling the movie, that you need to adjust numbers like this.

Increasing the standard deviation in sRGB space will usually also increase saturation. I think it is fine here, but you can decrease saturation if you want with "-modulate 100,75,100" or similar.

[salmen]

Magick! Never really thought about the statistics, but all seems logical, great explanation!
Will see if I can put something together tomorrow.

[snibgo]

Yes, we are tweaking both images so the bottom halves have matching statistics. My code matches only the mean and SD. I find these useful because they are readily understandable as brightness and contrast, and the tweaking is a simple linear adjustment with "level".

Other statistics are available, with more complex tweaking. We can easily adjust an image so its histogram matches the histogram of another, or change both to a common histogram. However, the result often looks "wrong".

[salmen]

This is great! But unfortunatly I am not much of a scripter. Especially the environment variables give me a headache.
Thought I'd start by just calculating the gain and bias for all channels and applying them to another photo. But dont know how to apply them?

Code: Select all

#!/bin/bash
mean1=$(identify -format "%[fx:mean]\n" file1.jpg)
mean2=$(identify -format "%[fx:mean]\n" file2.jpg)
sd1=$(identify -format "%[fx:standard_deviation]\n" file1.jpg)
sd2=$(identify -format "%[fx:standard_deviation]\n" file2.jpg)
gain=$(echo "$sd2/$sd1" | bc -l)
echo gain $gain
bias=$(echo "$mean2-$mean1*$gain" | bc -l)
echo bias $bias
#convert something I guess.. apply gain and bias to file2