[many images] Don't load all images at once?

[zebediah49]

I have (at the moment, I am still gathering more) 3000 5.5MPx images.

I want to average them.

Code: Select all

convert *.gif -average average.png

Works, ish. It works if I use a tiny subset of the images (60 of them), it takes 7 seconds and almost 3GB of memory

These images, even in full RGBA format, are only 21 MB each.. so I'm not sure where that overhead is coming from (60*21MB=1.2GB).

Either way, I'm just trying to average the images. This should be doable in less than 100MB, since with a running average one just needs a current image and a current sum image.

Can I tell imagemagick to do the average as it loads the images, or am I stuck using something else, writing something myself, or using some sort of dirty hack involving averaging sets, and then averaging the averages?

I am aware of the -limit options, but based on what I've seen thus far, the disk use required to use more than the ~8G I'm willing to allocate to this will take prohibitively long to do.

[magick]

You may want to use another utility. If you use ImageMagick, add '-limit memory 2mb -limit map 4mb' to your command line. The conversion will complete if you have enough temporary disk space.

[anthony]

You are asking IM to read all images in then average them.

I fyou don't want to do that, then you will need to average groups of equall number of images first, then averge the groups.
If you can not get equal number of images in each group, then you may need the added complication of weighting the 'odd sized' group.

An alturnative is to use HDRI, then simply add all the images together (in groups), and at the end divide by the total number of images.

Either solution may need some scripting to get working right,

The groupings in the first method could be done in parallel on different machines if speed becomes an issue.
Don't do parrellization on the same machine as IM is already threaded by default to take advantage of multi-core processors.

[fmw42]

If you write a script to loop over all your images, you can do a successive averaging using -compose Mathematics or better -compose blend. Unfortunately, I don't have time tonight to give you the correct formula for each successive combination. But I can try to do that tomorrow if you want me to follow up on that.

see
http://www.imagemagick.org/Usage/compose/#mathematics
http://www.imagemagick.org/Usage/compose/#blend

[fmw42]

Here is the unix script that will do the successive average.


cd yourimagedirectory
fileArr=(`ls`)
num=${#fileArr[*]}
convert ${fileArr[0]} tmp.mpc
echo "${fileArr[0]}"
i=1
while [ $i -lt $num ]; do
j=$((i+1))
new=`convert xc: -format "%[fx:100/$j]" info:`
old=`convert xc: -format "%[fx:100-$new]" info:`
composite -blend $old%x$new% tmp.mpc ${fileArr[$i]} tmp.mpc
i=$((i+1))
done
convert tmp.mpc tmp_ave.png
rm -f tmp.mpc
rm -f tmp.cache


For Windows, see http://www.imagemagick.org/Usage/windows/ (Sorry I am not a Windows user)

[anthony]

If you write a script to loop over all your images, you can do a successive averaging using -compose Mathematics or better -compose blend. Unfortunately, I don't have time tonight to give you the correct formula for each successive combination. But I can try to do that tomorrow if you want me to follow up on that.

I would be very careful with that method. At 5000 images, you may started to get quantum round of errors in your averaging.
At Q16 the integer size limit is 65535 so divisions is approaching 1/10 of the round off level.

You are still better off group averaging your images. say 100 images at a time to form 50 groups. That will avoid quantum effects. HDRI (floating point) is the other way to do this.

[fmw42]

If you don't want to do that, then you will need to average groups of equall number of images first, then averge the groups.
If you can not get equal number of images in each group, then you may need the added complication of weighting the 'odd sized' group.

This is very tricky to average groups. The for example 5 image (A, B, C, D, E) and average A+B, C+D+E

(A/2 + B/2) and (C/3 + D/3 + E/3) if averaged together with equal weights gives

A/4 + B/4 + C/6 + D/6 + E/6 which is incorrect as it should be A/5 + B/5 + C/5 + D/5 + E/5

The groups would have to have the same number of images in each average to make it work out easy (as Anthony points out above).

[anthony]

Equal numbers in each group is the best solution. But as I said, if one group did not have the same number of images that group would need a different weighting, based on the relative number of images in each group.
A 'blend' composition can be used to join two average images with un-equal image counts, you just need to work out the weightings, based on the number of images in each 'group average'.

Fred's 'incremental Average' actually does this, but requires some extreme weightings (which can cause quantum rounding effects). Basically each loop is merging an average of N images, with a average of 1 image (the original image).