Averaging (convert mean/median) 80000 16bit PNGs

[pinktank]

Hello All,
I'm trying to average a folder of 80000 16bit PNGS but running out of memory and harddrive space. The files are about 800GB. It also takes a million years. Is there a better way to do this with a running average or something? I'm on OSX

[fmw42]

You probably need to send your policy.xml file arguments to be able to process images using disk space if you are running out of RAM. Also you should do a running average. But that would require some scripting that is OS dependent. You would definitely run out of RAM trying to load all 80,000 PNGs. How much RAM do you have? What platform/OS? What version of ImageMagick?

See https://imagemagick.org/Usage/files/#massive

[pinktank]

32GB Ram, OSX High Sierra, ImageMagick 7.0.8-59 Q16 x86_64

I tried something like
find . -type f -name "*.png" | xargs -n 100 sh -c 'convert "$0" "$@" -evaluate-sequence mean outdir/"$0" '
but it gave the following error
convert: unable to open image 'outdir/./out0000001.png': No such file or directory @ error/blob.c/OpenBlob/3497.
convert: WriteBlob Failed `outdir/./out0000001.png' @ error/png.c/MagickPNGErrorHandler/1715.

[fmw42]

do you have outdir created? ImageMagick will not create a directory. But that is not going to do a running average because you have to divide by different numbers of images each time. see cumulative average at https://en.wikipedia.org/wiki/Moving_average and https://stackoverflow.com/questions/126 ... data-total and https://en.wikiversity.org/wiki/Moving_Average

For each new value:
new_average = (previous_average*previous_count + new_value) / (previous_count+1)

first_average = (0*0 + first_value)/(0+1) = first_value/1
second_average = (first_average*1 + second_value)/2
third_average = (second_average*2 + third_value)/3
etc

[pinktank]

Yeah I was just sharing it as another way to deal with memory issues since it would dump after each sequence.
My one concern with a moving average is that it is going to have a lot of rounding errors at 8000 16bit limited multiplications. Does Imagemagick carry that math out in 16bits or 64?

There are some tries here that I might look at.
https://www.imagemagick.org/discourse-s ... hp?t=33188
https://www.imagemagick.org/discourse-s ... hp?t=21279

[fmw42]

Computations are done to double precision (64-bits), I believe. But returning and storing in a variable may be limited. But you can use -precision to keep more decimal places if you use fx calculations. Or if you use bc, you can set the scale.

[snibgo]

There are many methods for calculating the mean image from a large set, eg all at once, or rolling average, or in batches of say 100, or some combination. How large are the images, in pixels? That determines how many will fit in memory at once.

I suggest you start work with small images, eg "-crop 1x1+0+0" from each image in your set. Then you can easily experiment with timings and precision. Finally, use your chosen method on the full-size inputs.

PNGs are only 16 bits/channel/pixel. "-evaluate-sequence" uses long double, which are 64-bits. I doubt that precision will be a problem.

[fmw42]

One way would be to use mogrify (see https://imagemagick.org/Usage/basics/#mogrify) with -scale 1x1! to average all your images to 1 pixel. The -scale does a simple average. See https://imagemagick.org/script/command- ... .php#scale . Then average all the 1 pixel results to get the complete average of the full set. If you need more precision in your average of each image to 1 pixel, then store the results as 1pixel float TIFFS (or MIFF or PFM) using an HDRI compiled ImageMagick. IM 7 is HDRI by default. See TIFF defines for floating point results at https://imagemagick.org/script/formats.php or https://imagemagick.org/script/command- ... php#define. See -define quantum:format=type

[snibgo]

I think the OP wants an average image, not a simple number. My suggestion for 1x1 images is only for development and testing.

[fmw42]

I think the OP wants an average image, not a simple number. My suggestion for 1x1 images is only for development and testing.

My misunderstanding. Thanks.