large number of tiff images and files
Posted: 2012-05-24T12:42:48-07:00
#!/bin/bash
#There are approx 10 tiff files with each image/scene/frame of 190x190 16bit-pixels. There are approx 30000 images per file.
#Plan: get center slice from each of the 30K images and concatenate all these slices into 1 image file. So will get 1 new image file per multi-image.tif file.
#pseudocode
#loop
# get multiimage.tif filename
# get number of images in file (nImages=identify...)
# loop 0,nImages
# make a filename with # to use for crop output
# get slice out of middle of current image and put in numbered file
# endloop - get next image in multiimage.tif
# Now have $nImages slice-files in tempDir and want to combine them all
# into 1 image file. There will be 1 new file per muti-image.tif file.
# Use convert to combine all ~30K slice-files into 1 new image file.
# Remove tempdir/* (all slice-files).
#endloop - get next multiimage.tif
mkdir "tempDir"
for file in imagedir/*.tif; do
nImages=`identify -ping -format %n $file`
fileonly=$(basename $file)
for (( i = 0 ; i < $nImages; i=i+1 )); do
orderFile=tempDir/$fileonly`printf "%06d" $i`.tif
convert -quiet -crop "1"x+"95" $file[$i] $orderFile
done
convert tempDir/* +append ${fileonly}_newfile_bydir.tif
rm -f tempDir/*
done
There are basically 3 commands. Can any of them be sped up?
1 is posted as bug report, 2 gets run 30K times to produce 30K files, 3 appends 30K imagefiles.
1) identify
2) convert -crop
3) convert *.tif +append
Here is an example tiff file with 29516 images in it (0-29515). It is 2.1GB ungzipped.
ftp://ftp.swri.org/pub/incoming/multi29 ... bit.tif.gz
I have 2 alternatives to 1) identify. tiffinfo requires installing another package. They are:
alt1)
#!/bin/bash
convert -quiet multi29515images16bit.tif[0] delme.tif
onesiz=`ls -ln delme.tif | awk '{print $5}'`
allsiz=`ls -lnH $file | awk '{print $5}'`
rm delme.tif
nguess=`expr $allsiz / $onesiz`
nlow=`expr $nguess / 1000 \* 1000`
nhigh=`expr $nlow + 1000`
ncount=`identify -format %n -quiet multi29515images16bit.tif[$nlow-$nhigh]`
nImages=`expr $nlow + $ncount`
nImages=`expr $nImages - 1`
alt2)
tiffinfo multi29515images16bit.tif 2>errs | grep TIFF | wc -l
convert -version
Version: ImageMagick 6.5.4-8 2009-10-24 Q16 OpenMP http://www.imagemagick.org
Copyright: Copyright (C) 1999-2009 ImageMagick Studio LLC
I have a speedup for 2) the convert section. It only works for i=i+1, that is, when doing ALL the images. If skipping images, not shown but would be something like for(;;i=i+$skip), then the code below would not be appropriate. Here is replacement which does 1000 images at a time.
On my computer, it takes 6.7 secs to do the 1st 1000 images and about 4minutes for all 29K instead of the 9+ hours of the original code.
for (( i = 0 ; i < $nImages; i=i+1000 )); do
j=`expr $i + 999`
convert -quiet -crop "1"x+"95" $file[$i-$j] tempDir/${fileonly}%06d.tif
done
#There are approx 10 tiff files with each image/scene/frame of 190x190 16bit-pixels. There are approx 30000 images per file.
#Plan: get center slice from each of the 30K images and concatenate all these slices into 1 image file. So will get 1 new image file per multi-image.tif file.
#pseudocode
#loop
# get multiimage.tif filename
# get number of images in file (nImages=identify...)
# loop 0,nImages
# make a filename with # to use for crop output
# get slice out of middle of current image and put in numbered file
# endloop - get next image in multiimage.tif
# Now have $nImages slice-files in tempDir and want to combine them all
# into 1 image file. There will be 1 new file per muti-image.tif file.
# Use convert to combine all ~30K slice-files into 1 new image file.
# Remove tempdir/* (all slice-files).
#endloop - get next multiimage.tif
mkdir "tempDir"
for file in imagedir/*.tif; do
nImages=`identify -ping -format %n $file`
fileonly=$(basename $file)
for (( i = 0 ; i < $nImages; i=i+1 )); do
orderFile=tempDir/$fileonly`printf "%06d" $i`.tif
convert -quiet -crop "1"x+"95" $file[$i] $orderFile
done
convert tempDir/* +append ${fileonly}_newfile_bydir.tif
rm -f tempDir/*
done
There are basically 3 commands. Can any of them be sped up?
1 is posted as bug report, 2 gets run 30K times to produce 30K files, 3 appends 30K imagefiles.
1) identify
2) convert -crop
3) convert *.tif +append
Here is an example tiff file with 29516 images in it (0-29515). It is 2.1GB ungzipped.
ftp://ftp.swri.org/pub/incoming/multi29 ... bit.tif.gz
I have 2 alternatives to 1) identify. tiffinfo requires installing another package. They are:
alt1)
#!/bin/bash
convert -quiet multi29515images16bit.tif[0] delme.tif
onesiz=`ls -ln delme.tif | awk '{print $5}'`
allsiz=`ls -lnH $file | awk '{print $5}'`
rm delme.tif
nguess=`expr $allsiz / $onesiz`
nlow=`expr $nguess / 1000 \* 1000`
nhigh=`expr $nlow + 1000`
ncount=`identify -format %n -quiet multi29515images16bit.tif[$nlow-$nhigh]`
nImages=`expr $nlow + $ncount`
nImages=`expr $nImages - 1`
alt2)
tiffinfo multi29515images16bit.tif 2>errs | grep TIFF | wc -l
convert -version
Version: ImageMagick 6.5.4-8 2009-10-24 Q16 OpenMP http://www.imagemagick.org
Copyright: Copyright (C) 1999-2009 ImageMagick Studio LLC
I have a speedup for 2) the convert section. It only works for i=i+1, that is, when doing ALL the images. If skipping images, not shown but would be something like for(;;i=i+$skip), then the code below would not be appropriate. Here is replacement which does 1000 images at a time.
On my computer, it takes 6.7 secs to do the 1st 1000 images and about 4minutes for all 29K instead of the 9+ hours of the original code.
for (( i = 0 ; i < $nImages; i=i+1000 )); do
j=`expr $i + 999`
convert -quiet -crop "1"x+"95" $file[$i-$j] tempDir/${fileonly}%06d.tif
done