Deleting blanks pages

[ylafont]

Code: Select all

time compare -metric rmse -subimage-search -dissimilarity-threshold 1 -similarity-threshold 0.98 white1.png \( -size 1x1 xc:black \) null:
64224 (0.979995) @ 10,10
real    0m16.691s
user    0m15.020s
sys     0m0.620s

thought I have posted those, sorry, however not sure it that will help. was running some test, and they all varied, out4.tif took over 5 minutes on another test as well as other pages. there is a results.txt file listing the results.

Code: Select all

/home/pi/tmp/TestScans/out1.tif
0 (0) @ 7,0
real    2m17.109s
user    0m26.180s
sys     0m6.750s

/home/pi/tmp/TestScans/out3.tif
0 (0) @ 1,0
real    6m18.873s
user    0m27.030s
sys     0m7.450s

/home/pi/tmp/TestScans/out4.tif
0 (0) @ 11,0
real    6m28.050s
user    0m25.990s
sys     0m8.550s

/home/pi/tmp/TestScans/out6.tif
0 (0) @ 10,0
real    1m58.936s
user    0m26.540s
sys     0m6.740s

Code: Select all

pi@PiScanner ~/tmp/TestScans $ convert -version
Version: ImageMagick 6.8.9-10 Q16 armv6l 2014-11-06 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2014 ImageMagick Studio LLC
Features: DPC OpenMP
Delegates: bzlib djvu fontconfig freetype jbig jng jpeg lcms lqr openexr png tiff x xml zlib

Link to TestScan Folder
https://www.dropbox.com/sh/dnarphzdop9u ... oqtma?dl=0

Was hoping the pi could handle this, did not want to move processing over to a another system. Not sure what can one expect for $35, but if i would have to guess a copier would have something that is less powerful to handle stufff like this.

[fmw42]

From what I can see none of your images are perfectly white. Some have regions of dark on the left side. So the compare technique is not going to work well, if there is even one dark pixel below gray(98%).

I think snibgo's suggestion of use the mean is likely going to be the best. You just need to find some threshold that works.

Here are some times on my Mac Mini using one thread and IM 6.8.9.10 Q16

# black border and some small amount of text
time convert out1.tif -format "%[fx:100*mean]" info:
98.8824
real 0m2.312s
user 0m0.771s
sys 0m0.088s


# nearly all white but some dark on the left side
time convert out2.tiff -format "%[fx:100*mean]" info:
99.9365
real 0m0.847s
user 0m0.767s
sys 0m0.069s


# full text
time convert out3.tif -format "%[fx:100*mean]" info:
92.6911
real 0m0.845s
user 0m0.763s
sys 0m0.072s


So you just need to decide what threshold you would consider mostly white.

The times are very reasonable.

[snibgo]

Looking at the means: the lightest odd-numbered page is out1.tiff, at mean=0.988824. The darkest even-numbered page is out6.tiff, at mean=0.999451.

So setting the threshold at 0.99 seems reasonable.

[fmw42]

You must have it backwards. A lighter image will have a higher mean value than a darker image.

I am not quite following the significance of even and odd pages.

[snibgo]

In these images, the odd-numbered pages (out1, out3, out5, ...) have text. Their mean values vary from 0.926911 to 0.988824. So the lightest page that has text has mean 0.988824.

The even-numbered pages (out2, out4, out6, ...) have no text. They are nominally blank. Their mean value vary from 0.999451 to 0.999878. Hence the darkest page that has no text has mean 0.999451.

So a threshold mean for determining whether a page is blank should be between 0.988824 and 0.999451, eg 0.99.

[EDIT: I wrote "0.999878" instead of "0.999451".]

[ylafont]

You must have it backwards. A lighter image will have a higher mean value than a darker image.

I am not quite following the significance of even and odd pages.

The scanner was set to scan in Duplex Mode Via the ADF, Hence there is blank page on every other page.


Fixing the Libtiff issure broke a few other things which i have been trying to fix. Will be trying out the mean value today i hope to see if it is quick.

[ylafont]

In these images, the odd-numbered pages (out1, out3, out5, ...) have text. Their mean values vary from 0.926911 to 0.988824. So the lightest page that has text has mean 0.988824.

The even-numbered pages (out2, out4, out6, ...) have no text. They are nominally blank. Their mean value vary from 0.999451 to 0.999878. Hence the darkest page that has no text has mean 0.999451.

So a threshold mean for determining whether a page is blank should be between 0.988824 and 0.999451, eg 0.99.

[EDIT: I wrote "0.999878" instead of "0.999451".]

Snibgo, Can you give the command you used to get the mean value?

[snibgo]

Code: Select all

convert out1.tif -format "%%[fx:mean]" info:

I did this in a Windows BAT script to get the means of all the images. Then I sorted them in order of the means.

Code: Select all

del testScans.lis

for %%F in (*.tif) do (
  echo %%F

  %IM%convert %%F -format "%%[fx:mean], %%f\n" info: >>testScans.lis
)

type testScans.lis

cSort /itestScans.lis /otestScans.srt /k0

type testScans.srt

cSort is my own unpublished program, but the job could be done with just Windows tools.

[ylafont]

Thanks, I already have a bash script to go through each of the pages. I got excited at first, results were coming back in few seconds until i realized the command only has one "%" instead of two in linux.


Mean method Took a consistent 13 seconds per page, which is better than the first method tried. I think that may be as fast as the Pi is going to get. not great, but will do for now. As soon as i get everything working, i will have to test on a faster machine to see the speed improvements. What were your results?

First method.

Code: Select all

pi@PiScanner ~/tmp/TestScans $ sudo ./premove.sh
/home/pi/tmp/TestScans
Total number of Scans is = 31
Pageinfo is  = /home/pi/tmp/TestScans/out10.tif  0 (0) @ 13,0 real 0m31.006s user 0m25.490s sys 0m1.390s
Pageinfo is  = /home/pi/tmp/TestScans/out11.tif  0 (0) @ 46,31 real 2m0.260s user 1m50.700s sys 0m4.330s
Pageinfo is  = /home/pi/tmp/TestScans/out12.tif  0 (0) @ 11,0 real 0m29.331s user 0m25.600s sys 0m1.460s
Pageinfo is  = /home/pi/tmp/TestScans/out13.tif  0 (0) @ 45,30 real 1m55.371s user 1m47.080s sys 0m4.020s
Pageinfo is  = /home/pi/tmp/TestScans/out14.tif  0 (0) @ 13,0 real 0m28.433s user 0m25.260s sys 0m1.380s
Pageinfo is  = /home/pi/tmp/TestScans/out15.tif  0 (0) @ 1020,19 real 1m24.192s user 1m18.430s sys 0m3.150s
Pageinfo is  = /home/pi/tmp/TestScans/out16.tif  0 (0) @ 17,0 real 0m28.522s user 0m25.500s sys 0m1.410s
Pageinfo is  = /home/pi/tmp/TestScans/out17.tif  0 (0) @ 44,24 real 1m39.214s user 1m32.520s sys 0m3.440s
Pageinfo is  = /home/pi/tmp/TestScans/out18.tif  0 (0) @ 9,1 real 0m31.169s user 0m28.220s sys 0m1.450s
Pageinfo is  = /home/pi/tmp/TestScans/out19.tif  0 (0) @ 43,28 real 1m48.548s user 1m42.060s sys 0m3.910s
Pageinfo is  = /home/pi/tmp/TestScans/out1.tif  0 (0) @ 7,0 real 0m27.968s user 0m25.300s sys 0m1.410s
Pageinfo is  = /home/pi/tmp/TestScans/out20.tif  0 (0) @ 13,0 real 0m28.206s user 0m25.570s sys 0m1.120s
Pageinfo is  = /home/pi/tmp/TestScans/out21.tif  0 (0) @ 44,30 real 1m52.480s user 1m45.740s sys 0m3.920s
Pageinfo is  = /home/pi/tmp/TestScans/out22.tif  0 (0) @ 17,0 real 0m27.939s user 0m25.110s sys 0m1.520s
Pageinfo is  = /home/pi/tmp/TestScans/out23.tif  0 (0) @ 43,39 real 2m19.050s user 2m11.230s sys 0m4.700s
Pageinfo is  = /home/pi/tmp/TestScans/out24.tif  0 (0) @ 14,0 real 0m27.869s user 0m25.410s sys 0m1.320s
Pageinfo is  = /home/pi/tmp/TestScans/out25.tif  0 (0) @ 45,37 real 2m14.986s user 2m6.510s sys 0m4.960s
Pageinfo is  = /home/pi/tmp/TestScans/out26.tif  0 (0) @ 12,0 real 0m28.287s user 0m25.280s sys 0m1.360s
Pageinfo is  = /home/pi/tmp/TestScans/out27.tif  0 (0) @ 47,36 real 2m12.562s user 2m5.520s sys 0m4.070s
Pageinfo is  = /home/pi/tmp/TestScans/out28.tif  0 (0) @ 5,0 real 0m27.976s user 0m25.590s sys 0m1.310s
Pageinfo is  = /home/pi/tmp/TestScans/out29.tif  0 (0) @ 54,28 real 1m48.675s user 1m41.980s sys 0m3.980s
Pageinfo is  = /home/pi/tmp/TestScans/out30.tif  0 (0) @ 5,0 real 0m30.793s user 0m27.970s sys 0m1.410s
Pageinfo is  = /home/pi/tmp/TestScans/out31.tif  0 (0) @ 274,0 real 0m28.867s user 0m26.130s sys 0m1.470s
Pageinfo is  = /home/pi/tmp/TestScans/out32.tif  0 (0) @ 2,0 real 0m27.887s user 0m24.950s sys 0m1.630s
Pageinfo is  = /home/pi/tmp/TestScans/out3.tif  0 (0) @ 1,0 real 0m27.996s user 0m25.660s sys 0m1.250s
Pageinfo is  = /home/pi/tmp/TestScans/out4.tif  0 (0) @ 11,0 real 0m30.246s user 0m27.920s sys 0m1.180s
Pageinfo is  = /home/pi/tmp/TestScans/out5.tif  0 (0) @ 49,39 real 2m18.218s user 2m10.410s sys 0m4.560s
Pageinfo is  = /home/pi/tmp/TestScans/out6.tif  0 (0) @ 10,0 real 0m28.019s user 0m25.180s sys 0m1.520s
Pageinfo is  = /home/pi/tmp/TestScans/out7.tif  0 (0) @ 2536,9 real 0m56.842s user 0m52.270s sys 0m2.240s
Pageinfo is  = /home/pi/tmp/TestScans/out8.tif  0 (0) @ 7,0 real 0m27.974s user 0m25.430s sys 0m1.250s
Pageinfo is  = /home/pi/tmp/TestScans/out9.tif  0 (0) @ 47,34 real 2m4.789s user 1m57.170s sys 0m4.360s

Mean Values

Code: Select all

pi@PiScanner ~/tmp/TestScans $ sudo ./premove.sh
/home/pi/tmp/TestScans
Total number of Scans is = 31
Pageinfo is  = /home/pi/tmp/TestScans/out10.tif  0.999597 real 0m13.760s user 0m13.210s sys 0m0.300s
Pageinfo is  = /home/pi/tmp/TestScans/out11.tif  0.95953 real 0m13.721s user 0m13.190s sys 0m0.290s
Pageinfo is  = /home/pi/tmp/TestScans/out12.tif  0.999742 real 0m13.749s user 0m13.270s sys 0m0.240s
Pageinfo is  = /home/pi/tmp/TestScans/out13.tif  0.936339 real 0m13.700s user 0m13.120s sys 0m0.340s
Pageinfo is  = /home/pi/tmp/TestScans/out14.tif  0.999759 real 0m13.771s user 0m13.260s sys 0m0.250s
Pageinfo is  = /home/pi/tmp/TestScans/out15.tif  0.934371 real 0m13.702s user 0m13.130s sys 0m0.330s
Pageinfo is  = /home/pi/tmp/TestScans/out16.tif  0.99982 real 0m13.753s user 0m13.220s sys 0m0.290s
Pageinfo is  = /home/pi/tmp/TestScans/out17.tif  0.964442 real 0m13.719s user 0m13.170s sys 0m0.310s
Pageinfo is  = /home/pi/tmp/TestScans/out18.tif  0.999878 real 0m13.740s user 0m13.200s sys 0m0.310s
Pageinfo is  = /home/pi/tmp/TestScans/out19.tif  0.951414 real 0m13.711s user 0m13.190s sys 0m0.280s
Pageinfo is  = /home/pi/tmp/TestScans/out1.tif  0.988824 real 0m13.743s user 0m13.160s sys 0m0.340s
Pageinfo is  = /home/pi/tmp/TestScans/out20.tif  0.99984 real 0m13.723s user 0m13.190s sys 0m0.310s
Pageinfo is  = /home/pi/tmp/TestScans/out21.tif  0.953579 real 0m13.716s user 0m13.190s sys 0m0.290s
Pageinfo is  = /home/pi/tmp/TestScans/out22.tif  0.999805 real 0m13.758s user 0m13.240s sys 0m0.280s
Pageinfo is  = /home/pi/tmp/TestScans/out23.tif  0.957244 real 0m13.725s user 0m13.200s sys 0m0.290s
Pageinfo is  = /home/pi/tmp/TestScans/out24.tif  0.999565 real 0m13.729s user 0m13.270s sys 0m0.230s
Pageinfo is  = /home/pi/tmp/TestScans/out25.tif  0.962744 real 0m13.721s user 0m13.200s sys 0m0.280s
Pageinfo is  = /home/pi/tmp/TestScans/out26.tif  0.999565 real 0m13.752s user 0m13.160s sys 0m0.350s
Pageinfo is  = /home/pi/tmp/TestScans/out27.tif  0.944369 real 0m13.710s user 0m13.200s sys 0m0.270s
Pageinfo is  = /home/pi/tmp/TestScans/out28.tif  0.999681 real 0m13.732s user 0m13.200s sys 0m0.310s
Pageinfo is  = /home/pi/tmp/TestScans/out29.tif  0.940447 real 0m13.704s user 0m13.200s sys 0m0.270s
Pageinfo is  = /home/pi/tmp/TestScans/out30.tif  0.999541 real 0m13.753s user 0m13.220s sys 0m0.290s
Pageinfo is  = /home/pi/tmp/TestScans/out31.tif  0.92885 real 0m13.701s user 0m13.260s sys 0m0.200s
Pageinfo is  = /home/pi/tmp/TestScans/out32.tif  0.999799 real 0m13.732s user 0m13.240s sys 0m0.270s
Pageinfo is  = /home/pi/tmp/TestScans/out3.tif  0.926911 real 0m13.699s user 0m13.200s sys 0m0.260s
Pageinfo is  = /home/pi/tmp/TestScans/out4.tif  0.999452 real 0m13.747s user 0m13.220s sys 0m0.280s
Pageinfo is  = /home/pi/tmp/TestScans/out5.tif  0.964521 real 0m13.698s user 0m13.230s sys 0m0.240s
Pageinfo is  = /home/pi/tmp/TestScans/out6.tif  0.999451 real 0m13.753s user 0m13.280s sys 0m0.230s
Pageinfo is  = /home/pi/tmp/TestScans/out7.tif  0.970369 real 0m13.732s user 0m13.210s sys 0m0.280s
Pageinfo is  = /home/pi/tmp/TestScans/out8.tif  0.999734 real 0m13.760s user 0m13.290s sys 0m0.230s
Pageinfo is  = /home/pi/tmp/TestScans/out9.tif  0.982718 real 0m13.714s user 0m13.240s sys 0m0.250s
pi@PiScanner ~/tmp/TestScans $

[snibgo]

My results seem the same as yours:

Code: Select all

0.926911, out3.tif
0.92885, out31.tif
0.934371, out15.tif
0.936339, out13.tif
0.940447, out29.tif
0.944369, out27.tif
0.951414, out19.tif
0.953579, out21.tif
0.957244, out23.tif
0.95953, out11.tif
0.962744, out25.tif
0.964442, out17.tif
0.964521, out5.tif
0.970369, out7.tif
0.982718, out9.tif
0.988824, out1.tif
0.999451, out6.tif
0.999452, out4.tif
0.999541, out30.tif
0.999565, out24.tif
0.999565, out26.tif
0.999597, out10.tif
0.999681, out28.tif
0.999734, out8.tif
0.999742, out12.tif
0.999759, out14.tif
0.999799, out32.tif
0.999805, out22.tif
0.99982, out16.tif
0.99984, out20.tif
0.999878, out18.tif

[ylafont]

My results seem the same as yours:

Code: Select all

0.926911, out3.tif
0.92885, out31.tif
0.934371, out15.tif
0.936339, out13.tif
0.940447, out29.tif
0.944369, out27.tif
0.951414, out19.tif
0.953579, out21.tif
0.957244, out23.tif
0.95953, out11.tif
0.962744, out25.tif
0.964442, out17.tif
0.964521, out5.tif
0.970369, out7.tif
0.982718, out9.tif
0.988824, out1.tif
0.999451, out6.tif
0.999452, out4.tif
0.999541, out30.tif
0.999565, out24.tif
0.999565, out26.tif
0.999597, out10.tif
0.999681, out28.tif
0.999734, out8.tif
0.999742, out12.tif
0.999759, out14.tif
0.999799, out32.tif
0.999805, out22.tif
0.99982, out16.tif
0.99984, out20.tif
0.999878, out18.tif

I was curious to know the time as well. just to use it as a reference point. thank you for all your assistance!

[snibgo]

It takes a total elapsed time of 11 seconds, including sorting the output. (The sort takes almost zero time). So about 0.34 seconds per image.

This is on a Windows 8.1 laptop, Intel core i7, which has 4 processors each with 2 cores, or perhaps 2 processors each with 4 cores, or something.

[ylafont]

about 0.34 seconds per image.

HUGE Difference! compared to my 13 seconds per page on the pi!

[snibgo]

Yup, my laptop does the whole batch of 32 before your Pi has done the first. The laptop probably weighs 32 times as much as the Pi and costs 32 times as much.

Perhaps if you increase the voltage to the Pi by 32 times it will go 32 times faster? (Only kidding -- don't do this!)

[ylafont]

Perhaps if you increase the voltage to the Pi by 32 times it will go 32 times faster? (Only kidding -- don't do this!)

No, but i have not played with the overclocking on it yet. It may help!