You might do double-blind tests. (Technically, maybe these are merely single-blind.)NicolasRobidoux wrote:(Hopefully I'm not suffering too badly from selection bias.)
When your script has created images, it assigns a random number to each, and renames each with its number. The script keeps a log. You view and compare only the numbered files.
If you have, say, ten images, each processed by algorithm A and algorithm B, you compare the numbered images pair-wise. Then you look at the log file. If you always favour one algorithm over another, you know you are on to something. If it is 50-50, you might then determine the strengths and weaknesses of each.