henrywho wrote:But you cannot avoid JPEG compression in the real world.
What the test suite does is pretty simple minded (but lots of people do it): Reduce a "gold standard" image, re-enlarge it, and compare. Although many scholarly articles treat such reconstruction tests as being conclusive, they are not. However, they
are informative. The rotation tests are also not really conclusive, but they are also informative. (We did not implement those.)
Although the suite (which will be called EXQUIRES: Evaluative eXtensible QUantitative Re-Enlargement Suite) will be fully configurable (you can plug in metrics, test images, upsamplers, downsamplers), I had to make judgement calls as to what "plain vanilla" should be, as well as which questions we'd try to answer in Adam Turcotte's thesis. One important judgement call is that I decided to only use the image geometry convention that ImageMagick uses: align corners.
I decided against JPEG "gold standard" images because I do not want upsamplers to be rewarded for reproducing JPEG artifacts. Of course this could be said of demosaicing artifacts and whatever noise is left in the gold standard, as well as the artifacts that come from mild box downsampling, but again, I had to make reasonable judgement calls to get things moving. (The clock is ticking...)
What would be interesting would be to downsample "gold standard" images (not JPEG), compress them with JPEG, re-enlarge to uncompressed TIFF (or PNG), and compare. What this would measure is the ability of the upsampler to "correct" JPEG artifacts, or at least robustly handle them. Alternatively, downsample a JPEG compressed version of the gold standard, re-enlarge the result, and compare to the gold standard. This would, to an extent, mimick re-enlarging an image downsized from that comes out of common digital camera toolchains.
Similarly, one could add noise to the downsampled images.
Manana.
But not necessarily by us: The PyPi suite will (hopefully soon) be packaged as a DEB, so it should install and configure painlessly on Debian, Ubuntu, and Mint.
By popular demand, Adam may add other OSes. Our hope is that it's going to become the standard software whenever a scholar decides to do re-enlargement tests.
If someone would like to be an alpha tester, contact me.