Resizing artifacts with Lanczos

[NicolasRobidoux]

IMHO, "diagonal noise" is something which comes from using filters with negative lobes in a tensor (orthogonal) way: Basically, if l(x) is a function which alternates between positive and negative, l(x)l(y) will give you a checkerboard. With a (say) vertical line, you don't get the checkerboard, because it is averaged out vertically. Unfortunately, a translation parallel to a diagonal line maps the checkerboard to itself, which means that one of its two high frequencies is not averaged out somewhat like it is with a vertical (horizontal) line: They both are there, fully.

This shortcoming is exactly what averaging radially ("cylindrically") avoids.

[anthony]

They are so new that they've only been tested for enlarging, but I'm really excited about them!

Make sure you have the very latest IM before testing, you will get a very blurry result!

[anthony]

Warning: If you call these filters through -resize, you will not get their cylindrical versions. You need to call them through distort or -fx to get the desired effect.

Note that while -fx will use the same resampling functions as distort, It does NOT apply EWA filtering. It specifically turns off EWA filtering as it does not have any of the 'scaling' factors need for distortion/resize filtering. As such -fx only applies -interpolate settings.

-fx is often used to demonstrate distortions and resizing, but only for the pixel lookup aspect of the process. It does not provide any form of filtering beyond unscaled interpolation of the pixel lookup.

Apologies to Nicholas if I implied that it did.

[NicolasRobidoux]

Apologies to Nicholas if I implied that it did.

Thanks for the heads up (I get easily confused, so no apologies needed).

[adx]

Wow. It'll take me some time to get my head around all this. And I thought I was fairly well obsessed with this sort of stuff (but very part time, I shouldn't really be doing it).

Building IM under Windows to test out the new filtering is a bit beyond me, so I've made myself a nice new Ubuntu VM to work in, might take a while to get round to though.

I hadn't given the "diagonal noise" issue much more thought than thinking it might be because there's more energy left to see in the higher diagonal frequencies after downsampling a 'noisy' image. I don't yet know if it applies to the (semi) square reconstruction from an LCD at 1:1 (which should tend to compensate?) because I only really notice it in upsized images. I remember the look well from some time in the 90s when Fuji photo prints went digital - the picky lab people thought I was a bit mad but could eventually see it and didn't know what to do about it. (I now know it's because they scan film at half resolution to print 6*4s, if you request a full res CD then the Frontier prints come out perfect...) Anyway my idea was to try filtering with a proper 2D radial kernel slightly larger (as in higher frequency) than normal to avoid wasting the corners of the frequency response while allowing some aliasing energy to spill into the result, or some variation on that theme.

Another idea I've wondered about (and half tried) is using an elliptical shaped frequency response (I don't think anything to do with EWA) to better utilise the corners, while increasing the horizontal frequency limit to take advantage of extra horizontal resolution on an LCD from the RGB subpixels (needs thought given to any jpeg chroma subsampling, and only display on the correct orientation of LCD). I found it sort of works but kind of doesn't because RGB is really RG with a gap where blue is(n't), so behaves almost like a vertical stripe unless it is strongly blue coloured (in which case it can be shifted halfway left or right of centre). Getting carried away.

I also noticed that in the original post, the ringing is (unsurprisingly) more visible inside the A for the nonlinear resize and outside the A for linear. I just calibrated my monitor to get a better look - LCDs (mine anyway) clip the whites badly which may be swaying the result. Perhaps we're more used to "inside ringing" on still images which are traditionally resized nonlinearly, and "outside ringing" on video sources which have a good 50 years maturity over digital still imaging and where gamma is the first lesson you learn. Something else I am yet to try is to compromise between linear and nonlinear space - it would be particularly nice if a power of 2 worked better than 2.2 for example.

But I kind of can't see the justification for high-lobed filters, especially when downsizing. There is nothing in the original image capture process that spreads information that far apart (in a downsampling there are a lot of filter taps even for a 1 lobe filter thus a multi-lobe filter covers a lot of pixels), so that type of filtering has no relevance to the original image information (Nyquist bandlimiting and aliasing are both destructive to information so it can't be argued there is any "correct" approach) - and beyond the optical blur that our eyes are used to seeing, no relevance to what's going on in there either. So I think extreme brick-wall filtering is more of a technical curiosity. Sound is diffrent, because our ears do a highly linear mechanical frequency domain transformation (and even then people get uppity about microseconds of ringing). I don't frequent these sorts of forums though so I'm sure this has all been argued out before!

Until I get some time, here's a bit more on some of this in my post here:
http://meesoft.logicnet.dk/support/view ... p?pid=4827

[NicolasRobidoux]

Anyway my idea was to try filtering with a proper 2D radial kernel slightly larger (as in higher frequency) than normal to avoid wasting the corners of the frequency response while allowing some aliasing energy to spill into the result, or some variation on that theme.

'Sounds an awful lot to me like distort Lanczos and the other filters tuned for EWA used with -define filter:blur < 1.

[NicolasRobidoux]

Maybe you'll find the following interesting.

(Don Munsil is a video processing expert of some notoriety.)

Some Don Munsil quotes from forum discussions with other experts:

From http://www.freelists.org/post/opendtv/M ... ampling,12:

> The electronic sinc-like low-pass filter can have a sharper
> transition band than the optical gaussian filter, and this
> implies that it will create ringing. This ringing contributes
> to the sharpness impression, but it is unnatural too...

Overshoot is unnatural, but if one only uses a single negative lobe (on a
sinc or sinc-like filter) and the over- and preshoot doesn't get too deep it
contributes to the perception of edge sharpness. Certainly a completely
unsharpened image looks soft to most people.

> That is not a good filter for down-sampling ! You would need
> at least a 12-taps polyphase FIR filter for decent performance.
> You could use a transposed polyphase filter with fewer taps
> (between 4 and , see our patent US5892695. The bi-cubic is
> not good enough for transposing, unless you apply some further
> tricks. Better to use a polyphase filter with lookup tables.

Well, I beg to differ on the first point. If you like, you could use a
two-lobe windowed sinc, but for all intents and purposes, Catmull-Rom is
equivalent to two-lobe Lanczos-windowed sinc. The two curves overlay almost
perfectly. Yes, if you use a bicubic for downsampling you are venturing
outside the original interpolation math because Catmull-Rom was not intended
to be a low-pass filter, but it does work fine. I just use it because it's
equivalent in performance to a windowed sinc but is faster to calculate.
Given that the calculation of the filter kernels is a tiny portion of the
whole scaling operation, it's a false economy, but I just don't see any
reason to switch to windowed sinc.

And yes, I agree completely that you need quite a few taps and a polyphase
filter to get acceptable results. The current scaler I'm working on uses as
many taps and phases as are mathematically necessary, as it's not intended
to be real-time.

> Probably, but it is always a trade-off between sharpness and
> (aliasing) artefacts. Sometimes less sharpness gives better
> pictures. I find that one of the potential attractions of
> HDTV: to give a bit more sharpness for a lot less artefacts.

Agreed.

From http://www.freelists.org/post/opendtv/M ... ampling,15:

From: "Jeroen Stessen" <jeroen.stessen@xxxxxxxxxxx>
> You can't judge a filter like that... You should judge the
> flatness of the passband and the deepness of the stopband,
> which can only be done in the frequency domain. Even a tiny
> variation in the time or place domain can give a huge
> difference in the stopband. A good filter should be designed
> by looking in the frequency domain in the first place. By just
> windowing some "perfect" filter in the time domain, especially
> if the window length is too short, you get inferior filters.

I disagree almost completely. I don't want to start a huge argument, but I
feel pretty strongly about this, having spent an inordinately long time
researching and implementing an image scaling engine.

Evaluating image scaling filters in the frequency domain is basically
worthless. I mean that without any hyperbole at all - the frequency response
of a digital image resizing filter does not predict in any meaningful way
how well it performs, either aesthetically or as a model of how real-world
image gathering works.

If we imagine scaling to be a physical simulation of swapping one image
sensor for another image sensor with more or fewer pixels, it's clear that
any interpolation or resampling operation is not in any way modeling this.
Fundamental to any resampling mathematics is the treatment of a pixel as a
point sample, and pixels are clearly not point samples. (There are a few
researchers who have looked at iterative methods of constructing models of
likely distributions of photons across pixel sites, and then "repacking"
them into new rectangles representing the new virtual CCD sensors, but such
methods are (so far) computationally expensive and don't necessarily produce
any more pleasing results than traditional interpolation methods.)

That's not to say that interpolation and resampling strategies are worthless
for scaling; they work quite well. It's just that we shouldn't imagine that
they represent some kind of "correct" interpretation of image scaling.

Moreover, the filters with the very best frequency response are absolutely
terrible for scaling, and the better the frequency response the worse the
performance. One of the primary artifacts of very long filters is excessive
ringing, and the longer the filter the further the ringing extends from an
edge. This ringing is not a real-world image phenomenon. You don't start to
see little halos around objects as you move closer to them or further away.

The primary reason frequency response is irrelevant to digital scaling is
because we do not evaluate images in frequency space. The eye measures light
amplitude directly at randomly scattered sites on the retina. There is no
direct measurement of "spacial frequency". This is in contrast to the ear,
which is a physical frequency analyzer. Thus for audio, frequency response
of resampling filters is essential, and measuring the frequency response
tells you how well the filter will perform. For images this is just not
true.

There is a pervasive assumption in image processing (implicit in many of the
standard texts) that image resampling is essentially identical to audio
resampling, just in two dimensions instead of one. It's not.

In practice, a good video scaling filter will have a relatively
decent-looking frequency response, but that's a side effect, not a goal.
Improving the frequency response will more than likely reduce the final
image quality, not improve it.

Needless to say, not every one agrees with the "frequency response does not matter, exact filter details don't matter, just, mostly, the general shape and
zero crossings," but somehow this seems to have some connection to Mitchell doing well for EWA (just like Don Munsil likes Catrom for orthogonal downsampling) even though it has nothing to do with cylindrical filtering.

More or less the same opinions were expressed here, without the benefit of strong/well educated dissenting opinions:
http://archive2.avsforum.com/avs-vb/arc ... 60922.html

Now, a personal comment: Every time I read that "Sinc is the ideal filter" in the context of image resampling, my grumpiness meter goes up one notch. I gather that, minus grumpy, you agree.

[NicolasRobidoux]

One last comment: Filters without negative lobes ("monotone" methods) will not be affected as much by color profile shenanigans.

[NicolasRobidoux]

I hadn't given the "diagonal noise" issue much more thought than thinking it might be because there's more energy left to see in the higher diagonal frequencies after downsampling a 'noisy' image.

(To quote a recent Disney movie) It's not noise, it's artifacts (assuming we are talking about the same thing).

Enlarging a "perfectly sharp" dark grey line on a light grey background should show more "diagonal noise" than doing the same with a vertical one. (Kinda sounding like Yogi Berra just now!)

Yogi Berra quote of the day:

In theory there is no difference between theory and practice. In practice there is.