Legacy ImageMagick Discussions Archive

Hi!

I was testing how LCD sub-pixel awareness could enhance images when they are resampled to smaller sizes. This is like ClearType, but for photos. For that effect I used the following commands, you might find useful.

For normal resizing:
And considering the LCD as a silkscreen of RGB triplets with triple the horizontal resolution:
This results in ugly aliasing in some images, due to the "Point" sampling. There you should either use some horizontal blur, changing the first matrix into '0.25, 0.5, 0.25', or use Mitchell instead of Point. The results are less sharp, but more tempered so to say. The test image Rings1.jpg renders beautifully with a full Mitchell pipeline, and delivers a ugly rainbow otherwise.

Normal photos seem to work better with Point, or Point with a bit of horizontal blurring.

Here is a sub-pixel-resize example, with Point and no blurring ...



... and the same, with normal Mitchell resize.



The difference is subtle, but if you put each image in a browser tab and switch between them it should be apparent. The sub-pixel resampled shows more detail.

Of course, it only works on LCDs (or Plasmas) when 1 pixel in the image equals 1 pixel in the screen, that is, only at 1:1. Otherwise the sub-pixel sampled image is actually worse.

This has been discussed before in the forums (a long time ago!)

IM does not currently understand or use sub-pixel LCD rendering. It is also a patented technique making it difficult for any FOSS (free use, open source) program to use.

Well, I was not asking for any "support", since it's already supported with the use of current operators I was only discussing how to do it, and optimize it, within the current API

BTW, I know it's patented, but that's a bunch of specific implementations, not the idea in general. Both Apple, Adobe, Microsoft and FreeType feature their own implementations, and there is no patent dispute I am aware of right now.

Afetr re-organising your command so it is more readable (step by step)
The first convolve is useless, it does nothing. But you have it there so you can add horizontal blur, that is understandable.

The -depth 16 also is of no use, as it is only an input and output setting.

I would whoever save using PNG not JPEG. Even -quality 100 is lossy with JPEG, unless you use the J2 (JPEG2000) format specifically.

I see you are making correct use of gamma in the handling too. which is good to see, something that I would like to see more automatic in IMv7, (to look at in late stage development).

I am not certain exactly how it is working, but I don't think -filter point (whcih simply removes columns) is right. To me the sharpness may only be caused by the aliasing from using Point filter. But I would need to review exactly how LCD sub-sampling works to know for certain what that operation should be (it has been some time).

Do you have a link to the specific technique detail?

Untill I do review LCD sub-sampling that is all I can say for the moment.

Perhaps others would like to comment.

Hi anthony!

As you said the first convolve is there only to add horizontal blur if needed, otherwise it's useless.

Depth 16 is critical for gamma 0.5 followed by gamma 2.0, or otherwise ugly posterization will occur. 8 bits is far from enough for a transformation as aggressive as y=x^2 followed by y=sqrt(x). That is unless you use the hdri enabled version of imagemagick, which uses floating point. I presume most folks don't have hdri enabled (I have) so I included that.

I concur Point filter is not correct from a digital signal processing perspective. The effect is as if the LCD matrix was overlaid directly over an higher resolution image without any anti-alias filter, like a Bayer-array without anti-alias filter. It is well known that results in color moiré in some situations (basically fabrics with a sharp lens...), but also results in sharper output in most cases. For those borderline cases were it fails miserably I recommend either full Mitchell pipeline (theoretically correct, but only eliminates LCD color fringing without increasing apparent resolution) or a bit of horizontal blur (works in practice, and is sharper).

Please note that in the general case (theoretically correct, shall we say) the resolution is not increased, only the slight color fringing normally seen at sharp edges in LCD images is eliminated, resulting in a very subtle effect. That slight fringing is the result of not accounting for the fact that color channels are in average shifted 1/3 pixel in relation to each other in an LCD. Resolution cannot be increased, of course, since your screen still has the same number of pixels in the end, but by allowing moiré you can increase the apparent resolution in a controlled way, just as in a bayer array without anti alias filter does.

I would like to see others comment on this. I don't have a link to the specific technique detail since I came up with it myself yesterday. Of course I have previous generic knowledge in the field of image processing and practical digital photography, otherwise I couldn't come up with all this ideas and interpretations in such a short time.

Here's my writeup about a simple form of subpixel-aware resizing. It may not be the best you can do, but it was surprisingly easy to implement the "offset" feature that makes it possible.

Depth 16 is critical for gamma 0.5 followed by gamma 2.0, or otherwise ugly posterization will occur.

But it is 16 bit by default -- in memory which is where it counts. The -depth does not change in memory bit quality, it is only for IO not for memory which is the compile time quality.

And thanks for the re-cap as to how LCD pixel sub-sampling works.

Hmm from my bookmarks from when I looked at the previously, these are my links..

Email detailing the method... (it starts with a scale up!)
http://lists.nongnu.org/archive/html/fr ... 00087.html

Freetype code for it...
http://git.savannah.gnu.org/cgit/freety ... ftlcdfil.c

From http://gidden.net/tom/2006/05/29/lcd-resampling/

from a Blog by Tom Gidden, on LCD sub-sampling wrote:Lossy compression breaks any results, leaving us with large images when JPEG would be more appropriate. Since the antialiasing manifests as a colour fringe, the lossy JPEG algorithm munges the detail completely. Worst still, the "extra" colour in the image can "stain" the normal JPEG artifacts

So I was right -- don't use JPEG for results!

Toms example 'testbed' page
http://gidden.net/junk/lcdresample/
provides a nice way of 'flipping' between photoshop resize and sub-pixel sampling resize. Looking at the sharp edge of the almost perfect vertical lines shows clearly the sharpening that it achieves. Makes a good testing image for this!

Though I somehow have the feeling that the photoshop resize did its resize linearly, without gamma being taken into account. But then IM does not take gamma into account by default either. Basically the edges of the photoshop resize just do not seem as 'smooth' as they should be!

And while it works well for the image, it fails badly for lines. Probably because Tom used various interpolated resize, rather than a better Mitchell resize filter.

Thanks Jason S for the pointer to your write up on this. It is better than anything I previously found.

Hello all!

Jason, your method, which I saw a few hours after starting experimenting myself, basically eliminates color fringing. I'm not so sure it increases perceived resolution. In fact it may decrease it a little bit. An improvement still over no-lcd-aware rendering imho.

Meanwhile I found a way to only use decent resize methods and keeping good sharpness, while still color-defringing. It's not as sharp at Point resampling in the end, but much more fool proof:
Here is a result:



Rationale for the changes:

You need a 3x larger image to use small & simple shift matrices, but Mitchell, which is a good theoretical window for sinc with little losses, does not deliver visually sharp output, so let's not fight it and let's use a 3x larger image on both axis instead of horizontal only. Too many pixels never hurt, cpu and memory being cheap. This allows us to later Lanczos for final downsampling, which keeps things far sharper and is still robust for all images I've thrown at it.

Concerning the use of JPEG:

Anthony, while JPEG basically sucks it's standard. In the past I did a study to evaluate JPEG adequacy for high quality image transmission, so I know a bit about this, and I can share some useful facts here.

In the study I used a few jpeg encoders, including ImageMagick and cjpeg. At 100% quality with no color subsampling you get rounding errors only, since all frequency components are transmited, which translates into some added noise.

This noise keeps signal averages and basically does not introduce any statistical bias, and this holds even when you lower quality. That's because of a specific property in DCT I can't recall now (linearity?...).

At 100% quality those errors generated noise at a similar level to the errors already introduced by using 8 bits quantization anyway. Summing both errors it's as if you got precision similar to using 7 bit sampling, but without the posterizing effects.

This noise (8 bit + jpeg_100) also happens to be similar (in terms of magnitude) to shot noise level with a sensor featuring a well depth of ~32k electrons if you encode with gamma 2.0. Analysis was simplified by the fact that the square root transform applied to poisson distributed countings (shot noise...) has the nice property of making noise a constant, since in those countings noise=sqrt(signal).

The noise introduced if also of a magnitude similar to the dithering needed to properly encode using 8 bits anyway.

To keep things simple, by using JPEG at 100% quality with no color subsampling you basically introduce a very subtle noise at a magnitude similar to noise already present in any properly encoded 8 bit image. It's also lower than noise already present in most photographic images. So it's adequate even for very high quality transmission. Don't bother, you won't be able to see the difference from a tiff or a PNG. Well... except maybe in a high brightness DICOM calibrated medical display, but even then I have sincere doubts.

When not to use JPEG: in intermediate steps of image editing, when you don't want to introduce errors at each saving of your work. You should probably also use 16 bit sampling, because otherwise you get similar errors by just using 8 bits.

As a footnote I may add that jpeg supports 12 bits per channel, but I've never seen it used that way. 12 bit / channel jpeg with gamma 2.0 encoding would be way better than any existing image sensor in terms of noise, featuring noise similar to a well depth of 8 million electrons, while even very large sensors with very large pixels have at most 1 million electrons or so.

It just occurred to me you can also combine both methods, Point and Lanczos, in the final step.

This is again not fool proof, requiring horizontal blur sometimes, so for general use, not requiring human evaluation of the results, I still think the method I stated in the previous post is the best.

This new one does deliver nice and sharp results with most images though.

Code:
Example:

Even sharper, while accepting aliasing in both axis.

This aliasing can be controlled using blur and keeping the final ratio at 2:1 vertically and 3:1 horizontally. A different blur factor for each axis might be needed for best results. This must me tested by trial and error, since best results vary from image to image.

With this final method, and a bit of intermediate blur if needed, I presume we are getting quite close to the maximum detail per pixel you can express in a LCD.

Code:

Many apologies for the lateness of my response, but what may I ask has been patented? LCD rendering? Who actually has the patent on this? I figured that rendering was vague enough of a concept that no one could have "invented it."

Anyway, I digress. . .I'm very new to the world of photo retouching -- let alone LCD sub-pixel rendering. I'm technologically inept in some areas. Can someone help me out? Feel free to PM me. Thanks, all!

SmudgeToolKitteh, quite frankly I've never seen this applied to images except for Jason S.

My method is different since it also accepts a controllable amount of aliasing (just use more or less gaussian blur, or a specific kernel for filtering in the intermediate steps), benefiting sharpness. It's also quite practical and easy to implement. ImageMagick already supports it, as you can see

I couldn't care less about patents... Otherwise I would have patented this myself

What has been patented is IIRC specifically Microsoft's ClearType font rendering. Other subpixel rendering technologies, like the ones used in the X server from x.org and FreeType, are available for free under an open source license, are not patented. Don't bother, just use it

This guys describe something similar to what I'm doing, and was published in 2002.

http://ics.ele.tue.nl/~dehaan/pdf/90_jsid2003.pdf

Including the aliasing vs sharpness tradeoffs. Very interesting... I'm reinventing the wheel because someone forgot it long ago... I know of no programs actively doing this to enhance image output, except for fonts ans simple line art.

They also mention the real resolution does not improve, just the perception. Aliasing must be accepted for the sharpness to increase, and moves more into colors, away from luminance. Just as I said... oh well!

I don't know if there is any related patent.

Rodrigo, this is really sweet and works well. Very interesting!

Im a bit stymied now how I would convert this into a general purpose shell script without hard coding the sizes. Is this possible with a single command? I just want to give it the final size of the largest dimension, similar to "-resize 640".

towolf, it should be trivial to make a perl script invoking imagemagick (or using perlmagick) with the right parameters. Do you want me to do it?

Say, such a script could take screen dimensions and downsample an entire image collection for slideshow viewing with the best quality possible, fitting the images into the screen. This only makes sense at the output stage, or near output, since unless you are looking at images 1:1 it only makes things worse!

What I was thinking the other day is if this thing could be accelarated using OpenGL or something like that... making possible real time downsampling of ultra-high-definition-tv, or gaming... but I'm no expert in the relevant tecnologies.