Resize and resample in alpha-multiplied light

[NicolasRobidoux]

In this post, I will give a simple example that justifies the use of alpha-multiplied light when resampling images (resizing, for example).
-----
When resizing and resampling images, the issue arises as to whether one should resample the channels independently, or not.
In ImageMagick (and GEGL and Image Worsener and, I am sure, many other libraries and tools), the channels are not resampled independently.
Only one channel is resampled independently of the others: the alpha (transparency) channel.
The pixel values of the color channels are first multiplied by the alpha (transparency) channel's pixel value, then linear resampling is performed, and finally the final color channel values are obtained by "unmultiplying" the results of resampling the alpha-multiplied color channels by dividing them by the alpha value at the same location, with some exception handling when alpha is close to zero. That is, the color (non transparency) channels are resampled taking transparency into account.
Of course, when the image is opaque, that is, when alpha is 1 throughout the input image, the color channels end up being resampled independently. Actually, the same goes for any image with a constant transparency channel (with alpha=.5 throughout, for example, which corresponds to a half-transparent image). This post, really, only concerns images with an active transparency channel, meaning an image with varying transparency.
In order to explain why one should resample using alpha-multiplied light, I will use a very simple "1D" greyscale image, with only two channels (grey, and alpha) and two pixels (pixel "0" and pixel "1").
So, the left input pixel has greyscale value g0 and transparency a0, and the right input pixel has color g1 and transparency a1.
Let's see what happens if we enlarge this 2x1 image to 3x1, inserting a pixel halfway between the two with (bi)linear interpolation.
If I resample the two channels independently, this gives the following values for the new pixel:

Code: Select all

a = (a_0+a_1)/2 and g = (g0+g1)/2

since (bi)linear interpolation is the same as averaging in the current situation.
On the other hand, if I use alpha-multiplied light, the new pixel values are

Code: Select all

a = (a_0+a_1)/2 and g = ((a0*g0+a1*g1)/2)/a

which is more complicated, and clearly needs love if a happens to be 0.
Why in the world would this be better?
Here is why:
Suppose that we know ahead of time that we actually want to use a constant background color of value b. Then, there is no reason to resample two channels: We may as well flatten before resampling, and then resample a one channel image. What this means is that the final, "flat", result we actually want is

Code: Select all

G = (a0*g0+(1-a0)*b+a1*g1+(1-a1)*b)/2 = (a0*g0+a1*g1)/2+(1-(a0+a1)/2)*b

Do we get the same result if we flatten after resampling?
Let's see:
With the image enlarged keeping the channels separate, the result of flattening after resampling is

Code: Select all

a*(g0+g1)/2+(1-a)*b = G - (g1-g0)*(a1-a0)/4

that is, there is an error proportional to both the local variation in color value, and the local variation in transparency value. (Basically, the deviation from what we want is proportional to the product of two slopes: the alpha channel slope, and the greyscale channel slope. In the language of numerical analysis, this means that resampling channels independently is only first order accurate at halfway points, even though (bi)linear interpolation is a second order accurate resampling method at any location.)
On the other hand, with the image enlarged using alpha-multiplied light, the result of flattening after resampling is

Code: Select all

a*(((a0*g0+a1*g1)/2)/a)+(1-a)*b = G

as desired.
-----
The above discussion used a simple filter (bilinear) and sampling location for the sake of keeping the algebra simple. The general situation, with a more complex linear filter and other sampling locations, leads to the same conclusion:

Resample using alpha-multiplied light

Which is what ImageMagick does.
Technically: If a linear filter returns a white image when fed a white image, flattening to a constant background after applying the filter is the same as flattening before, provided alpha-multiplied light is used.
Note: For best results, don't clamp intermediate results: Carry negative and "whiter than white" color and transparency pixel values around until the image is flattened. This, of course, is not possible without HDRI with ImageMagick, or when storing images in unsigned integer formats. But it's something to keep in mind: Don't clamp unless (or until) you have to.

[henrywho]

Last week I was resampling some large web banners (with transparency) into small paletted (8bit) icons. I also ended up with flatten-then-resample:

Code: Select all

convert big_banner_with_alpha.png -set colorspace srgb +repage \
-colorspace lab -background white -flatten \
-define filter:window=Quadratic -define filter:filter=Jinc -define filter:lobes=3 \
-gamma 1.25 -distort resize ${W}x${H}! +repage -gamma 0.8 \
-ordered-dither o8x8,25,11,11 -colorspace srgb -depth 8 -transparent white \
-quality 95 png8:small_icon_with_transparency.png

This is a simplified command-line because the actual one has the graphics and text splitted from two clones, and the above gamma-thickening trick is applied to the text portion only.

And the LAB ordered dither is found somewhere in this forum and it really help bringing down the file size.

[NicolasRobidoux]

Henry: You remind me that I forgot to emphasize that it's only when you are flattening to a constant color background that you can expect to get the same result by flattening before or after (discounting alpha values really close to zero, which are generally harmless).

Rule of Thumb:
If possible, flatten before resampling (or resizing).
In other words, flatten as early as possible.

[NicolasRobidoux]

Here is a more "mathematically abstract" justification of the use of premultiplied alpha when resampling.

Notation:
All products (meaning, all multiplications) are those that "obviously make sense" in the present context. Same with divisions. For example, A*R is the image obtained by multiplying, at every pixel location, the alpha value stored in the alpha channel A by the value stored in the R channel, and C*R=c*R provided C is a constant image channel with value c, with C*R and c*R equal to the image with pixel values equal to c times the R pixel value at each pixel location.

Assumptions:
The resampling filter filter has the following properties:
1) filter is linear, meaning that if R and S are image channels and t is a "scalar" (plain number), then

filter(R+S) = filter(R)+filter(S)
and
filter(t*R) = t*filter(R)

2) filter "preserves contants", meaning that if C is an image channel with constant pixel value c,

filter(C) = C

where there is some abuse of notation, since the image channel C that appears in the left of the equation may not have the same dimensions as the one that appears on the right.

When Property 1) holds, Property 2) is a consequence of a simply explained property: Resampling a white image gives a white image (assuming that white is defined by a nonzero pixel value, which is pretty much universal).

3) Intermediate pixel values are not clamped if they happen to overshoot.

Consequence: Provided divisions by null alpha values do not occur, flattening to a constant background pixel value c before resampling gives the same result as flattening after resampling using premultiplied alpha values.

Proof:

Let A be the alpha (transparency) channel, C be a non-transparency image channel with constant value c, and R be an arbitrary non-transparency image channel.

Here is the result of first flattening to the "color" c, then resampling:

filter(A*R+(1-A)*C)

Here is the result of flattening after resampling, using premultiplied alpha:

(filter(A))*((1/filter(A))*filter(A*R)) + (1-filter(A))*C
= filter(A*R)) + C - C*filter(A)
= filter(A*R)) + filter(C) - c*filter(A)
= filter(A*R)) + filter(C) - filter(c*A)
= filter(A*R + C - c*A)
= filter(A*R + (1-A)*C)
= same

[anthony]

You also have to consider if vitural pixels or other edge effects. When creating a Distort Resize Operator I needed to use alpha to remove the effect of virtual pixels. Unfortuantally if the image already has alpha channel, then I need to do extra processing to also preserve that alpha channel.

IMv7 should have a number of methods to improve this situation (multiple alpha channels, OR operators that know if a pixel is vitual or not (or read masked) and thus ignore (or not) any contribution), but they are not in production use as far as I know, just hooks for future use at this time.

[NicolasRobidoux]

A similar type of result:

Gamma inserted in a linear pipeline preserves color lines, in the sense that the property of having proportional colors is preserved by the combination.

Justification:
(R,G,B) is the input image. The pipeline is the composition of a filter1 (a linear color map), a gamma transformation, and filter2 (another linear color map).

filter2( ( filter1(l*(R,G,B)) )^gamma )
= filter2( ( l*filter1((R,G,B)) )^gamma )
= filter2( l^gamma * (filter1((R,G,B))^gamma )
= l^gamma * filter2( (filter1(R,G,B))^gamma) )

which says that proportional inputs give proportional outputs (with a gamma mapped proportionality constant). The above of course ignores that, often, linear pipelines clip during intermediate stages.