Merging layers, recovering original tones

[imagebloko]

I have been thinking about how best to explain this problem... it isn't a problem with ImageMagick on the command line, but more a problem of how best to *use* ImageMagick... I'm afraid I've run out of ideas about how to approach the problem due to a lack of experience

Please allow me to briefly describe a *simplification* of the scenario which I hope captures the salient points:

Imagine a glass plate (PG) which is not perfectly transparent, will have reflections on it, may be smoked glass etc. (i.e. not necessarily uniform).

This plate PG is held above two adjacent surfaces (S1) and (S2) (aligned centrally between them, and photographed. Visually in ASCI : [BLUE |Glass ][ Glass| GREEN]
The surfaces are for "chromakey" i.e. maximum Blue and maximum Green.

I have an Image mask for the area which PG covers, lets call this IM. Using this I can extract just the PG part of the picture.

I would now like to render a new image which shows PG above S1' and S2' where S1' is now Red (not Blue) and S2' is now White (not Green). To do this I take the masked part of PG and layer it on top of the new S1 and S2.

Naturally, I now have a Blue and Green looking glass plate surrounded by Red and White... so the goal is to remove the Blue and Green from the masked part of PG, whilst probably applying some transparency on it.. so that when I place it on S1' and S2' it looks correct (i.e. you see the Red/White distored by the non-perfect transparency of the glass).

When I try to set the Blue colour to transparent, and then the Green colour to transparent in the masked image ... the glass plate does layer nicely on S1' and S2' but it has a strange yellow tinge.. try as I might I can't find a solution in that direction (which sort of doesn't surprise me because if you remove the blue/green channels how can you get back to a "grey" like appearance?).

If on the other hand, I try to use combinations of difference operators, or subtraction, I can't get any closer to my end result.

I'm totally lacking the relevant sense of theory... and after reading through a LOT of ImageMagick examples, I'm still not finding my way...

Is this possible, or is information about the inherent "transparency" of the glass simply being lost through the taking of a photo (which lacks that transparency information) such that it can't be reconstructed in anyway?

I'm sorry this is such a long problem description... it just shows that I don't really understand the problem myself!

Bottom line : Is something like this at all possible?

[anthony]

Perhaps a small example would help us understand.

Note blue and green do not go to transparent. transparency is a completely separate effect.

Images are not real world things. and IM is not a raytracer whcih does model the real world.



Now the glass is a filter that removes 'attenuates' some for the light going though it. That is a multiplication (a composition operator). If it attenuated all the light, the result is black.

You would then ADD the 'reflected' light. That is ambient/phong/relectied light.

These are all percentages. For example 40% of the light form behind, + 30% of the ambient/phong/reflected light, whcih is a composite blend (attenuated addition) of the reflected and transmitted light. Rarely does those percentages add to 100% unless the object is either a light source of some kind.

So what you need to do is work out the image of the transmitted light (using distorts and multiply as appropriate for fiter and lens effects), and separately work out the reflected/ambient effects, then blend (add) the images together.

[imagebloko]

Thanks for the reply Anthony,

I fully appreciate what you're saying about the distinction between physical modeling and image processing - I was hoping that I could simply use some sort of logical image processing, otherwise I'm stuffed

If I simplify the example even further it seems so trivial, but I'm still stuck.

Generate a test image:
Take a background with a known colour (e.g. max green).
Overlay a second rectangle which covers half the area, this rectangle is for the sake of argument, 100% black and 50% transparent, so we see a "darkened" green shining through.

We now know the original background green colour, and we see the second rectangle as an area of darker green.

Writing this is seem impossible to me that one can use what we know about the original background colour, to arrive at a situation where the second rectangle appears as 100% black, and 50% transparent again - because the colour and the transparency are two different things. Is it really impossible to recover the "original" state of the second rectangle before it was merged with the background layer in the original image?

Thanks

[fmw42]

Is this what you are trying to do:



convert zelda3.png \( -size 128x64 xc:"rgba(0,255,0,0.5)" xc:none -append \) \
-compose over -composite zelda3_half_green.png

[anthony]

Writing this is seem impossible to me that one can use what we know about the original background colour, to arrive at a situation where the second rectangle appears as 100% black, and 50% transparent again - because the colour and the transparency are two different things. Is it really impossible to recover the "original" state of the second rectangle before it was merged with the background layer in the original image?

As long as some color is still present, and you know exactly how the color was modified, it is posible to recover the original image (with only minimal degrading). It is only when say one color was completely removed that you will be unable to recover.

For example if the overlay acts like a 'red filter' removing all green and blue colors, then of course you will not be able to revover any of the green and blue color channel values.

For a 50% transparency, whcih basically multiplys all the colors values by 50%, then you can recover the original colors by multipying by 200%. Not a problem.


As mentioned you really need to know how the colors were affected, and some how reverse the effect.

One possible method for recovery is to create something like a HALD colormap of the effect.
http://www.imagemagick.org/Usage/color_mods/#hald-clut
And then create a program to generate a 'reverse hald' colormap to reverse the effect of the color changes. But at this time I don't think anyone has looking into creating HALD colormaps (reversed) from specific color changes. Such a program would need to use a sort of 3-dimensional sparse color fill algorithm from a list of color changes seen. It would be a very useful program to have.

[anthony]

Is this what you are trying to do:

I believe he is wanting to take your second image, and recover the first image!

[imagebloko]

Yes... it's more like I'm trying to recover the second image...

Although, as I say, the problem remains that I am handling photos of semi-transparent objects (with differing degrees of transparency) which are photographed against a known (bright green) background.

I want to algorithmically replace the bright green background with different coloured backgrounds. Obviously this works fine when the objects in front of the background are opaque, but I'm having problems with semi-transparent objects...

If I replace the green background with a red background, no matter how I process things, the semi-transparent foreground object doesn't look as it would were it photographed directly against a red background... it is always tinged with unexpected colours (I think I understand why, but don't understand how to remove them).

Do you see where I'm coming from? Sorry my explanations have been so lousy.

[fmw42]

Do you know the degree of semi-transparency and the exact background color?

Perhaps you could post a link to an example image and describe what you want to change it to and any information you have about the transparency, background color, etc.

[imagebloko]

Only the the exact colour of the original background is known.

I'm trying to make an idiot proof frame into which various glass samples (some of which include inlays) can be placed, and somebody just has to press the shutter. Later a little script is/was basically supposed to take all these images and produce new images with a variety of backgrounds, which are then automatically fed into a "configurator" module (it's actually all a bit more complex than that, but it doesn't matter here).

The main point is that it needs to look after itself. Nothing can be "calculated" or measured per "glass inlay", it either works by some form of subtraction from the known reference point in the system (i.e. the original background colour) or it doesn't work at all.

I keep trying permutations with a simple test case : just a 60% black rectangle with 50% transparency partially "overlayed" over a bright green background in Inkscape, then rendered to an image. I then try to make the green portion transparent/replace its colour, in such a way that the original rectangle appears either as it should against a new background colour. Frustratingly hard to achieve... the "Replace with Alpha" function in the GIMP gets me closest of all the things I've tried, but it still isn't right.

Thanks very much for your continued interest and patience with this daft little problem. It's much appreciated, at least I will get an idea wether or not it's just me being thick

[anthony]

We need to some how determine the color of an object, and the amount of transparency.

Because there are two aspects one photo and one reference background will NOT do the job. There is not enough data to make a determination!

For example how can you determine if that green patch on the object is an opaque green, or a transparent/reflection of the green background?

We will thus need at least four images using two (or more) very different background colors (for example 'near white', green, and black) and reference images (with and without the object) , using the same photo conditions (turn of any camera auto-level handling). To be able to work out the correct color/transparency of the object.

Object must be unmoved relative to camera in the appropriate images. So the order of photos probably should be background 1 reference (photo1), with object (photo 2), change background (photo 3), remove object (photo 4).

Basically it would give enough data points to determine how the object effects the background color.

That is from that we can determine...

• did this pixel replace the background completely (it is opaque), because it produced the same color regardless of background.
• Or did the background value remain the same as its reference (no object) image, and is thus fully-transparent.
• or some measure in-between, as the color changed but had some other color component included, and thus semi-transparent, or reflective component.

Actually for our purposes using images of pure white and pure black would work best!
Though using red/blue/green screens could be used to limit the effect of any camera/software auto-levelling that may effect results. However more than two background colors (as well reference (no object) images will be needed for any object you want to do this for.

[imagebloko]

Thanks Anthony, that's what I suspected, but you propose something interesting there.

When I say I'm building a frame, what I mean is a rigid (aluminium) frame around 1x1x1 meter with a "half-pipe" on which the object is mounted, with absolutely immovable fixed mounting points for lighting and two cheap Canon A495 cameras (which are superglued in place, sacrificed for this project ).

The stability is certainly there... now to the background colour... perhaps sufficient colours already exist, but if not it might be possible to make that happen....

As it is, the glass panes are actually accessories which slot into a custom extruded aluminium profile. These profiles have slots for slipping in LED lighting, and custom "slot covers" which themselves may be solid or transparent. The point is that there are a ridiculous amount of permutations possible (around 38 Million), and my goal is to reduce the number of photos which must be taken to a minimum, where my program (which already calculates all the permutations I want and detects/gathers files with appropriate naming for processing by ImageMagick - the point at which I hit the roadblock) should then assemble a few thousand combinations from those base images.

One of the variables is the coating of the profile which holds the glass panes.

My plan was therefore, to take a "base profile" sprayed bright blue, fixed to the bright green background of the "half pipe", and to take photos of the relevant combinations against these.

This means that photos of the glass plates will contain two known backgrounds - the blue and the green background. A base picture of the background with the two colours can be taken, but I think you actually mean that for each pixel position we would need two photos?

In this case, are you implying that if I take a set of photos against the blue profile and the green background, and I were to repeat the process against e.g. a blue background and a green profile, that taken together the "glass plate" part of the image could be successfully (realistically) programatically layered on top of arbitrary other profiles (of arbitrary colour)?

[anthony]

Blue and green may not be a good combination. You will not get a good 'red' value handling in the input data.

This was why I suggested a white and black (non-reflective matt black) backgrounds, as you get a full difference in all three color channels.

If everything including ambient lighting is fixed (no nearby window) then you can just take two base images for a processing run, and then just two images (with background change and object untouched) with each background.

I would like to see this type of image processing worked out as it would be a useful method, and can be made independent on the exact background colors, as long as you get a good change of color values for all three RGB channels. Actually I have been thinking of this problem for a number of years (on and off). This is why I could give you a good solution. However I have never had actual image data to play with to work out the image processing needed.

In this case, are you implying that if I take a set of photos against the blue profile and the green background, and I were to repeat the process against e.g. a blue background and a green profile, that taken together the "glass plate" part of the image could be successfully (realistically) programatically layered on top of arbitrary other profiles (of arbitrary colour)?

Yes it can be done. as long as teh two colors provide a large change in RGB values. For example Blue and a strong Yellow, or Green and Purple (maroon), or black nad white. The backgrounds need to be Matte (non reflective) and be able to be changed without moving or touching the object, as if the object moves (relative to camera and lighting) it will produce false readings.

[imagebloko]

If this interests you, that is fantastic. I can see it is an interesting problem.

The frame will be finished, and the first images taken, in the next few weeks. We have built the background of the frame in such a way that we can slip different colours of card into the "halfpipe" so experimenting with different backgrounds won't be a problem. I can also arrange to get multiple samples of the base profile to experiment with different colours there.

I'm really doing this out of interest and for fun myself (the "budget" for this little experiment has long been and gone - which I expected), so I won't be working on it with incredible fury and intensity. Having said that, fury and intensity is often not conducive to good research

What's the best way to post images to the forum? I see no option. Should I upload them to my own server and then just hotlink them or is there a better (more persistent) alternative?

[anthony]

Just load them on a server somewhere. Your own or one of the commercial ones.
I myself am using DropBox at the moment but there are many others.

Then post the URL, or for small images use a the Img format around the posted URL.

Small images are better. If you can crop a smaller area from the images for experimental purposes.

[imagebloko]

Hi,

The basic frame is now ready. I took some quick pictures with my (poor quality) phone camera, but I think they serve to demonstrate the point.

Next week I will setup the frame with the lighting / cameras exactly as we want it, then I'll upload some sample images with different profiles and backgrounds at which point the image processing fun can begin

This gives an overview (showing a sample profile mounted on the board) :

Notice the three rectangular "lips" (top and bottom) which extend above the board. We can fit any colour card onto the backing board and it will be held in place by force against these lips.

Here you can see the two cameras mounted on arms (the arms can be moved into almost any position, and then fixed securely) :


In this photo you can see the profile is raised from the board - however, this is optional. It can be raised an arbitrary height, or set against the board.
The board itself has a flattened section in the center so that the profile *will* sit snuggly against it :


From behind, you can see the reinforcing of the board (it is very strong indeed) and the switches to control the four lights independently.
The lights can also be repositioned along the edges of the frame, but also the distance of the light from the edge can be altered, including both "pitch and yaw" :


Here's a close up of one of the camera mountings... I hope it shows how many degrees of freedom in the positioning there are :