Is there a way to clean chroma key processed image?

[lp1051]

Hi,

I was trying many ways, testing, searching, but nothing gave me satisfactory result, so I'd like to ask for help with this one.

I'm using example image found on web.
https://dl.dropboxusercontent.com/u/15391592/input.jpg

I processed this image with primatte chroma keying and got this one:

https://dl.dropboxusercontent.com/u/15391592/result.png

Don't look at the right bottom black cloud, it is just due to bad lighting of the example image. My problem and question is about all the other mess. If you zoom in the image and check left top or left bottom areas, you can see some pixels after the chroma keying and almost gradient of semitransparent pixels in the bottom. What I'm trying to do is to clean (make fully transparent) everything outside the man's body and the speech table.
So far, the best I could come up with is making pixels with similar color transparent, using draw matte and floodfill:

Code: Select all

convert result.png -fill none -fuzz 10% -draw "matte 1,1 floodfill" -draw "matte 1,719 floodfill" result-draw.png

The result is then here: https://dl.dropboxusercontent.com/u/153 ... t-draw.png. As you can see, the gradient in the left bottom area is gone, but not all of the bad pixels. Using too high fuzz factor is slowly cutting the man's hair, so I cannot use too high factor. So now I don't know how to improve the selection of pixels to make transparent. Any ideas how to clean the mess up?

Thanks for all suggestions,
Luk

Btw. I was trying to compose difference of the input and result image to get some kind of mask, but it obviously doesn't work.

[fmw42]

try this, where I remove the shadows by appending the first and last columns, expanding to full size, then -compose divide on the green channel and then do -fuzz to change green to transparent. This is still a bit of outline of green.

This would have worked much better if your input was not jpg, but png or tiff (any non-lossy compression). JPG does not maintain constant (background) colors very well.

Code: Select all

convert input.jpg \
\( -clone 0  -channel g -separate +channel \) \
\( -clone 1 -crop 1x720+0+0 +repage \) \
\( -clone 1 -flop -crop 1x720+0+0 +repage \) \
\( -clone 2 -clone 3 +append -resize 1280x720! \) \
-delete 1,2,3 \
\( -clone 0 -clone 1 +swap -compose divide -composite \) \
-delete 0,1 \
-channel rgba -fuzz 30% -fill none -opaque "rgb(100,255,130)" \
input_trans_bck.png

http://www.fmwconcepts.com/misc_tests/b ... ns_bck.png

Or if you do not want the addition brightening of the image

Code: Select all

convert input.jpg \
\( -clone 0  -channel g -separate +channel \) \
\( -clone 1 -crop 1x720+0+0 +repage \) \
\( -clone 1 -flop -crop 1x720+0+0 +repage \) \
\( -clone 2 -clone 3 +append -resize 1280x720! \) \
-delete 1,2,3 \
\( -clone 0 -clone 1 +swap -compose divide -composite \
-channel rgba -fuzz 30% -fill white +opaque "rgb(100,255,130)" -fill black +opaque white \) \
-delete 1 \
-alpha off -compose copy_opacity -composite \
input_trans_bck2.png

http://www.fmwconcepts.com/misc_tests/b ... s_bck2.png

[lp1051]

Thanks a lot for your feedback!
To answer why JPG? I expect to receive the images from digital camera (together with some iptc metadata), and the most common format is JPG there, or?
However, I tried to convert the image from jpg to tiff or png prior the primatte cleaning, and there was major improvement, so thanks a lot for that hint!
Your first example cleans out all the mess around the person, keeps better his suit color, but also modifies too much his tie, finger nails and cut his hair off, which I wanted to prevent in some way.

So now, when using TIFF and after primatte cleaning I get this https://dl.dropboxusercontent.com/u/153 ... m-tiff.png(notice the black cloud is light grey now) and after using the

Code: Select all

convert result.png -fill none -fuzz 30% -draw "matte 1278,719 floodfill" result-draw.png

I get this: https://dl.dropboxusercontent.com/u/153 ... fuzz30.png. Which removes all the mess in the area around the person, but also the top of his hair, which is what makes me mad:)
Btw. -fuzz 29% is almost perfect, even keeping the hair, but there are still some few messy pixels around...

But since the improvement from converting JPG to TIFF or PNG is so vital, is there some better way for conversion than simple

Code: Select all

convert input.jpg input.tiff

? I mean if I could improve the green background during the conversion, without deforming the other colors or anything that would make all colors in the green background more similar (sort of like decrease color deviance in the green areas)?

[fmw42]

The second example cleans the green screen and does not modify the colors. Try it.

With regard to JPG, once you have jpg there is no point in converting to PNG or TIFF. The data is already lost from the JPG compression.

[snibgo]

The image comes from a video. If you want to process the video, a video editor may serve you better. If you want to process just a single frame from the video, don't save it to a lossy 8-bit format like JPEG.

[fmw42]

Using more of the background gives a better result than my previous example 2.

Code: Select all

convert input.jpg \
\( -clone 0  -channel g -separate +channel \) \
\( -clone 1 -crop 200x720+0+0 +repage \) \
\( -clone 1 -flop -crop 100x720+0+0 +repage \) \
\( -clone 2 -clone 3 +append -resize 1280x720! \) \
-delete 1,2,3 \
\( -clone 0 -clone 1 +swap -compose divide -composite \
-channel rgba -fuzz 25% -fill white +opaque "rgb(100,255,130)" -fill black +opaque white \) \
-delete 1 \
-alpha off -compose copy_opacity -composite \
input_trans_bck3.png

http://www.fmwconcepts.com/misc_tests/b ... s_bck3.png


Doing the above, but using some morphology on the mask helps even more.

Code: Select all

convert input.jpg \
\( -clone 0  -channel g -separate +channel \) \
\( -clone 1 -crop 200x720+0+0 +repage \) \
\( -clone 1 -flop -crop 100x720+0+0 +repage \) \
\( -clone 2 -clone 3 +append -resize 1280x720! \) \
-delete 1,2,3 \
\( -clone 0 -clone 1 +swap -compose divide -composite \
-channel rgba -fuzz 25% -fill white +opaque "rgb(100,255,130)" \
-fill black +opaque white -morphology open octagon:2 \) \
-delete 1 \
-alpha off -compose copy_opacity -composite \
input_trans_bck4.png

http://www.fmwconcepts.com/misc_tests/b ... s_bck4.png

[snibgo]

Here is my version. When the background has high saturation "chroma-key", finding it is fairly easy. But it creates more edge problems. Windows BAT scripts.

Step 1: Make a background green across the whole image. I ignore the lightness of Lab colorspace -- I'm looking for a particular hue and saturation, regardless of lightness. I reduce this to a small number of colours, expecting the chroma-key will occupy just one slot. Assuming coord (0,0) is chroma-key, I fill from there. (Other techniques can be used for more mobile subjects.)

Code: Select all

%IM%convert ^
  greenBack.jpg ^
  -alpha Set ^
  ( +clone ^
    -blur 0x5 ^
    -colorspace Lab -channel R -evaluate set 50%% +channel ^
    -colorspace sRGB ^
    +dither -colors 30 ^
    -fuzz 15%% ^
    -fill None ^
    -draw "matte 0,0 replace" ^
    -channel A -negate +channel ^
  ) ^
  -compose CopyOpacity -composite ^
  -blur 0x40 ^
  -alpha off ^
  gb_1.png

There are "creases" where the subject is in front of the chroma-key. That is no problem.

Step 2: Find the difference between the original and the green-screen. Blurring smooths JPEG compression problems. "-grayscale Brightness" gives the lightest of the three channels. "-level" makes it a black/white mask with small anti-alias. For other sources, adjust the level to get a clean mask.

Code: Select all

%IM%convert ^
  ( greenBack.jpg -blur 0x1 ) ^
  gb_1.png ^
  -compose Difference -composite ^
  -grayscale Brightness ^
  -level 20,25%% ^
  gb_2.png

Step 3: Apply the mask opacity to the original.

Code: Select all

%IM%convert ^
  greenBack.jpg ^
  gb_2.png ^
  -compose CopyOpacity -composite ^
  gb_3.png

Step 4: Green has bled into the source, most noticable in the hair and around the hands. Blowing up the image, we can see it occurs everywhere. I call a separate script to de-bleed the edge. This de-bleeding is colour-blind. It ignores the actual colour of the chroma-key. If it was more intelligent, it could probably handle the hair properly.

Code: Select all

call %PICTBAT%deBleedEdge gb_3.png 8 gb_4.png

Where the guts of deBleedEdge.bat is:

Code: Select all

%IM%convert ^
  %INFILE% ^
  ( -clone 0 -alpha Extract ) ^
  ( -clone 1 ^
    -morphology Distance "Euclidean:7,%EDGE_W%^!" ^
  ) ^
  ( -clone 1-2 -compose Difference -composite ) ^
  ( -clone 0,2 ^
    -compose CopyOpacity -composite ^
    -blur 0x%BLUR_SIG% ^
  ) ^
  ( -clone -1,3 ^
    -compose CopyOpacity -composite ^
  ) ^
  -delete 1-4 ^
  -compose Over -composite ^
  %OUTFILE%

Step 5: De-bleeding has pixelated the edges, so we feather the transparent/opaque boundary inwards for the finished result.

Code: Select all

%IM%convert ^
  gb_4.png ^
  ( +clone -alpha extract ^
    -morphology Distance "Euclidean:7,2^!" ^
  ) ^
  -compose CopyOpacity -composite ^
  gb_out.png

The hair isn't perfect. Tweaking the above might improve it, but at the expense of other areas. Perhaps a "hair-detector" mask is needed.

The shiney lecturn has reflected green. A matt spray (available in all good photo studios) would cure that.

[lp1051]

Thanks a lot both of you, you gave me more than I asked:)
My problem is that I have to use the primatte cleaning script that produces the results as shown in my examples, so although you showed me great alternatives, I cannot really apply them. That's why was my question if there is a way to clean the mess from already processed image.

With regard to JPG, once you have jpg there is no point in converting to PNG or TIFF. The data is already lost from the JPG compression.

Yes, it's true, but still the cleaning script I'm running gave lot better result when using jpg converted to tiff, as shown on the black cloud in the right bottom area. So perhaps there was some color adjustment in the conversion, or color changes in the resulting TIFF?

I will try to test more with the approaches you showed me, if I could use them for the final cleanup. Or do you have some alternatives about how to clean this image: , in another way than my -fuzz xx% -draw "matte... floodfill"?

Thank you!

[snibgo]

Hmm. Primatte is a professional product that costs hundreds of dollars. If it isn't giving you results as good as the methods from Fred or me, have you asked them why? Do they have support forums?

If converting to (16-bit) tiff gives better results than 8-bit jpg, I suspect Primatte chooses its internal format from your input file.

You might find that pre-processing your image with IM helps Primatte to make better results. If so, then I suggest Primatte needs to improve their product.

[lp1051]

Yes, they provide support, but the contact person has holiday now. I've tried all possible options and their combinations to improve the results, but many photos with unbalanced light or deformed green screen background ends up with similar results like the examples I showed. Also the c++ code is too advanced for me to edit, so I was trying to use IM for fine tuning, as I anyway use IM for some other tasks within the application.
But it's true, yours and Fred's solutions can in many ways compete with the primatte method:)

[fmw42]

If you are stuck with Primatte, then why even ask this question on the IM forum. Perhaps I misunderstand what you mean by stuck with Primatte?

[lp1051]

I'm searching for ways to improve the results of some photos after using primatte. Is it wrong to come here and ask for help using IM? Or is the only right way to ask on primatte forum/support?
After I received such a valuable tips from you and snibgo about the green screen removal itself, I realized I should just show the result images, without mentioning anything about the input and use of primatte. As it moved the focus from the main question elsewhere. I just thought it would be better to describe the whole situation... I'm sorry if it feels like I'm using IM only for some cleaning after primatte, but my actual problem is exactly that - what can I use for cleaning after primatte and how?

[fmw42]

OK. Lets start again with just your Primatte result. Sorry we misunderstood that you need to process that result and not start from the original. I will try to get back later today or tomorrow with ideas. Others can jump in, too.

[fmw42]

try this, where result.png is your result from your other software

Code: Select all

convert result.png \
\( -clone 0 -alpha off \) \
\( -clone 0 -alpha extract \) \
\( -clone 1 -clone 2 -compose multiply -composite \
-channel rgba -fuzz 0.9% -fill none -opaque black \
-alpha extract -morphology erode octagon:2 -morphology close octagon:3 \) \
-delete 1,2 -alpha off -compose copy_opacity -composite new_result.png

[lp1051]

Thanks a lot for the new hint. It helped me to see more clearly the value of -alpha option and I could move forward with this image. But then I found out with other photos, that primatte is actually translating colors and probably if they match or are in range of the green background, they are made semi-transparent, although they're part of the person's body. So I'm now back at where I was, as I cannot simply extract the alpha channel and copy opacity or change mask with the processed image.
In that way, the -draw "matte 0,0 floodfill" approach worked a little better as it stopped as soon as it reached the person's body contour, but with the price of eating up the hair with higher fuzz factor.
So as one of my last attempts - is there some way to improve the -draw matte floodfill concept, other than fuzz factor? I couldn't find anything that would help, but perhaps some of the morphology patterns could help? So far, I couldn't find any good... Any idea?