Netflix's research team announces an idea to improve the chromakey technology that cuts out the green background with ``magenta lighting''

Chroma key synthesis is a technology that shoots a movie with a single color screen as the background, separates only the background color and synthesizes another image there, generally a green screen called 'green screen' is often used for chromakey compositing. A Netflix researcher has announced an idea to improve such chromakey synthesis technology.

[2306.13702] Magenta Green Screen: Spectrally Multiplexed Alpha Matting with Deep Colorization

https://arxiv.org/abs/2306.13702

Magenta Green Screen: Spectrally Multiplexed Alpha Matting with Deep Colorization
https://dmsm.github.io/magenta-green-screen/

Netflix says its magenta approach beats green screen
https://techxplore.com/news/2023-07-netflix-magenta-approach-green-screen.html

Chroma key synthesis itself was used in the movie ' The Wizard of Oz ' released in 1939, and as film shooting technology progressed, chroma key synthesis technology has also been refined. However, even after more than 80 years since the release of 'The Wizard of Oz', the chromakey synthesis technology is not perfect.

Appropriate and balanced lighting is required for natural compositing of footage. For example, shadows cast on the background disrupt the consistency of the background color, making it very difficult to accurately isolate the motion of fine objects such as hair and feathers. In addition, separating only the background color has the limitation that the same color cannot be used for costumes and props.

Netflix researchers have announced a new approach to chromakey compositing called 'magenta green screen (MGS)'. MGS is not a green screen, but the point is to illuminate the actors acting in front of the background with red and blue LEDs. Since digital cameras basically capture colors in three colors, red, green, and blue, illuminating the subject with red and blue makes it easier to clearly separate the background and subject. The research team said, 'Our method respects the claim that 'the transparency of an image is as important as its color',' and measured the transparency from the green tones of the shots. , AI can be used to enable more natural and high-quality synthesis.

You can see what kind of technology MGS actually is by watching the demo movie below.

Magenta Green Screen: Spectrally Multiplexed Alpha Matting with Deep Colorization (DigiPro 2023)-YouTube


A scene from the studio I shot. A woman and a man in colorful shirts are standing in front of a green screen, holding a glass bottle, and in the center is a metal sphere that reflects the surroundings and a color pattern table. The colors of the lighting in the entire studio are red and blue, so the overall color is reddish purple (magenta).

The RAW image taken with the camera looks like this.

After color space processing, the contrast between the background and the person became clear.

From here, separate red/blue tones (left) and green tones (right).

Return the red and blue images to normal tones, and measure the transparency from the green tones with

Based on the extracted image of the person and transparency information, machine learning is used to adjust the color tone to the state of shooting with white lighting.

The following is where the background is synthesized and adjusted with this data. However, the motion blur that occurs when the bottle is shaken or moved violently leaves a green part.

Therefore, when we further optimized for motion blur using machine learning, the green part disappeared. The research team states that the alpha channel data obtained with MGS can prepare a high-quality training data set that can be useful for AI learning that enables more natural synthesis. It's better than