Google succeeded in removing the background in real time from the movie being filmed without chroma key composition

If you look at people live on YouTube live etc., there are times when you see another screen such as game screen on the background of the distributor. How to do this shootingChromakey synthesisAlthough it can be realized by using such as, it takes time and labor preparation such as stretching a wall of green color in advance as a background part. Therefore, in order to eliminate such time-consuming preparations, Google has developed technology to remove only the background from the real-time video being shot.

Research Blog: Mobile Real-time Video Segmentation
https://research.googleblog.com/2018/03/mobile-real-time-video-segmentation.html

According to Google, the "segmentation" technology that separates this person from the background is "Convolution neural network (CNN)"It was usedMachine learningIt is said that it is realized with, and even if you look at the actual demonstration image, you can see that people can be set in a different background with little sense of incongruity.


In order to enable highly accurate "segmentation", we are learning that hundreds of thousands of image data is learned, and "learned" images such as "hair" "neck" "skin" etc. , It seems that information was added for each part.


This technology is assumed to be used on smartphones,LSTMYaGRUThere is a disadvantage that the load is too high if processing to remove the background is done with. Therefore, the researcher calculates and finds the position of a person for each frame from the idea that "Movie is a collection of images of one piece called a frame, and almost no change occurs in adjacent frames" Instead of calculating the position of the current person taking into consideration the position of the person who was in the previous frame, we are reducing the calculation load.


Google is also devised to reduce the computational burden, so that it is possible to apply this segmentation technology even for movie shooting of 100 FPS or more for iPhone 7 and 40 FPS or more for Pixel 2, realizing high-speed processing with smartphones It seems that it became possible. In addition, we plan to incorporate this segmentation technology into YouTube's "story" format, but the concrete timing is not clear.