Composite technology "Headset Removal" to show VR headset as translucent and convey the wearer's expression

Oculus RiftAlthough it is possible to enter the world of virtual reality (VR) or mixed reality (MR) when using a VR headset like that, the surrounding people who do not wear a headset experience what the user is like It is hard to understand whether there is. Therefore, Google is trying to "remove the headset from the image synthesizing the virtual reality world and user's movement"

Research Blog: Headset "Removal" for Virtual and Mixed Reality
https://research.googleblog.com/2017/02/headset-removal-for-virtual-and-mixed.html

Google Research and Daydream Labs: Headset Removal - YouTube


The VR headset takes people who are wearing the headset to another world. But for those who do not have a headset, I do not know what the wearer is doing.


So standing in front of the green screen ......


If you track women's movements with a camera ......


You can synthesize images of the virtual world and women and see what kind of experiences women are experiencing.


"But I can not see the face of a woman!" Another man gets confused.


That's why currently developed by Google is "Headset Removal" technology.


Headset Removal first imports 3D data of the person wearing the headset. The stereoscopic feeling of the face seen from the front ... ...


left


I will move the face, such as the right side.


Then, we will follow points that move on the display only by eye line. This tracks the movement of the line of sight.


Data actually taken in is like this. Blinks and gaze movements are important, so it seems that the final image created by this will be "as if it is alive" finish. Note that it takes less than 1 minute to capture 3D data of the face and learn expression data such as blink and eye direction.


Using this 3D model, you can synthesize images as if you do not have a headset on.


When another man uses Headset Removal, it looks like this.


When watching the image, the movement of eyes is certainly natural, so it looks as if you are wearing a transparent headset.


A man who hands on the headset.


Removing it is not a transparent headset, you can see that the image is projected.