Devices using Oculus Rift are being developed that can reflect their expressions in real time to CG

The immersive VR headsetOculus RiftHas never been realisticVR (virtual reality) spaceAlthough it is a device that can experience the system, a system that incorporates sensors that read the movement of the face in real to this and reflect it in CG is developed. With this device, it will be possible to reflect your own expression in real time in the world while enjoying the VR space.

Hao Li - publications [Facial Performance Sensing Head - Mounted Display]
http://www.hao-li.com/Hao_Li/Hao_Li_-_publications_%5BFacial_Performance_Sensing_Head-Mounted_Display%5D.html

You can check the operation of the system with the following movie.

Facial Performance Sensing Head - Mounted Display (SIGGRAPH 2015) - YouTube


A man with a large open mouth with Oculus Rift wearing. On the display before that, the face that reproduced the expression similar to men with CG is displayed.


Even though a man tried to distort his mouth sideways, the face of CG seems to be tracking properly although there is some time lag.


Male wears are made by Oculus Rift and Intel, which is ahead of the arms extending therefrom3D gesture camera.


This camera and also provided by IntelIVCAMYou can track the expression in the lower half of the face using software called so.


In addition, Oculus Rift has a pad part that hits the face with eight "pulling sensors" built in, so that the movement of muscles in the upper half of the face can be converted into signals.


The position where the sensor is mounted is indicated around yellow.


The movement of the face of the user is converted into CG by computer usually based on the RGB image by the camera, the depth information by the 3D gesture camera, and the information by the 8 pulling sensors.


In addition to the facial expressions, Oculus Rift seems to be able to create richer expressions by tracking the movement of the face (head) and reflecting it in CG.


With this technology, I express my expression by CGAvatarIt will be possible to reflect in near real time, communication such as online chat will be expressive rich.


Since avatar is CG, it is possible to use animal's face besides human's face. It seems possible to use wider characters distinctly.


Although there was a similar device conventionally just to grasp facial expressions, the merit of this device seems to be that it is a compact system. By using VR headset, it may be the day when anyone can communicate with expressive avatar while immersing in realistic images.