Dec 01, 2021 20:00:00

Disney develops a method that combines 3D face map and AI to create a 'natural face'

By utilizing CG and VFX in video works, it is possible to create effects that cannot occur in reality, or to make objects that cannot be made as props appear. Disney's AI research department has further developed this technology by combining AI with a 3D face map that does not have eyes, mouth, hair, etc., to create a 'natural face' like a living human being. going.

Rendering with Style: Combining Traditional and Neural Approaches for High-Quality Face Rendering | Disney Research Studios

https://studios.disneyresearch.com/2021/11/30/rendering-with-style-combining-traditional-and-neural-approaches-for-high-quality-face-rendering/

Disney Combines CGI With Neural Rendering to tackle the'Uncanny Valley' --Unite.AI
https://www.unite.ai/disney-combines-cgi-with-neural-rendering-to-tackle-the-uncanny-valley/

The following movie explains how the technology developed by Disney is.

Rendering with Style Combining Traditional and Neural Approaches for High Quality Face Rendering --YouTube


The technology for capturing human faces has already evolved to a fairly high level, but ...

Details such as the eyes, mouth, and hair have not been reproduced.

In the 2016 movie

Rogue One: A Star Wars Story , another actor played to re-appear Wilhuff Tarkin, who died in 1994 in his previous film , Peter Cushing. Using the motion capture of the face as a material, the method of synthesizing Mr. Cushing's face with CG was adopted. While the Takin reproduced by this method received high praise, some people pointed out that 'it feels strange to talk or move.'

In any case, there is a large gap between the model generated from the human capture and the photorealistic human, which requires manual work by a skilled artist. So Disney has developed a method to create a 'natural face' that can withstand video works by combining 3D 'face skins' with elements such as eyes and mouth generated by neural rendering.

The research team used StyleGan2, an image generation algorithm developed by NVIDIA. StyleGan2, which is an improvement of the original StyleGan announced in 2018, can generate eyes and mouths that are much more realistic than CG.

The research team says that by combining 3D face skins with details such as eyes and mouth generated by StyleGan2, you can create a 'human-like face'.

The new method can respond to various changes in facial expressions ...

It can also respond to changes in face angle and lighting.

There are still problems such as it is difficult to make the 'hairstyle' consistent when moving the generated face, so further research is needed to actually use it for video works etc., but the research team said that this approach is ' It also claims to be a potential technique for generating 'face image datasets'.