Tracking human facial expressions in real time and becoming a two-dimensional character becomes like this

Building a deformable three-dimensional two-dimensional model based on a hand-painted "picture" and allowing you to move the picture freely as it is "Live2D"Is currently"Live2D Creative Award 2014"We are holding a contest called Live2D, and we were recruiting original movies created using Live2D from March 27, 2014 until June 1.

One of the entries of this "Tracking your facial expression in real time and becoming a Live2D character! 【Live2D_2014】"By tracking the human face in real time, we can change the facial expression of the character created by Live2D to match the change of human expression and succeed in becoming a 2D character.

Tracking your facial expression in real time and becoming a Live2D character! 【Live2D_2014】 - YouTube


This is a character created using Live2D. As for this character, the entire body also moves in all directions, front and back, left and right while changing facial expression very finely.


The facial expressions and body movements of this character are changing in real time by tracking human faces and movements.


When thinking that "Does your facial expression really change to the human face?", The 3D model of the human face currently tracking in real time is displayed on the right side of the screen. It is said that the expression and movement of the character are linked with this, but how much expression and movement can be reproduced?


First move the face up, down, left and right.

right


left


Up


under


Although the human face model is moving a little big, the character succeeded in following it and facing all directions up and down, left and right.

Furthermore, if the model person moves the body, it is also possible to move the upper body of the character. Although it says that the character's body also moves, it does not seem to be able to move too much.



Furthermore, the shape of the mouth can be expressed. From a pocket-sized look ... ...


Ninmari. The 3D model that expresses the human face is pretty smooth, but the character has a bitter laughter expression as if its mouth corners have risen a little.


A vague look.


My mouth is Parka.


Serious looking


A smile


Angry face


In this way, it is possible to track the human expression and let the character take various facial expressions.


With this, it will become possible to "deliver full of characters" using Skype · Nico · USTREAM etc.


The following applications were used to create this movie.

· Live2D
· Unity
·FaceShift Studio

Although the facial expressions are still hard and the movements are still small, I hope that you can see characters that will show you various facial expressions like human beings, like these technologies progressing to become your favorite character I can not help hoping for a future that will become.