A chat bot 'Hank' that recognizes the other's facial expression and can interact with nature using a neural network appeared

The research team of the University of Toronto developed a chat bot " Hank " which reads the facial expression of the partner and can respond appropriately to each. Speech recognizable AI that can be interacted so far has only talked about the meaning of the word, but in the actual communication the research team is to take away the intention from "the expression of the opponent" I succeeded in developing a neural network model that responds differently by analogizing the mood from the expression of the partner even if it is the same question. I tagged avatar and named it "Hank", and I release a movie showing how various dialogues are actually carried out.

A Face-to-Face Neural Conversation Model
http://www.cs.toronto.edu/face2face

A researcher at the University of Toronto announced "Neural network model capable of responding in natural language while reading opponent's expression" during conversation. 250 neural networks are used for learning, and only the moment when one face is projected from the movie is gathered and used as the data set "MovieChat". In addition, the " OpenFace " of the deep learning library is also used to detect the movement of the partner's face.

Recurrent neural network (RNN) encoders and decoders are used to reflect the movement of facial muscles in verbal conversation, in particular the decoder consists of two layers. The lower layer is responsible for spoken responses and rough opponent facial expression recognition, and the other layer contains subtle gestures, and by passing through this layer the output of the neural network is transformed into a more natural one It is said that it will be.

You can see how Hank of the chat bot created using this neural network model talks to you by watching the following movie.

face2face_v2 on Vimeo


Hank can interact with the image of one's face.

Against the lines saying "It is a secret that I saw him last night ..."

"I asked him to kill me," he said, returning that kind of line.

Talking about a woman in a movie 's scene saying "You see it being confused, just say a damn fellow ..."

As if it was just one of the performers, Hank who answers smoothly, "It's not easy, but it's not a big problem."

Hank reads the expression of the opponent and shows different responses accordingly.

For example, if you say "Thank you" with no expression ... ...

"It looks like someone has told you," Hank says.

Speaking of "Thank you" with a smile ... ...

"Here it is!" Respond with a smile.

Stop the mouth and say "Thank you" ......

"At least I feel good."

Tilt your face and say "Thank you" ......

"What's wrong?" Hank was concerned.

In addition, Hank can orally reply to text.

When asking "What are you talking about?" ...

"I am talking about you," responding to nature.

When asking "Where do you like Christmas?" ...

"This is the first time for me to experience Christmas ... so ...".

When asking, "Did you know that Christmas illumination is in the form of red chili pepper?" ...

It is possible to accept and respond humorously, "That seems to be a cockroach."

In the following movie you can see how you talk to Hank of chatbot in real time.

face2face_ demo on Vimeo


In the movie, Hank's developer answered "Please tell me how this will change the world easily" and replied, "This is the first step to make human clones digitally" It is.