Research is under way to make computer understand "non verbal communication" by reading human body language

When humans communicate, they read information from "body language" such as hand shake, facial expressions, body movements as much as information from words uttered from mouth. "Non-verbal communication ability" which enables human beings to learn and read naturally, but research is being carried out to make computers even acquire the same ability.

Cracking the elaborate code - The Verge
http://www.theverge.com/2016/12/7/13857144/social-vr-carnegie-mellon-panoptic-studio-facebook-oculus-toybox

This research is conducted by Carnegie Mellon University's research team. A movie of The Verge who visited the laboratory and covered special equipment was released.

Cracking the code of body language to build lifelike robots - YouTube


Spherical special structure installed in the research facility. This reads human movements in 3D "Panoptic Studio("Panoramic studio").


The inside is like this. Like a soccer ball, it is a spherical body composed of pentagon and hexagonal panels, and you can see that there are large and small black spots on each panel.


These points are where large and small cameras are installed. Twenty-four small VGA cameras and one HD camera are built in each panel, and the cameras are designed to be arranged as evenly as possible when assembled. There are 480 VGA cameras and 30 HD cameras that are used throughout the studio, as well as the "Kinect Sensor" which can measure the distance to the object.


Reporter of The Verge dancing in the studio. The camera and sensor read this movement ... ....


When 510 images and sensor information are read and processed, ...


Data captured body motion in 3D was generated. Although it is obvious if you look at the movie, innumerable colorful lines show the direction in which the body moved, and the length of the line shows the speed of movement. On this screen, it seems that the dancing reporter and the photographer shooting it are captured.


The image from each camera reaches 600 GB per minute.


The total amount of data captured so far reaches as much as 1 petabyte.


As a technology to 3D capture human motion, "motion capture" which is used in the movie world etc. is well known.



"Goram (smeargol)" which appears in the movie "Lord of the Rings" is also a motion capture and reproduction of the movement which the actual human being played, including the expression of the face.


However, since motion capture needs to wear special suits with multiple balls, it will not be usable in the real world.


The excellent point of Panoptic Studio is that you can capture and move data of your body in detail, even without wearing special costumes. When we capture a situation where several people are talking, ...


The movement of three people was converted to 3D data.


Even if there are multiple people, it seems that they can be converted to data without major problems.



When humans communicate with someone, they are doing a variety of movements, such as poorly rubbing, moving hands, distributing their eyes, and they are one of the signs of emotions and atmosphere.



Hanbyul Joo, a research assistant who says that Panoptic Studio's purpose is to make computer understand body language by capturing such data and converting it into data.


It is deep learning of artificial intelligence (AI) which is cutting-edge computer technology, as well as make it possible to read expressions from the data obtained in this way.


First of all, let AI teach you what kind of reaction the actual person shows, so that gradually let AI learn to make your own judgment. This is the same thing that AI learns to drive by automatic driving technology which Google and others are conducting research.


Computers are now acquiring "reading lipics" by guessing what they are talking about by reading only the movement of the mouth of the person speaking.


And further, the computer is trying to be able to recognize the information from other than words by reading the motion of the human body.


This technology is going to be utilized to a great extent in the future as robots come into people's lives.


When a robot enters the welfare field such as nursing care and the other life, it becomes very important that the robot can do the same thing as human communication.


As before, it may be the day when robots communicate using "body" just as human beings do.