Nov 06, 2019 06:00:00

AI will be developed that reflects the movement in the movie in just one photo and creates a realistic movie

A research team from

NVIDIA , known for its GPU and AI development, announced that it has developed 'AI that creates a realistic movie that reflects the movement in a movie in a single photo.'

Few-shot Video-to-Video Synthesis
https://nvlabs.github.io/few-shot-vid2vid/

In recent years, with the development of AI technology, it has become possible to create new fictitious movies and photos from datasets such as movies and photos. However, the existing model requires a large number of data sets related to the subject, and the learned model is difficult to generalize widely to other targets. Therefore, the NVIDIA research team developed `` AI that extracts only the movement in the input movie and combines it with one photo that the model has not learned so far to create a realistic movie '' .

You can see what the newly developed AI looks like by watching the following movie.

Few-Shot Video-to-Video Synthesis (NeurIPS 2019)-YouTube


On the left side of the screen is an abstract motion movie that was previously input to the model. In the top movie, a stickman dances, in the second, the face drawn with a line talks about something, and in the third, a landscape seen from a car traveling on the road is shown.

Combining these movies and the photos under the caption 'example' using a model developed by the research team ...

People and landscapes in the photo were combined with the movements represented by the abstract movie, creating a realistic movie.

The people in the output movie are dancing and scolding with almost no sense of incongruity, and I don't think there is only one photo with the original data.

Also, if you replace the input image with another one ...

The output movie is also a different person and landscape. In this model, it is possible to create a new movie based on a simple movie that has been input and combined with many other photos.

Even if the input images are different poses and people ...

The dance is as smooth as the input movie.

Also, the motion input movie does not necessarily have to be abstract, and a live-action dance movie can be combined with

Michelangelo 's David image . It is quite surreal that the sculptures dance and bounce.

Also, a person's face photo is combined with a news reporter movie ...

It is also possible to create a movie in which a person with a face photo speaks smoothly.

The input image can be made into a painting such as

Mona Lisa, and a movie spoken by the person in the painting can be created.

You can also combine abstract movies with real-world landscape photos. The input movie on the left has no specific information about each object or building, but it can be guessed from the example image and output a movie-like movie.