Google announces ultra-high quality video generation AI ``Lumiere'', with natural connections between frames, ``generate videos from text and images'', ``animate part of a photo'', ``specify and process a part of a video'', etc. Functions such as Tenkomori
Google's research team announced the video generation AI ' Lumiere ' on Wednesday, January 24, 2024. With Lumiere, you can generate videos with natural connections between frames that look natural, and you can generate videos by inputting text and images. Operations such as moving are also possible.
Lumiere
There are many AIs that can generate videos by inputting text and images, but many of these AIs generate videos by ``generating multiple frames and interpolating between the frames.'' , the connections between frames tend to be unnatural. On the other hand, Lumiere uses a method that ``generates the entire video frame at once'', making it possible to generate videos with natural connections between frames and less discomfort. The frame rate of videos that can be generated is 16fps, and the video length is 5 seconds. Play the video below to see an example of a video generated with Lumiere.
Lumiere - YouTube
Lumiere not only allows you to input text and images to generate a video according to the input content, but also has functions such as generating a video while maintaining the image style, and replacing part of the video with the generated video. . An overview of Lumiere's various functions is as follows.
◆Generate a video of a different subject while maintaining the image style
In the example below, a video of a different subject such as a duck or a butterfly is generated while maintaining the 'sticker-like' style of 'sticker-like image depicting a tree.'
Below is an example of how to generate a video while maintaining the 'melting gold' style.
◆Process videos with AI
In addition to text and images, it is also possible to input videos and process them with AI. In the example below, the original video was modified with prompts such as 'Made of wooden blocks,' 'Origami art,' 'Made of toy blocks,' and 'Made of flowers.'
◆Processing a part of the video
You can also select only a part of the video and process it. In the example below, only the woman's body from the neck down is selected and processed with the prompts ``She is wearing a gold strapless gown'' and ``She is wearing a striped strapless dress.''
In addition, in the example below, the chick's torso is selected and processed with the prompt ``Wearing a bathrobe'', and the top of the chick's head is selected and processed with the prompt ``Wearing a party hat.'' Masu.
◆Move part of the image
You can also do things like ``make the butterfly's wings flap in the image'' or ``make the water surface in the image ripple.''
◆Complete missing parts of the video
You can also black out part of the video and supplement the blacked out part with the generated video.
Technical details of Lumiere can be found in the paper below.
[2401.12945] Lumiere: A Space-Time Diffusion Model for Video Generation
https://arxiv.org/abs/2401.12945
Related Posts:
