Meta develops video generation AI 'Movie Gen', which in addition to generating videos from text also supports 'specifying characters', 'adding background music' and 'editing existing videos'
On October 4, 2024, Meta announced the video generation AI ' Movie Gen. ' Movie Gen is capable of generating videos from text, generating videos from text by specifying characters in images, editing existing videos, and adding background music and sound effects to videos, and is said to be able to generate higher quality videos than competing video generation AIs.
Meta Movie Gen
How Meta Movie Gen could usher in a new AI-enabled era for content creators
https://ai.meta.com/blog/movie-gen-media-foundation-models-generative-ai-video/
◆Generate videos from text
The following video was generated using Movie Gen from the text 'A red-faced monkey with white fur is bathing in a natural hot spring. The monkey is playing in the water with a miniature sail ship in front of it, made of wood with a white sail and a small rudder. The hot spring is surrounded by lush greenery, with rocks and trees.' All the elements included in the text are included, and there is little AI-like discomfort.
Video generated by Meta's video generation AI 'Movie Gen' - YouTube
◆ Generate videos by specifying the characters
Movie Gen also allows you to input photos of people and specify them as characters. The following video was generated by inputting the image in the upper right and the text: 'A man is doing a scientific experiment in a lab with rainbow wallpaper. The man has a serious expression and is wearing glasses. He is wearing a white lab coat with a pen in the pocket. The man pours liquid into a glass beaker and a cloud of white smoke blooms.'
Videos generated by specifying the characters using Meta's video generation AI 'Movie Gen' - YouTube
◆Edit existing videos
You can also use Movie Gen to edit existing videos. The following four-part video is the original video in the upper left, the video edited with the text 'Change the VR headset to binoculars' in the upper right, 'Remove the VR headset' in the lower left, and 'Add bubbles and futuristic effect' in the lower right.
Edit existing videos with Meta's video generation AI 'Movie Gen' - YouTube
◆Add background music and sound effects to existing videos
You can also add background music and sound effects to existing videos. The timing of the background music and sound effects will be automatically adjusted to match the video. For example, the following video has background music and sound effects generated from the text 'Rain pours against the cliff and the person, with music playing in the background.'
Video with background music and sound effects added by Meta's video generation AI 'Movie Gen' - YouTube
◆Comparison with other video generation AI
In an A/B test comparing Movie Gen's performance with other video generation AIs such as Runway Gen3, OpenAI Sora, Kling 1.5, and ID-Animator, Movie Gen was found to be more highly rated than its rivals.
◆Movie Gen research papers
Movie Gen's research paper is available at the following link:
Movie Gen: A Cast of Media Foundation Models
(PDF file) https://ai.meta.com/static-resource/movie-gen-research-paper
Related Posts:
