AI ``StableVideo'' that maintains consistency between frames from movies and text and generates highly practical movies
Since the appearance of Stability AI's `` Stable Diffusion '' in August 2022, image generation AI has made rapid progress and can now generate not only images but also movies. However, objects and backgrounds drawn in movies created by generation AI change their shape and color drastically, so it can be said that they are not practical. ' StableVideo ' announced by Zhejiang University and Microsoft's research team introduces the concept of time into the text-driven diffusion model, making it possible to generate stable and highly practical movies.
rese1f.github.io/StableVideo/
GitHub - rese1f/StableVideo: [ICCV 2023] StableVideo: Text-driven Consistency-aware Diffusion Video Editing
https://github.com/rese1f/StableVideo
A video consists of a series of still images (frames). StableVideo maintains consistency between frames, allowing you to pass information from one generated frame to the next to maintain consistency and generate stable movies.
Below is a movie generated by actually loading the ship's movie (left) into StableVideo and prompting the text 'A Red Ship' and 'Sunset'. There are no flickering movie frames or distorted colors or objects.
Two movies generated by reading the movie of the ship into 'StableVideo' and entering text - YouTube
A movie in which a car runs (left) and based on it, ``A Rusty Car in Dessert'' and ``A Graffiti Car in Miami'' The movie generated with the prompt is like this.
Two movies generated by reading the video of the car running in 'StableVideo' and entering text - YouTube
The following is a movie of 'A White Swan' and 'A Duck' generated from a black waterfowl movie.
A movie that reads a black waterfowl movie in 'StableVideo' and turns it into a swan or a duck - YouTube
StableVideo's repository is published on GitHub, and StableVideo's pre-trained models are distributed on HuggingFace. Also, the movie for the sample is published on DropBox .
lllyasviel/ControlNet · Hugging Face
https://huggingface.co/lllyasviel/ControlNet
Related Posts:
