Sep 16, 2022 20:00:00

I tried using 'stable-diffusion-videos' that can create movies with image generation AI 'Stable Diffusion'

AI '

Stable Diffusion ' released to the public in August 2022, ' an astronaut riding a horse ' and ' elon musk as dr strange ' If you enter a character string such as , it will automatically generate an image corresponding to it. Since ' stable-diffusion-videos ' that can create movies using such Stable Diffusion has been released, I actually tried using it.

GitHub - nateraw/stable-diffusion-videos: Create ???? videos with Stable Diffusion by exploring the latent space and morphing between text prompts
https://github.com/nateraw/stable-diffusion-videos

For stable-diffusion-videos, a notebook that runs on the online execution environment Google Colaboratory (Google Colab) is open to the public, and you can try it for free even on a low-spec PC.

Before using the notebook, you need to issue a token on the Stable Diffusion page on the AI-related community site Hugging Face. First, access the Stable Diffusion page published on Hugging Face and log in. If you don't have an account, create a free account on the registration page .

CompVis/stable-diffusion-v1-4 Hugging Face
https://huggingface.co/CompVis/stable-diffusion-v1-4

On the above page, check 'I agree to the license' and click 'Access repository'.

Then click the user icon in the upper right and select Settings.

Click 'Access Tokens' in the left column.

Click 'New token'.

A 'Create a new access token' pop-up will appear. Enter a name in Name, select 'read' in Role, and click 'Generate a token'. Since there is no particular name specified, this time it is 'test'.

Then, an access token was issued as follows. Copy the token by clicking the icon to the right of the token.

Next, while logged into your Google account, access the following Google Colab notebooks.

stable_diffusion_videos.ipynb - Colaboratory

https://colab.research.google.com/github/nateraw/stable-diffusion-videos/blob/main/stable_diffusion_videos.ipynb

A warning will appear, but click 'Run Anyway'.

If you place the cursor over the [ ] part, a play icon will be displayed, so basically press this play icon in order from the top to proceed with the introduction.

In the 'Authenticate with Hugging Face Hub' item, if you click the play icon and execute it, an access token input field will be displayed. Paste the access token obtained from Hugging Face here and click 'Login'.

Run the App will install stable-diffusion-videos.

Also, you need to mount Google Drive to save the generated movie, and you will be asked for permission to access Google Drive. Click 'Connect to Google Drive'.

Select the account logged into Google Colab.

Click 'Allow'.

Clicking the 'Launch' play icon will launch the web UI on your Google Colab notebook. Image generation is also possible from here, but it is narrow, so access the URL displayed at the top.

The web UI looks like this. Click the 'Videos!' tab at the top, set it and click 'Send' OK.

The setting items are as follows.

prompt1: first image
seed1: Seed value for generating the first image
prompt2: second image
seed2: seed value for generating the second image
schedule: Sampler
guidance_scale: CFG scale
num_inference_steps: number of inference steps
num_walk_steps: number of steps to generate, the higher this is, the longer the video
do_loop: Make loop video
disable_tqdm: Hide generation progress bar
use_lerp_for_text: use

linear interpolation
output_dir: output destination directory

Google Colab shows a progress bar when running movie generation on stable-diffusion-videos.

It took about 3 minutes to generate 10 images for use in the movie, and a 'movie that changed from blueberry spaghetti to strawberry spaghetti' was generated.

So the following is the `` movie where blueberry spaghetti turns into strawberry spaghetti '' generated by `` stable-diffusion-videos ''.

``Movie where blueberry spaghetti turns into strawberry spaghetti'' created with ``stable-diffusion-videos'' where AI automatically generates movies-YouTube


If you want to save the movie locally, right click on the movie and select 'Save video as' OK.

This time I made a movie where Beethoven's face changes to Mozart. Set prompt1 to 'beethoven', prompt2 to 'mozart', guidance_scale to 10, num_inference_steps to 60, num_walk_steps to 15.

Below is the ``movie that morphs from Beethoven to Mozart'' that I actually made. The total number of steps was 30, and it took about 3 minutes to make a movie of only 1 second.

I made a Mozart morphing movie from Beethoven with 'stable-diffusion-videos' that can make movies with image generation AI 'Stable Diffusion'-YouTube