I tried the combined technique of ``ControlNet'' & ``Stable Diffusion'' that allows you to specify poses and compositions and quickly generate your favorite illustration images Review



With the advent of image generation AI such as Stable Diffusion, an environment where you can easily output your favorite images is being prepared, but it takes a long time and experience to generate illustrations with your favorite poses with only text (prompt) instructions. is. If you use the newly developed '

ControlNet ' together with the image generation AI, you can easily generate illustrations that specify your favorite poses, so I actually installed it and checked how to use it.

GitHub - Mikubill/sd-webui-controlnet: WebUI extension for ControlNet
https://github.com/Mikubill/sd-webui-controlnet

How to generate AI illustrations by specifying poses and compositions with ControlNet|ddPn08|note
https://note.com/ddpn08/n/n7fce838499e7

·table of contents
◆ 1: Install ControlNet extension to 'Stable Diffusion web UI (AUTOMATIC 1111 version)'
◆2: Download ControlNet model data
◆3: Image generation by extracting skeletal data from reference images using ControlNet
◆ 4: Make your own pose with a design doll and generate an image

◆ 1: Install ControlNet extension to 'Stable Diffusion web UI (AUTOMATIC 1111 version)'
ControlNet is a neural network that can improve the quality of generated images by providing additional information such as poses, depth information, and contours when inputting prompts to image generation AI. This time, ControlNet will try to use the extended function for 'Stable Diffusion web UI (AUTOMATIC 1111 version)', so first follow the article below to install 'Stable Diffusion web UI (AUTOMATIC 1111 version)'.

Image generation AI ``Stable Diffusion'' works even with 4 GB GPU & various functions such as learning your own pattern can be easily operated on Google Colabo or Windows Definitive edition ``Stable Diffusion web UI (AUTOMATIC 1111 version)'' installation method summary - GIGAZINE



After starting 'Stable Diffusion web UI (AUTOMATIC1111 version)', click 'Extension', then click 'Install from URL', and enter ' https://github.com/Mikubill/sd-webui ' in the URL input field. -controlnet and click Install.



When the installation is completed, click 'Installed', confirm that 'sd-webui-controlnet' is in the extension list at the bottom of the screen, and click 'Apply and restart UI'.



If 'ControlNet' is displayed at the bottom of the 'txt2img' tab, the installation of the extension is complete.



◆2: Download ControlNet model data
Next, access the link below and download the ControlNet model data.

webui/ControlNet-modules-safetensors at main
https://huggingface.co/webui/ControlNet-modules-safetensors/tree/main

Moderators are lined up at the link destination. This time, I want to use the function 'Generate image by specifying pose', so click 'control_openpose-fp16.safetensors'.



Then click 'download' to download the model data (file size 689 MB).



OK if you can download the following files.



Next, move 'extensions' → 'sd-webui-controlnet' → 'models' in the installation folder of 'Stable Diffusion web UI (AUTOMATIC1111 version)', and move the downloaded model data into the 'models' folder. increase.



When the model data exists in the 'models' folder, preparation is complete.



◆3: Image generation by extracting skeletal data from reference images using ControlNet
In order to generate an 'image with a specified pose' using ControlNet, it is necessary to prepare a reference image of the pose. A reference image with a simple background is better. This time I prepared the following images.



To generate the image, first click 'ControlNet' placed at the bottom of the 'txt2img' tab.



Then, the setting items of 'ControlNet' are displayed.



To generate an 'image with a specified pose', check 'Enable', select 'openpose' in 'Preprocessor', and select the downloaded model data in 'Model'. In addition, enter the resolution of the reference image.



Next, drag and drop the reference image to the part marked 'Drop image here'.



It is OK if the reference image is displayed as follows.



Then enter the prompt and click Generate. This time, I entered 'girl, yoga'.



After waiting for a while, the image will be output together with the 'skeletal image of the pose'.



The image generated this time looks like this. I was able to output an image with the exact same pose as the reference image.



◆ 4: Make your own pose with a design doll and generate an image
You can generate an 'image with a specified pose' using the above procedure, but it takes time to find a reference image of your favorite pose. Therefore, I will create a reference image of my favorite pose using the free posing tool ' Design Doll '. To use Design Doll, first access the following link.

Design Doll | Next Generation Drawing Doll

https://terawell.net/ja/index.php

Then click 'Download'.



When the following screen is displayed, click 'Download' to download the installer (file size 43.2MB).



After downloading the installer, double-click to launch it.



Click 'Next'.



Click 'Next'.



Click 'Install'.



When the installation is complete, check 'Run DesignDoll' and click 'Finish'.



When the following screen is displayed, click the button on the lower right.



Then click Install next to the version you want to install. This time I installed version 5.6.0.0.



When the installation is complete, Design Doll will start. By dragging the legs and hands of the doll displayed on the screen, you can move each part, so you can pose it as you like.



This time, I created the following pose with both hands on the waist.



When the pose creation is completed, click 'File' and click 'Export image'.



Enter the file name and select PNG or JPG from the image format. Specify the resolution and click 'Save'.



This will output an image like the one below.



When the pose image is completed, drag and drop it into the reference image specification area of 'ControlNet'.



It is OK if the image is displayed as follows.



Then enter the prompt and click 'Generate'. This time I simply entered 'girl'.



You have now generated an image with your preferred pose specified.



After that, you can get closer to your favorite image by devising the model data and prompts.



If you show only the upper waist as a reference image as shown below, only the pose above the waist will be reflected in the generated image.



The generated result looks like this. Just by creating poses and compositions with tools such as design dolls, you can reflect them in the generated images, so you can generate your favorite images fairly intuitively.



in Review,   Software,   Art, Posted by log1o_hf