A method of generating high-quality images by combining the image generation AI ``Stable Diffusion'' with ``LoRA that collects bad examples'' has appeared, and a demo that can be easily tried has also been released, so I tried it
I Made Stable Diffusion XL Smarter by Finetuning it on Bad AI-Generated Images | Max Woolf's Blog
https://minimaxir.com/2023/08/stable-diffusion-xl-wrong/
LoRA is a mechanism that makes it easier to generate images that include your favorite patterns and clothes by collecting reference images and learning additional patterns and clothes. For example, by applying the LoRA file learned from the image of ' live-action Sonic with teeth ' to SDXL 1.0, you can generate a large number of images of 'live-action Sonic with teeth' as follows.
Stable Diffusion has a mechanism to call LoRA's pattern by including a 'trigger word' in the prompt. Mr. Wolf said that as a method to solve problems such as ``the human body collapses'', ``mysterious watermarking'', and ``cutting out at strange positions'' that occur frequently in image generation by AI, ``LoRA that learned mistakes that are common in image generation AI and generate an image by specifying the created LoRA trigger word as a negative prompt.
Mr. Wolf first uses SDXL 1.0 to identify ``low quality'', ``error'', ``missing or extra fingers'', ``cropped'', ``watermark ( Using prompts containing words such as 'watermark' and 'ugly', a large number of broken images such as the following are generated. In addition, I created a LoRA file '
The following three images all input the same prompt 'A wolf in Yosemite National Park, chilly nature documentary film photography', while the left side is 'Do not use sdxl-wrong-lora and do not specify a negative prompt', the center is The image on the right is generated under the conditions of 'Enter 'wrong' at the negative prompt without using sdxl-wrong-lora' and 'Use sdxl-wrong-lora and enter 'wrong' at the negative prompt'. Even if you just type 'wrong' into the negative prompt without using sdxl-wrong-lora, the background is finely drawn. The fur of the drawn wolf is also becoming more fluffy.
Mr. Wolf has also prepared a demo that can verify the quality improvement effect of generated images by sdxl-wrong-lora. To try the demo, access the link below while logged into your Google account.
sdxl_wrong_comparison.ipynb - Colaboratory
Running the demo is as easy as clicking the play buttons on the left side of the script, starting from the top left.
If the processing is successful to the end, two images of Mario are generated as follows. The image on the left is generated under the condition that 'sdxl-wrong-lora is not used and negative prompt is not specified', and the image on the right is 'using sdxl-wrong-lora and entering 'wrong' at the negative prompt'. , other conditions being identical. Comparing the images, the image on the right using sdxl-wrong-lora depicts Mario's eyebrows and gloves more accurately than the image on the left.
By rewriting the contents of the part marked 'prompt', you can generate an image with your favorite prompt. Below is an image generated with the prompt 'hacking girl, japanese anime, illustration' as a test. In the image on the left, there are problems such as ``the background is messy and I don't know what is drawn'' and ``the shape of the display is broken'', but the image on the right using sdxl-wrong-lora In the image, in addition to the neat city buildings drawn in the background, the display failure has also been resolved.
As a result of generating other images, images using sdxl-wrong-lora were generally of higher quality.
In some cases, the composition changed greatly as shown in the image below.
Note that the demo below can only generate high quality images using sdxl-wrong-lora, not comparison images.
sdxl_image_generation.ipynb - Colaboratory
https://colab.research.google.com/github/minimaxir/sdxl-experiments/blob/main/sdxl_image_generation.ipynb
Also, sdxl-wrong-lora is published at the following link.
minimaxir/sdxl-wrong-lora Hugging Face
https://huggingface.co/minimaxir/sdxl-wrong-lora
Related Posts:
