A method of generating high-quality images by combining the image generation AI ``Stable Diffusion'' with ``LoRA that collects bad examples'' has appeared, and a demo that can be easily tried has also been released, so I tried it

`` Stable Diffusion XL 1.0 (SDXL 1.0) '' released in July 2023 not only can generate higher quality images than existing Stability AI models, but also supports adjustment of images generated by LoRA. . Data scientist Max Wolf devised a method to further improve the quality of images generated by SDXL 1.0 using LoRA created by collecting 'bad examples', and released LoRA files and demos that anyone can try.

I Made Stable Diffusion XL Smarter by Finetuning it on Bad AI-Generated Images | Max Woolf's Blog

LoRA is a mechanism that makes it easier to generate images that include your favorite patterns and clothes by collecting reference images and learning additional patterns and clothes. For example, by applying the LoRA file learned from the image of ' live-action Sonic with teeth ' to SDXL 1.0, you can generate a large number of images of 'live-action Sonic with teeth' as follows.

Stable Diffusion has a mechanism to call LoRA's pattern by including a 'trigger word' in the prompt. Mr. Wolf said that as a method to solve problems such as ``the human body collapses'', ``mysterious watermarking'', and ``cutting out at strange positions'' that occur frequently in image generation by AI, ``LoRA that learned mistakes that are common in image generation AI and generate an image by specifying the created LoRA trigger word as a negative prompt.

Mr. Wolf first uses SDXL 1.0 to identify ``low quality'', ``error'', ``missing or extra fingers'', ``cropped'', ``watermark ( Using prompts containing words such as 'watermark' and 'ugly', a large number of broken images such as the following are generated. In addition, I created a LoRA file '

sdxl-wrong-lora ' with the generated image specified as training data and the trigger word set to 'wrong'.

The following three images all input the same prompt 'A wolf in Yosemite National Park, chilly nature documentary film photography', while the left side is 'Do not use sdxl-wrong-lora and do not specify a negative prompt', the center is The image on the right is generated under the conditions of 'Enter 'wrong' at the negative prompt without using sdxl-wrong-lora' and 'Use sdxl-wrong-lora and enter 'wrong' at the negative prompt'. Even if you just type 'wrong' into the negative prompt without using sdxl-wrong-lora, the background is finely drawn. The fur of the drawn wolf is also becoming more fluffy.

Mr. Wolf has also prepared a demo that can verify the quality improvement effect of generated images by sdxl-wrong-lora. To try the demo, access the link below while logged into your Google account.

sdxl_wrong_comparison.ipynb - Colaboratory


Running the demo is as easy as clicking the play buttons on the left side of the script, starting from the top left.

If the processing is successful to the end, two images of Mario are generated as follows. The image on the left is generated under the condition that 'sdxl-wrong-lora is not used and negative prompt is not specified', and the image on the right is 'using sdxl-wrong-lora and entering 'wrong' at the negative prompt'. , other conditions being identical. Comparing the images, the image on the right using sdxl-wrong-lora depicts Mario's eyebrows and gloves more accurately than the image on the left.

By rewriting the contents of the part marked 'prompt', you can generate an image with your favorite prompt. Below is an image generated with the prompt 'hacking girl, japanese anime, illustration' as a test. In the image on the left, there are problems such as ``the background is messy and I don't know what is drawn'' and ``the shape of the display is broken'', but the image on the right using sdxl-wrong-lora In the image, in addition to the neat city buildings drawn in the background, the display failure has also been resolved.

As a result of generating other images, images using sdxl-wrong-lora were generally of higher quality.

In some cases, the composition changed greatly as shown in the image below.

Note that the demo below can only generate high quality images using sdxl-wrong-lora, not comparison images.

sdxl_image_generation.ipynb - Colaboratory

Also, sdxl-wrong-lora is published at the following link.

minimaxir/sdxl-wrong-lora Hugging Face

in Review,   Software, Posted by log1o_hf