May 01, 2024 16:00:00

'StoryDiffusion' is an AI model that automatically generates story-driven manga

With the copyright of

manga using AI-generated images causing controversy in the United States, and the emergence of AI services that suggest manga settings and plots, AI is beginning to be used not only to generate pictures and illustrations but also to create manga works. A research team from China's ByteDance and Nankai University has announced an AI model called 'StoryDiffusion' that can output manga with a story.

StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation
https://storydiffusion.github.io/

StoryDiffusion can output cartoon characters with a consistent look. For example, the following shows the results of outputting a robin, a squirrel, and a hedgehog character from the top with prompts such as 'wake up in bed,' 'eat breakfast,' and 'be on the road,' from left to right. You can see that the character shapes and designs are almost the same.

It is possible to maintain consistency even with multiple characters.

With these features, StoryDiffusion can generate comics with a story. For example, the following comic was generated on the theme of 'When you win the Turing Award.' The story is about three men, as shown in the image on the right, who meet and get to know each other at the awards ceremony after receiving the news of their awards. Although the shape of the hands is a bit worrisome, the appearance of the characters is consistent.

There is also a cutely illustrated manga called 'The Girl and the Squirrel.'

To generate a series of images that fit the story, the research team divided the story text into several prompts and generated images in bulk based on those prompts, allowing them to generate multiple cohesive images.