A fierce man who runs Stable Diffusion on Xbox has appeared, using ONNX to reduce the dependence on Python to zero, NVIDIA's CUDA, AMD's ROCm, Apple's CoreML, Qualcomm's QNN, Microsoft's DirectML, Windows, Linux, macOS, Android, iOS, WebAssembly, etc. are all OK, and integration into almost any application is possible
Stable Diffusion is an AI that generates new images based on characters and images, but Python has been required as an execution environment. In the midst of this, a stalwart appeared who succeeded in running on Xbox by eliminating the dependence on Python by utilizing the ONNX model and ONNX Runtime and implementing it in C++.
An application called 'Unpaint' whose large icon on the left was created this time. It does not exist in the store at the time of article creation, and the user needs to sign the package himself to install it. It can work on both Xbox Series X and Xbox Series S. The large screen on the left shows the Xbox Series X output.
It looks like this when you start it. I am typing the prompt character by character using the virtual keyboard.
After entering the prompt, one image was generated in about 20 seconds.
After that, images are generated at a pace of one every 20 to 30 seconds.
The ' ONNX ' project played a major role in running Stable Diffusion on the Xbox. There are many cases where a specific environment is required for the execution environment or framework to operate an AI model, but if a model operates only in a single environment, multiple models may require different environments. It quickly becomes difficult to combine models. ONNX is a project led by Microsoft to solve such problems, aiming to improve interoperability by running any model on any tool platform.
One of the ONNX projects is ONNX Runtime , which is a tool that allows ONNX format models to run on any platform such as NVIDIA's CUDA, AMD's ROCm, Apple's CoreML, Qualcomm's QNN, Microsoft's DirectML. Any OS such as Windows, Linux, macOS, Android, iOS etc. is OK. It also supports WebAssembly.
This time, the model of Stable Diffusion converted to ONNX model is operated with C++ using ONNX Runtime to realize operation on Xbox. Since the entire image generation process is performed in one process, it can be installed simply and on a small scale. It seems that only about 30MB capacity is used except for using 2.5GB for saving models.
Also, the code to call Stable Diffusion in C ++ is published on GitHub of this Xbox app author. In addition to 'txt2img' and 'img2img' that generate images from characters and images, 'inpainting' that corrects part of the image and 'safety checker' that checks safety are available. Since Stable Diffusion can be used from any project that can import C++ or C functions, it is a very useful tool for those who are considering integration into real-time CG applications such as game development. That's it.
Related Posts:
