A fierce player has appeared who runs Stable Diffusion on Xbox. By using ONNX, dependency on Python is zero, and it works on NVIDIA's CUDA, AMD's ROCm, Apple's CoreML, Qualcomm's QNN, Microsoft's DirectML, and Windows, Linux, macOS, Android, iOS, WebAssembly, etc., making it possible to integrate it into almost any application.
Stable Diffusion is an AI that generates new images based on text and images, but it requires Python as its runtime environment. However, someone has now successfully implemented it on Xbox by using ONNX models and ONNX Runtime, implementing it in C++, and eliminating the dependency on Python.
The large icon on the far left is the newly created app, 'Unpaint.' At the time of writing, it is not available on the Xbox Store, and users must sign the package themselves to install it. It is said to work on both Xbox Series X and Xbox Series S. The output of the Xbox Series X is displayed on the large screen on the left.
When you start it up, it looks like this. You can enter the prompt one character at a time using the virtual keyboard.
After entering the prompt, an image was generated in about 20 seconds.
After that, images are generated at a rate of one every 20 to 30 seconds.
The ONNX project played a major role in running Stable Diffusion on Xbox. AI models often require specific environments for their execution environments and frameworks to run, but if a model can only run in a single environment, combining models can become difficult if multiple models require different environments. ONNX is a Microsoft-led project aimed at solving this problem, with the goal of increasing interoperability by allowing any model to run on any tool or platform.
One of the ONNX projects is the ONNX Runtime , a tool that allows ONNX format models to run on any platform, including NVIDIA's CUDA, AMD's ROCm, Apple's CoreML, Qualcomm's QNN, and Microsoft's DirectML. It also supports any operating system, including Windows, Linux, macOS, Android, and iOS. It also supports WebAssembly.
In this example, a Stable Diffusion model converted to an ONNX model was run in C++ using ONNX Runtime, enabling it to run on Xbox. Because the entire image generation process is carried out in a single process, it can be installed simply and on a small scale. Excluding the 2.5GB required for saving the model, it only uses around 30MB of space.
The Xbox app creator has also published the code for calling Stable Diffusion in C++ on GitHub. It includes functions like 'txt2img' and 'img2img,' which generate images from text and images, as well as 'inpainting,' which edits parts of an image, and a 'safety checker' to verify safety. Since Stable Diffusion can be used in any project that can import C++ or C functions, it's a valuable tool for those considering integrating it into real-time CG applications, such as game development.
Related Posts:
