A new inference engine called 'Colibrì' has emerged that can run the massive AI 'GLM-5.2,' with 744 billion parameters, on a regular PC with 25GB of memory.
Colibrì, an inference engine that runs the large-scale language model 'GLM-5.2,' which has 744 billion parameters, on a general-purpose PC with 25GB of memory, has been released. Instead of loading the entire model into VRAM or RAM, it uses a mechanism that reads only the necessary data from the SSD as needed.
GitHub - JustVugg/colibri: Run GLM-5.2 (744B MoE) on a 25GB-RAM consumer machine — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦 · GitHub
While there are increasing examples of massive AI models being released as open models for free, these models require hundreds of gigabytes of memory to actually run, making them difficult to execute on a home PC. GLM-5.2 is Z.ai's flagship model designed for long-term tasks and boasts high performance, capable of stably handling contexts of up to 1 million tokens. However, with 744 billion parameters, even with memory-saving techniques such as quantization, it is difficult to run on a home PC. Colibrì is an inference engine that makes GLM-5.2 runnable even on PCs with limited memory, and its developers claim to have been able to run GLM-5.2 on a PC with a 12-core CPU and 25GB of memory.
GLM-5.2, a Chinese model surpassing Claude Opus 4.7, has been officially announced. It has even outperformed Claude Fable 5 in some tests and is available for download as an open model - GIGAZINE
Colibrì's developer, vforno, noted that GLM-5.2 employs a structure called 'Mixture-of-Experts (MoE).' In MoE, instead of using all 744 billion parameters each time a question is answered, only a selection of 'experts' appropriate to the input are chosen for calculation. Approximately 40 billion parameters are actually activated when GLM-5.2 generates one token.
Colibrì compresses approximately 17 billion parameters, including an attention mechanism that identifies important parts of a text and an embedding layer that converts words into numerical data, into 4 bits and stores them in RAM as approximately 9.9 GB of data. Meanwhile, the 21,504 experts whose selections change depending on the input are stored on storage such as an SSD as approximately 370 GB of data. Only the experts needed during answer generation are loaded from storage.
To reduce the number of times the SSD is accessed, an LRU cache is provided that keeps recently used experts in RAM. Furthermore, a learning cache is included that records frequently called experts from the usage history and preferentially fixes them to available RAM. The usage history is recorded for each interaction, and the next time the system is started, the frequently used experts will be preferentially loaded into available RAM.
The execution engine is built around a C language file of approximately 1300 lines, and during inference execution, it has no dependencies on external libraries and does not require Python or external numerical computation libraries. A GPU is also not required, and the main requirements are 'Linux or Windows Subsystem for Linux 2 (WSL2)', 'OpenMP-compatible GCC compiler', 'AVX2-compatible CPU', '16GB or more of RAM', and 'a local NVMe SSD capable of storing approximately 370GB of models'.
However, while it can run on less memory, its processing speed is extremely slow. On the PC used for development, which had a 12-core CPU and 25GB of memory, the token generation rate was approximately 0.05 to 0.1 tokens per second with an empty cache. With an empty cache, approximately 11GB of data needs to be read from the SSD for each token, so even short answers can take several minutes. vforno also explained that the goal was not speed, but 'to make it work no matter how slow.'
The system also implements 'Speculative Decoding using Multi-Token Prediction,' which predicts multiple candidate tokens in advance and verifies them together. While this may reduce the number of reads when the necessary experts are stored in the RAM cache, it may actually slow things down if the cache is empty by calling extra experts. Furthermore, it is stated that at the time of writing, full-scale measurement of the accuracy degradation due to 4-bit quantization and the implementation of a sparse attention mechanism for efficiently processing long contexts have not yet been completed.
vforno is seeking support in the form of benchmark results on PCs equipped with high-speed SSDs and large amounts of RAM, quality assessments, and contributions to code improvements.
Related Posts:
