Aug 26, 2026 07:00:00

Unsloth has developed 'Unsloth Dynamic 3.0 GGUF,' a next-generation quantization method that reduces memory usage while maintaining AI performance as much as possible.

Unsloth is a company that works on quantizing AI models, and it has become customary for Unsloth to release quantized versions of open models released by AI companies such as Google and Alibaba within a few days. Now, Unsloth has developed a new quantization method called 'Unsloth Dynamic 3.0 GGUF (hereinafter referred to as Dynamic 3.0)' and has released details of it. Dynamic 3.0 has already been applied to Qwen3.8-27B and has succeeded in reducing memory usage while suppressing performance degradation.

Unsloth Dynamic 3.0 GGUFs | Unsloth Documentation
https://unsloth.ai/docs/basics/dynamic-3.0-ggufs

Unsloth Dynamic 3.0 GGUF | Unsloth Documentation
https://unsloth.ai/docs/jp/ji-ben/dynamic-3.0-ggufs

Quantizing AI models works by reducing the file size and thus memory usage of the model at the expense of computational precision. However, quantized models tend to perform worse than the original models. Therefore, development is underway to reduce file size while maintaining performance as much as possible. Unsloth's Dynamic 3.0 has been able to maintain higher performance compared to the previous Dynamic 2.0.

The graph below shows the file size and performance of Qwen3.8-27B at different quantization densities. The green line represents the results of the Dynamic 3.0 quantization model, demonstrating that performance degradation is minimized when the density is reduced.

At the heart of Dynamic 3.0 are improved methods and new features. A prime example is the adoption of the imatrix calibration dataset, which has been collected from diverse sources and optimized for improving agent coding, chat, and multilingual performance. Furthermore, various quantization techniques have been introduced to improve layer selection and maintain model quality. Notably, it employs only conventional post-training quantization (PTQ), without using newer techniques such as QAT (Quantization Aware Training) or QAD (Quantization Aware Distillation), which distills knowledge from the original model after quantization. The imatrix files are available for community testing, evaluation, and use, and researchers and developers are encouraged to use the imatrix calibration dataset to create variations and fine-tune Qwen 3.8.

Disk space requirements were also reduced by quantizing Qwen3.8-27B with Dynamic 3.0. The smaller quantization model UD-Q2_K_XL reduced disk space by approximately 500MB by removing the MTP module. The Q4_0 MTP module can also be used if needed. Furthermore, smaller UD-1bit quantization models like UD-IQ1_S have been developed, achieving an 89% size reduction while maintaining approximately 72% Top-1 accuracy at a size of 6.2GB without MTP. UD-Q2_K_XL maintains a size of 9.83GB despite having approximately 8% higher Top-1 accuracy compared to the next best model, making it possible to create working applications, even with minor JavaScript bugs, under conditions where previously only broken web applications could be created.

The evolution of evaluation metrics is another important factor. Traditionally, Top-1 accuracy, used to evaluate the accuracy of machine learning models, is

argmax for a single prediction and has limitations in accurately measuring actual inference ability. Therefore, Unsloth created a dataset consisting of 300 holdout examples not included in the calibration dataset, using Terminal-Bench 2.1, DeepSWE, Harbor, MathArena 2025-26, non-Latin characters, and long prompts, and performed 'greedy' argmax decoding of 32 tokens between BF16 and the fully quantized model. This metric, named 'Divergence-300 @32,' helps to evaluate whether overfitting has occurred and how similar the quantized output is to the BF16 trajectory across multiple tokens. In other words, it is like KL Divergence , which extends Top-1 accuracy to 32 tokens, and can be said to be a more practical evaluation method than Top-1 accuracy.

The graph below illustrates the results of quantizing Qwen3.8-27B with Dynamic 3.0 using various quantization models for 'Divergence-300 @32'. Upon closer inspection, it can be seen that the accuracy drops sharply from approximately 25% to 8-10% when transitioning from UD-Q2_K_XL to UD-IQ2_S. This decrease in accuracy can lead to malfunctions in tool calls and non-thinking modes, so 1-bit quantization is not recommended for agent applications.

The main problems and mitigation strategies when using the low-precision quantized version of Qwen3.8-27B are as follows:

- Excessive loops : Quantization models less than UD-Q2_K_XL tend to have many loops. In this case, it is always recommended to use 'presence_penalty = 1.5 (or higher)'.
• Empty responses : In 1-bit quantization models, it is recommended to always enable thinking mode, at least at low levels of inference. In non-thinking mode, the model may not generate a response.
- Agent Use Cases and Tool Calls : Using models for tool calls is not recommended. Highly quantized models only retain general knowledge, which can lead to issues such as 'tool call failure,' 'repeated calls,' or 'no call at all.' While they can be used for short questions about general knowledge, it is best to use UD-Q2_K_XL.

Unsloth also performed KL Divergence benchmarks for all providers and reported on Top-1 accuracy and average KL Divergence. Data on quantization of Qwen3.8-27B with Dynamic 3.0 shows up to a 10% improvement in Top-1 accuracy for the same disk capacity, especially at smaller quantization sizes. However, there is still room for improvement with larger models.

Overfitting is suppressed by completely separating the calibration dataset and the test dataset and eliminating leaks as much as possible. When the KL Divergence test was run on these unseen datasets, it was found that because it is a pure PTQ that does not use QAD/QAT, the risk of overfitting is lower compared to other QAD/QAT approaches. In addition, 'Divergence-300 @32' also uses an unseen dataset consisting of 300 prompts obtained from DeepSWE, Terminal Bench, etc., so it is another indicator that the Dynamic 3.0 method does not overfit.