Aug 04, 2026 12:52:00

Cloudflare has released information on how to streamline its high-speed inference service for large-scale AI models.

Cloudflare offers ' Workers AI, ' an inference service that uses open AI models. An article explaining methods for improving the inference efficiency of the AI used in Cloudflare Workers AI has been published on Cloudflare's official blog.

Smaller, faster, safer: running Kimi and GLM at scale | The Cloudflare Blog
https://blog.cloudflare.com/smaller-faster-safer-models/

Cloudflare Workers AI deploys large-scale AI models such as Moonshot AI's Kimi series and Z.ai's GLM series. Because large-scale AI models consume a lot of memory, measures to reduce memory consumption are essential for deploying high-speed inference services.

Workers AI Models · Cloudflare Workers AI docs
https://developers.cloudflare.com/workers-ai/models/

One of the memory consumption reduction techniques Cloudflare employs is 'quantization.' In the case of GLM-5.2, it quantizes to INT4 before deploying its inference service. The FP8 version of GLM-5.2 has a file size of 705GB, but the INT4 version reduces it to 421GB.

Generally, AI models are said to perform worse as the computational precision is reduced, but according to Cloudflare, there is no noticeable performance degradation in the INT4 version of GLM-5.2. The benchmark results for the FP8 version of GLM-5.2 and the INT4 version of GLM-5.2 are as follows, showing that even when quantized to INT4, the performance degradation is suppressed compared to FP8.

benchmark	index	FP8	INT4
GSM8K	Exact match	94.39%	93.56%
GSM8K	Flexible	94.24%	93.48%
ARC-Easy	Comfort	86.62%	86.15%
ARC-Easy	Acc (norm)	84.51%	85.19%
ARC-Challenge	Comfort	64.93%	64.85%
ARC-Challenge	Acc (norm)	67.24%	66.64%
MMLU	Average	86.60%	86.54%
MMLU-Pro	Exact	80.80%	80.47%
mcxams (internal protocol)	Passed	62/63	62/63

During decoding, the model weights need to be read from GPU memory, so processing speed improves as the model size decreases. For this reason, the INT4 version of GLM-5.2 can process faster than the FP8 version. However, the FP8 version is faster for prefilling up to outputting the first token.

Number of simultaneous requests	FP8(tok/s)	INT4(tok/s)	difference
1	60	92	+55%
8	425	513	+21%
16	683	825	+21%
32	994	1267	+27%
64	1672	1933	+16%

Cloudflare also implements KV cache quantization. KV cache is a technology that speeds up processing by saving already computed content and skipping recalculations of the same content. The inference engine ' SGLang ' normally stores KV cache in BF16, but Cloudflare increases the amount of KV cache that can be stored in memory by quantizing the KV cache to FP8. In the case of Kimi K2.6, the amount of context that can be stored in memory increases from approximately 686,000 tokens to approximately 1,370,000 tokens.

Quantizing the KV cache increases the processing required for reading, slightly reducing processing speed. The table below summarizes the processing speed for each number of requests. Quantizing the KV cache to FP8 slightly reduces processing speed, but it can handle more requests simultaneously than BF16.

Number of simultaneous requests	BF16 KV cache (tok/s)	FP8 KV cache (tok/s)
1	137	125
8	731	689
16	1106	1028
32	1558	1489
64	Insufficient memory	2192

Cloudflare aims to support more users at a lower cost by continuing to explore quantization with NVFP4 and refining KV cache quantization.