Smaller, faster, safer: running Kimi and GLM at scale

Cloudflare says it is using three serving optimizations for Moonshot’s Kimi and Z.ai’s GLM models on Workers AI: FP8 quantization of KV caches, INT4 compression of GLM weights, and integrity checks for shared cache pages. The changes are intended to fit more requests into GPU memory and reduce serving costs without materially changing model accuracy. On Kimi K2.6, FP8 KV caches doubled available context capacity from roughly 686,000 to 1.37 million tokens and enabled higher peak throughput after BF16 ran out of memory, though FP8 was slightly slower at the same concurrency. For GLM 5.2, INT4 reduced the checkpoint from 705 GB to 421 GB and improved decode throughput by 16% to 55% in the reported tests, while slowing compute-bound prefill; Cloudflare therefore uses different precisions for decode and prefill. The cache-integrity layer aborts requests if page mappings do not match and added less than 1% to measured throughput and p95 latency. Cloudflare reports running and benchmarking the work with SGLang and plans to expand FP8 KV caching, test NVFP4 weights on Blackwell GPUs, and further reduce the cost of integrity checks.