TurboQuant: Google’s 3-Bit KV Cache Compression Cuts LLM Memory 6x

The Memory Wall Holding Back LLM Deployment

Every time a large language model processes a long conversation or document, it builds a key-value (KV) cache — a running memory of all previous tokens that the attention mechanism references. For models like Llama 3.1 8B handling 128K-token contexts, this cache alone can consume 40 GB of GPU memory, often exceeding the space taken by the model weights themselves. That memory footprint directly limits how many users a single GPU can serve simultaneously and how long the context window can stretch.

Google Research has now demonstrated a way to compress that cache by 6x with near-zero quality degradation. Their algorithm, TurboQuant, was published on March 25, 2026 and will be formally presented at ICLR 2026 in Rio de Janeiro on April 25. The paper (arXiv: 2504.19874) was authored by Amir Zandieh, Majid Daliri, Majid Hadian, and Vahab Mirrokni.

How TurboQuant Works: Rotation Plus Residual Correction

TurboQuant is elegantly simple at its core. The algorithm uses a two-stage pipeline that compresses each KV vector from 16-bit floating-point down to approximately 3 bits per coordinate.

Stage 1 — PolarQuant. Each KV vector is multiplied by a random orthogonal matrix. This rotation spreads the energy uniformly across all coordinates, transforming the distribution into a predictable Beta distribution. Because the distribution is known mathematically, an optimal set of quantization buckets can be precomputed using the Lloyd-Max algorithm — once, ahead of time, for all models.

Stage 2 — Quantized Johnson-Lindenstrauss (QJL). A 1-bit sketch of the residual quantization error is computed and stored alongside the quantized vector. This error-correction step recovers most of the information lost during scalar quantization, pushing the overall compression to near-lossless levels.

The critical advantage is that TurboQuant is entirely data-oblivious. The same precomputed codebook works for every model, every layer, and every attention head. There is no calibration dataset, no fine-tuning pass, and no model-specific tuning required. This makes it a genuine drop-in replacement for the standard FP16 KV cache.

Benchmark Results: Near-Zero Quality Loss at 6x Compression

Google evaluated TurboQuant across five standard long-context benchmarks — LongBench, Needle-in-a-Haystack (NIAH), ZeroSCROLLS, RULER, and L-Eval — using Gemma, Mistral, and Llama 3.1 8B Instruct models.

The results are striking. At 3.5 bits per coordinate (TQ3.5), the algorithm achieves absolute quality neutrality — perplexity change is under 0.5% for Llama 3 and Mistral models. On the Needle-in-a-Haystack benchmark, TurboQuant maintains 100% retrieval accuracy through 104,000 tokens, matching full-precision performance exactly. At its most aggressive setting (TQ3, 3 bits), it delivers 4.9x compression versus FP16, storing each 128-value vector in just 52 bytes.

On NVIDIA H100 GPUs, 4-bit TurboQuant achieves up to 8x faster attention-logit computation compared to 32-bit unquantized keys. The practical implication is immediate: a 40 GB KV cache shrinks to roughly 6.7 GB, freeing enough memory to serve multiple concurrent requests or extend context windows dramatically on the same hardware.

How TurboQuant Compares to Existing Methods

TurboQuant enters a field with several established KV cache compression approaches, but it occupies a unique position.

KIVI, published at ICML 2024, introduced asymmetric 2-bit quantization and became the standard baseline, achieving 2.6x memory reduction. TurboQuant more than doubles that compression ratio while matching or exceeding KIVI’s quality — at 3.5 bits, TurboQuant scores 0.997 on the Needle benchmark versus KIVI’s 0.981 at 2 bits.

The vLLM inference engine already supports FP8 KV cache quantization natively, delivering roughly 2x compression versus BF16. It is production-ready today but offers far less compression than TurboQuant.

NVIDIA’s KVTC, also being presented at ICLR 2026, takes a different approach using PCA-based decorrelation and entropy coding to achieve an impressive 20x compression — but with a measurable accuracy penalty of less than 1 percentage point. TurboQuant trades lower compression for genuinely zero quality loss, a trade-off many production systems will prefer.

Market Shock: Memory Chip Stocks Rattled

The financial markets reacted swiftly to TurboQuant’s implications. The day after Google published the research blog, SK Hynix shares fell 6.23% and Samsung Electronics dropped 4.8% on the Korea Exchange. Japan’s Kioxia fell nearly 6%, while Micron and Sandisk declined in US trading.

The logic is straightforward: if AI workloads need 6x less memory per request, demand growth for HBM and DRAM chips could decelerate. Analysts, however, pushed back on the panic. Memory demand is driven by many factors beyond KV cache size, and lower per-request memory could enable more deployments overall — expanding the total addressable market rather than shrinking it.

The Production Gap: Research to Reality

As of April 2026, Google has not released an official implementation of TurboQuant. The community has filled the gap with multiple open-source implementations — PyTorch versions, Triton GPU kernels, a llama.cpp integration discussion, and even an Apple Silicon MLX port — but none carry Google’s endorsement or have been battle-tested at scale.

For engineering teams evaluating TurboQuant, the path to production involves integrating these community kernels into existing serving stacks like vLLM or SGLang, then validating quality on their specific model and workload. The algorithm’s data-oblivious nature makes this simpler than most quantization methods — there is no per-model calibration step to worry about — but kernel-level optimization for different GPU architectures remains active work.

Sources & Further Reading

• TurboQuant: Redefining AI Efficiency with Extreme Compression — Google Research Blog
• TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate — arXiv
• Google AI TurboQuant Memory Chip Stocks Samsung Micron — CNBC
• Google’s TurboQuant Compresses LLM KV Caches to 3 Bits — Tom’s Hardware
• TurboQuant: Reducing LLM Memory Usage With Vector Quantization — Hackaday
• Google TurboQuant AI Memory Compression Pied Piper — TechCrunch