Author: Santosh Jaiswal (@hellojais)
Paper: TurboQuant (ICLR 2026)
Library: mlx-optiq / mlx-vlm by Prince Canuma and contributors

TL;DR

I validated Google Research and DeepMind’s TurboQuant (ICLR 2026) on an Apple M5 Max with 128GB unified memory. Running Qwen-7B at a 32k context window, KV cache RAM dropped from 4.2 GB to 0.85 GB (a 5x reduction), with 100% Needle-in-a-Haystack retrieval accuracy preserved. TurboQuant is data-oblivious: no retraining, no model modification, works with any Transformer. For local and agentic workflows, that is a significant practical result.

The Question

I recently picked up a MacBook Pro M5 Max (128GB unified memory, 40-core GPU) with one question in mind: how far can you push local AI inference before you hit the memory wall?

The KV cache is usually where that wall appears. For long-context tasks (32k tokens and beyond), the cache alone can consume several gigabytes of RAM, grow linearly with context length, and either crash your run or force you onto cloud hardware. This is especially painful for agentic workflows that maintain long tool-use chains across many turns.

Perfect timing: just days after I got the machine, Google Research and DeepMind dropped TurboQuant at ICLR 2026. It targets exactly this bottleneck.

What TurboQuant Does

KV cache quantization is not new, but most approaches have a catch: they require calibration data, fine-tuning, or model-specific modification. TurboQuant is data-oblivious: it compresses the KV cache on the fly, at inference time, with no access to training data and no changes to model weights.

The mechanism in brief:

  1. Rotate the data into a distribution that is more predictable (lower effective entropy per dimension).
  2. Quantize aggressively to 3.5-bit in my experiment.
  3. Add a 1-bit correction path that handles the residual rounding error from step 2.

The rotation step is the key insight. Raw key/value tensors have high variance in a small number of dimensions and low variance in most others, a structure that conventional uniform quantization handles poorly. After rotation, the variance is spread more evenly, and a fixed low-bit quantizer wastes far fewer bits on noise.

The math is non-trivial (the paper is worth a read; I used NotebookLM to work through it), but the practical implication is clean: you get aggressive compression without a calibration pipeline.

The Setup

Hardware: MacBook Pro M5 Max, 128GB unified memory, 40-core GPU Model: Qwen-7B Context window: 32,000 tokens Quantization library: mlx-optiq (Apple Silicon implementation by Prince Canuma and contributors)

A big shout-out to the OSS community here. An mlx-optiq implementation landed almost immediately after the paper dropped, which made validation on Apple Silicon surprisingly straightforward.

Baseline: Standard FP16 inference, 32k context, no KV cache compression.

Test: Needle-in-a-Haystack: embed a specific fact inside a 32,000-token document, then ask the model to retrieve it. This is a standard long-context stress test: if KV cache compression degrades the representation of distant tokens, retrieval fails.

Results

Configuration KV Cache RAM Retrieval accuracy
FP16 baseline (no compression) 4.2 GB 100%
TurboQuant 3.5-bit 0.85 GB 100%

RAM reduction: 4.2 GB → 0.85 GB (5x)

The compression held up exactly where it needed to. The Needle was found instantly. Retrieval accuracy stayed at 100% despite the aggressive 3.5-bit quantization. No degradation on the task that matters most for long-context agentic work: remembering what was said earlier.

Why This Matters for Local Agentic Workflows

The standard objection to running serious AI locally is memory. A long tool-use chain (where the model maintains context across dozens of function calls, intermediate results, and user turns) can easily push 20k-50k tokens of context. At FP16, that KV cache fills RAM fast.

5x compression changes that calculus. What previously required 20 GB of KV cache memory now fits in 4 GB. On a machine with 128GB unified memory, this means you can run significantly longer chains, larger models, or multiple concurrent sessions before hitting the wall.

More broadly: with 128GB of unified memory and this kind of quantization arithmetic, a high-end laptop starts behaving like a memory cluster. The gap between “what you can run locally” and “what requires a server rack” is closing faster than the GPU roadmaps suggest, because the constraint was never compute. It was memory bandwidth and capacity, and those are now being compressed away at the software layer.

Caveats

A few things to note before extrapolating:

  • Single model, single task. This was Qwen-7B on a retrieval benchmark. I have not tested TurboQuant across different model families, quantization targets, or generation tasks. The 5x figure is real but specific to this configuration.
  • The 1-bit correction path has overhead. TurboQuant is not free: the rotation and correction operations add latency. For throughput-sensitive applications, this matters. I did not benchmark generation speed in this experiment.
  • mlx-optiq is a community implementation. It may not match the reference implementation exactly. I used it because it was the only option available for Apple Silicon at the time.

What’s Next

A few directions I want to explore:

1. Longer contexts. 32k is a moderate test. I want to see how the compression holds at 64k and 128k, where the memory savings matter even more and the per-token representation becomes thinner.

2. Generation quality metrics. Needle-in-a-Haystack tests retrieval but not generation fluency. MMLU or a perplexity sweep across quantization levels would give a more complete picture of accuracy-compression tradeoffs.

3. Larger models. 7B at FP16 is comfortably within 128GB. With 5x KV compression, running 32B or 70B with long contexts locally becomes plausible. I want to test the practical ceiling.

Resources

Resource Link
📄 Paper TurboQuant (ICLR 2026)
💻 Library mlx-optiq / mlx-vlm

If you’ve run TurboQuant on other hardware or model families, I’d be curious what numbers you’re seeing. Reach out at hellojais@gmail.com or @hellojais on GitHub.