01 The Problem: Memory Bottlenecks in AI
Modern AI models operate on vectors — lists of numbers encoding meaning. When processing long conversations, they store these in a Key-Value (KV) cache — a fast memory bank preventing redundant computation.
The problem: caches grow enormous. A large model processing a 128,000-word document may need hundreds of gigabytes of GPU memory. More cache = less room for the model = slower speeds = higher costs.
Traditional quantization introduces memory overhead — extra bookkeeping adding 1–2 bits per number, partially defeating the purpose. TurboQuant eliminates this entirely.
02 What is Vector Quantization?
Quantization maps high-precision values to a smaller set. Like reducing a photo from 16 million colors to 256 — recognizable, far smaller. In AI: instead of a 32-bit float (4 bytes), store a 4-bit integer (0.5 bytes) — 8x compression.
03 QJL: The 1-Bit Magic Trick
The Quantized Johnson-Lindenstrauss (QJL) algorithm projects high-dimensional vectors into lower dimensions while preserving distances. Like a 3D shadow on a 2D wall that still captures the shape.
QJL reduces each number to a single sign bit: +1 or −1. Maximum compression, zero memory overhead.
Apply a random projection. Record only the sign (+1 or -1). Pair with the full-precision query to accurately compute attention scores.
04 PolarQuant: A New Angle on Compression
Traditional Cartesian quantization must measure data range and store that measurement — adding overhead. PolarQuant converts vectors to polar coordinates (radius + angles). Neural network angles follow predictable patterns, so the grid is already fixed. No measurement needed, no overhead.
Analogy: instead of "3 East, 4 North," say "5 blocks at 37°." The circle never changes — nothing to write down.
Group coordinate pairs → convert to (radius, angle) → recursively repeat on radii → store a few bits per angle + one final radius. Zero normalization constants needed.
05 TurboQuant: Putting It All Together
Stage 1 — PolarQuant: Randomly rotate the input vector, apply PolarQuant. Captures bulk information in a few bits, zero overhead.
Stage 2 — QJL (1 bit): Apply QJL to the remaining tiny error. Mathematically eliminates bias, keeping attention scores accurate.
Result: 3-bit total — provably optimal, no training required, no accuracy loss.
06 Results & Performance
Tested on Gemma, Mistral, and Llama-3.1-8B across five long-context benchmarks covering question answering, code generation, and summarization.
07 Why TurboQuant Matters
Backed by mathematical proofs showing it operates near the theoretical lower bound for distortion — it cannot be significantly improved upon in its class.
A 6–8× KV cache reduction means 6–8× more users served, or 6–8× longer documents processed, on the same hardware.
3 bits. Zero accuracy loss. No retraining. Provably optimal. Already running in production AI systems.