Learn · The KV Cache
The KV cache
A model generates text one token at a time, and every step attends to the whole prefix. Without a key/value cache it would re-project every earlier token again and again. Generate tokens below and watch the wasted work pile up quadratically — then see the cache flatten it to a straight line.
Without a cache
Every step recomputes K and V for every token in the prefix. The whole grid lights up as re-projected work.
With a KV cache
Earlier rows are stored once and reused (shown greyed as cached). Only the new token's K and V are computed each step.
Compute cost
Counting K/V row computations only — an illustrative operations count, not FLOPs.
Press “Generate next token” to start decoding.
Cumulative work as the sequence grows
The magenta curve bends upward (quadratic); the cyan line stays straight (linear). The gap is the work the cache saves.
What's actually going on
What K and V are
Each token is projected into a key (how it can be matched) and a value (what it contributes) inside every attention layer. Keys and values only depend on that token and its position, so once computed they never change.
What gets cached (and what doesn't)
The K and V rows for past tokens are stored and reused. The query is not cached — each step it's just the one new token asking “who should I attend to?” against the whole cached history.
Cheap compute, costly memory
Skipping the re-projection turns per-step cost from growing to constant, so long generations get dramatically cheaper. The catch: the cache itself grows with sequence length, so memory becomes the new bottleneck.
This is an illustrative model. The counters tally K/V row computations to show the O(n²) vs O(n) gap — they aren't FLOPs. Real attention still reads all cached K/V every step (that part stays linear per step); the saving is not re-projecting them from the hidden states each time. See generation step by step in autoregressive decoding, how attention masks the future in causal attention, or browse more on the Learn shelf.