The cache grows once per generated token

During autoregressive generation, the model retains key and value tensors from earlier tokens so it does not recompute them for every next token. That cache grows with sequence length and consumes serving memory that could otherwise hold more requests.

Eviction policies keep a subset of token states according to recency, attention-derived importance, fixed landmarks, or another rule. The engineering question is which information can be removed without damaging the tasks the system must perform.

EVICTION FORECASTThe policy predicts which retained state will matter laterA removed token cannot support a future decoding step.
The KV cache reaches its memory budgetKeeping every prior token state is no longer possible.
The policy scores retained statesIt may use recency, attention, landmarks, or a learned signal.
The runtime separates candidatesThe score becomes a keep-or-remove decision.
Generation continues under the smaller cacheLater tokens reveal whether the forecast was sound.
removed state stays irrelevant
Memory pressure fallsThe task continues without needing the discarded state.
removed state becomes relevant
Task quality can fallThe model can no longer attend to that information.

An eviction policy is useful only when task-level tests challenge the states it prefers to remove.

Eviction, compression, and paging solve different constraints

PagedAttention changes how retained cache blocks are placed. Quantization stores cache values with fewer bits, while eviction removes token states entirely.

A smaller context input can avoid creating some states in the first place, but that is context selection rather than cache eviction.

A name at the start of a conversation is a hard test

If a long support conversation introduces an account identifier early and asks for it again much later, an eviction policy that keeps only recent tokens may lose the relevant state. A credible evaluation varies the position and type of required evidence instead of verifying a fixture constructed to match the policy.

The KV-cache eviction analysis explains the accuracy risks without presenting an unsupported hardware benchmark. The common misuse is publishing a memory or latency gain without the model, sequence distribution, hardware, policy settings, and task-level quality result needed to interpret it.

Attention weight is not the same as future importance

A token that receives little attention now may become essential after the topic changes or the user refers back to it. Policies based on recent attention therefore make a prediction about future use, not an objective judgment that a token is irrelevant.

Evaluation should include delayed references, information near different positions, and tasks that depend on several separated facts. A policy that succeeds only when the fixture places important tokens where the policy already prefers carries no evidence beyond its construction.

Continue with these glossary entries: