Context Rot: Why LLM Performance Degrades as Context Grows

TL;DR

Context rot is the measured decline in LLM output quality as input context grows, even when the added tokens are relevant. The term entered common use after Chroma's 2025 technical report showed that model performance on identical questions degrades as the surrounding context lengthens, across every model family tested. It is the reason a million-token context window does not solve memory: the window holds more, but the model uses what it holds less reliably. This guide covers what the research actually measured, the mechanisms behind it, and the three engineering responses that work.

What did the research actually show?

Chroma's technical report on context rot evaluated frontier models on tasks where the answer was present in context and only the context length varied. The pattern that gives the phenomenon its name: performance falls as length rises, and the fall is not graceful.

The findings that hold up and are worth citing:

1. Degradation happens on relevant context, not only filler. Models got worse even when the additional tokens belonged to the task, which rules out "just filter the junk" as a complete fix.

2. Distractors amplify it. Content that resembles the answer without being the answer hurt more than neutral padding, and the effect grew with length.

3. Structure interacts with it in unintuitive ways. In some conditions, models performed differently on logically ordered context versus shuffled context, which suggests position effects rather than pure comprehension.

4. Every family showed it. The effect appeared across the vendors tested, differing in degree, not in kind.

Anthropic's own engineering guidance on context management states the underlying premise plainly: attention is a finite budget, and every token spends some of it. The needle-in-a-haystack benchmarks that vendors publish measure retrieval of one planted fact, which is the easy case; context rot shows up in the realistic case, where the model must weigh many partially relevant facts.

Why does it happen?

Three mechanisms, all consequences of how transformers process input:

1. Attention dilution. Attention weights are distributed across every token in the window. More tokens means less attention per token, so the signal that matters competes with everything else you sent.

2. Position effects. Models attend unevenly across the window, and content in the middle of long contexts is recalled worse than content at the edges, the pattern usually cited as "lost in the middle" from Liu et al.'s 2023 paper.

3. Distractor confusion. As near-miss content accumulates, the probability that the model anchors on a plausible-but-wrong passage rises. This is why appending your whole wiki to a prompt can produce worse answers than appending nothing.

The uncomfortable implication for agent builders: an agent that accumulates history without pruning gets worse at its job over a long session while costing more per call. Cost and quality degrade together.

What actually works against context rot?

The engineering responses, ranked by how much of the problem each addresses:

1. Send less, better. The highest-leverage response is selection: send the facts the task needs rather than the documents that mention them. This is what a memory layer does structurally, resolving raw material into compact facts at write time so the context assembled at read time is small and non-redundant. Sentra's published Terminal-Bench 2.1 evaluation is one measurement of this approach: 41.2% fewer tokens with accuracy rising from 83.37% to 88.31% across 445 trials, improvement on both axes at once, which is what you expect when the mechanism is less-but-better context rather than a smarter model.

2. Compaction and summarisation. Agent harnesses, including Claude Code, compress older history as sessions grow. This bounds the rot but loses detail, and what the summary drops is unrecoverable.

3. Position engineering. Put instructions and the most decision-relevant facts at the start or end of the prompt, never buried in the middle. Cheap, real, and limited.

4. Sub-task isolation. Delegate long searches to subagents with fresh windows so the parent context stays small. This contains rot per-context rather than fixing it.

What does not work: buying a bigger window. The window ceiling and the attention budget are different constraints, and only the first one is for sale.

Context rot, context windows and memory compared

FAQ

The decision rule

Treat context as a budget, not a warehouse. If a token does not change the model's decision, it is spending attention and money to make the answer slightly worse. The teams that beat context rot are not the ones with the biggest windows, they are the ones whose pipelines resolve knowledge before the prompt instead of inside it.