Shipping reliable AI apps, part 7 of 7
Context Rot Explained
Models read long prompts worse than short ones. Why context rot happens, what the research shows, how to measure it in your app, and what to trim.
After this video
You can test your own app for context rot with a needle test and a focused-versus-full run, and trim what the model reads.
The short version
The more tokens you send a model, the less accurately it recalls any one of them. That's context rot, and it shows up across models. A bigger window only says what fits, not what gets read well. Distractors (text on the same topic that looks like the answer) and facts buried in the middle of a long prompt make it worse. Measure it in your own app with a needle test and a focused-versus-full run, then trim what the model reads.
What the context is
Everything the model reads in one call: the system prompt, the chat so far, tool results and documents. The model doesn't remember the last turn, so the whole pile goes in again on every call, and it only grows.
The help-desk bot. Forty messages in, the bot resends all thirty-nine before it. The customer said "annual plan" in message three, and the bot still quoted the monthly policy.
Why it happens
In a transformer every token can look at every other token, so doubling the tokens roughly quadruples the pairs to weigh. Attention gets stretched thin, and models learn mostly from shorter texts. Think of it as an attention budget that every added token spends.
What the research shows
Chroma tested eighteen models with the same task at different input lengths: performance dropped as inputs grew, unevenly. On chat histories, every model did clearly better with about 300 relevant tokens than with about 113,000. NoLiMa found eleven of thirteen long-context models fell below half their short-prompt score at 32,000 tokens.
Distractors and position
Even one distractor lowers accuracy, and longer inputs make distractors hurt more. Lost in the Middle found answers at the start or end of a long prompt did best and the middle sagged. Put long documents first and the question last.
Layout fix. Put the key customer facts in a short block right above the question, and ask the model to quote the lines it uses before answering.
Measure, then trim
Log input tokens per call against the turn number. Run a needle test: one fact, worded differently from the question, at three lengths and three positions. Run your eval set twice, focused and full. Then trim: retrieve with pointers instead of pasting, clear old tool results, keep a notes block, compact very long runs, and hand big digging to a sub-agent.
The notes block. The bot reads one note that says "plan: annual" instead of forty messages. When a fact changes, update the note, so the stale fact can't distract.
Check yourself
Your chat bot sends the full history every turn and starts missing facts from early messages. What is going on, and why does a bigger context window not fix it?
Context rot: as the tokens in the context grow, the model recalls them less accurately, across all models. The window only says what fits, not what gets read well.
How would you test whether long context hurts your own app?
A needle test: hide one fact from your app at a few lengths and positions, ask the same question worded differently, and score it; and run your eval questions twice, once with only the relevant parts and once with everything. A clear win for the focused version means context rot.
Name three ways to trim an agent's context, and say which one fixes a fact that changed.
Retrieve with pointers instead of pasting, clear old tool results, keep a notes block of key facts, compact near the limit, or use a sub-agent with a clean window. The notes block, updated when a fact changes, removes the stale fact.
Sources
Every claim in this video was checked against these pages. Open one to see the claims it backs and the exact words they rest on.
anthropic.com/…/effective-context-engineering-for-ai-agents
- Context rot: as tokens in the context window increase, the model's recall of that context decreases.
as the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases.
- Context rot emerges across all models; some degrade more gently.
while some models exhibit more gentle degradation than others, this characteristic emerges across all models.
- Context is a finite resource with diminishing returns; LLMs have an 'attention budget' every token depletes.
every new token introduced depletes this budget by some amount, increasing the need to carefully curate the tokens available to the llm.
- In a transformer every token attends to every other token: n squared pairwise relationships for n tokens.
which enables every token to attend to every other token across the entire context. this results in n² pairwise relationships for n tokens.
- As context length increases, the model's ability to capture these pairwise relationships gets stretched thin.
as its context length increases, a model's ability to capture these pairwise relationships gets stretched thin
- Models learn attention patterns from training data where shorter sequences are more common than longer ones.
models develop their attention patterns from training data distributions where shorter sequences are typically more common than longer ones.
- Good context engineering finds the smallest possible set of high-signal tokens for the outcome.
good context engineering means finding the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome.
- Just-in-time context: keep lightweight identifiers (file paths, queries, links) and load data at runtime with tools.
maintain lightweight identifiers (file paths, stored queries, web links, etc.) and use these references to dynamically load data into context at runtime using tools.
- Tool result clearing is one of the safest, lightest forms of compaction.
one of the safest lightest touch forms of compaction is tool result clearing
- Compaction: summarize a conversation nearing the limit and start a new context window with the summary.
compaction is the practice of taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new context window with the summary.
- Structured note-taking: the agent writes notes outside the context window and pulls them back in later.
structured note-taking, or agentic memory, is a technique where the agent regularly writes notes persisted to memory outside of the context window.
- Sub-agents handle focused tasks with clean context windows.
specialized sub-agents can handle focused tasks with clean context windows.
- A sub-agent may use tens of thousands of tokens but returns a condensed summary, often 1,000-2,000 tokens.
each subagent might explore extensively, using tens of thousands of tokens or more, but returns only a condensed, distilled summary of its work (often 1,000-2,000 tokens).
- Context windows of all sizes are likely to remain subject to context pollution.
it's likely that for the foreseeable future, context windows of all sizes will be subject to context pollution and information relevance concerns
- Context is the set of tokens included when sampling from an LLM.
context refers to the set of tokens included when sampling from a large-language model (llm).
- Context rot: as tokens in the context window increase, the model's recall of that context decreases.
research.trychroma.com/context-rot
- Modern LLMs commonly have input context lengths in the millions of tokens.
it is common for modern llms to have input context lengths in the millions of tokens.
- Chroma's context rot report evaluated 18 LLMs.
in this report, we evaluate 18 llms
- Chroma held task complexity constant and varied only the input length.
our experiments hold task complexity constant while varying only the input length
- Model performance degrades as input length increases, even on simple tasks, in non-uniform ways.
model performance degrades as input length increases, often in surprising and non-uniform ways.
- In Chroma's LongMemEval test the focused prompts (only the relevant parts) averaged about 300 tokens.
focused prompts average to ~300 tokens
- The full LongMemEval prompts averaged about 113k tokens.
these prompts average out to ~113k tokens.
- Across all models, performance was significantly higher on focused prompts than full prompts.
across all models, we see significantly higher performance on focused prompts compared to full prompts.
- A naive chat memory includes the full chat history in the prompt for following chats.
a naive approach would be to include the full chat history into the prompt for following chats.
- Irrelevant context forces the model to find what is relevant and reason, two tasks at once.
adding irrelevant context adds the additional step of identifying what is relevant, forcing the model to perform two tasks simultaneously.
- A distractor is topically related to the needle but does not quite answer the question.
distractors are topically related to the needle, but do not quite answer the question
- Even a single distractor reduces performance vs the needle alone; four distractors compound it.
even a single distractor reduces performance relative to the baseline (needle only), and adding four distractors compounds this degradation further.
- The impact of distractors amplifies as input length grows.
the impact of distractors and their non-uniformity amplifies as input length grows across models
- Some models answer confidently but wrongly when distractors are present.
often generating confident but incorrect responses when distractors are present.
- Some models abstain when uncertain, saying no answer can be found.
tend to abstain when uncertain, explicitly stating that no answer can be found.
- Models score near-perfectly on needle-in-a-haystack, which mostly tests direct lexical matching.
because these models achieve near-perfect scores on widely adopted benchmarks like needle in a haystack (niah)
- Lower needle-question similarity makes performance degrade faster with length.
lower similarity needle-question pairs increases the rate of performance degradation.
- Chroma found no notable variation by needle position on its simple needle task.
testing across 11 needle positions, we find no notable variation in performance for this specific niah task.
- Chroma's context rot report was published July 2025.
chroma technical report july 14, 2025
- Modern LLMs commonly have input context lengths in the millions of tokens.
arxiv.org/…/2502.05167
- NoLiMa tested 13 LLMs claiming 128K+ context; at 32K, 11 dropped below 50% of their short-context baselines.
at 32k, for instance, 11 models drop below 50% of their strong short-length baselines.
- NoLiMa tested 13 LLMs claiming 128K+ context; at 32K, 11 dropped below 50% of their short-context baselines.
arxiv.org/…/2307.03172
- Performance is often highest when relevant information is at the beginning or end of the context and degrades in the middle.
performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts
- Lost in the Middle was published in 2023 (TACL).
accepted for publication in transactions of the association for computational linguistics (tacl), 2023
- Performance is often highest when relevant information is at the beginning or end of the context and degrades in the middle.
arxiv.org/…/2307.03172v3
- With the answer in the middle, GPT-3.5-turbo did worse on multi-document QA than with no documents (closed-book).
when relevant information is placed in the middle of its input context, gpt-3.5-turbo's performance on the multi-document question task is lower than its performance when predicting without any documents
- Lost in the Middle compared 10-, 20- and 30-document settings in multi-document QA.
the 10- and 20-document settings both fit in the context window of gpt-3.5-turbo
- With the answer in the middle, GPT-3.5-turbo did worse on multi-document QA than with no documents (closed-book).
platform.claude.com/…/claude-prompting-best-practices
- For long inputs, place long documents near the top of the prompt, above the query and instructions.
place your long documents and inputs near the top of your prompt, above your query, instructions, and examples.
- Queries at the end can improve response quality by up to 30% in Anthropic's tests.
queries at the end can improve response quality by up to 30 percent in tests, especially with complex, multidocument inputs.
- The long-context layout advice is for inputs of 20k+ tokens.
when working with large documents or data-rich inputs (20k+ tokens), structure your prompt carefully to get the best results
- Asking the model to quote relevant parts first helps it focus on relevant content.
ask claude to quote relevant parts of the documents first before carrying out its task.
- For long inputs, place long documents near the top of the prompt, above the query and instructions.
platform.claude.com/…/token-counting
- Anthropic's token counting endpoint is free to use (rate limited).
token counting is free to use but subject to requests per minute rate limits
- Anthropic's token counting endpoint is free to use (rate limited).
platform.claude.com/…/working-with-messages
- The Messages API is stateless: the app sends the full conversation history every call.
the messages api is stateless, which means that you always send the full conversational history to the api.
- The Messages API is stateless: the app sends the full conversation history every call.
Spotted a mistake? Say so in the video's comments. Fixes are listed on corrections.