Why a Bigger Context Window Makes Claude Worse
We will talk about Context Windows a lot in this post, make sure you have read the previous post in the series: Looking Through Claude's Context Window about what Context Windows actually are before steaming ahead!
What is Context Rot?
As models get more powerful, and context windows grow, it would be easy to think that these frontier models are becoming more capable. Why wouldn't you, if you can put more tokens in and the model can output more tokens, that opens up the possibility to have more words, files, references etc when you make your query. Surely that would mean a better, well informed response from the model as well. At least that is what we are sold.
Recent research has found it to not be the case. As the context length grows, the model actually becomes worse at recalling tokens within the context, this is known as 'Context Rot'. You might assume that the model is accurate across the board no matter how many tokens are inputted. You might even think that the issue only arises when the context grows substantially, say when comparing 500k input tokens to 1M. Unfortunately it is much worse than that. Below we will look at 3 studies performed on the topic to dive a little deeper into what actually happens to our beloved models as these Context Windows grow, and some tips on how best to avoid rot setting in.
Bigger context != better performance
Most LLM providers use a benchmark called Needle in the Haystack (NIAH) to show model performance in a growing Context Window. This test involves giving a sentence of text (needle) to find within a long text (haystack), while this does measure how a model performs lexical matching on a growing context, it does not really reflect how context affects model performance in the real world. In their study, Chroma points out that when we interact with LLMs, the increased Context Window we use might also mean we are asking for more complex reasoning*. It's not about how the model finds the token you require, it's what it needs to do with that information which affects the model's accuracy. The question they wanted to answer was: 'Is the cause of degrading accuracy larger inputs, or simply harder problems?' With that in mind Chroma modified the NIAH test so that the difficulty was a constant, varying only the length of the Context Window to show whether purely increasing amounts of input tokens affected model accuracy.
What they found was that even with the stable difficulty level, model accuracy reduced across the board as the amount of input tokens increased, even on the simplest tasks. This shows that while a bigger Context Window might seem like it unlocks more potential, it could be reducing the effectiveness of your model's output, hindering what might have been a better answer if it just used fewer, more focused tokens.
So what about if we treat the Context Window as something dynamic, rather than static? Would continuously monitoring and updating the context as the session grew longer help with recall accuracy?
The Context as a living document
Yao et al did an experiment to find out whether dynamically consolidating the Context Window provided better recall over long Context Windows compared with a static management approach. This included the use of a dedicated Context Manager (agent) whose sole job was to manage and consolidate the context at runtime, actively rewriting the context, whilst the separate execution agent was the one taking action. Splitting this up into what they coined ARC showed that actively managing the context rather than passively had large improvements on long-horizon deep research agents, albeit on a much smaller scale of tokens and models compared to the current leading models**. But I think it's worth a mention here because the idea is incredibly clever. It's best to think of this with an analogy:
A detective (Execution Agent) and a lieutenant (Context Manager) are trying to solve a case, they have case notes (Interaction Memory) and a checklist which they use to keep track of what they need to ask the suspect or witness. When speaking with someone, the detective reads the checklist and the existing case notes to determine which actions to take. After the interaction, the lieutenant will create a summary of what was said and append it to the notes. The next interrogation, the detective goes in with the full transcript of the previous talk, the checklist and notes and takes actions based on those. As the different interactions happen, the lieutenant will actively notice that recent events or information have changed earlier assumptions and that information in the notes are now outdated, so he rewrites the notes to match, and also updates the checklist so that the detective knows what he needs to ask next time. In this light it is clear to see that having a dynamic, active way of managing the notes allows the detective to stay focused on relevant information and avoids him reading the notes and making an incorrect assumption based on evidence that is no longer valid.
While we might not have access to a trained Context Manager, or even a dynamic way to consolidate the Context Window yet (yes, /compact <message> exists - it's in my previous post, but that isn't dynamic management, it's triggered manually by us). This research sheds some light on how important it is for us to think about the context as this living memory and that we must keep it updated regularly with the correct information to avoid the model producing invalid output. But what if the problem was not recalling something you said, but noticing that part of the context contained malicious commands a bad actor had injected in there?
Hiding the payload
It's not only recall that's affected by a larger Context Window. Another study by AI safety researchers at Anthropic looked at what happens when you use a model as a monitor: handing it a full agent transcript and asking it to flag any malicious actions inside. As those transcripts grow, monitors get noticeably worse at their job.
They ran two experiments. In the first, they took existing transcripts where an agent is asked to configure a firewall. It completes the task, but also quietly disables the firewall for certain IP addresses. Subtle, but potentially devastating. On its own, Claude Opus 4.6 with thinking caught that 98.6% of the time. Prepend 800k tokens of unrelated, entirely benign activity in front of the same transcript, and recall fell to 88%. Nothing about the attack changed, only the amount of harmless material the model had to sift through before reaching it.
The second experiment went further and injected a blatant payload into a real Claude Code transcript (a curl | sh injection), Opus 4.6 with thinking caught it 99.7% of the time with 100k tokens. At 800k, that dropped to 69%. They also found position matters enormously: attacks buried in the middle of a transcript are hardest to spot, and this affects non-thinking models far worse. With 500k tokens of padding before the payload and 100k after, non-thinking detection collapsed to 5%. Think about that for a second. 600k tokens is well within the current Context Window limits of frontier models. If somehow a bad actor was able to inject that payload into your context and you didn't have model 'thinking' enabled, it's close to a guarantee it will be missed!
Interestingly they conclude with the fact that you can add reminders into the prompt to help reduce attack success rate for free (well almost, < 3% input tokens, but it didn't cause false positives to increase which is a big win), but it doesn't really hold up against more nuanced injections, only on the obvious needle-injection commands (think curl | sh payloads).
First steps to avoiding context rot
These three articles have shown us that growing Context Windows do negatively affect the recall accuracy of models, not only with remembering what has been said to the model during conversation, but also how capable the model is at finding something malicious within a context, which is much scarier.
We know that context rot exists, but the exact mechanism causing context rot isn't fully known, and as the research shows it isn't a solved problem. As we have seen, researchers have done the experiments to see what improves it or makes it worse, and made some claims, but there is no real 'this is why it happens' just yet.
So what can we do about it? Dynamically managing the Context Window with an agent may unlock a lot of potential into improving the recall accuracy of our models at larger Context Windows - the problem here is that is a hard thing to do, and we can only use the tools available to us. Claude sub-agents allow us to get some of the way there, the Context Manager did two things, summarize each turn and actively consolidate the whole context when it saw things going awry. If you consistently delegate work to sub-agents, and they return a summarized version of their context to the main context, that is essentially the first part of the Context Manager's job done.
The second part is on you. Making sure that you compact your context regularly when the prompt starts to go in a different direction and making sure you use a specific summarize command when doing so (e.g. /compact focus on why context engineering is important) should help keep the LLM focused on what currently matters.
Reducing the token limit for the Context Window auto-compaction is also something which may benefit you, if this post has shown us anything, it's that bigger Context Windows are not always better. You can tune this in your Claude Code settings with the /autocompact command, try a few values and see if you notice any improvement to model accuracy yourself over long conversations.
Old information in the context is not only wasted space, but detrimental to the performance of our models. The Chroma team pointed out that we must use context engineering effectively to get the most out of our time with AI, and for us the simplest way to do that is to manage our Context Window effectively.
* Chroma do have skin in the game, their product would be invalid if there was not a scalability problem with Context Windows, that does not invalidate what they claim, it just explains their motivations.
** This research was performed on non-frontier models (the context manager was initialized from Qwen3-14B), so not on the flagship LLMs you and I use regularly. Also important to note this research was done on a fraction of the tokens, 32K to be exact, so we must take this paper with a small pinch of salt.
