Context Bloat: Too Much Context Makes Agents Worse

Imagine you are building a customer-support agent. At turn two, it has everything it needs to do the job. By turn twenty, the prompt also carries every failed attempt, three versions of the customer’s address, two complete policy documents, a dozen raw tool results, and schemas for tools this step cannot even call. The customer asks a simple question, and the agent answers with the old address. The current one was in the prompt — buried in everything else.

This is context bloat: the context keeps growing because the system can carry more, not because the next decision needs more. The visible costs are latency and token spend. The one that is easier to miss is reliability: the agent starts making mistakes it did not make when the conversation was short, and the team feels safe the whole time because no information was ever thrown away.

None of this means long context is bad. An agent may genuinely need to read a full contract, follow a long investigation, or work across a large codebase. The problem starts when the context keeps carrying material long after it stopped helping.

A large context is not always a bloated context

The context sent to a model is usually much more than the user’s latest message. It can include system instructions, tool definitions, retrieved documents, conversation history, summaries, memory, prior model output, and raw tool results. In an agent loop, most of that material is assembled again before every model call.

Some large inputs are necessary. If the task is to compare every clause in a 100-page agreement, the agreement is not bloat. If the agent carries the same agreement through ten later calls that only need a date and a renewal rule, it probably is.

There is no token count that separates healthy from bloated. A 4,000-token prompt full of stale instructions can be worse than a relevant 40,000-token document. The question to ask about each piece of context is whether it helps the model make the next decision. Token count is just the warning light that tells you to go look.

It helps to keep three ideas separate, because they often get treated as one. The context window is the model’s maximum capacity. The request context is what your application sends in one call. The agent’s durable state is what the workflow must remember across calls. Durable state has to live somewhere, but that somewhere does not have to be inside every request.

Why more context can make an agent worse

A fact can be in the prompt and still be hard for the model to use. That is the central finding of Lost in the Middle: in the models tested, answers were often better when the relevant evidence sat near the beginning or end of a long input, and worse when it sat in the middle. The fact always fit in the window. What changed was whether the model used it reliably.

You should not treat that result as a universal law, though. Models have improved, and a 2025 Google study found that Gemini 2.5 Flash handled simple fact retrieval near the context limit without the same middle-position drop. The fair reading of both results is that long-context performance depends on the model and, more importantly, on the task — which is exactly why you should measure it on your own workload instead of trusting either headline.

And real agent work is harder than clean fact retrieval. The agent has to infer which old message matters, tell the current address from the previous one, connect a tool result to a policy, and ignore documents that look relevant but are not. The NoLiMa benchmark was built to test this harder case: it removes the obvious word overlap between the question and the relevant evidence, so the model has to make a semantic connection instead of matching a phrase. At 32,000 tokens, ten of the twelve models tested fell below half of their short-context baseline.

The Context Rot experiments push the same point further: long-context performance changes with length, distractors, semantic similarity, and how the information is arranged. Having the fact somewhere in the window is not enough. The model also has to find it among everything else you sent.

Old facts compete with current facts

An agent conversation is full of corrections. The customer changes an address, a tool retry returns newer data, a plan is abandoned, a policy is superseded by a more specific rule. If the context keeps every version with the same apparent weight, the model has to work out which one is current on every single turn, and sometimes it gets that wrong.

Instructions conflict in the same way. A broad system rule, a workflow-specific rule, a retrieved policy, and an old assistant plan may all point in slightly different directions. Adding another paragraph that says “always follow the latest information” does not remove the conflict; it just hands the model the job of resolving it.

You pay for the history again

In a multi-turn loop, the whole input is processed — and billed — again on every call. If the prompt grows from 8,000 tokens to 40,000 across five steps, the final 40,000 understates the bill, because you paid for all the earlier versions too. Latency grows the same way. When a step gets slow, requests time out, retries pile up, and some users give up before the answer arrives.

Prompt caching helps with the cost and latency of a stable, repeated prefix, and you should use it. But the model still has to reason over everything you send, so cached noise is only cheaper noise. Bloat also works against the caching itself: constantly changing histories and tool results shrink the stable prefix that can be reused.

Open the traces and watch the context grow

You do not need a sophisticated memory architecture to find the first problem. Open a few complete production runs and read the input to each model call. The growth pattern is usually obvious.

Then compare successful and failed production cohorts — groups of real runs that share the same workflow version, task type, model, or outcome. If failed runs carry much larger tool results, repeat more history, or retrieve more documents than comparable successful runs, you have something worth testing. One strange trace can reveal a bug; cohorts tell you whether it is a pattern.

Repeated tokens are not automatically bad. Stable instructions and tool contracts may need to appear on every call, and a stable prefix caches well. What you are looking for is repeated content whose value has expired: a tool result that was already consumed, an abandoned plan, a full document after the relevant passage is known, a schema for a tool the current step cannot call.

Build a smaller context on purpose

The most useful change is usually to separate the working state from the transcript. The transcript is a record of what happened; the working state is what the next step needs to know. They do different jobs, and nothing forces them to live in the same prompt.

A compact state for the support example might look like this:

What makes this work is the structure, not just the smaller size. Current facts are marked as current, superseded ones as superseded. Completed actions are separated from proposed ones. Open questions stay open instead of quietly disappearing. And the evidence is referenced rather than pasted in full, so the agent can go back to it when it needs to.

Keep the stable foundation small

The system prompt should carry durable instructions, not every lesson the team has ever learned. Tool definitions should cover the tools the current workflow can actually use. If twenty tools exist but this step can call three, loading all twenty schemas costs tokens and gives the model seventeen extra ways to pick the wrong one.

Keep stable material at the beginning and avoid reordering it without a reason. That makes the prompt easier to inspect, improves cache reuse, and makes it visible when a supposedly stable instruction changes between versions.

Retrieve evidence when the decision needs it

Do not preload an entire knowledge base because one page might matter later. Retrieve for the current question, rerank the candidates, and pass the smallest complete passage that supports the decision. If the first result is insufficient, let the agent ask for more.

The same rule applies to tools. A tool built for an agent should return what the decision needs, not everything the database has. If the model needs the account status, balance, and last payment date, return those three fields with a reference to the full record. Do not send 117 unused fields and then tell the model to ignore them.

Compact at a real boundary

Compression is most reliable when a phase has actually ended: research is complete, a plan is approved, a tool mutation succeeded, or the task is handing off to another agent. At that point, write the facts and decisions into a structured artifact, keep references to the source evidence, and start the next phase with a clean context.

Sub-agents are a clean way to draw this boundary. When a step needs a large exploration — reading a long document, calling a verbose API, searching a codebase — run it as a sub-agent with its own context, and have it return just the fields the parent needs plus a reference to the full output. The parent’s context stays small, and the details stay one lookup away. The same idea works inside a single agent: give tool results a token budget, and when a payload exceeds it, store the payload and return a reference instead of pasting it into the context.

This is also the practical theme of Anthropic’s context-engineering guidance: context is a finite resource that has to be curated while the agent runs. A long-running agent does not need one endlessly growing conversation; it needs a reliable way to carry state between focused pieces of work.

Be careful what you remove

A smaller context is not automatically a better one, because every trimming technique fails in its own way. A summary can drop the one exception that changes a decision. Truncation can cut the message where the user gave permission while keeping the plan that depends on it. Retrieval can miss a document because its wording does not resemble the query. You want to remove the history that stopped mattering and nothing else, and none of these techniques guarantees that on its own.

So some information should never be left to a summarizer: the current user goal, permissions, irreversible actions already taken, policy constraints, questions that are still open, and where the important facts came from. Keep these in explicit fields and validate them when they change. For financial or otherwise consequential actions, keep the record in your application’s own state, not in the model’s summary.

Keep the raw trace too. You still need it for debugging, for audit, and for improving the compaction logic itself. It just does not need to be sent with every model request. If the agent later needs a source, it should be able to retrieve the referenced message, document, or tool result.

There is a real tradeoff here. Compressing loses detail, and keeping everything loses focus. You have to decide which loss is safer for the decision at hand, and keep a way back to the raw evidence for the cases where you get it wrong.

Test the smaller context on the work your agent actually does

A prompt that is 40% shorter is not a result by itself. If the smaller context causes more retries, more escalations, or wrong actions, you have made the agent worse and probably more expensive. Measure the outcome, the path the agent took, and the cost per successful run.

Take a versioned set of known cases and a sample of recent production traces. Run the existing context assembly as the baseline. Then change one source of context at a time: replace old history with working state, trim tool payloads, reduce retrieved documents, or expose fewer tools. Changing one thing at a time is what tells you which change helped and which one removed information the agent needed.

Make the long-context tests harder than asking for a sentence that shares the question’s exact words. Move critical evidence to the beginning, middle, and end. Plant a plausible but outdated fact. Ask the same question without reusing the document’s wording. Repeat the run. These tests borrow the useful lessons from Lost in the Middle, NoLiMa, and Context Rot without pretending that any benchmark predicts your production workflow.

Then compare production cohorts after release. Watch task success, tool-call correctness, retries, latency, cost per success, and corrections caused by stale facts. Watch the misses too: cases where the smaller context no longer contained enough evidence. You will know the change is working when the agent becomes more focused without becoming less informed.

Send the model what the next decision needs, keep everything else where the agent can get it back, and judge the change on real runs rather than on the token count.