How Much Context Does an AI Agent Actually Need? The Goldilocks Problem
An AI agent needs exactly enough context to answer the task at hand — and no more. Too little, and it guesses, hallucinates, or gives a generic answer. Too much, and it loses track of what matters, slows down, and costs more per call. The right amount is the scoped, curated middle: the specific facts, documents, and history relevant to the current request, retrieved on demand rather than dumped wholesale into the prompt.
There is no universal token number. The answer depends on the task. A one-line code fix needs a few files; a policy question needs the relevant policy, not the whole handbook. Two failure modes bracket the sweet spot: too little context makes the model guess, while too much triggers context rot and lost-in-the-middle errors on top of higher cost and latency. The reliable way to land between them is curated, scoped retrieval — a discipline known as context engineering. This guide walks through the Goldilocks problem and how to hit the middle.
In this guide
- What does context mean for an AI agent
- Why does too little context make AI give bad answers
- Why does too much context also hurt
- Does a bigger context window fix the problem
- How do you hit the Goldilocks middle
- Worked example: sizing context for three tasks
- Common mistakes when sizing agent context
- How to measure whether you have the right amount
- Where does a shared context layer fit
What does “context” mean for an AI agent?
Context is everything an AI agent sees when it generates a response: the user’s question, the system instructions, prior conversation, retrieved documents, tool outputs, and any data injected into the prompt. It is the model’s entire working memory for that single call.
The model has no other knowledge of your situation. If a fact is not in the context, the agent either guesses from its training data or makes something up. If you want a deeper definition, see our guide on what agent context actually is.
A useful way to think about it: the model’s training gives it general fluency about the world, but the context is the only place it learns about you. Your release date, your refund window, the function signature in the file you are editing — none of that is in the weights. It has to arrive in the prompt for that specific call, or it does not exist as far as the model is concerned. “How much context does an agent need?” is therefore really the question “how much of your situation does this particular task depend on?” — and that varies enormously from one task to the next.
Why does too little context make AI give bad answers?
With too little context, the model fills the gaps with its training priors — plausible-sounding generalities that may be wrong for your case.
Ask an agent about “our refund policy” without giving it the policy, and it invents a reasonable-but-fictional one. This is the root cause of most generic or wrong AI answers. The fix is not a bigger model. It is giving the model the specific facts it needs.
The hallucination trap
When a model lacks grounding, it does not say “I don’t know.” It produces the statistically likely answer. That often reads as confident and correct, which makes the error harder to catch.
What “too little” looks like in practice
Under-context shows up in recognizable patterns. The agent answers in generalities (“a typical onboarding process includes…”), it asks no clarifying questions when a human would, or it confidently states a specific-sounding fact that turns out to be invented — a phone number, a version, a date. A coding agent given only the file you asked it to change, but not the interface it implements, will write code that compiles in isolation and breaks against the real type. Each of these is the same root cause wearing a different costume: a fact the task needed was simply never in the window.
The tempting fix is to hand the agent more — paste in the whole repo, the entire policy manual, every prior message. That over-corrects straight into the opposite failure, which is the rest of this guide.
Why does too much context also hurt?
Adding more context past a point makes answers worse, not better. Three failure modes appear:
- Context rot — accuracy drops as the input grows, well before the window is full. See why more context can make AI worse.
- Lost in the middle — models reliably use facts at the start and end of a prompt, but drop information buried in the middle of long prompts.
- Cost and latency — every token is billed and processed, so a bloated prompt is slower and more expensive for no accuracy gain.
A landmark Stanford study, Lost in the Middle (Liu et al., 2023), showed model accuracy follows a U-shaped curve relative to where the key fact sits in the prompt. More recently, Chroma’s context-rot research (2025) found that all 18 frontier models tested degraded as input length grew. Anthropic frames the same idea in its guidance on effective context engineering for AI agents: context is “a critical but finite resource,” and adding more of it does not reliably buy better behaviour.
A back-of-the-envelope cost picture
The cost side is easy to underestimate. Token billing is roughly linear in input length, and latency rises with it too. Consider an agent that runs a multi-step task across, say, ten model calls. If each call carries an extra 30,000 tokens of “just in case” context that the task never uses, that is 300,000 wasted input tokens for a single task — paid for, processed, and very possibly harmful to accuracy because it dilutes attention. Now multiply by every task, every day. The bloated prompt is not a free insurance policy; it is a recurring tax that buys negative accuracy.
| Symptom | Likely cause | Direction to move |
|---|---|---|
| Vague, textbook-style answers | Too little context | Add the specific missing fact |
| Confident but invented specifics | Too little grounding | Retrieve the real fact |
| Drops a fact that was in the prompt | Too much context (rot / middle) | Trim and reorder |
| Slow and expensive, no accuracy gain | Bloated prompt | Prune unused content |
Does a bigger context window fix the problem?
No. A larger window raises the ceiling on how much you can include, but it does not make the model use that content well.
A model with a 200K-token window can degrade badly at 50K tokens. The constraint is attention quality, not capacity. This is why does more context improve LLM answers has a counterintuitive answer: often, no.
How do you hit the Goldilocks middle?
You hit the middle by selecting context, not dumping it. The disciplines that do this:
- Context engineering — the practice of deciding what goes into the window and what stays out.
- Context window management — budgeting tokens across instructions, history, and retrieved data.
- Context pruning — actively removing stale or irrelevant content before each call.
The common thread: retrieve the minimum sufficient context for the task, freshly scoped to the current request.
A simple rule of thumb
Start with the task. Ask: what specific facts would a competent human need to answer this? Give the agent those — and skip the rest. If the answer is wrong, add the missing fact, not the whole corpus.
This “competent colleague” test is worth internalizing. If you handed the task to a smart new hire, you would not photocopy the entire company wiki onto their desk. You would point them at the two documents that matter and let them ask if they need more. Good context for an agent looks the same: a tight, relevant working set, with a retrieval path open for the rare case where the agent genuinely needs to pull in more.
Worked example: sizing context for three tasks
The right amount is easiest to see by walking concrete tasks from under-context to over-context to the middle.
Task 1 — “Fix the off-by-one bug in paginate().” Too little: just the function body, so the agent cannot see the caller’s expectations and guesses the intended bounds. Too much: the entire repository pasted in, burying the relevant file among hundreds. The middle: the function, its tests, and the one or two call sites — perhaps a few hundred to a couple of thousand tokens.
Task 2 — “Does our refund policy cover digital goods after 14 days?” Too little: no policy at all, so the model invents a plausible-sounding rule. Too much: the entire employee handbook, where the refund clause sits in the lossy middle and gets skipped. The middle: the refund section, retrieved on demand — a few hundred tokens that decisively answer the question.
Task 3 — “Summarize this 80-page contract’s termination clauses.” This task genuinely needs a lot of material in view, so the window will be large — but even here, more is not automatically better. Feeding all 80 pages when only the termination and notice sections matter invites the model to lose the relevant clauses among boilerplate. The middle: retrieve the termination-related sections and summarize those.
The pattern across all three: start from the task, not from a token target. Two of these tasks need very little; one needs a lot. The number follows the task.
Common mistakes when sizing agent context
- Treating the window size as a target. A 200K window is a ceiling, not a goal. Filling it does not make the agent smarter — it usually makes it worse.
- Pasting whole documents. Almost every document is mostly irrelevant to any one query. Trim to the passage that answers the question.
- Never pruning history. Long sessions accumulate dead turns. Old exchanges that no longer bear on the current step are pure noise — see context pruning.
- Adding context to fix a wrong answer without diagnosing why. If the answer was wrong because a fact was missing, add that fact. If it was wrong because the fact was buried, removing noise is the fix, not adding more.
- Assuming a bigger model rescues a bloated prompt. Larger models reason better over what they see, but they still drop facts in over-long inputs.
How to measure whether you have the right amount
You do not have to guess. A few cheap signals tell you which side of the curve you are on. If answers are vague or invented, you are likely under-contextualized — add the specific fact and re-test. If the agent ignores a fact you know is in the prompt, you are likely over-contextualized — shorten and reorder so the fact sits near an edge. Track input token count per call alongside answer quality; when quality plateaus or dips while tokens keep climbing, you have passed the peak. The disciplined version of this is to build a small evaluation set of representative tasks and vary the context you feed, watching where accuracy turns over.
Where does a shared context layer fit?
Most teams hit the Goldilocks problem because their knowledge is scattered across docs, wikis, tickets, and files, with no clean way to scope it per query.
A unified context layer solves this by letting AI tools pull just the relevant slice of company knowledge on demand — scoped, curated, and shared across tools — instead of copy-pasting walls of text into a chat. Pairing that scoped retrieval with persistent agent memory carries the right context across sessions. A scoped, MCP-based layer is exactly what we’re building at CtxFlow, if you want to see where it goes next.
FAQ
How many tokens of context does an AI agent need? There is no fixed number. It depends entirely on the task. A small code edit may need a few hundred tokens; a research summary may need several thousand. The goal is the minimum sufficient context, not a target token count.
Is more context always better for an AI agent? No. Beyond the point of sufficiency, more context degrades accuracy through context rot and lost-in-the-middle effects, while raising cost and latency. Selection beats volume.
Why does my AI give generic answers even with a big prompt? Likely because the specific fact it needs is buried or missing. Models drop information in the middle of long prompts and lean on training priors when grounding is weak. Scope the context tighter.
Does a bigger context window solve context problems? No. A bigger window increases capacity but not attention quality. Models can degrade at a fraction of their stated window size, so curation still matters.
What is the difference between context and memory? Context is the working set for a single call. Memory persists across sessions. Good agents combine scoped per-call context with durable agent memory so they stay consistent over time.
How do I know if my agent has too much context? Watch for facts that are present in the prompt but absent from the answer, rising token cost with flat or falling accuracy, and slower responses. Those are over-context signals. Shorten the prompt, move key facts to the edges, and prune unused history and spent tool outputs before each call.
Should I just buy the largest context window available? Only if a specific task genuinely needs more material in view. A bigger window raises the ceiling but not attention quality, and models degrade well below their advertised limits. For most tasks, scoped retrieval into a modest window beats dumping everything into a huge one.
Does the right amount of context change between models? The exact thresholds shift — newer models tolerate longer inputs better — but the shape holds across all of them: helpful up to sufficiency, harmful past it. The safe discipline is model-independent: retrieve the minimum sufficient context per task rather than tuning to one model’s quirks.