Context engineering: the discipline that replaces prompt engineering

Sep 08, 2026

The model doesn’t see your corpus, your memory, or your tools: it sees a context window you assembled. Deciding what goes in it, in what order, and what to leave out is the real discipline, and most teams underinvest in it.

The problem

A team has done everything right upstream. Retrieval is sharp: hybrid search, a reranker, the Article 3 stack. Memory works. The tools are well-built. And the agent still gives mediocre answers, because by the time all of that lands in the context window, the window holds eighteen retrieved chunks, a 4,000-token memory dump, six verbose tool descriptions, and the system prompt. The one paragraph that actually answers the question is sitting at position eleven of eighteen, buried in the middle where the model half-ignores it.

Nobody wrote a bad prompt. The prompt template is fine. The failure is in what got assembled into the context and in what order: too much retrieved, too little curated, the key fact buried, the budget spent on chunks that didn’t earn their place. That assembling is the discipline, and “prompt engineering” was always a misleading name for it: the leverage was never in the wording of the instruction, it was in selecting and ordering everything else the model reads. The knowledge layer (the part of an agent system that decides reliability in production, and an engineering discipline in its own right) has its final, decisive step right here, where everything retrieved and remembered gets composed into the few thousand tokens the model actually sees.

How context engineering works

The spine: treat the context window as a finite budget you allocate deliberately, not a bucket you fill until retrieval stops returning things.

The reason this is a discipline and not a formatting chore is that more context is not better context: it actively degrades past a point well short of the advertised window. Two findings make this concrete. “Lost in the Middle” (Nelson Liu and co-authors, TACL 2023, [arXiv 2307.03172](https://arxiv.org/abs/2307.03172)) showed models use information at the start and end of a long context far better than information in the middle. Position, not just presence, determines whether a fact gets used. And NoLiMa (Ali Modarressi and co-authors, [arXiv 2502.05167](https://arxiv.org/abs/2502.05167), ICML 2025) showed how early the rot sets in: at 32k tokens, 11 of the 13 models they tested dropped below half of their own short-context accuracy, long before the million-token windows the vendors advertise. The window being big enough to hold everything is not the same as the model using everything in it, and that gap is the entire reason context engineering exists.

The moves that follow from this are concrete. Compress memory rather than dumping it: a summarized state, not a raw transcript. Trim retrieval to the reranked top handful, not the top-50; this is where the reranker from Article 3 pays off a second time, as a context-budget tool. Prune tool descriptions to the ones plausibly relevant to the current step, because every unused tool spec is tokens spent on noise. And order deliberately: put the load-bearing content where the model attends, at the edges rather than buried mid-context.

In practice

Take the buried-answer scenario. The naive assembly and the engineered one start from the same retrieval and end very differently:

```text
NAIVE (≈9k tokens, answer at position 11 of 18):
  system prompt
  ALL 18 retrieved chunks, raw similarity order
  full 4k-token memory dump
  all 6 tool descriptions
  question

ENGINEERED (≈3k tokens, answer up front):
  system prompt
  question                       ← stated early, so retrieval is read in its light
  top-4 reranked chunks          ← 18 → 4, the reranker doubles as a budget tool
  1-paragraph memory summary     ← 4k → ~200 tokens, compressed not dumped
  (tool descriptions: only the 2 the current step can use)
```

Same upstream pipeline, a third of the tokens, and the fact that answers the question is no longer in the dead zone. The engineered version is also cheaper and faster on every call, because tokens are the bill and the latency. None of this touched the model, the retrieval, or the prompt wording. It’s pure allocation, and it’s the cheapest large reliability win most teams have left on the table.

Two vendor primitives make the allocation cheaper to run. Anthropic’s prompt caching (launched August 2024) lets you keep the stable part of the context (system prompt, tool specs, a long reference doc) cached across calls at a fraction of the input cost, so good context engineering stops being something you re-pay for on every turn. And OpenAI’s Structured Outputs (August 2024) constrains the model’s response to a schema, which keeps the output tight and parseable instead of spending tokens on prose you have to clean up downstream. The field has a name for this now: Anthropic’s own Effective context engineering for AI agents (September 2025) framed it explicitly as the successor discipline to prompt engineering, “curating and maintaining the optimal set of tokens (information) during LLM inference”, which is roughly the moment the industry agreed the old name was wrong.

Where it shines

Two settings where context engineering is the highest-leverage thing you can do.

The first is agents with many tools and long histories: exactly the agentic systems from Article 5. The more capable the agent, the more competes for the window: tool specs, prior steps, retrieved evidence, memory. Here, disciplined allocation is the difference between an agent that stays coherent over a long task and one that degrades as its own context fills with its own exhaust. Pruning what each step actually needs is not optional at this scale.

The second is high-volume production where cost and latency are the constraint. When you’re paying per token across millions of calls, the difference between a 9k-token context and a 3k-token one that answers as well is a direct, recurring cut to the bill and the p99. Context engineering is one of the few reliability moves that makes the system cheaper while making it better, which is why it’s an easy sell to the person holding the budget.

Where it breaks

It’s a discipline with real failure modes, not a free lunch.

Over-compression drops the load-bearing detail. Summarize memory too aggressively and you delete the one fact the next turn needed; trim retrieval too hard and the answer wasn’t in the four chunks you kept. The budget can be cut too far, and the failure looks exactly like a retrieval miss even though retrieval was fine.

Context poisoning persists and compounds. Once a wrong fact (a bad memory, a hallucinated tool result) is in the context, it gets carried forward and the model treats it as ground truth, and in an agentic loop it contaminates every subsequent step. A single bad item in a long-running context is more dangerous than a single bad retrieval, because it sticks around.

Ordering is model-dependent and shifts under you. The “put it at the edges” rule comes from how today’s models attend, and that behaviour changes across model versions: a context layout tuned for one model cohort can underperform on the next. It’s a moving target, which is why it has to be measured, not assumed.

When not to use it

There isn’t really a “don’t” here: every system that puts text in a context window is doing context engineering, the only question is whether deliberately or by accident. The honest version of “when not to” is when not to over-invest: if your context is small and stable (a short system prompt and one retrieved document), elaborate budget machinery is premature, and you should spend the effort on retrieval quality instead. The discipline scales with the complexity of what’s competing for the window. A simple agent doesn’t need a context manager; a multi-tool agent with memory absolutely does.

Combinations and hybrids

Context engineering is the stage every other technique in this series feeds into. Retrieval (Articles 2–4) produces the chunks; the reranker decides which few survive the budget. Memory (Article 6) produces the history; compression decides how much of it lands. The agentic loop (Article 5) produces intermediate results; context management decides what carries forward and what gets dropped between steps. It’s the convergence point, which is also why it’s where upstream mistakes become visible, since a sloppy retrieval or an unbounded memory shows up here as a bloated, low-signal window. Get the earlier stages right and context engineering is allocation; get them wrong and it’s damage control. The next two articles turn to keeping the knowledge itself correct: editing it, and proving it works.

Production checklist

Budget the window explicitly: decide token allocations per source (system, tools, memory, retrieval) rather than letting whatever’s returned fill it.
Measure the position of the answer, not just its presence: instrument whether load-bearing content lands at the edges or the dead middle.
Compress memory on the write or assembly path; never dump a raw transcript into context.
Prune tool descriptions per step to the plausibly-relevant set; unused specs are pure token cost.
Cache the stable prefix (system prompt, tool specs, reference docs) so good context isn’t re-paid on every call.
Re-tune ordering on model upgrades: the attention profile shifts; what was optimal last cohort may not be this one.
Alert on context bloat: track tokens-per-call and flag when a window is creeping toward the degradation zone the benchmarks warn about.

The take

“Prompt engineering” was always a misnomer for the real discipline: deciding what lands in the context window, in what order, and what to leave out. The model doesn’t see your pipeline. It sees the few thousand tokens you assembled, and more of them is not better past a limit that arrives far earlier than the advertised window. Budget the context like the scarce resource it is, put the load-bearing facts where the model actually reads, and measure it on every model upgrade. It’s the rare reliability win that makes the system cheaper and faster at the same time, which is exactly why the teams that underinvest here are leaving the easiest gains on the table.

I take on a small number of advisory engagements each year for teams hitting exactly these problems. Reach out if that’s you.

The Knowledge Layer

Discussion about this post

Ready for more?