Context engineering for production LLM systems

Every scale production AI failure traces back to what the model saw, not how the prompt was phrased. The retrieval returned the wrong document. The tool catalog ate 40,000 tokens of overhead before the user said hello. The chat history drowned the current turn's intent. The long context window hit its own degradation curve weeks before anyone measured it. All four are context problems. Prompt engineering is the narrow subset that only moves the instruction string.

The discipline that replaced it has a name now and a working set of techniques. Anthropic's engineering team published its first full guide to context engineering for AI agents on September 29, 2025, alongside Claude Sonnet 4.5, and defined it as the practice of curating and maintaining the most useful tokens during inference. Chroma Research had set the stage three months earlier with its context rot study. By 2026, context engineering is the discipline a US B2B engineering lead can point to in code review, and the gap between teams that treat it as a discipline and teams that treat it as better prompting is the gap between a demo and a system a production team can run.

This post is for the engineer who is building the system and the operator who is paying for it. The sections below name the six layers of context every production agent call assembles, the three techniques teams actually ship for long-horizon work, the implementation decisions that separate a disciplined context layer from an ad-hoc one, and the failure modes each pattern prevents. Where the mechanism is deeply technical, the sentence after it translates for the non-engineer reading along.

The first instinct, through 2024 and most of 2025, was to buy a bigger context window. Google shipped a 2 million token window on Gemini 1.5. Anthropic shipped a 1 million token window on Claude. The assumption was that if the model could see more, it would make better decisions. In practice, teams discovered the inverse: the longer the context, the less reliably the model used any part of it.

The Chroma Research context rot study, published by Kelly Hong, Anton Troynikov, and Jeff Huber in July 2025, documented the pattern across 18 production models from Anthropic, OpenAI, Google, and Alibaba. The same model, given the same task, degraded in accuracy as the input grew longer, with the degradation curve varying by model and task but present in every single run. On the LongMemEval benchmark, which uses prompts averaging 113,000 tokens against a focused 300-token baseline, every model tested performed significantly better on the focused prompt. On a repeated-words task scaling from 25 to 10,000 input words, Gemini models began producing random words around the 500 to 750 word mark; Qwen3-8B started producing random outputs around 5,000 words.

Three details from that study matter for a production team. First, distractors in the context compounded the drop, with one distractor already visible in the error curve and four compounding it. Second, haystacks with shuffled sentences produced better results than logically structured ones across all 18 models, which suggests that models are not reading context the way the team thinks they are. Third, position in the window mattered less than total length, which flips the earlier lost-in-the-middle intuition on its head: the problem is not where in the window the key fact sits, it is how many tokens are competing for the model's attention.

What this means for a team planning a production system: the context window is a shared attention budget, not a container. Every irrelevant document in a RAG retrieval, every tool schema the agent cannot actually call, every chat message from the morning that no longer matters costs the model accuracy on the request the user is making right now. The job of context engineering is to spend that attention budget on tokens that will actually be used, and to prove it with evaluation.

In plain English for the operator: a model with a bigger window is not a model that reads more carefully. It is a model that has more ways to get distracted. The teams shipping reliable production AI in 2026 budget what the model sees as deliberately as they budget what it costs.

Every production agent call assembles its context from six distinct sources, each with a different lifecycle and a different failure mode. A disciplined team names them and tests them separately.

Instructions are the first layer: the versioned system prompt, the hard rules, the output schema the response must conform to. These change rarely and are stored in version control, not in a prompt library UI. A system prompt that still carries edge-case rules from last quarter is a prompt that is competing with current rules for attention; mature teams audit theirs on the same cadence as the eval suite.

Task context is the second layer: the typed fields describing what this call is for. Tenant ID, user ID, workflow node, request type. In a disciplined system this is a Pydantic or Zod schema the orchestration layer validates before any model is called. Untyped task context is where cross-tenant data leaks start.

Conversational memory is the third layer, and the one most teams implement loosest. The pattern that holds up in production has three parts: a short raw window of the most recent turns (usually three to five), a running summary of older turns, and extracted facts the agent needs to carry forward (user preferences, resolved sub-tasks, named entities). Dumping the full chat history into every call is where the cost curve breaks first and the quality curve breaks second.

Retrieved knowledge is the fourth layer: the output of RAG, graph queries, or tool-returned documents. A production retrieval layer applies metadata filters before vector search, hybrid keyword and vector matching, re-ranking against the actual query, and freshness windows that drop stale content. The compressed output is a small set of ranked chunks with citations, not full documents, and almost always carries a provenance tag so the audit log can trace what the model saw. The failure modes here are the same ones [RAG systems hit in production](/blog/why-rag-fails-in-production): a confident wrong answer because the retrieval returned a related document that was not the right document.

Tool context is the fifth layer: the catalog of tools the model can call. The number is the key design decision. OpenAI's own guidance to agent builders is to keep the tool count per agent under 20. Industry reviews have found that tool schemas across roughly 90 tools can consume more than 50,000 tokens before the user has said anything, which is attention the model does not spend on the request. Mature teams scope the tool catalog per workflow node, not per agent: the node that drafts an email has draft and preview tools; the node that sends has send; neither has both.

Operational context is the sixth and most-skipped layer: remaining budget, prior failures in this run, approval state, step count. A system that reruns a failed tool call without reading its own operational context is a system that will loop. A system that lets a 50-cent request burn through its budget on retries is a system without operational context in production.

In plain English for the non-engineer: the model does not read your Slack channel. It reads a package your code assembles for it on every call. The quality of the output depends on what that package contains at least as much as on the model you chose. The six layers are the parts of that package you can control.

For a long-horizon task, where the agent runs across many turns or tool calls, the three techniques that hold up in production are compaction, structured note-taking, and multi-agent architecture. Anthropic's September 2025 guidance documents all three, and the trade-offs are specific enough to pick between.

Compaction is a continuous summarization pass over the message history. When the context approaches a threshold (teams typically set it at 60 to 70 percent of the window), a cheap classifier or a dedicated summarization call rewrites older turns into a condensed summary, drops raw tool results that have been consumed, and keeps architectural decisions, unresolved sub-tasks, and the most recent few turns in full. Claude Code's open implementation keeps the five most recently accessed files in raw form after compaction. The design question is what to preserve under pressure: errors the agent has seen so it does not repeat them, decisions the agent has committed to so it stays consistent, and sub-task state so it can resume. Everything else is summary. Compaction suits a long back-and-forth where the state is conversational.

Structured note-taking is the alternative when the state is not conversational but iterative. The agent writes persistent notes to a file (NOTES.md, TODO.md, or a typed JSON state object) outside the context window, and rereads them after a reset or at the start of each turn. This is what Anthropic's memory tool, in public beta since 2025, supports. The design question is what the note schema contains: a to-do list the agent can tick through, decisions already made, open questions, and references to files or external resources. Structured notes suit multi-session work with clear milestones, where resume-from-NOTES.md is a more reliable restart point than replaying the whole chat.

The multi-agent split is the third option. A lead agent handles planning and synthesis in one context window; sub-agents explore specific sub-tasks in their own clean windows and return a condensed summary (Anthropic's guidance names 1,000 to 2,000 tokens as the right size). The lead never sees the sub-agent's raw context; it reads only the summary. This suits parallel research, hierarchical workflows, or any task where one agent would otherwise accumulate a working set too large for any single window to hold accurately. The failure mode is specific and worth naming: a sub-agent that returns a 10,000-token summary has not helped; it has moved the problem up a level. Teams shipping this pattern budget the sub-agent's output and reject anything over the cap, which forces the sub-agent to prioritize.

In plain English for the operator: a model that runs for an hour sees more tokens than it can hold in attention. Compaction, notes, and sub-agents are the three ways teams keep the agent's working memory focused. If a vendor's agent has none of them, the agent will degrade past a certain horizon length, and the horizon is usually shorter than the demo.

The implementation discipline that separates a context layer from a prompt dump has four specific moves.

First, a typed context package. The context assembled for a model call is a schema object (Pydantic in Python, Zod in TypeScript), not a string the orchestration layer concatenates. The schema has typed fields for each of the six layers above, with versioned instruction references, a token budget, and a validation step. A context package is testable, loggable, and diffable; a concatenated string is none of those things. The pattern that holds up: every model call carries a trace of the context package it ran on, logged with a hash so audit can reconstruct what the model saw at any point in the past.

Second, per-node context, not per-agent. In a workflow graph, each node (classify intake, retrieve policy, draft response, approve, send) gets its own context package with its own instructions, its own tool catalog, and its own retrieval scope. The intake classifier does not need the sender's full history; the policy retriever does not need write tools; the send step does not need the full draft chain. This is where tool-use failure modes get prevented structurally, and it is the same discipline [agent architecture patterns](/blog/ai-agent-architecture-patterns) applies at the macro level.

Third, token budgets with a drop policy. Every context package has a budget per section, assigned in that order: instructions and output schema first (because without them nothing else works), task context second, recent messages third, retrieved knowledge fourth. When the budget is tight, the lowest-value content drops first, which almost always means older retrieved chunks or older conversational turns. The budget is not aspirational; it is enforced at assembly time with a hard limit, and the assembly code either drops content or returns an error. Teams that ship this discipline also budget model calls, tool calls, wall-clock time, and spend per tenant, which is what keeps a runaway loop from producing a surprise invoice. The ceiling is code, not a dashboard, in the same pattern [pilot with guardrails](/blog/pilot-with-guardrails-deep-dive) names.

Fourth, the tool-schema tax. Every tool in the agent's catalog adds its JSON Schema to the context, and the sum grows faster than most teams model. The MCP ecosystem makes this worse before it makes it better: the Model Context Protocol standardized how agents connect to tools in late 2024, and by 2026 Anthropic's registry alone lists more than 10,000 published MCP servers. Each server exposes multiple tools; a team that connects five commonly used MCP servers can end up with 60 or more tools in a single agent's catalog without noticing. The fix is scoping per node, as above, and auditing the tool catalog token cost on the same cadence as the eval set.

In plain English for the technically literate operator: the agent carries a tax on every call, paid in attention and dollars. The tax is proportional to how many tools, documents, and chat messages the system sent along for the ride. A disciplined team knows the number before the invoice arrives.

Context engineering is not a universal defense. It prevents a specific set of production failures and leaves others alone. Naming them separately is how a team decides what to build first.

What it prevents: wrong-but-confident answers, because retrieval was similarity-matched instead of filtered and re-ranked; dangerous side effects, because the write tool was in the catalog of a node that should not have had it; prompt injection via retrieved content, because retrieved passages are delimited as evidence and tool authorization is enforced in code, not in the prompt text; runaway cost from full-history replay, because conversational memory is compacted; loops on permanent errors, because operational context tracks prior failures and the budget caps retry count; contradictions from an overgrown system prompt, because instructions are versioned, modular, and audited; and the more-context-is-safer retrieval-dumping anti-pattern, because retrieval is budgeted and compressed.

What it does not prevent: a wrong model choice for the task, a stale knowledge base, a system prompt that is wrong on its merits, a tool that is wrong on its merits, or an eval set that does not match production traffic. These are different disciplines. Context engineering lives alongside [production AI evaluation](/blog/how-to-evaluate-ai-systems) and alongside model selection; it does not replace either. A vendor who claims that moving to context engineering will fix an eval gap is selling the wrong thing.

The second category worth naming separately: hallucination without a context cause. A model that invents a reasonable-sounding fact when the context does not contain it is a model problem, usually fixable by grounding (retrieval that cites) or by refusing low-confidence answers. Context engineering reduces the frequency of this failure, because the retrieval layer brings the right source, but it does not eliminate it, and a team that routes hallucinations into the context backlog instead of the eval backlog will not close the gap.

An engineering team or vendor claiming to do context engineering should be able to answer four specific questions in code, not in slides.

What is in the context package for a given model call, and can you show the schema? If the answer is a string template, it is prompt engineering with extra steps. If the answer is a typed object with named fields for instructions, task, memory, retrieval, tools, and operational state, it is context engineering.

What is the token budget per section, and what drops first when the budget is tight? If the answer is that the limit has not been hit yet, the system has not been run on real traffic, or it is being run at a cost the team is not watching. If the answer names specific caps per section and a specific drop order, the budget is enforced in code.

Which tools does each workflow node have access to, and can you show the catalog? If every agent has every tool, scope was never designed in, which means excessive agency is one prompt-injection payload away. If the catalog is scoped per node with a documented list, the control is structural.

How is the context package logged, and can an auditor reconstruct what the model saw on last Tuesday's 3:17 PM run? If the answer is that the prompt that was sent is on file, the audit stops at the prompt string. If the answer includes retrieval source IDs, instruction versions, tool catalog version, and token counts per section, the discipline is operating.

Context engineering is not a slide and it is not a vocabulary refresh. It is the code that assembles, budgets, and logs what the model sees on every call, and the test harness that proves that assembly produces the behavior the business signed off on. The teams shipping production AI in 2026 build this layer first and treat the model choice as a tuning parameter inside it. The teams still shipping demos are tuning the prompt and wondering why real data breaks what the demo answered.

If this gap is where your pilot is stalling, Stallwart does production AI engineering for US B2B teams that need the context layer and the evaluation harness that turn a demo into a system someone can run without the engineer who wrote it. The deliverable is code the team can audit, with the context schema, the budgets, and the eval set the system depends on in version control.