Context Layers: Giving Al the Information It Needs

In the first article of this series, we argued that the model is the easy part of a production AI system — everything around it is the product. This article zooms into the single most consequential piece of that “everything”: the context window, and the six distinct layers of information that compete for space inside it.

Every token a model sees arrives through one of six channels: system instructions, retrieval, tool outputs, conversation history, structured memory, or live state. Most teams treat these as one undifferentiated blob of “the prompt.” Production teams treat them as an architecture — because each layer has a different cost, a different half-life, and a different way of failing. Anthropic’s engineering team frames the discipline precisely: the question is not what words do I use? but what configuration of context is most likely to produce the behavior I want?

This piece maps the six layers: what each one is for, what it costs, how fast it goes stale, how it fails — and how to compose all six into one context window without collision.

The Six Layers at a Glance

Think of the context window as a stack. At the bottom sits the most stable information (who the agent is); at the top, the most volatile (what is true right now). Every layer in between trades stability for freshness.

The order matters — not just conceptually but literally. Research on long-context behavior shows models attend most reliably to the beginning and end of their input, with recall degrading in the middle — the well-documented lost in the middle effect. Stable framing belongs at the start of the window; the freshest, most decision-relevant information belongs near the end, closest to the question being asked.

Layer 1: System Instructions

System instructions are the agent’s constitution: identity, rules, tool definitions, output format, and refusal boundaries. They are written by engineers, reviewed like code, and changed through releases — never at runtime.

When to use it: for anything that must be true in every interaction. Behavioral rules, safety boundaries, tone, schemas. If a fact only matters sometimes, it does not belong here — every token in this layer is paid on every single call.

Cost profile: the most expensive layer per unit of information, because it is always present. The saving grace is prompt caching, which lets providers reuse the processed prefix across calls at a fraction of the price — but only if the layer stays byte-stable. Teams that inject timestamps or per-user data into system prompts silently destroy their cache hit rate.

Half-life: months. This is the slowest-moving layer in the stack.

Failure mode: ossification. Instructions accumulate (“also never do X…”) until the prompt becomes a sediment of patches that no one dares refactor, and rules begin to contradict each other. The symptom is an agent that behaves erratically on edge cases because it is obeying two incompatible sentences written six months apart.

Layer 2: Retrieval

Retrieval — the pattern introduced by the original RAG paper — fetches knowledge on demand: documentation, policies, product data, prior tickets. Content is chunked, embedded, and searched at question time, so the model sees only the slice of the corpus that matters now.

When to use it: for knowledge too large to fit in the window and too changeable to bake into instructions. Retrieval is the difference between an agent that knows about your business and one that knows your business.

Cost profile: moderate per call (only retrieved chunks are paid for), plus standing infrastructure — embedding pipelines, a vector store, reindexing jobs. The hidden cost is quality engineering: chunking strategy, hybrid search, and reranking decide whether the right passage actually arrives.

Half-life: hours to days — exactly as fresh as the last reindex. A perfect retrieval system over a stale index is a confident liar.

Failure mode: silent irrelevance. Retrieval rarely errors; it returns something, and the model weaves whatever arrives into a fluent answer. Wrong-but-plausible chunks are the root of most production hallucinations attributed to “the model.”

Layer 3: Tool Outputs

Tool outputs are what the agent just learned by acting: the API response, the query result, the file contents. Increasingly these arrive through standardized connectors via the Model Context Protocol, rather than bespoke glue code.

When to use it: whenever the answer must reflect a system of record. Reading the actual order status beats remembering it, retrieving it, or guessing it — tool outputs are ground truth at the moment of execution.

Cost profile: deceptively high. A single verbose API response can be tens of thousands of tokens, and in a multi-step loop each output is re-sent on every subsequent call. Production agents truncate, filter, and summarize tool results before they enter the window — returning the twelve fields the task needs, not the four-hundred-field payload.

Half-life: minutes — or less. A stock quote, a seat availability, a build status: each is true when fetched and decaying immediately after.

Failure mode: flooding. An unfiltered tool result shoves everything else out of the effective window, and the agent forgets its own plan because a JSON blob is sitting where its reasoning used to be.

Layer 4: Conversation History

Conversation history is the session’s running transcript — every user turn, assistant reply, and intermediate step. It is what makes an exchange feel coherent: pronouns resolve, corrections stick, plans persist.

When to use it: always, but never raw for long. As sessions grow, history must be actively managed — rolling summarization, pruning, and compaction that preserve decisions and discard filler.

Cost profile: the compounding layer. Unmanaged, history makes every call more expensive than the last, and cost grows quadratically over a session’s lifetime as the transcript is re-sent with each turn.

Half-life: the session itself. Yesterday’s transcript is rarely worth its tokens tomorrow — its durable facts belong in memory (Layer 5), not in replay.

Failure mode: drowning the signal. The decision made in turn 3 sits in the degraded middle of the window at turn 40 — technically present, effectively invisible. The model contradicts an earlier commitment not because it “forgot,” but because attention over a bloated transcript is a lossy channel.

Layer 5: Structured Memory

Structured memory is what survives after the session ends: user preferences, entity facts, past decisions, learned failures — extracted, typed, and stored deliberately rather than replayed verbatim. It is the long-term memory in the classic agent taxonomy: written by an explicit process, retrieved selectively, and owned like data.

When to use it: for anything that should be true next session. “Customer prefers concise answers.” “Refund policy exception approved on 2026-05-12.” “This fix already failed twice.” Memory is the difference between an agent with users and an agent with relationships.

Cost profile: cheap to read (a handful of compact facts), costly to maintain — extraction pipelines, deduplication, conflict resolution, and access control. Memory is a database with opinions, and it needs the same tenant isolation and retention discipline as any other store.

Half-life: weeks to months, with explicit expiration. A memory system without TTLs and consolidation becomes an archive of stale certainties — worse than forgetting, because it is confidently wrong.

Failure mode: the outdated fact stated with conviction. Retrieval failures look like ignorance; memory failures look like knowledge. That makes them harder to catch and more corrosive to user trust.

Layer 6: Live State

Live state is the runtime’s snapshot of now, injected fresh into every call: the current time, the user’s role and permissions, feature flags, environment, task progress, remaining budget. It is never cached, never remembered, never retrieved — always regenerated.

When to use it: for anything where a stale value is not merely unhelpful but wrong. Permissions read from memory instead of live state is how an agent honors an access grant that was revoked an hour ago.

Cost profile: the cheapest layer — a few dozen structured tokens — and the highest value-per-token in the stack.

Half-life: seconds. By design, live state has no memory at all.

Failure mode: absence. Nothing crashes when live state is missing; the model simply fills the gap with an assumption (“it is probably still business hours”, “the user can probably see this document”). The bug report will blame the model. The defect is in the layer that wasn’t there.

Three Dimensions That Separate the Layers

Put the six layers side by side and the architecture almost designs itself. Each occupies a distinct position on three axes — what it costs, how fast it decays, and how it breaks.

The half-life axis deserves special attention, because it dictates refresh machinery: a layer’s half-life tells you how often something must rebuild it, and what happens when nothing does.

And the failure modes are the debugging map. When a production agent misbehaves, the fastest diagnostic question is: which layer failed? Contradictory behavior points at instructions; plausible-but-wrong facts at retrieval; forgotten plans at history management; confident staleness at memory; unwarranted assumptions at live state.

Composing the Window: One Budget, Six Claimants

Knowing the layers is half the job. Production systems must assemble them — per call, under a hard token budget, without letting the layers collide. Three rules govern the composition.

Rule 1: Position by stability. Stable layers first, volatile layers last. System instructions open the window (and stay byte-identical for caching); live state and the freshest tool outputs land nearest the user’s question, where attention is strongest. This single ordering decision serves the cache, the attention curve, and staleness precedence simultaneously.

Rule 2: Budget explicitly. Give every layer an allocation and an enforcement mechanism — history gets compacted to fit its share, retrieval returns its top-k and no more, tool outputs are summarized past a threshold. A window without budgets is a window where the loudest layer wins, and the loudest layer is never the most important one.

Rule 3: Resolve collisions by freshness. The layers will disagree: memory says the customer is on the Pro plan, the tool output says Enterprise; instructions say “always offer a call,” live state says it is 3 a.m. in the user’s timezone. Composition needs an explicit precedence order — as a rule, live state and tool outputs (ground truth now) override memory and retrieval (beliefs about the past), and task-specific context overrides general instructions. When the conflict is material, the agent should surface it rather than silently pick a side.

Staleness is handled the same way: each layer carries its refresh obligation. If retrieval’s index predates the last policy change, or memory’s fact has outlived its TTL, the composition layer should treat it as suspect — not equal — evidence.

Where This Lands in Practice

This layered view is the architecture behind the context engineering we described in the production guide: prompt engineering polishes one layer; context engineering orchestrates all six. It is also the design principle behind context-aware evaluation in Omdena’s own tooling — Umaku’s premise is that an AI system judging real work needs the business context, project context, and live project state stacked correctly before it reasons at all.

The practical takeaway is a checklist. For each layer, a production team should be able to answer: who writes it, what it costs per call, when it was last refreshed, and what happens when it is wrong. Six layers, four questions each — twenty-four answers that describe an architecture, not a prompt.

Models will keep getting better at reasoning. But no model can reason its way out of a context window that is missing the fact it needed, drowning it in noise, or asserting last month’s truth. Feed the model deliberately, layer by layer — that is the work.