Your agent's problem isn't the model. It's the knowledge layer.

Most agent failures aren't model failures or prompt failures — they're failures in the layer underneath. This series is about that layer.

Jun 02, 2026

The pattern

I’ll start with a pattern. You’ve probably seen it.

An LLM agent demos well. The team builds it against a few hand-picked examples, ships it to a stakeholder meeting, the stakeholder is impressed, and the project gets the green light for production. Three months in, the agent is quietly turned off and replaced with a form. Nobody calls it a failure; they call it a V1.

When you dig into why, the failure is almost never the model. The team didn’t pick the wrong LLM — they were probably on Claude Opus 4.7, GPT-5.5, or Gemini 3.5 Flash, and any of those would have been fine (even an overkill in most cases). The failure is almost never the prompt either; modern models tolerate prompt sloppiness in a way they didn’t two years ago. The failure is, almost every time, in the layer between the corpus of company knowledge and the model: which documents got retrieved, what the memory remembered (or forgot), how the context was packed, what got cached, what was stale, what was missing.

Most people now call it the knowledge layer. Pinecone calls its Nexus a “knowledge engine”. Microsoft markets Fabric IQ as a “unified intelligence platform”. Google’s April 2026 rebrand from Dataplex Universal Catalog to Knowledge Catalog put the word in the product name. What’s lagged is the engineering discipline around it — most teams still treat it as three separate plug-ins (RAG, memory, context engineering) bolted onto a model rather than one coherent job. This series makes the case that this — treated as one job, not three — is what decides whether your agent survives production.

This first piece is the map: what the knowledge layer is, why it deserves to be treated as its own engineering discipline, and the three places I keep seeing teams break in production. This article series will go deep on the techniques — what they do, where they shine, where they break, and how to choose between them.

What the knowledge layer actually is

Strip an agent system down and you find three things stacked on top of each other.

There’s the model — the LLM you’re renting from OpenAI, Anthropic, Google, or whichever provider you prefer. There’s the prompt — the instructions you give it, the role you put it in, the format you ask the output in. And underneath those, doing the work that’s much less visible, there’s the knowledge layer — everything that has to happen for the model to know what’s actually in your company, your conversation history, your last invoice, your evolving SOPs.

In practice the knowledge layer is four sub-problems that compose into one:

Storage. Where the corpus lives. The vector-database era is the canonical answer — Pinecone, Weaviate, Qdrant, or Chroma if you’re starting small; Milvus or Vespa at scale. But the most common starting point in real enterprises isn’t a dedicated vector database at all — it’s pgvector on the Postgres they already run, which means no new system to operate, transactional consistency with the rest of the data, and a migration path to a purpose-built store only if and when scale forces it. “Just use pgvector” is the right first answer more often than any vendor will tell you. The graph side has its own canon: Neo4j, Memgraph, TigerGraph for typed-relation corpora. And in 2026 a new tier appeared — dedicated knowledge engines explicitly built for agents: Pinecone Nexus (May 2026, ships with a query language called KnowQL and compile-time artifacts), Microsoft Fabric IQ (announced at Ignite 2025, semantic layer over OneLake with ontology + graph + agents), Google Knowledge Catalog (the April 2026 rebrand of Dataplex Universal Catalog, building dynamic context graphs for Gemini agents). The schema and abstractions you commit to here are very hard to change later.
Updating. How the storage stays fresh when the world moves on — when a policy is rewritten, when a product is discontinued, when a customer changes their preferences. Historically this was a custom ETL pipeline somebody on the team owned. In 2025–2026 it’s slowly becoming a vendor concern — Pinecone Nexus’s Context Compiler, Mem0‘s dynamic memory extraction, Letta‘s streaming-update primitives — but most enterprises still hand-roll it. This is the sub-problem most teams discover six months in.
Retrieval. How the right slice of the corpus shows up at the right moment — what gets pulled when the user asks “is acupuncture covered by my health plan?”, and what gets ignored. Embedding-model choice (text-embedding-3-large from OpenAI, Cohere Embed 4, Voyage 3) and reranker choice (Cohere Rerank 3.5, Voyage Rerank) make most of the difference, with hybrid sparse-dense techniques (BM25 + vectors) and graph-augmented retrievers (LightRAG, KAG) cleaning up the rest. Most “RAG isn’t working” conversations turn out to be retrieval conversations.
Context construction. How the retrieved material gets ordered, weighted, and packed into the model’s context window. The state of the art shifted twice in 2024–2025 — first with Anthropic’s Contextual Retrieval (September 2024), which argued context construction is its own engineering discipline, then with Anthropic’s Effective context engineering for AI agents essay and the broader industry adoption of prompt caching, structured outputs, and (with Claude Managed Agents in April 2026) cross-session memory primitives. The model only sees what context construction decides to show it, so this step quietly governs the quality of every answer.

The four are tightly coupled. A change in storage forces a change in retrieval. A change in retrieval breaks context construction. A change in update policy can quietly poison both. Teams that treat them as four separate engineering problems eat the cost of integrating them by accident.

Why it’s one discipline, not three

I want to argue that the knowledge layer is worth treating as one engineering job rather than three separate ones (RAG, memory, context engineering). Three reasons.

The first is that the failures cluster here. When agents fail in production, they fail because retrieval pulled the wrong document, because the corpus was stale, because the conversation history attributed a fact to the wrong session, or because the relevant chunk got drowned out by ten less-relevant chunks in a stuffed context window. These are different surface symptoms of the same underlying engineering problem: getting the right information in front of the model at the right moment.

The second is that the techniques compose. GraphRAG and a typed memory system aren’t alternatives; they’re tools you reach for at different points in the same pipeline. Treating each technique as a separate engineering investment is what produces the demo-to-production gap — the team ships RAG, finds it isn’t enough, layers on memory, finds that doesn’t fit either, plugs in a reranker, plugs in a graph, and the architecture quietly becomes a graveyard of half-finished plug-ins.

The third is that the discipline is recognizably different from prompting and from model choice. Prompting is how to talk to the model. Model choice is which model to rent. The knowledge layer is what does this model know about your business and how does it stay current. Those are different jobs, and putting them in the same bucket — the way “let’s add RAG” used to be a complete project plan — is part of why so many enterprise agent projects stall a year in.

The industry has been converging on this view in public. Anthropic published Contextual Retrieval in September 2024, essentially an argument that context construction is its own engineering problem, not a free side-effect of dropping a vector store next to your prompt. Microsoft’s GraphRAG paper, five months earlier, made the same case for structured knowledge. By the time Anthropic shipped Claude Managed Agents in April 2026 — multiagent orchestration, persistent state — and demoed cross-session “dreaming” at Code with Claude the following month, the knowledge layer wasn’t a side project anymore; it was the developer surface. The terminology and the products are converging fast. What’s still missing in most enterprise teams is the engineering culture around treating this as one coherent job rather than three plug-ins.

The model is the chef. The prompt is the recipe. The knowledge layer is the pantry — and the pantry is where most kitchens go wrong.

The three places it breaks

Three failure modes account for most of what kills enterprise agents in production. They show up across verticals, model choices, and frameworks.

Stale corpus. The data was right when it was loaded. Then a policy changed, a product was rebranded, a fee schedule was updated — and nobody told the retrieval system. The agent keeps citing the old version with full confidence because, from its perspective, the old version is the source of truth. The cost of this failure isn’t only the wrong answer; it’s that the user now has to fact-check the agent every time, which destroys the time-saving the agent was supposed to deliver in the first place.

Wrong retrieval. The right document exists in the corpus, but it didn’t surface for the user’s question. Maybe the question was phrased differently from how the document was written; maybe the embedding model — text-embedding-3-large, Cohere Embed 4, Voyage 3, take your pick — treats two semantically-related phrases as unrelated; maybe the chunking sliced the relevant passage in half. The agent then either falls back on the model’s pre-trained knowledge — which is generic and often wrong for your domain — or fabricates an answer from the closest-but-incorrect chunks it did manage to retrieve.

Wrong context construction. Retrieval surfaced the right chunk, but it was packed in among ten less-relevant chunks, or it was at the bottom of a long context window where models attend less, or the system prompt was so long that the retrieved content was effectively buried. The model had the right information in front of it and still gave the wrong answer because the context wasn’t constructed to let it see clearly.

The pattern across all three is that the model and the prompt were fine, and a smarter model or a cleaner prompt wouldn’t have helped. The fix is at the knowledge-layer level.

The map

Here’s how the four sub-problems compose into a pipeline.

The interesting moves all happen along that horizontal axis: choosing a storage abstraction (vector store, typed graph, hybrid), choosing an update strategy (re-index, delta, event-driven), choosing a retrieval method (dense, sparse, hybrid, agentic), choosing a context construction policy (reranking, compression, structured assembly). The next nine articles in this series each go deep on one of those decision points.

The landscape

Before the techniques, here’s how to approximately position them mentally. Two axes do most of the work:

Horizontal — how structured the underlying knowledge is. Raw vectors on the left (the semantic-similarity world), typed graphs and ontologies on the right (the relational world).
Vertical — how much agency the system has at retrieval time. A fixed pipeline at the bottom (the agent doesn’t decide what to fetch), an agentic loop at the top (the agent plans and refines retrieval as part of its reasoning).

The axes are a loose synthesis of the architectural breakdown in the SoK: Agentic RAG paper (Mishra et al., arxiv 2603.07379, March 2026) and the Agentic RAG Survey (Singh et al., arxiv 2501.09136, revised April 2026), both of which organize current paradigms along similar structure-and-agency dimensions. The placements on the map aren’t precise — vendors move on it as they ship — but it’s the first thing I sketch when a team asks “where does our stack sit?” The most interesting movement in 2025–2026 has been up and to the right: from naive RAG in the bottom-left to compiled knowledge engines and ontology-backed agent platforms in the top-right.

What this series will cover

Briefly, and not as a table of contents.

Articles 2–5 walk through the retrieval surface — naive RAG, advanced RAG (the production version with reranking, hybrid search, query rewriting), GraphRAG for when relationships matter more than similarity, and agentic RAG for when retrieval becomes a decision the agent makes rather than a pipeline it runs. Article 6 covers agent memory systems — Mem0, Letta, Zep, LangMem (the four-vendor landscape that consolidated through 2025–2026), plus research-side proposals like A-MEM and MemPalace — and the question of what kind of memory your agent actually needs. Article 7 is on context engineering, which I think is the discipline that quietly replaces prompt engineering for serious teams. Article 8 critiques knowledge editing — the model-weight-surgery techniques the academic world is excited about, and why they’re wrong for almost every enterprise update problem. Article 9 is on evaluation, the part most teams skip and most regret skipping. Article 10 is the practical selection guide: given your corpus, your update frequency, your latency budget, and your team’s depth, here’s what to actually build.

On the community and open-source side, the conversation has been just as alive — and worth keeping an eye on even if you don’t deploy any of these. Andrej Karpathy’s LLM Wiki gist (April 2026) sketched a pattern for compounding personal knowledge that pre-compiles a maintained wiki of interlinked entity pages instead of re-retrieving raw chunks at query time. MemPalace (April 2026) applied the ancient method-of-loci spatial metaphor to long-term memory and pulled 47k GitHub stars in two weeks, prompting a critical-analysis paper from Dey and Viradecha that, fairly, attributed most of the retrieval performance to the underlying embeddings rather than the spatial metaphor. A-MEM (February 2025, AGI Research) extended the Zettelkasten idea into agentic memory with dynamic indexing. MAGMA (January 2026) experimented with multi-graph agentic memory. None of these are production stacks yet, but they’re where the next set of ideas tends to come from.

The articles can be read in isolation, but they reward being read in sequence.

What it costs you when you get this wrong

Rough numbers, because the buyer reading this needs them.

These are rough. I haven’t seen a team that escaped them by working harder on prompts.

The take

If you’re building agents and you’re spending most of your engineering effort on prompts and model selection, you’re working on roughly twenty percent of the problem. The other eighty percent — the part that decides whether your agent survives contact with real users and real data — lives in the knowledge layer. Get it right and the model and prompt choices become easy. Get it wrong and no amount of model upgrading or prompt tuning will save you.

The knowledge layer is the part of an agent system that decides reliability in production, and it’s an engineering discipline in its own right. That’s the claim the rest of this series defends.

New articles land here first. Subscribe for the full series. I take on a small number of advisory engagements each year for teams hitting exactly these problems. The fastest way to reach me is on LinkedIn; reach out if that’s you.

The Knowledge Layer

Discussion about this post

Ready for more?