An agent handles a customer over three sessions across two weeks. Session one, the customer explains they’re on the enterprise plan and their main pain is API rate limits. Session two, a different day, they ask a follow-up, and the agent greets them like a stranger, re-asks which plan they’re on, and re-explains the rate-limit basics it already walked through. Session three, the customer mentions they downgraded last week, and the agent cheerfully keeps citing enterprise-tier limits, because nothing told it the earlier fact had expired.

Everything in this series so far retrieves from a corpus that sits still. Memory is the part of the knowledge layer that has to remember things that happened: facts that accrue from the interaction itself, change over time, and contradict what was true last week. That’s a different write path, and the knowledge layer (the part of an agent system that decides reliability in production, and an engineering discipline in its own right) gets a lot harder once it has to update, not just retrieve. The vendors building agent memory have each bet on a different abstraction for that write path, and the bet they made determines what your agent is good and bad at remembering.

How agent memory works

Strip away the branding and there are three abstractions underneath every memory product. Most real systems combine two of them, and in my view every product I’ve looked at is a bet on which of the three to lead with, not a fourth kind. (The cognitive-science taxonomies add procedural memory, the learned skills and routines. It’s real, and it’s starting to show up as a feature, LangMem’s prompt-updating memory and Mem0’s procedural memory type among them, but no product leads with it yet, so it stays out of this tour.)

Semantic memory is the vector approach: extract the durable facts from a conversation, embed them, and retrieve the relevant ones later by similarity. It’s RAG pointed at the interaction history instead of a document corpus. Episodic / working memory keeps a running, self-editing representation of what happened (a summarized history, a persona block, a scratchpad) that the agent rewrites as the conversation grows, so the context window holds the gist rather than the raw transcript. Temporal knowledge graph memory stores facts as typed nodes and edges, each stamped with when it became true and when it stopped. Of the three, it’s the one that handles the downgrade in the opening story by construction: superseding a fact is a first-class, dated operation that keeps the old value in the timeline, rather than an update pass that has to notice the collision and get it right.

In practice: the vendor tour

This is the most vendor-heavy article in the series, because the memory space is where the abstraction choice is most visible in the product. Four worth knowing, each leading with a different bet.

Mem0 leads with semantic memory plus lightweight extraction. It sits next to your agent’s model calls: after each exchange it pulls the salient facts out and stores them as embeddings, and before each answer it recalls the relevant ones by similarity. It’s the vector bet, productised, with a focus on keeping the recalled context small. Mem0’s own paper (Prateek Chhikara and co-authors, Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory, [arXiv 2504.19413](https://arxiv.org/abs/2504.19413), April 2025, later published at ECAI 2025) frames the win as token efficiency: store a compact set of extracted facts rather than stuffing whole histories into context, and it reports a 91% lower p95 latency and over 90% lower token cost than a full-context baseline on the LoCoMo benchmark. Even the vector bet is being pulled toward time: Mem0 rewrote its extraction and retrieval algorithm in April 2026 (single-pass extraction, multi-signal retrieval), then in May 2026 added temporal reasoning and memory decay on top of it, which tells you where the pressure in this space is. It’s the easiest of the four to drop in, and the right default when “remember what this user told me” is most of what you need.

Letta leads with episodic / self-editing memory, and it’s the direct descendant of the MemGPT paper (Charles Packer and co-authors, MemGPT: Towards LLMs as Operating Systems, [arXiv 2310.08560](https://arxiv.org/abs/2310.08560), October 2023). The core idea is borrowed from operating systems: treat the context window like RAM and an external store like disk, and let the agent itself decide what to page in and out, editing its own memory blocks (a persona, a running summary, key facts) as the conversation evolves. The bet is that an agent which manages its own context beats one handed a fixed retrieval, and it’s the strongest fit for long-running conversational agents that need a coherent sense of the whole relationship, not just a bag of recalled facts.

Zep leads with the temporal knowledge graph, built on its open-source engine Graphiti. Every fact extracted from the conversation becomes a typed edge in a graph with validity timestamps, so the system knows not just what is true but when it was true and what superseded it (Preston Rasmussen and co-authors, Zep: A Temporal Knowledge Graph Architecture for Agent Memory, [arXiv 2501.13956](https://arxiv.org/abs/2501.13956), January 2025). This is the abstraction that handles change correctly (the downgraded customer, the moved deadline, the role that changed), and it’s where the memory question and the GraphRAG question from Article 4 turn out to be the same question asked of a different corpus.

A-MEM is the research frontier rather than a product, and it’s the one I find most interesting. Wujiang Xu and co-authors (A-MEM: Agentic Memory for LLM Agents, [arXiv 2502.12110](https://arxiv.org/abs/2502.12110), February 2025, published at NeurIPS 2025) borrow the Zettelkasten note-taking method: each memory is an atomic note that the system links to related notes and re-indexes as new memories arrive, so the memory structure evolves instead of sitting in a flat store. It’s a bet that the organisation of memory should be dynamic: the same instinct behind the schema-evolution problem, which is one of the genuinely unsolved problems in the area, and what draws me to it.

The temporal-graph case is the easiest to motivate. Take a media-analytics agent tracking a developing story: the signature question is “how did this narrative evolve over six weeks, and who introduced each turn?” That’s hard to answer with semantic memory alone, because the answer isn’t a set of recalled facts, it’s a timeline of who-said-what-when with each claim stamped and linked. A temporal graph stores exactly that; a vector store flattens it into a pile of similar-looking sentences with no order. The abstraction you pick is the set of questions you can answer.

Where it shines

Two situations where memory earns a dedicated system rather than a longer prompt.

The first is long-running relationships: an assistant a user returns to across days or weeks, a sales agent tracking a deal, a support bot that should know the customer’s history. Here the value is continuity: not re-asking, not contradicting, building on what came before. The opening story is this case failing, and it’s the one buyers feel most directly because it reads as the agent being dumb even when the underlying model is excellent.

The second is questions about change over time: anything where when a fact was true matters as much as the fact. Compliance histories, evolving narratives, account states that supersede each other, “what did we know and when.” This is the temporal-graph sweet spot, and it’s common in finance, journalism, and any regulated domain where an out-of-date fact stated confidently is a real liability, not a cosmetic miss.

Where it breaks

Memory is harder than retrieval, and the failure modes are quieter.

Stale and contradictory memory. The single worst failure: the agent confidently recalls a fact that was superseded, like the enterprise-tier limits after a downgrade. Semantic memory is especially prone to this, because two contradictory facts (”on the enterprise plan”, “on the pro plan”) embed to nearly the same vector, and a plain vector store has no native notion of one replacing the other. The vendors know it: Mem0’s paper describes an LLM pass that decides whether each new fact adds, updates, or deletes an existing memory, but that pass is another extraction call that can get the call wrong. If your domain has facts that expire, an abstraction without a time dimension will quietly serve the dead one.

Extraction errors compound. Every memory system that extracts facts from conversation can extract the wrong fact (mis-attribute a preference, capture a hypothetical as a commitment), and once a wrong memory is written, it gets recalled and reinforced. Unlike a bad retrieval, which is transient, a bad memory persists and poisons future answers until something deletes it.

Unbounded growth and recall noise. Memory accumulates, and a store that never forgets eventually recalls ten marginally-relevant facts for every useful one, drowning the signal and inflating the context bill. Deciding what not to remember (and what to forget) is an unsolved design problem that most teams discover only after the store is already bloated.

No standard for evaluation. Until 2024 there wasn’t a widely shared yardstick for any of this. LoCoMo (Adyasha Maharana and co-authors, [arXiv 2402.17753](https://arxiv.org/abs/2402.17753), ACL 2024) was the first purpose-built one to get wide pickup, and LongMemEval (Di Wu and co-authors, [arXiv 2410.10813](https://arxiv.org/abs/2410.10813), October 2024, ICLR 2025) put the harder numbers on it: 500 questions testing information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention across long histories, on which commercial chat assistants and long-context models showed a 30% accuracy drop.

The measurement problem has since moved rather than gone away. MemDelta (Kuan Wang, [arXiv 2606.29914](https://arxiv.org/abs/2606.29914), June 2026, preprint) re-ran memory systems against RAG and full-context baselines on LongMemEval, changing one component at a time. Swapping only the embedding model shifted accuracy by about 6 points, enough to flip the winner, and on some question types a managed memory layer matched plain retrieval at roughly fifty times the cost. The recommendation that follows, fix the embedding model across comparisons and report write-path cost before crediting the architecture, is the right one. If you’re not measuring memory against something like these, you’re shipping on vibes.

When not to use it

If your agent is stateless by nature (every query is self-contained, no continuity across turns matters, the corpus answers everything), then a memory system is complexity you don’t need, and you should stay with the retrieval stack from the earlier articles. A single-shot Q&A bot over a documentation set has no use for episodic memory; bolting one on just adds a write path to maintain and a new way to be wrong. Memory earns its keep when the interaction carries information the corpus doesn’t: when “what did this user tell me last Tuesday” is a question your agent actually needs to answer.

Cost / latency / setup effort:

Combinations and hybrids

The clean split in practice: use a vector store for “what does the corpus say” (RAG, Articles 2–3), and a memory system for “what do I know about this user / this session / this developing situation.” Most production agents need both, and the good ones keep them separate: corpus knowledge and interaction memory have different lifecycles, different freshness needs, and different failure modes, and fusing them into one store muddies all three. Inside memory, the strong hybrids pair episodic working memory (for the coherent gist of the conversation) with a temporal graph (for the facts that must stay correct as they change). That is, not coincidentally, the same vector-plus-graph pattern Article 4 landed on for retrieval. And the agent deciding what to commit to memory is itself a retrieval-as-decision problem, straight out of the agentic-RAG article. The techniques in this series are less a menu of alternatives than a set of parts that keep recombining.

Production checklist

• Give facts a time dimension if anything in your domain expires: valid-from / valid-to, or you will serve superseded facts with confidence.

• Log every memory write with its source turn, so a wrong memory can be traced to the exchange that created it and deleted.

• Set an eviction / forgetting policy up front (decide what ages out and how) rather than discovering the bloat after the store is huge and the recall is noisy.

• Separate corpus retrieval from interaction memory; don’t let one store do both jobs.

• Measure recall quality against a memory benchmark (LongMemEval- or LoCoMo-style), not just anecdotes: track multi-session and temporal questions specifically, and hold the embedding model fixed when you compare systems.

• Alert on contradiction: when a new memory conflicts with a stored one, surface it rather than silently keeping both.

• Cap and monitor memory-write cost; every extraction is a model call, and at scale the write path can cost more than the reads.

The take

Agent memory isn’t one technique: it’s three abstractions wearing vendor logos, and the one you pick decides which questions your agent can answer. Vector memory for “remember what they told me,” episodic memory for “keep a coherent sense of the whole relationship,” temporal graphs for “track what’s true and when it changed.” Most real agents need a blend, all of them need a forgetting policy and a time dimension, and none of them should be shipped without measuring recall against something harder than your own optimism. Pick the abstraction that matches the questions you actually get asked, and budget for the write path, because remembering is more expensive than retrieving.

I take on a small number of advisory engagements each year for teams hitting exactly these problems. Reach out if that’s you.

What to read next

This series:

• Article 1 — Your agent’s problem isn’t the model, it’s the knowledge layer

• Article 2 — Naive RAG: the baseline you’ll always benchmark against

• Article 3 — Advanced RAG: what you actually run in production

• Article 4 — GraphRAG and friends: when entities and relationships beat similarity

• Article 5 — Agentic RAG: when retrieval becomes a decision, not a pipeline

• Article 6 — Vector vs graph vs episodic: a tour of agent memory systems

• Article 7 — Context engineering: the discipline that replaces prompt engineering (coming soon)

External:

• Packer et al. (October 2023), MemGPT: Towards LLMs as Operating Systems. https://arxiv.org/abs/2310.08560. The OS-for-context idea behind Letta and self-editing memory.

• Rasmussen et al. (January 2025), Zep: A Temporal Knowledge Graph Architecture for Agent Memory. https://arxiv.org/abs/2501.13956. Memory as a temporal graph; the open-source engine is Graphiti.

• Xu et al. (February 2025), A-MEM: Agentic Memory for LLM Agents. https://arxiv.org/abs/2502.12110. Zettelkasten-style atomic memory that re-links as it grows.

• Wu et al. (October 2024, ICLR 2025), LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. https://arxiv.org/abs/2410.10813. The benchmark to measure any of this against.

• Wang (June 2026, preprint), MemDelta: Controlled Baselines and Hidden Confounds in Agent Memory Evaluation. https://arxiv.org/abs/2606.29914. Why memory-vs-RAG comparisons flip when you change one component, and why write-path cost belongs in the report.

The question nobody benchmarks honestly

Every team building a retrieval system hits the same fork, and almost nobody has numbers to settle it. Do you run plain vector search? Add reranking and hybrid search? Build a knowledge graph? Wrap the whole thing in an agent that decides what to fetch? Or skip retrieval and stuff the documents into a long-context window, now that the windows are big enough to hold them?

The honest answer is “it depends,” and the thing it depends on is almost never the cleverness of the technique. It is the shape of your corpus and the shape of the questions your users actually ask. I wanted that answer with numbers attached, so, together with Sasa Mirkovic, I built a system where the same question is answered five different ways over one real corpus, and we measured what each approach buys and what it costs: accuracy, citation quality, hallucination rate, temporal correctness, latency, and dollars per query, side by side.

If you have been following this series, you have met these architectures one at a time: naive RAG in article 2, the production-grade flat stack in article 3, GraphRAG in article 4, and agentic retrieval in article 5 (the fifth arm, long-context stuffing, is the one everyone argues about without needing an article). This is the post they were building toward, where the arguments stop and the measurement starts. The project is open source, linked at the bottom, and there is a screen-capture walkthrough of the running system further down.

The frame the whole thing rests on, and the through-line of everything I write: the knowledge layer, how an agent stores, updates, retrieves, and assembles what it knows, is the part that decides whether it is reliable in production. This is five architectures for that layer, measured on which parts actually pay.

One corpus, built deep

“One real corpus” should mean something, so here are the actual numbers. The corpus is Serbian statutory law from the official state legal-information system (PIS); the texts are public and excluded from copyright by statute, so there is no employer or client data anywhere near this. It holds 33 seed statutes plus a six-act shared backbone (the Constitution and the codes everything else leans on), spanning ten everyday-law clusters: employment and contributions, family and inheritance, real estate and construction, consumer law, traffic, personal tax and freelancing, data and digital, company life, civil disputes and enforcement, and health and welfare. From those laws the pipeline then walks the delegation graph and fetches every reachable implementing bylaw (about 475 materialize with article structure; roughly 550 more fetch as HTML but carry no article structure and stay graph-only stubs) plus 21 Constitutional-Court decisions. That is 536 acts with full text and 25,675 retrievable units: one chunk per article per version slice, including 3,257 dated version slices, around 910 extracted HTML-table annexes, and around 167 vision-OCR’d image annexes.

Every arm queries the same two stores:

• A vector plus full-text index (LanceDB, gemini-embedding-2 embeddings, BM25 for the lexical leg), so a query can land on the exact text in force on a given date.

• A typed temporal knowledge graph in plain SQLite: eleven edge types, including article-to-article cross-references, term-aware definition links, the law-to-bylaw delegation edge, and a constitutional-validity edge that models a court strike as a first-class temporal death.

Two invariants run through everything. First, no claim without provenance: every retrievable unit carries its law, article, version, and an ELI URL, and an answer with no citations is only legal when it abstains. Second, correctness is time-indexed: statutes amend, and a confidently cited repealed article is worse than no answer, so every unit carries validity dates.

Hold onto one set of numbers from that graph: the article-to-article cross-reference web has a mean out-degree of 0.57, 81% of article nodes have no outgoing reference at all, and only 264 of its edges (2.2%) cross from one law to another. That is most of the story of Finding 2.

The setup: a ladder, not a demo

The five arms are not five random systems. They are a ladder, where each rung adds exactly one class of capability over the one below it:

• Context-Stuffing is the “why retrieve at all” arm, run as an oracle: it is handed every act the gold answer cites, a priori, and stuffs them whole into the window. That makes it a ceiling, not a deployable system (knowing which acts to read is precisely the problem retrieval exists to solve), so it is reported but kept out of the ranking. Think of it as an open-book exam where someone else has already found the right pages.

• Naive RAG is the floor: embed the query, take the top-8 vector hits, generate. Deliberately dumb.

• Advanced RAG is the production flat stack: hybrid search (dense vectors plus BM25, fused with Reciprocal Rank Fusion), a temporal filter that drops articles not in force on the asked-about date, a listwise LLM reranker, and a diversity cap so one chatty bylaw cannot crowd out the governing article.

• Advanced + Graph takes advanced’s exact output as its seeds, then adds typed-graph traversal on top: cross-reference and definition hops, a delegation-gated law-to-bylaw hop, and the constitutional-validity path. It is a strict superset of advanced, pinned by a unit test: turn the structure off and it reproduces advanced exactly, so any difference between the two is attributable to structure and nothing else.

• Agentic is a bounded ReAct loop (at most 10 steps) over seven typed tools that cover everything the other arms can reach: search, fetch an article, expand references, pull delegated bylaws, check constitutionality, read a law’s validity timeline.

Everything else is held constant: same corpus, same chunking, same generator (Gemini 3.6 Flash), same prompts, temperature zero. Retrieval strategy is the only treatment. Grading is a cross-family judge, Claude Opus 5 grading Gemini’s answers against gold, precisely because the first objection to any LLM-graded benchmark is “the model graded itself.” A second-family juror grades nearly every item too (293 of the 295 gradable pairs), and the agreement between them is reported.

Three honesty notes before the table, because they bound everything in it. This is a single-seed run, so every number is a point estimate on 60 items. The judge-juror agreement came out at α = 0.764, below the 0.80 bar I would want; a second juror pass made the disagreement diagnosable (81% of it sits on the correct-versus-partial boundary, so ordinal α reads 0.842, and the ranking with every engineered-versus-naive p-value survives re-derivation from the second judge’s labels), which bounds the caveat without dissolving it: margins of two or three items are still not safe to read; margins of sixteen-plus are. And the generator is a floating model alias with a run-to-run flip rate of about 6% of items (quantified on its predecessor alias), so any difference of four items or fewer sits inside the drift envelope. I will point at these caveats exactly where they bite.

The scoreboard

Sixty situational, first-person client questions (”I’m buying a flat and the contract goes to the notary...”), three difficulty tiers, five arms, 300 graded answers:

Accuracy counts only answers the judge marked fully correct; correctness gives half credit for partially correct ones. Provenance F1 is deterministic citation precision and recall against the gold articles, no judge involved. The intervals are Wilson 95% on the accuracy proportion, item-sampling only. Every engineered arm’s interval overlaps every other’s; only naive separates cleanly.

Deployable ranking: Agentic 0.87 > Advanced + Graph 0.77 ≈ Advanced 0.73 > Naive 0.48. Five things in that table are worth stopping on.

Finding 1: the retrieval ladder pays, and this one has receipts

Every engineered arm beats the naive floor decisively: paired exact McNemar p = 0.0003 or better on all three comparisons, with naive losing 16, 18, and 24 of the discordant items respectively. Naive’s confidence interval is the only one that does not overlap the rest of the board.

The cheapest step is still the biggest bargain: naive to advanced is +0.25 accuracy for about a cent and a half more per query. Hybrid search, a reranker, a temporal filter, and a diversity cap. No new model, no graph, no agent. If your RAG system is underperforming and you have not done these four things, that is almost certainly your problem, not the model. This is the least glamorous finding, and given the caveats above it is also the only one with margins wide enough that no plausible amount of judge disagreement or generator drift can touch it.

Finding 2: the typed graph added nothing, and the scope of that sentence is the finding

In the GraphRAG article I said a graph earns its keep when relationships are first-class in the questions your users ask. The corollary under test here is whether the corpus has the connection density to traverse. Here is what happened when I measured mine: Advanced + Graph came out at 0.77 against advanced’s 0.73, which sounds like a win until you run the paired test. Two items in one direction, four in the other, p = 0.69. In the previous campaign the graph arm sat nominally below advanced; scoring corrections landed in between, so the two campaigns are not strict replicates, but an ordering that will not stay put is exactly what a null effect looks like. This graph layer, on this corpus, neither beats nor loses to strong flat retrieval.

The instrumentation is what makes that sentence worth publishing. The superset design rules out “you built the graph arm wrong”: structure-off provably reproduces advanced, so the null delta is attributable to the structure itself. And the decomposition locates why it is null, stage by stage:

At the construction stage, the cross-reference web barely exists. Serbian statutes rarely cite each other by article number: the code of obligations has roughly 1,100 articles and only 12 of them carry a numbered cross-reference at all, none pointing outside the code. Corpus-wide, 81% of article nodes are isolated and 2.2% of reference edges cross laws. You cannot manufacture those edges without fabricating law. At the retrieval stage, the one dense structure the corpus does have, the law-to-bylaw delegation spine, hits a different wall: a single law can delegate to as many as 79 materialized bylaws, and the graph hop picks which ones to search in arbitrary stored order rather than by relevance. Ranking that fan-out by the query would fix it, but ranking by the query is retrieval, the thing the flat backbone already does over the whole corpus. That is the sharpest form of the result: on this corpus, the typed edges add no retrieval power that good flat retrieval doesn’t already have.

Two scope notes, both load-bearing. This is a claim about this graph implementation on this corpus, not about graph structure in general; the fan-out problem is a fixable engineering property, and fixing it is deliberately left to a held-out test so the negative result stays clean. And on the three constitutional-validity questions, where the validity edges genuinely exist, the graph arm was the only fixed-pipeline arm to go 3 for 3 (the agentic loop managed it too). Three questions proves nothing, but it points where structure’s remaining home turf is: validity- and definition-aware traversal, not cross-reference multi-hop.

The durable lesson survives all the caveats: whether a graph helps is a property of your corpus topology, not of the technique. Count your cross-references before you build one.

Finding 3: the agent is the headline, for the quality profile rather than the accuracy lead

Agentic tops the accuracy column at 0.87, a 13-point lead over advanced. I am deliberately not headlining that number, because it does not survive multiple-comparison correction: p = 0.0386 uncorrected, against a Bonferroni threshold of 0.0083 across the six pairwise tests, and against the graph arm it is nowhere near significance. On accuracy, the honest verdict is “promising, unproven at this sample size.”

What the data does support is the quality profile, and it is consistent across every axis I measured. Best deployable correctness (0.93). Best deployable provenance F1 (0.61). Misgrounding and hallucination rates as low as any arm’s on the board, the oracle included (0.07 and 0.07, versus naive’s 0.22 and 0.20). Zero over-abstentions: it never refused an answerable question, while naive refused seven. It is the only deployable arm that holds up on the hard tier (16 of 20, level with the oracle, though the oracle’s sixteenth hard item rests on a re-run cell the report discloses). And the price of all that is about 2.3× advanced’s cost ($0.068 vs $0.030 per query) and under twice the median latency, 26 seconds against 15.

That is a much better trade than the agentic folklore suggests. Retrieval-as-a-decision is a trustworthiness purchase: for a legal, medical, or financial assistant where the expensive failure is a confident wrong answer with a plausible citation, the safety delta is worth double the spend. For an internal FAQ bot it is not. That is the actual decision, and now it has a price tag.

Finding 4: the ceiling closed, and what that actually means

The oracle, handed every act the gold answer cites, scores 0.92. Two readings of that number are wrong and one is useful.

Wrong reading one: “long context killed RAG.” The oracle’s input assumption is the problem statement: someone has to know which acts the answer rests on, and that act selection is what retrieval does. The previous campaign measured its worth directly, as a disclosed same-day correction: an oracle stuffing only the single primary act scored 0.42, and handing over the full gold act set moved it to 0.93. Knowing which documents to read was worth more than any retrieval-stage refinement measured in this project. Wrong reading two: “so use the oracle’s setup anyway.” It costs $0.197 per query, 12.8× naive and 2.9× agentic, and it is temporally blind by construction: it stuffs current consolidated text, so on questions where the in-force answer is a superseded version it structurally cannot cite the right law.

The useful reading is narrower. The gap between the best deployable arm and the ceiling has closed to within what this benchmark can measure: agentic versus oracle is three items against six, p = 0.51, and that is a statement about statistical power, not proof of parity. Still, a bounded agent over typed tools, paying a third of the oracle’s price and keeping temporal awareness, is at least keeping pace with perfect act selection at this sample size. If that holds up at more seeds, it is the best argument I know for spending your budget on the knowledge layer rather than on a bigger window.

Finding 5: temporal correctness turned out to be a property of the knowledge base, not the model

The most transferable result is also the one I have to report most carefully. On questions dated into the past, where the correct answer is a superseded version of an article, what decides the outcome is whether the arm’s retrieval layer can reach and prefer the right version slice. Naive and the oracle index only current text, so where the in-force text at the asked date is a superseded slice, they cannot cite it, structurally. The three arms with a validity filter cannot cite an out-of-force one. Neither outcome could have come out otherwise, which is why I am calling this a mechanism demonstration, not a measurement: my dated tier held three items, and after two of them were passed by all five arms, exactly one item did the discriminating.

Full disclosure, because it belongs next to this number: that one discriminating item was itself corrected after arm behaviour had been analysed, an exception to the benchmark’s blind-authoring rule that the report states rather than hides, and it is a second reason no temporal effect is claimed here. There is also a methodological trap inside this that anyone building a temporal benchmark will hit: a per-citation validity metric ends up measuring the arm’s retrieval configuration, not its temporal reasoning, and an arm can pass it by citing the wrong law entirely. In my view this methodological finding will outlive the scoreboard. If your corpus has versions, effective dates, or amendment history, temporal correctness has to be built into storage and retrieval as validity intervals and an as-of filter. No prompt and no bigger model fixes it downstream, and measuring it takes a tier of materially amended articles I have not built yet.

Watch it work

Numbers are one kind of evidence; the system showing its work is another. I recorded a short screen-capture walkthrough of the running app, and it is embedded here on the canonical version of this post:

What it walks through, in order: a real question asked with an as-of date and the arm selector; the answer with citation chips and validity badges (green “in force since”, red “ceased to be valid”); the Runs drill-down for that exact answer, with per-stage latency, cost, and, for the agentic arm, the full tool trace of what the agent searched, fetched, and expanded before answering; the Benchmark tab, where the whole five-arm matrix in this post can be re-run with any capability toggled off; and the Corpus topology panel, the sparsity numbers behind Finding 2. Every number on those screens comes from the same telemetry rows the published report is computed from. There is no demo mode.

What this actually costs to run

Generating the full 300-answer campaign cost $20.91 in API calls across the five arms; the cross-family grading runs on top of that.

What this is, and what it isn’t

This is a build-in-public launch of a real open-source project, built in cooperation with Sasa Mirkovic (https://github.com/SasaMirkovic), who co-designed the ingestion pipeline and the knowledge base every arm retrieves over and advised on the retrieval architecture and the evaluation design. The evidential position is stated rather than implied. The three caveats from the top (single seed, below-threshold judge agreement, generator drift) ride every number above; the 16-to-24-item margins behind Finding 1 survive them, the small margins do not. The gold answers are source-authored and mechanically checked for grounding in the statute text they cite, but not yet reviewed by practising lawyers. No external system (LightRAG, Microsoft GraphRAG) has been run on this corpus yet, so the results say what each architecture buys relative to the others here, not how any of them compares to a published system. The hardened tier, with three-plus seeds, expert-validated gold, and external baselines, is the next phase, and it feeds the benchmark paper this project exists to support. The full report, including the sections titled “what to distrust” and “what this evaluation does not establish,” ships in the repo.

For an enterprise the translation is direct. Swap Serbian statutes for your regulated, amendment-heavy corpus (policies, contracts, compliance manuals, anything with versions and effective dates) and the decision rules transfer as-is: do the boring retrieval engineering first, count your cross-references before you build a graph, budget the agentic loop for the queries where being confidently wrong actually costs you, and put temporal validity in the knowledge base, because nothing downstream can recover it.

Run it yourself

The repo is public at https://github.com/RatkoNikolic/srb-legal-rag and small (~6 MB): the five instrumented arms, the eval harness, the 60-question golden set (CC BY 4.0, reusable in published work with attribution), 224 tests, the web UI, and six design docs that explain every decision with the literature behind it. The code is Apache-2.0. Setup instructions are in the README, and building your own knowledge base is three commands: a rate-limited crawl of the official source (about 7 minutes), a local graph build (seconds), and embedding (about $1.50 of API calls). Re-running the full benchmark needs a Gemini key plus an Anthropic key for the cross-family judge; generation alone ran me about $21, with grading on top.

One thing to know before you build, because it is unusual and deliberate: the repository ships the machinery, not the corpus. No statute text is distributed, and your rebuilt knowledge base will not byte-match the one these numbers were measured on. That is a measured fact, not a guess: a fresh rebuild produced 24,652 units against the published 25,675, with hundreds of article chunks existing only on one side. Serbian law amends continuously (the source moves under you), the annex layer needs its own paid extraction step whose vision-OCR portion is model output that varies even on identical inputs, and borderline documents flip between parseable and not. Comparing arms within your own build stays valid; comparing your numbers to mine does not. The exact knowledge base behind the published campaign, the full chunk set, the typed graph, and the vector index, is available on request (email in the repo README): it is too large to ship in the repo, and given the probabilistic construction and the drift in the source materials, handing over the measured artifact is the only honest way to let someone run the same comparison these numbers came from.

I take on a small number of advisory engagements each year for teams building retrieval systems that have to hold up in production. Reach out if that’s you.

What to read next

This series:

• Article 1 — Your agent’s problem isn’t the model, it’s the knowledge layer

• Article 2 — Naive RAG: the baseline you’ll always benchmark against

• Article 3 — Advanced RAG: what you actually run in production

• Article 4 — GraphRAG and friends: when entities and relationships beat similarity

• Article 5 — Agentic RAG: when retrieval becomes a decision, not a pipeline

• Article 6 — Vector vs graph vs episodic: a tour of agent memory systems

• Article 7 — Context engineering: the discipline that replaces prompt engineering (coming soon)

External:

• Xiang et al. (2025), When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation. [https://arxiv.org/abs/2506.05690](https://arxiv.org/abs/2506.05690). Root-causes graph underperformance at the construction stage, the same decomposition Finding 2 runs.

• Xiao et al. (2025), GraphRAG-Bench: Challenging Domain-Specific Reasoning for Evaluating Graph Retrieval-Augmented Generation. [https://arxiv.org/abs/2506.02404](https://arxiv.org/abs/2506.02404). The construction, retrieval, generation stage split and the structural stats (degree, isolated share) this benchmark reports.

• Han et al. (2025), RAG vs. GraphRAG: A Systematic Evaluation and Key Insights. [https://arxiv.org/abs/2502.11371](https://arxiv.org/abs/2502.11371). The flat-versus-graph comparison precedent: measured evidence that graph RAG does not dominate flat RAG.

• Fan et al. (2026), Can LLMs Time Travel? Enhancing Temporal Consistency in Legal Agentic Search through Reinforcement Learning. [https://arxiv.org/abs/2605.25920](https://arxiv.org/abs/2605.25920). Point-in-time legal retrieval as a first-class problem, the axis this benchmark could only demonstrate, not measure.

• Yang et al. (EMNLP 2018), HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. [https://arxiv.org/abs/1809.09600](https://arxiv.org/abs/1809.09600). The origin of gold-context-as-ceiling: its gold-supporting-fact setting is “an upper bound on performance”, which is exactly how the oracle arm is used here.

• srb-legal-rag: [https://github.com/RatkoNikolic/srb-legal-rag](https://github.com/RatkoNikolic/srb-legal-rag). The code, the question sets, and the full campaign report behind every number in this post.

A team ships a support assistant over a good advanced-RAG stack: hybrid search, a reranker, metadata filters, the works. It answers single-fact questions well. Then a user asks: “I’m moving from the Berlin office to the Madrid one mid-contract: what changes for my health cover, my notice period, and my equity vesting?” The pipeline does what it always does: embeds the question, pulls the top chunks, stuffs them in. It comes back with a confident paragraph about health cover and silently drops the other two. Three questions went in; one came out, and nobody flagged the gap.

The pipeline isn’t broken. It did exactly what a pipeline does: one retrieval, one shot, no second look. The failure is that nothing in the system ever decided this question needed three separate lookups, or noticed that the retrieved context only covered one of them. That deciding is the thing agentic RAG adds. The knowledge layer (the part of an agent system that decides reliability in production, and an engineering discipline in its own right) stops being a fixed conveyor belt and becomes a loop the model drives: retrieve, look at what came back, decide whether it’s enough, and go again if it isn’t.

How agentic RAG works

The spine is one idea: make retrieval a tool the model calls inside a reasoning loop, instead of a step that always runs once before the model sees anything.

The pattern traces back to ReAct (Shunyu Yao and co-authors, ReAct: Synergizing Reasoning and Acting in Language Models, [arXiv 2210.03629](https://arxiv.org/abs/2210.03629), ICLR 2023), which interleaved reasoning steps with tool actions so a model could think, act, observe the result, and think again. Retrieval is just one of those actions. Two later papers sharpen it for RAG specifically. Self-RAG (Akari Asai and co-authors, [arXiv 2310.11511](https://arxiv.org/abs/2310.11511), ICLR 2024) trains the model to emit “reflection tokens” that decide whether to retrieve at all and then critique whether the retrieved passage actually supports the answer: retrieval on demand, with a built-in quality check. Adaptive-RAG (Soyeong Jeong and co-authors, [arXiv 2403.14403](https://arxiv.org/abs/2403.14403), NAACL 2024) adds a complexity classifier up front: trivial questions get answered with no retrieval, single-hop questions get one pass, and only genuinely multi-hop questions pay for the full iterative loop. The recurring move in all three is the same: spend retrieval effort in proportion to how hard the question actually is.

In production, this is usually built as a small state machine rather than a free-for-all. LangGraph, the graph-based agent framework from the LangChain team (1.0 in October 2025, now well into its 1.x line), is the de-facto way teams wire these loops as explicit nodes and edges with a retry budget, precisely so the “loop until satisfied” can’t run away. The broader industry has been moving the same direction: the Model Context Protocol (open-sourced by Anthropic in November 2024, adopted by OpenAI and Google through 2025, and donated to the Linux Foundation’s Agentic AI Foundation in December 2025) standardised how an agent reaches tools and data sources, which is exactly the plumbing an agentic retriever needs to query a vector store, a graph, a SQL database, and a third-party API through one interface.

In practice

Take the relocation question from the top. A static pipeline runs one retrieval. An agentic one does something closer to this:

User: moving Berlin → Madrid mid-contract. Health cover, notice period, equity vesting?

Agent (plan):    three sub-questions, different policy areas. Decompose.
  → retrieve("health insurance transfer between EU offices")     ✓ found
  → retrieve("notice period change on internal relocation")      ✓ found
  → retrieve("equity vesting on country transfer")               ✗ weak match
Agent (judge):   vesting result is thin. Re-query with a narrower term.
  → retrieve("RSU vesting schedule employee relocation")         ✓ found
Agent (judge):   three areas covered. Compose.

Four retrievals, two planning steps, one re-query when a result came back weak, versus the single shot a pipeline would have taken. That’s the whole trade in one trace. The agent caught the thin vesting result and went back for it, which is the failure the static pipeline shipped silently. It also cost roughly three-to-five times the model calls of a one-shot answer, because every plan, judge, and re-query is another round-trip to the model. Agentic RAG buys reliability on hard questions and pays for it in latency and tokens. That sentence is the whole article, and the discipline is knowing when the trade is worth it.

Where it shines

Two situations make the extra cost pay for itself.

The first is multi-hop and multi-part questions: anything that needs more than one fact, retrieved separately and combined. The relocation question is one; “which of our vendors fail the new compliance rule, and what’s the remediation for each” is another. A single retrieval can’t serve these because the sub-answers live in different documents and a one-shot top-*k* will over-weight whichever sub-question embeds strongest. The agent’s ability to decompose, retrieve per part, and notice a gap is the entire value.

The second is mixed-source corpora where the right source depends on the question. When some answers live in unstructured docs, some in a SQL table, some behind an API, and some in a knowledge graph, a static pipeline that always hits the vector store is wrong most of the time. An agent can route: read the question, pick the tool, and fall back to another source when the first comes up empty. This is common in enterprise settings where “the knowledge” is genuinely spread across systems, and it’s where retrieval-as-a-decision earns its keep over retrieval-as-a-step. The pattern has gone product-shaped, too: Microsoft’s Azure AI Search agentic retrieval (public preview in 2025; the core moved to GA in the April 2026 API, with the LLM query-planning step itself still in preview) ships exactly this loop as a managed service, with an LLM planning subqueries, running them in parallel, and merging the results. Microsoft has since wrapped the same engine into Foundry IQ (announced November 2025, in preview), pitched, in Microsoft’s own words, as a managed “knowledge layer” for agents.

Where it breaks

Be honest about the failure modes, because they’re nastier than a pipeline’s.

• The cost and latency multiply. Three-to-five times the model calls is the rough rule, and it lands on every hard question, not just occasionally. A question that took 2 seconds and one model call now takes 6–10 seconds and five, and the bill scales with it. On a high-traffic assistant that delta is not a rounding error: it’s the difference between a viable unit economic and an unviable one.

• Loops that don’t terminate. The signature agentic failure: the agent judges the results insufficient, re-queries, judges again, and never converges, burning budget and latency on a question it’s never going to answer. Every production loop needs a hard iteration cap and a “give up gracefully” branch, or it will eventually find the input that spins forever.

• Confident wrong routing. When the agent decides not to retrieve (Self-RAG’s whole premise is that sometimes it shouldn’t), it can decide wrong and answer a factual question from parametric memory, hallucinating where a retrieval would have saved it. The decision to skip retrieval is itself a new failure surface that a static pipeline simply doesn’t have.

• Harder to debug. A pipeline has one retrieval trace. An agent has a branching tree of plans, tool calls, and judgements, and when the answer is wrong you have to reconstruct which decision went wrong. The observability burden is real, and teams underestimate it.

When not to use it

If your questions are overwhelmingly single-fact lookups over one corpus (the FAQ-shaped workload from the naive-RAG article), agentic RAG is pure overhead. You pay 3–5× the cost and the added latency to wrap a decision around a question that only ever needed one retrieval, and a good advanced-RAG stack will beat it on both cost and speed. The loop earns its place only when a meaningful share of real questions are multi-hop, ambiguous, or span sources. Reach for it when you can point to the specific questions a pipeline keeps getting wrong; don’t reach for it because “agentic” is the word of the season.

Cost / latency / setup effort:

Combinations and hybrids

Agentic RAG isn’t a replacement for the earlier techniques: it’s a controller that sits on top of them. The tools the agent calls are naive RAG, advanced RAG, and GraphRAG; the agent just decides which one to use and whether to go again. The most effective real systems pair an agentic loop with a complexity gate in the Adaptive-RAG spirit: cheap questions bypass the loop entirely and get a one-shot answer, and only the hard ones pay for the full plan-retrieve-judge cycle. That gate is what keeps the cost honest: without it, you pay the 3–5× tax on every trivial lookup. Pair it the other way and agentic RAG also feeds the next article: an agent that decides what to remember across turns is the bridge from retrieval into memory.

Production checklist

• Cap the loop. A hard maximum on retrieval iterations plus a graceful “I couldn’t find enough” branch. Non-negotiable, or you will ship the infinite loop.

• Budget per query. Track and cap model calls and tokens per answer; alert when a single question blows past the expected envelope.

• Log the full decision tree (every plan, tool call, observation, and judgement), not just the final retrieval. This is the only way to debug a wrong answer in a branching system.

• Instrument the retrieve/skip decision separately; measure how often the agent chooses not to retrieve and how often that choice was wrong.

• Gate by complexity so trivial questions never enter the loop. Measure what fraction of traffic actually needs it.

• Watch p99 latency and tail cost, not the averages; the agentic tail is where both the budget and the user patience go.

• Evaluate end-to-end, not per-retrieval: a loop can make three good retrievals and still compose a bad answer, so score the final result against a golden set.

The take

Agentic RAG turns retrieval from a fixed step into a decision the model makes, and that’s genuinely more powerful on the questions that need it: multi-hop, ambiguous, multi-source. But power you pay for on every query is a liability, not a feature. Wrap the loop in a complexity gate, cap it hard, instrument every decision, and spend the 3–5× only where the reliability gain is real. The teams that win with agentic RAG are the ones who know exactly which questions justify the loop, and route everything else around it.

I take on a small number of advisory engagements each year for teams hitting exactly these problems. Reach out if that’s you.

What to read next

This series:

• Article 1 — Your agent’s problem isn’t the model, it’s the knowledge layer

• Article 2 — Naive RAG: the baseline you’ll always benchmark against

• Article 3 — Advanced RAG: what you actually run in production

• Article 4 — GraphRAG and friends: when entities and relationships beat similarity

• Article 5 — Agentic RAG: when retrieval becomes a decision, not a pipeline

• Article 6 — Vector vs graph vs episodic: a tour of agent memory systems

• Article 7 — Context engineering: the discipline that replaces prompt engineering (coming soon)

External:

• Yao et al. (ICLR 2023), ReAct: Synergizing Reasoning and Acting in Language Models. https://arxiv.org/abs/2210.03629. The reasoning-plus-acting loop everything here is built on.

• Asai et al. (ICLR 2024), Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. https://arxiv.org/abs/2310.11511. Retrieve-on-demand with a built-in quality check.

• Jeong et al. (NAACL 2024), Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity. https://arxiv.org/abs/2403.14403. Spend retrieval effort in proportion to question difficulty.

• Singh et al. (2025), Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG. https://arxiv.org/abs/2501.09136. The field map: where the ReAct-descended patterns sit in the broader agentic-RAG taxonomy.

• LangGraph (LangChain): https://github.com/langchain-ai/langgraph. The production-standard way to wire these loops as explicit, bounded state machines (docs now at docs.langchain.com).

Here is a question that breaks every RAG pipeline in this series so far: “which of our suppliers are connected to the legal entities named in this dispute?” The answer isn’t sitting in a chunk anywhere. It’s a path — supplier to parent company to subsidiary to the named party — assembled from facts that live in four different documents. Naive RAG retrieves the four documents independently and has nothing to join them. Advanced RAG reranks those four documents more precisely and still has nothing to join them. The retrieval got better; the question stayed unanswerable, because similarity search returns passages, and the answer is a traversal.

That’s the line that divides this article from the previous two. Vector search hands you the ten business cards that look most like your query. A graph hands you the org chart. When the question is “what does this document say,” the business cards are what you want. When the question is “how are these things connected,” you need the org chart — and building, storing, and querying that org chart is a different region of the knowledge layer, the part of an agent system that decides reliability in production and is an engineering discipline in its own right. This is where structure stops being optional.

How GraphRAG works

The spine: extract entities and relationships from the corpus into a graph, optionally summarize clusters of that graph ahead of time, and at query time either traverse the graph directly or retrieve over those summaries.

Microsoft’s GraphRAG — the paper that named the pattern, Edge and co-authors’ From Local to Global: A Graph RAG Approach to Query-Focused Summarization ([arXiv 2404.16130](https://arxiv.org/abs/2404.16130), April 2024) — does it in two passes. First, an LLM reads the corpus and extracts an entity-and-relationship graph. Then it partitions that graph into communities of closely-related entities using the Leiden algorithm, and pre-writes a summary of each community. A “global” question — “what are the recurring themes across all these incident reports?” — gets answered by map-reducing over the community summaries rather than over raw chunks, which is something flat RAG simply cannot do, because no single chunk contains the global answer.

The dominant production pattern looks slightly different from the research paper. Most teams store a property graph in Neo4j, Memgraph, or an embeddable store like Kùzu; chunks become document nodes linked to the entities they mention; and retrieval combines an LLM writing a Cypher query against the graph with ordinary vector search over the chunk embeddings. Neo4j’s neo4j-graphrag-python SDK and its Knowledge Graph Builder are the de-facto reference here, and LlamaIndex’s PropertyGraphIndex exposes the same idea framework-agnostically.

In practice

The example I’ll use is corporate ownership and supply relationships — the kind of corpus where the relationships are the point, not the prose.

Companies own stakes in other companies, supply each other, and share directors — and those links change as deals close. So the schema is genuinely typed: Company and Person nodes, OWNS edges carrying a stake percentage, SUPPLIES edges, DIRECTOR_OF edges, and a validity stamp on every edge so the graph knows which relationships held on a given date. The base vocabulary isn’t invented — it maps onto established models like the Financial Industry Business Ontology (FIBO) and legal-entity identifiers (LEI), which keeps the schema defensible rather than ad hoc.

Now the question “which suppliers does Beta Retail depend on, and which of those came under new ownership after 2024-01-01?” becomes a short traversal instead of an impossibility:

```cypher
MATCH (b:Company {name: 'Beta Retail'})-[:SUPPLIES*1..3]->(dep:Company)
MATCH (parent:Company)-[o:OWNS]->(dep)
WHERE o.since > date('2024-01-01')
RETURN dep.name, parent.name, o.since ORDER BY o.since
```

Flat RAG can’t express “depend on through the supply chain” — there’s no similarity score for a dependency path. The graph makes it a path query, and crucially it returns provenance: which company, which relationship, valid on which date. For a due-diligence or compliance answer, an uncited or out-of-date link is worse than no answer, and that’s exactly the failure a typed temporal graph is built to prevent.

Where it shines

Two situations make the cost worth it.

The first is relationship and multi-hop questions — anything of the form “how is X connected to Y,” “what depends on Z,” “trace the chain from A to B.” Supply-chain and ownership graphs, compliance and KYC (”is this counterparty linked to a sanctioned entity through any chain of ownership?”), the supplier-dependency case above. These questions are common in finance, law, and regulated industries, and they’re precisely the ones similarity search can’t touch. KAG, Lei Liang and co-authors’ framework from Ant Group ([arXiv 2409.13731](https://arxiv.org/abs/2409.13731), September 2024), leans hard into this — it adds logical-form reasoning over the graph and reported a 19.6% relative F1 gain on the multi-hop benchmark HotpotQA over a strong graph-RAG baseline (HippoRAG), which is the kind of lift you only get when the structure is doing real work.

The second is global sensemaking over a whole corpus — “what are the main themes,” “summarize the recurring risks across a thousand reports.” This is what GraphRAG’s community summaries were built for: the answer isn’t in any one document, it’s a property of the whole set, and the pre-computed summaries are how you reach it without stuffing the entire corpus into a context window.

Where it breaks

Be honest about this, because the hype skips it. A knowledge graph is expensive and brittle in three specific ways.

It’s expensive to build. Entity-and-relation extraction means an LLM call over every chunk of the corpus, and on a large corpus that indexing bill is real — easily orders of magnitude more than the few cents naive RAG cost in Article 2. Microsoft’s own answer to this is LazyGraphRAG (November 2024), which defers the community summarization until query time and cuts indexing cost down to roughly vector-RAG levels while keeping most of the global-query quality — Microsoft reports more than 700× lower cost on global queries than full GraphRAG. LightRAG (Guo and co-authors, EMNLP 2025 Findings, [arXiv 2410.05779](https://arxiv.org/abs/2410.05779)) attacks the same problem from the freshness side, with dual-level retrieval and incremental graph updates so you don’t re-extract the world every time a document changes.

It’s hard to keep fresh. A flat vector store updates by embedding the new document; a graph updates by re-extracting entities, reconciling them against what’s already there, and fixing up edges — which is a much heavier operation, and the one teams underestimate most.

The schema is the hard part, and it can’t be frozen. This is where I spend my research time, so I’ll be opinionated: the genuinely unsolved problem in enterprise knowledge graphs isn’t extraction, it’s schema evolution. You design a typed schema, and then the world hands you an entity type you didn’t anticipate — a new regulatory instrument, a new category of counterparty, a relationship that doesn’t fit your existing edge types. The agent ingesting that new document has to decide, under cost, whether to coerce the new thing into an existing type or propose extending the schema. Get that decision wrong in one direction and the graph rots into a mess of mis-typed nodes; wrong in the other and it fragments into a thousand one-off types nobody can query. Almost nobody has a principled control policy for that decision yet — it’s one of the genuinely unsolved problems in the area.

When not to use it

Most enterprises don’t need a knowledge graph — and that’s worth saying out loud before anyone spends a quarter building one. If your questions are overwhelmingly “what does this document say about X” rather than “how is X connected to Y,” a graph is expensive overkill, and a good advanced-RAG stack will beat a half-maintained graph every time. The graph earns its keep only when relationships are first-class in the questions your users actually ask, and you have the engineering capacity to keep it fresh. A stale graph is worse than no graph, because it answers a connection query with false confidence.

Cost / latency / setup effort:

Combinations and hybrids

In production, GraphRAG is almost never the whole answer — it’s the half that handles relationships, sitting next to a vector store that handles “what does this say.” The standard hybrid lets an agent route: similarity search for descriptive questions, graph traversal for relational ones, often both fused for questions that have a descriptive and a relational part. That routing decision — does this question need the graph at all — is itself a retrieval-as-decision problem, which is the subject of the next article on agentic RAG. The heuristic to remember: vectors for “what does this say”; graphs for “how are these connected.” Most real systems need a little of both, and the engineering is in knowing which question is in front of you.

Production checklist

• Version the schema explicitly and treat a schema change as a migration, not a config tweak — log when types are added and why.

• Track extraction precision/recall on a labelled sample; a graph built from sloppy extraction is confidently wrong, which is the worst failure mode.

• Stamp every node and edge with provenance and validity dates — which document, which version, in force when — so a traversal can return a citation, not just an answer.

• Instrument graph freshness: how stale is the graph relative to the corpus, and alert when re-extraction falls behind ingestion.

• Reconcile entities deliberately — decide your dedup/merge policy for “is this the same entity” up front; entity resolution drift is silent and corrosive.

• Measure how often the graph is actually needed — if most queries route to the vector side, that’s a signal you may not have needed the graph.

• Keep a coerce-vs-extend policy for new types, even a crude one, so schema growth is a decision and not an accident.

The take

GraphRAG isn’t a better RAG — it’s a different tool for a different question. When the answer is a passage, use vectors; when the answer is a path, you need a graph, and you should go in clear-eyed about the build cost, the freshness burden, and the fact that the schema is the part nobody has fully solved. Most teams don’t need one. The ones that do — law, finance, compliance, anywhere relationships are the product — get answers from it that no amount of reranking would ever surface.

New articles land here first. Subscribe for the full series. I take on a small number of advisory engagements each year for teams hitting exactly these problems. The fastest way to reach me is on LinkedIn; reach out if that’s you.

What to read next

This series:

• Article 1 — Your agent’s problem isn’t the model, it’s the knowledge layer

• Article 2 — Naive RAG: the baseline you’ll always benchmark against

• Article 3 — Advanced RAG: what you actually run in production

• Article 4 — GraphRAG and friends: when entities and relationships beat similarity

• Article 5 — Agentic RAG: when retrieval becomes a decision, not a pipeline

• Article 6 — Vector vs graph vs episodic: a tour of agent memory systems

• Article 7 — Context engineering: the discipline that replaces prompt engineering (coming soon)

External:

• Edge et al. (April 2024), From Local to Global: A Graph RAG Approach to Query-Focused Summarization. https://arxiv.org/abs/2404.16130 — the paper that named GraphRAG.

• Guo et al. (EMNLP 2025 Findings), LightRAG: Simple and Fast Retrieval-Augmented Generation. https://arxiv.org/abs/2410.05779 — graph RAG with incremental updates; the freshness answer.

• Liang et al. (September 2024), KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation. https://arxiv.org/abs/2409.13731 — logical-form reasoning over the graph for professional domains.

• Microsoft Research (November 2024), LazyGraphRAG: setting a new standard for quality and cost. https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/ — the cost answer, and the most-cited recent move in this space.

The naive pipeline from the last article gets a team about 70% of the way there, and then it stalls. The pattern is always the same: the demo answers were good, the corpus grew, the questions got more varied, and recall quietly fell off a cliff. A query that names an exact product code returns ten semantically-fuzzy chunks and misses the one document that actually mentions the code. A user phrases a question nothing like how the document was written, and the right chunk never clears the similarity threshold. The team adds more documents hoping to help, and makes it worse — more haystack, same needle.

None of this is a model problem, and a bigger embedding model won’t fix it. It’s the knowledge layer — the part of an agent system that decides reliability in production and is an engineering discipline in its own right — hitting the limits of pure top-*k* vector search. Advanced RAG is the set of moves that push past those limits. There are four worth knowing, and the discipline is in knowing which failure each one fixes and what it costs you, because none of them is free.

The four moves

Here’s the naive pipeline with the four upgrades dropped into place — the Article 2 baseline with four new boxes wired in.

Move 1 — Hybrid search (BM25 + vectors)

The failure it fixes: pure vector search is blind to exact tokens. Embeddings encode meaning, so a part number like SKU-44918, a statute reference, or a rare acronym embeds into roughly the same region as its neighbours and gets lost. Keyword search — BM25, the decades-old probabilistic ranking function that still backs Elasticsearch and OpenSearch — is exact where vectors are fuzzy. Hybrid search runs both and combines the result lists, usually with Reciprocal Rank Fusion, which scores a document by the reciprocal of its rank in each list and needs no tuning to merge them.

What it costs: a second index to build and keep in sync, and a fusion step. Operationally modest — pgvector plus a Postgres full-text index, or a single store that does both, keeps it to one system. Anthropic’s Contextual Retrieval write-up (September 2024) put a number on the payoff: contextual embeddings alone cut their top-20 retrieval failure rate by 35%, and adding contextual BM25 took the cut to 49%. Skip it when your corpus has no exact-match vocabulary — pure prose with no codes, IDs, or names — where keyword matching adds noise instead of signal.

Move 2 — Reranking

This is the highest-leverage single move, and the first one to reach for when retrieval is “almost there.”

The failure it fixes: the top-*k* chunks from a similarity search are ranked by a cheap approximation — cosine distance between independently-computed embeddings, which never actually compared the query and the chunk against each other. A reranker is a cross-encoder: it reads the query and each candidate chunk together and scores genuine relevance. You retrieve a generous longlist (say top-50 cheaply), then rerank to a precise shortlist (top-5). It’s a recruiting funnel — a cheap CV screen gets you fifty plausible candidates, and the expensive interview reads each one carefully before you pick five.

In practice: Cohere’s Rerank 4 (*Introducing Rerank 4*, December 2025 — Fast and Pro variants, a 32k context window, 100+ languages) and Voyage’s rerank-2.5 are the two production defaults. A realistic before/after on a mid-sized corpus: retrieve-then-rerank takes recall@5 from around 0.6 to 0.85, and p99 latency from roughly 200 ms to 600 ms. In the Anthropic numbers above, adding reranking on top of contextual hybrid search took the failure-rate cut from 49% all the way to 67%.

What it costs: an extra network call per query and that latency bump — the reranker has to score every candidate, so 80–150 ms is typical on top of retrieval, more if your longlist is large. Skip it when your corpus is so small that the top-*k* is already the whole relevant set, or when sub-200 ms latency is a hard product requirement you can’t spend.

Move 3 — Query rewriting

The failure it fixes: the user’s phrasing and the document’s phrasing don’t match, and one short query can’t express a multi-part question. Three sub-moves handle this. Expansion rewrites a terse query into a fuller one before embedding. Decomposition splits a compound question — “compare our refund policy for EU and US customers” — into separate retrievals that get merged. And HyDE (Hypothetical Document Embeddings, from Gao and co-authors’ Precise Zero-Shot Dense Retrieval without Relevance Labels, [arXiv 2212.10496](https://arxiv.org/abs/2212.10496), December 2022) has the model write a fake ideal answer first and embeds that instead of the question — on the theory that an answer looks more like the target document than the question does.

What it costs: an extra LLM call before retrieval even starts, which adds latency and a little money to every query, and introduces a new failure surface — a bad rewrite retrieves confidently for the wrong question. In practice this is the move to add last, after hybrid search and reranking, because its wins are real but smaller and its cost lands on every single query. Skip it for short, well-formed queries over a corpus written in the user’s own vocabulary — an internal tool where people already ask in the documents’ language gains little.

Move 4 — Metadata filtering

The failure it fixes: the naive pipeline ranks on similarity alone and has no notion that last year’s fee schedule is superseded by this year’s, or that a document belongs to a tenant the user can’t see. Metadata filtering attaches structured fields — valid_from, tenant_id, document_type, access tags — to each chunk and filters on them before or during retrieval. This is also the cheapest, most boring, and most-skipped reliability win in the whole pipeline.

What it costs: almost nothing at query time, and real discipline at index time — someone has to populate the metadata correctly and keep it correct, which is process work rather than ML work. Skip it essentially never. Even a single valid_from date filter pays for itself the first time it stops the agent from quoting a retired policy, and access-control filtering isn’t optional the moment your corpus spans more than one permission boundary.

Where the four leave you

Cost / latency / setup effort (deltas over naive RAG):

Read the table as a menu, not a recipe. The order of adoption that holds up in production, and that the costs above argue for — call it the cost-order rule: metadata filtering first (cheap, always worth it), then reranking (highest leverage), then hybrid search (cheap insurance against exact-match misses), then query rewriting last (real but smaller wins, paid on every query). And to put a boundary on the whole menu: all four together are typically a two-to-three-week retrofit on an existing pipeline, not a rebuild.

The stack a competent team converges on

Not as a prescription — as a checkpoint to measure your own choices against. After the dust settles, most teams running serious RAG land in the same place: a single store doing both vector and keyword search, Reciprocal Rank Fusion to merge them, a cross-encoder reranker (Cohere Rerank 4 or Voyage rerank-2.5) trimming the longlist to a tight shortlist, metadata filters enforcing recency and access control, and query rewriting added only where the query/document vocabulary genuinely diverges. The embedding and reranker models underneath keep moving — Voyage shipped its mixture-of-experts voyage-4-large in January 2026, the rerankers turn over every few months — but the shape of the stack has been stable for a while now, and that stability is the signal that this is the production answer, not a fashion.

If your stack looks materially different from that and you can say why — a latency budget that rules out reranking, a corpus with no exact-match vocabulary that rules out BM25 — you’re fine. If it looks different and you can’t say why, that’s worth a look.

Combinations and what’s next

Advanced RAG is still retrieval over a flat corpus — it makes the retrieval sharper, but it can’t answer a question whose answer isn’t sitting in some chunk. When the question is about relationships between entities — “which suppliers connect to the parties in this dispute?” — no amount of reranking helps, because the answer is a path through a graph, not a passage in a document. That’s where GraphRAG comes in, and it’s the next article. And when retrieval needs to become a decision the agent makes — query, judge the results, refine, query again — you’ve crossed into agentic RAG, two articles out. Advanced RAG is the substrate both of those assume you’ve already got right.

Production checklist

• Measure recall@k before and after every move on a held-out golden set — never add a reranker or a rewriter on faith; prove the delta.

• Log the full retrieval trace: rewritten query, both retrieval lists, fused order, reranker scores, final shortlist. When an answer is wrong, this is where you see which move failed.

• Watch the latency budget per stage — query rewriting and reranking each add a measurable slice; track p99, not just the average, because the tail is what users complain about.

• Keep the longlist size honest — reranking a 200-candidate list is slow and rarely beats reranking 50; tune it.

• Treat metadata as a first-class schema, version it, and validate it at index time; silent metadata rot defeats the filter quietly.

• Alert when the reranker’s top score is low — same principle as naive RAG, but now you have a more trustworthy signal to threshold on.

• Re-evaluate the embedding and reranker models quarterly — they turn over fast, and a model swap is often a bigger win than another pipeline tweak.

The take

Advanced RAG isn’t a different technique from naive RAG — it’s the same pipeline with four targeted repairs, each aimed at a failure you can name. Add them in cost order, measure every one against your baseline, and stop when the numbers stop moving. The team that does this beats the team chasing a bigger model almost every time, because they’re fixing the layer where the failures actually live.

New articles land here first. Subscribe for the full series. I take on a small number of advisory engagements each year for teams hitting exactly these problems. The fastest way to reach me is on LinkedIn; reach out if that’s you.

What to read next

This series:

• Article 1 — Your agent’s problem isn’t the model, it’s the knowledge layer

• Article 2 — Naive RAG: the baseline you’ll always benchmark against

• Article 3 — Advanced RAG: what you actually run in production

• Article 4 — GraphRAG and friends: when entities and relationships beat similarity

• Article 5 — Agentic RAG: when retrieval becomes a decision, not a pipeline

• Article 6 — Vector vs graph vs episodic: a tour of agent memory systems

• Article 7 — Context engineering: the discipline that replaces prompt engineering (coming soon)

External:

• Anthropic (September 2024), Introducing Contextual Retrieval. https://www.anthropic.com/news/contextual-retrieval — the clearest published numbers on what hybrid search + reranking buys you.

• Gao et al. (December 2022), Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE). https://arxiv.org/abs/2212.10496 — the query-rewriting idea most worth understanding.

• Cohere (December 2025), Introducing Rerank 4: Cohere’s most powerful reranker yet. https://cohere.com/blog/rerank-4 — the current production-default reranker and what a cross-encoder adds.

• Voyage AI (January 2026), The Voyage 4 model family. https://blog.voyageai.com/2026/01/15/voyage-4/ — where the embedding frontier underneath all of this currently sits.

A team builds a support assistant that answers questions over about three hundred internal policy pages. It works for weeks. Then someone asks whether the company covers relocation costs for a transfer between two offices, and the assistant confidently quotes the parental relocation benefit — wrong policy, right neighborhood. The page with the correct answer exists. It just never makes it into the model’s context.

That is the naive-RAG failure in one sentence: the answer is in the corpus, the model is fine, and the retrieval step quietly hands over the wrong paragraph. Before you can fix that — with reranking, hybrid search, graphs, any of the machinery the rest of this series covers — you have to understand the pipeline that produced it. Naive RAG is the first and simplest thing you build in the knowledge layer, the part of an agent system that decides reliability in production and is an engineering discipline in its own right. Everything else in this series is a response to where naive RAG breaks.

How naive RAG works

The spine is four steps: split the documents into chunks, turn each chunk into a vector, store the vectors, and at query time pull the few chunks closest to the question and paste them into the prompt.

Each step is a single choice. Chunking is usually fixed-size — say 500 tokens with a 50-token overlap so a sentence straddling a boundary survives in at least one piece. Embedding is one model call per chunk; the common picks are OpenAI’s text-embedding-3-large, Cohere’s Embed 4 (shipped April 2025, with a 128k-token context window that lets you embed a 200-page document in one pass), or Voyage’s voyage-4-large (January 2026, which Voyage bills as the first production embedding model built on a mixture-of-experts architecture). Storage, for most teams starting out, is pgvector on the Postgres they already run — no new system to operate — or Chroma if they want something embedding-native and local. Retrieval is a cosine-similarity search for the top k chunks, k usually between 3 and 8. That’s the whole thing. You can stand it up in an afternoon.

In practice

Here’s the shape of it, stripped to pseudo-code:

# index time — runs once, then on updates
for doc in corpus:
    for chunk in split(doc, size=500, overlap=50):
        vec = embed(chunk, model="text-embedding-3-large")
        db.insert(text=chunk, vector=vec, source=doc.id)

# query time — runs per question
q_vec   = embed(user_question, model="text-embedding-3-large")
hits    = db.search(q_vec, top_k=5)            # cosine similarity
context = "\n\n".join(h.text for h in hits)
answer  = llm(f"Answer using only this context:\n{context}\n\nQ: {user_question}") 

The economics are why this is everyone’s starting point. Embedding a 300-page handbook is roughly 200k tokens; at text-embedding-3-large prices — about $0.13 per million tokens — indexing the whole corpus costs you a few cents, once. Per query you pay one tiny embedding call plus one model call, and a cosine search over a few thousand chunks on pgvector returns in well under 50 ms. The latency a user feels is almost entirely the model generating the answer, not the retrieval. For a corpus this size, naive RAG is close to free and close to instant.

Where it shines

Two situations where I’d push back on anyone who wants something fancier.

The first is FAQ-shaped corpora — a body of mostly self-contained question-and-answer pairs or short policy statements, where the answer to any given question lives in one place. An internal IT helpdesk over a couple of hundred wiki pages, a product-support bot over a structured knowledge base: the unit of retrieval and the unit of answer line up, so pulling the single closest chunk usually pulls the right one. Adding a graph or an agentic loop here buys you latency and nothing else.

The second is low-volume, low-stakes internal tooling — the onboarding assistant, the “where’s the form for X” bot, the thing fifty employees hit a few times a week. The corpus is small, the cost of an occasional wrong answer is a follow-up Slack message rather than a compliance incident, and the engineering budget is better spent elsewhere. Naive RAG is the bicycle here: unglamorous, occasionally exactly enough, and the thing you measure every proposed upgrade against before you pay for the upgrade.

Where it breaks

The failure modes are specific, and they’re worth naming because every later technique exists to fix one of them.

Chunk-boundary loss. A fixed-size splitter has no idea what a sentence or a clause means; it cuts at a token count. If the answer to a question spans the seam between chunk 7 and chunk 8, similarity search may surface only one half. Counterintuitively, fancier chunking doesn’t reliably fix this — Renyi Qu and co-authors, in Is Semantic Chunking Worth the Computational Cost? (NAACL 2025 Findings, [arXiv 2410.13070](https://arxiv.org/abs/2410.13070)), found that semantic chunking’s extra compute often didn’t beat plain fixed-size chunks on real documents. The baseline is stubborn, which is exactly why it’s the baseline.

Semantically-close-but-wrong matches. Embeddings encode topical similarity, not factual identity. Ask about paternity leave and the closest vectors may be the maternity-leave chunks, because the model considers them near-synonymous — which is how the support assistant in the opening example quotes the wrong relocation policy. The retrieved chunk looks relevant, scores well, and is wrong, and the model has no way to know.

No sense of recency or version. Naive RAG ranks on similarity alone. If your corpus contains last year’s fee schedule and this year’s, both embed to nearly the same vector, and the pipeline has no notion that one supersedes the other — so it may hand the model the stale one, or both at once with no signal about which to trust.

No multi-hop, no relationships. Any question that requires joining two facts — “which of our vendors are subsidiaries of companies we’ve flagged?” — is dead on arrival, because each fact lives in a different chunk and similarity search retrieves them independently, with nothing to connect them.

When not to use it

The moment any of these is true, naive RAG stops being the right baseline and becomes a liability: the corpus updates frequently with facts that supersede each other, the questions require synthesis across multiple documents or reasoning over relationships, or a wrong answer is expensive — legal, medical, financial. If a confident wrong answer costs you a customer or a compliance finding, the few cents you saved on the retrieval pipeline is the wrong place to economize.

Cost / latency / setup effort:

Combinations and hybrids

Nobody runs naive RAG in production for long, but almost everyone runs something built on top of it. The upgrades are additive, and each targets one failure mode above. A reranker — Cohere Rerank, Voyage rerank — re-scores the top k so the genuinely-relevant chunk beats the merely-similar one; that’s the cheapest single fix and it’s the subject of the next article. Hybrid search bolts a keyword index (BM25) onto the vector search so exact terms like a product code or a statute number stop getting lost in semantic fog. Metadata filtering on a valid_from date is how you teach the pipeline about recency. And Anthropic’s Contextual Retrieval (September 2024) attacks chunk-boundary loss directly, by prepending a short LLM-generated description of each chunk’s context before embedding it. The point is that naive RAG isn’t a competitor to any of these — it’s the substrate they all assume.

Production checklist

• Log every retrieval: the question, the k chunks returned, their similarity scores, and which document each came from. Without this you cannot debug a single wrong answer.

• Hold out a golden set of 50–100 real questions with known-correct source chunks, and measure recall@k on it before and after any change.

• Alert on low top-score queries — when the best chunk’s similarity is below a threshold, the model is probably about to improvise. Flag it rather than answer it.

• Instrument the “no good chunk” path explicitly; decide whether the agent says “I don’t know” or falls back to the model’s parametric knowledge, and make that a deliberate choice, not an accident.

• Track chunk size and overlap as config, not as buried constants — they are the parameters you’ll tune most.

• Re-index on a schedule and log when each chunk was last refreshed, so a stale corpus is visible rather than silent.

• Watch p99 latency as the corpus grows; the cliff comes later than you’d think on pgvector, but it comes.

The take

Naive RAG is not a strawman and it’s not a beginner’s mistake — it’s the control group. Build it first, measure it honestly, and make every fancier technique earn its place against those numbers. Most of the time the upgrade is worth it; some of the time the bicycle was already enough, and the only way to know is to have ridden it first.

New articles land here first. Subscribe for the full series. I take on a small number of advisory engagements each year for teams hitting exactly these problems. The fastest way to reach me is on LinkedIn; reach out if that’s you.

What to read next

This series:

• Article 1 — Your agent’s problem isn’t the model, it’s the knowledge layer

• Article 2 — Naive RAG: the baseline you’ll always benchmark against

• Article 3 — Advanced RAG: what you actually run in production

• Article 4 — GraphRAG and friends: when entities and relationships beat similarity

• Article 5 — Agentic RAG: when retrieval becomes a decision, not a pipeline

• Article 6 — Vector vs graph vs episodic: a tour of agent memory systems

• Article 7 — Context engineering: the discipline that replaces prompt engineering (coming soon)

External:

• Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020). https://arxiv.org/abs/2005.11401 — the paper that named the pattern.

• Qu, Tu & Bao (NAACL 2025 Findings), Is Semantic Chunking Worth the Computational Cost? https://arxiv.org/abs/2410.13070 — the evidence that fixed-size chunking is a hard baseline to beat.

• Merola & Singh (April 2025), Reconstructing Context: Evaluating Advanced Chunking Strategies for Retrieval-Augmented Generation. https://arxiv.org/abs/2504.19754 — a careful look at where smarter chunking (late chunking, contextual retrieval) does and doesn’t pay off.

• Voyage AI (January 2026), The Voyage 4 model family: shared embedding space with MoE architecture. https://blog.voyageai.com/2026/01/15/voyage-4/ — where the embedding-model frontier sits right now.

I’ll start with a pattern. You’ve probably seen it.

An LLM agent demos well. The team builds it against a few hand-picked examples, ships it to a stakeholder meeting, the stakeholder is impressed, and the project gets the green light for production. Three months in, the agent is quietly turned off and replaced with a form. Nobody calls it a failure; they call it a V1.

When you dig into why, the failure is almost never the model. The team didn’t pick the wrong LLM — they were probably on Claude Opus 4.7, GPT-5.5, or Gemini 3.5 Flash, and any of those would have been fine (even an overkill in most cases). The failure is almost never the prompt either; modern models tolerate prompt sloppiness in a way they didn’t two years ago. The failure is, almost every time, in the layer between the corpus of company knowledge and the model: which documents got retrieved, what the memory remembered (or forgot), how the context was packed, what got cached, what was stale, what was missing.

Most people now call it the knowledge layer. Pinecone calls its Nexus a “knowledge engine”. Microsoft markets Fabric IQ as a “unified intelligence platform”. Google’s April 2026 rebrand from Dataplex Universal Catalog to Knowledge Catalog put the word in the product name. What’s lagged is the engineering discipline around it — most teams still treat it as three separate plug-ins (RAG, memory, context engineering) bolted onto a model rather than one coherent job. This series makes the case that this — treated as one job, not three — is what decides whether your agent survives production.

This first piece is the map: what the knowledge layer is, why it deserves to be treated as its own engineering discipline, and the three places I keep seeing teams break in production. This article series will go deep on the techniques — what they do, where they shine, where they break, and how to choose between them.

What the knowledge layer actually is

Strip an agent system down and you find three things stacked on top of each other.

There’s the model — the LLM you’re renting from OpenAI, Anthropic, Google, or whichever provider you prefer. There’s the prompt — the instructions you give it, the role you put it in, the format you ask the output in. And underneath those, doing the work that’s much less visible, there’s the knowledge layer — everything that has to happen for the model to know what’s actually in your company, your conversation history, your last invoice, your evolving SOPs.

In practice the knowledge layer is four sub-problems that compose into one:

• Storage. Where the corpus lives. The vector-database era is the canonical answer — Pinecone, Weaviate, Qdrant, or Chroma if you’re starting small; Milvus or Vespa at scale. But the most common starting point in real enterprises isn’t a dedicated vector database at all — it’s pgvector on the Postgres they already run, which means no new system to operate, transactional consistency with the rest of the data, and a migration path to a purpose-built store only if and when scale forces it. “Just use pgvector” is the right first answer more often than any vendor will tell you. The graph side has its own canon: Neo4j, Memgraph, TigerGraph for typed-relation corpora. And in 2026 a new tier appeared — dedicated knowledge engines explicitly built for agents: Pinecone Nexus (May 2026, ships with a query language called KnowQL and compile-time artifacts), Microsoft Fabric IQ (announced at Ignite 2025, semantic layer over OneLake with ontology + graph + agents), Google Knowledge Catalog (the April 2026 rebrand of Dataplex Universal Catalog, building dynamic context graphs for Gemini agents). The schema and abstractions you commit to here are very hard to change later.

• Updating. How the storage stays fresh when the world moves on — when a policy is rewritten, when a product is discontinued, when a customer changes their preferences. Historically this was a custom ETL pipeline somebody on the team owned. In 2025–2026 it’s slowly becoming a vendor concern — Pinecone Nexus’s Context Compiler, Mem0‘s dynamic memory extraction, Letta‘s streaming-update primitives — but most enterprises still hand-roll it. This is the sub-problem most teams discover six months in.

• Retrieval. How the right slice of the corpus shows up at the right moment — what gets pulled when the user asks “is acupuncture covered by my health plan?”, and what gets ignored. Embedding-model choice (text-embedding-3-large from OpenAI, Cohere Embed 4, Voyage 3) and reranker choice (Cohere Rerank 3.5, Voyage Rerank) make most of the difference, with hybrid sparse-dense techniques (BM25 + vectors) and graph-augmented retrievers (LightRAG, KAG) cleaning up the rest. Most “RAG isn’t working” conversations turn out to be retrieval conversations.

• Context construction. How the retrieved material gets ordered, weighted, and packed into the model’s context window. The state of the art shifted twice in 2024–2025 — first with Anthropic’s Contextual Retrieval (September 2024), which argued context construction is its own engineering discipline, then with Anthropic’s Effective context engineering for AI agents essay and the broader industry adoption of prompt caching, structured outputs, and (with Claude Managed Agents in April 2026) cross-session memory primitives. The model only sees what context construction decides to show it, so this step quietly governs the quality of every answer.

The four are tightly coupled. A change in storage forces a change in retrieval. A change in retrieval breaks context construction. A change in update policy can quietly poison both. Teams that treat them as four separate engineering problems eat the cost of integrating them by accident.

Why it’s one discipline, not three

I want to argue that the knowledge layer is worth treating as one engineering job rather than three separate ones (RAG, memory, context engineering). Three reasons.

The first is that the failures cluster here. When agents fail in production, they fail because retrieval pulled the wrong document, because the corpus was stale, because the conversation history attributed a fact to the wrong session, or because the relevant chunk got drowned out by ten less-relevant chunks in a stuffed context window. These are different surface symptoms of the same underlying engineering problem: getting the right information in front of the model at the right moment.

The second is that the techniques compose. GraphRAG and a typed memory system aren’t alternatives; they’re tools you reach for at different points in the same pipeline. Treating each technique as a separate engineering investment is what produces the demo-to-production gap — the team ships RAG, finds it isn’t enough, layers on memory, finds that doesn’t fit either, plugs in a reranker, plugs in a graph, and the architecture quietly becomes a graveyard of half-finished plug-ins.

The third is that the discipline is recognizably different from prompting and from model choice. Prompting is how to talk to the model. Model choice is which model to rent. The knowledge layer is what does this model know about your business and how does it stay current. Those are different jobs, and putting them in the same bucket — the way “let’s add RAG” used to be a complete project plan — is part of why so many enterprise agent projects stall a year in.

The industry has been converging on this view in public. Anthropic published Contextual Retrieval in September 2024, essentially an argument that context construction is its own engineering problem, not a free side-effect of dropping a vector store next to your prompt. Microsoft’s GraphRAG paper, five months earlier, made the same case for structured knowledge. By the time Anthropic shipped Claude Managed Agents in April 2026 — multiagent orchestration, persistent state — and demoed cross-session “dreaming” at Code with Claude the following month, the knowledge layer wasn’t a side project anymore; it was the developer surface. The terminology and the products are converging fast. What’s still missing in most enterprise teams is the engineering culture around treating this as one coherent job rather than three plug-ins.

The model is the chef. The prompt is the recipe. The knowledge layer is the pantry — and the pantry is where most kitchens go wrong.

The three places it breaks

Three failure modes account for most of what kills enterprise agents in production. They show up across verticals, model choices, and frameworks.

Stale corpus. The data was right when it was loaded. Then a policy changed, a product was rebranded, a fee schedule was updated — and nobody told the retrieval system. The agent keeps citing the old version with full confidence because, from its perspective, the old version is the source of truth. The cost of this failure isn’t only the wrong answer; it’s that the user now has to fact-check the agent every time, which destroys the time-saving the agent was supposed to deliver in the first place.

Wrong retrieval. The right document exists in the corpus, but it didn’t surface for the user’s question. Maybe the question was phrased differently from how the document was written; maybe the embedding model — text-embedding-3-large, Cohere Embed 4, Voyage 3, take your pick — treats two semantically-related phrases as unrelated; maybe the chunking sliced the relevant passage in half. The agent then either falls back on the model’s pre-trained knowledge — which is generic and often wrong for your domain — or fabricates an answer from the closest-but-incorrect chunks it did manage to retrieve.

Wrong context construction. Retrieval surfaced the right chunk, but it was packed in among ten less-relevant chunks, or it was at the bottom of a long context window where models attend less, or the system prompt was so long that the retrieved content was effectively buried. The model had the right information in front of it and still gave the wrong answer because the context wasn’t constructed to let it see clearly.

The pattern across all three is that the model and the prompt were fine, and a smarter model or a cleaner prompt wouldn’t have helped. The fix is at the knowledge-layer level.

The map

Here’s how the four sub-problems compose into a pipeline.

The interesting moves all happen along that horizontal axis: choosing a storage abstraction (vector store, typed graph, hybrid), choosing an update strategy (re-index, delta, event-driven), choosing a retrieval method (dense, sparse, hybrid, agentic), choosing a context construction policy (reranking, compression, structured assembly). The next nine articles in this series each go deep on one of those decision points.

The landscape

Before the techniques, here’s how to approximately position them mentally. Two axes do most of the work:

• Horizontal — how structured the underlying knowledge is. Raw vectors on the left (the semantic-similarity world), typed graphs and ontologies on the right (the relational world).

• Vertical — how much agency the system has at retrieval time. A fixed pipeline at the bottom (the agent doesn’t decide what to fetch), an agentic loop at the top (the agent plans and refines retrieval as part of its reasoning).

The axes are a loose synthesis of the architectural breakdown in the SoK: Agentic RAG paper (Mishra et al., arxiv 2603.07379, March 2026) and the Agentic RAG Survey (Singh et al., arxiv 2501.09136, revised April 2026), both of which organize current paradigms along similar structure-and-agency dimensions. The placements on the map aren’t precise — vendors move on it as they ship — but it’s the first thing I sketch when a team asks “where does our stack sit?” The most interesting movement in 2025–2026 has been up and to the right: from naive RAG in the bottom-left to compiled knowledge engines and ontology-backed agent platforms in the top-right.

What this series will cover

Briefly, and not as a table of contents.

Articles 2–5 walk through the retrieval surface — naive RAG, advanced RAG (the production version with reranking, hybrid search, query rewriting), GraphRAG for when relationships matter more than similarity, and agentic RAG for when retrieval becomes a decision the agent makes rather than a pipeline it runs. Article 6 covers agent memory systems — Mem0, Letta, Zep, LangMem (the four-vendor landscape that consolidated through 2025–2026), plus research-side proposals like A-MEM and MemPalace — and the question of what kind of memory your agent actually needs. Article 7 is on context engineering, which I think is the discipline that quietly replaces prompt engineering for serious teams. Article 8 critiques knowledge editing — the model-weight-surgery techniques the academic world is excited about, and why they’re wrong for almost every enterprise update problem. Article 9 is on evaluation, the part most teams skip and most regret skipping. Article 10 is the practical selection guide: given your corpus, your update frequency, your latency budget, and your team’s depth, here’s what to actually build.

On the community and open-source side, the conversation has been just as alive — and worth keeping an eye on even if you don’t deploy any of these. Andrej Karpathy’s LLM Wiki gist (April 2026) sketched a pattern for compounding personal knowledge that pre-compiles a maintained wiki of interlinked entity pages instead of re-retrieving raw chunks at query time. MemPalace (April 2026) applied the ancient method-of-loci spatial metaphor to long-term memory and pulled 47k GitHub stars in two weeks, prompting a critical-analysis paper from Dey and Viradecha that, fairly, attributed most of the retrieval performance to the underlying embeddings rather than the spatial metaphor. A-MEM (February 2025, AGI Research) extended the Zettelkasten idea into agentic memory with dynamic indexing. MAGMA (January 2026) experimented with multi-graph agentic memory. None of these are production stacks yet, but they’re where the next set of ideas tends to come from.

The articles can be read in isolation, but they reward being read in sequence.

What it costs you when you get this wrong

Rough numbers, because the buyer reading this needs them.

These are rough. I haven’t seen a team that escaped them by working harder on prompts.

The take

If you’re building agents and you’re spending most of your engineering effort on prompts and model selection, you’re working on roughly twenty percent of the problem. The other eighty percent — the part that decides whether your agent survives contact with real users and real data — lives in the knowledge layer. Get it right and the model and prompt choices become easy. Get it wrong and no amount of model upgrading or prompt tuning will save you.

The knowledge layer is the part of an agent system that decides reliability in production, and it’s an engineering discipline in its own right. That’s the claim the rest of this series defends.

New articles land here first. Subscribe for the full series. I take on a small number of advisory engagements each year for teams hitting exactly these problems. The fastest way to reach me is on LinkedIn; reach out if that’s you.

What to read next

This series:

• Article 1 — Your agent’s problem isn’t the model, it’s the knowledge layer

• Article 2 — Naive RAG: the baseline you’ll always benchmark against

• Article 3 — Advanced RAG: what you actually run in production

• Article 4 — GraphRAG and friends: when entities and relationships beat similarity

• Article 5 — Agentic RAG: when retrieval becomes a decision, not a pipeline

• Article 6 — Vector vs graph vs episodic: a tour of agent memory systems

• Article 7 — Context engineering: the discipline that replaces prompt engineering (coming soon)

External:

• Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. https://arxiv.org/abs/2005.11401 — the original RAG paper; useful background.

• Edge et al. (April 2024), GraphRAG: A Graph-Based Approach to Retrieval-Augmented Generation. https://arxiv.org/abs/2404.16130 — pioneer of “the knowledge layer needs structure” framing.

• Anthropic (September 2024), Introducing Contextual Retrieval. https://www.anthropic.com/news/contextual-retrieval — the most cited industry write-up making the context-engineering case.

• Anthropic Engineering (September 2025), Effective context engineering for AI agents. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents — the vendor-side “this is the discipline” essay, more practical than the 2024 piece.

• Singh et al. (revised April 2026), Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG. https://arxiv.org/abs/2501.09136 — the canonical survey of where the retrieval-as-decision wave landed.

• Du (March 2026), Memory for Autonomous LLM Agents: Mechanisms, Evaluation, and Emerging Frontiers. https://arxiv.org/abs/2603.07670 — the current state-of-the-field write-up for the memory side.