Agentic RAG: when retrieval becomes a decision, not a pipeline

Naive and advanced RAG retrieve the same way every time. Agentic RAG lets the model decide what to look up, whether the results are good enough, and when to stop. You pay for that judgement in latency

Jul 28, 2026

The problem

A team ships a support assistant over a good advanced-RAG stack: hybrid search, a reranker, metadata filters, the works. It answers single-fact questions well. Then a user asks: “I’m moving from the Berlin office to the Madrid one mid-contract: what changes for my health cover, my notice period, and my equity vesting?” The pipeline does what it always does: embeds the question, pulls the top chunks, stuffs them in. It comes back with a confident paragraph about health cover and silently drops the other two. Three questions went in; one came out, and nobody flagged the gap.

The pipeline isn’t broken. It did exactly what a pipeline does: one retrieval, one shot, no second look. The failure is that nothing in the system ever decided this question needed three separate lookups, or noticed that the retrieved context only covered one of them. That deciding is the thing agentic RAG adds. The knowledge layer (the part of an agent system that decides reliability in production, and an engineering discipline in its own right) stops being a fixed conveyor belt and becomes a loop the model drives: retrieve, look at what came back, decide whether it’s enough, and go again if it isn’t.

How agentic RAG works

The spine is one idea: make retrieval a tool the model calls inside a reasoning loop, instead of a step that always runs once before the model sees anything.

The pattern traces back to ReAct (Shunyu Yao and co-authors, ReAct: Synergizing Reasoning and Acting in Language Models, [arXiv 2210.03629](https://arxiv.org/abs/2210.03629), ICLR 2023), which interleaved reasoning steps with tool actions so a model could think, act, observe the result, and think again. Retrieval is just one of those actions. Two later papers sharpen it for RAG specifically. Self-RAG (Akari Asai and co-authors, [arXiv 2310.11511](https://arxiv.org/abs/2310.11511), ICLR 2024) trains the model to emit “reflection tokens” that decide whether to retrieve at all and then critique whether the retrieved passage actually supports the answer: retrieval on demand, with a built-in quality check. Adaptive-RAG (Soyeong Jeong and co-authors, [arXiv 2403.14403](https://arxiv.org/abs/2403.14403), NAACL 2024) adds a complexity classifier up front: trivial questions get answered with no retrieval, single-hop questions get one pass, and only genuinely multi-hop questions pay for the full iterative loop. The recurring move in all three is the same: spend retrieval effort in proportion to how hard the question actually is.

In production, this is usually built as a small state machine rather than a free-for-all. LangGraph, the graph-based agent framework from the LangChain team (1.0 in October 2025, now well into its 1.x line), is the de-facto way teams wire these loops as explicit nodes and edges with a retry budget, precisely so the “loop until satisfied” can’t run away. The broader industry has been moving the same direction: the Model Context Protocol (open-sourced by Anthropic in November 2024, adopted by OpenAI and Google through 2025, and donated to the Linux Foundation’s Agentic AI Foundation in December 2025) standardised how an agent reaches tools and data sources, which is exactly the plumbing an agentic retriever needs to query a vector store, a graph, a SQL database, and a third-party API through one interface.

In practice

Take the relocation question from the top. A static pipeline runs one retrieval. An agentic one does something closer to this:

User: moving Berlin → Madrid mid-contract. Health cover, notice period, equity vesting?

Agent (plan):    three sub-questions, different policy areas. Decompose.
  → retrieve("health insurance transfer between EU offices")     ✓ found
  → retrieve("notice period change on internal relocation")      ✓ found
  → retrieve("equity vesting on country transfer")               ✗ weak match
Agent (judge):   vesting result is thin. Re-query with a narrower term.
  → retrieve("RSU vesting schedule employee relocation")         ✓ found
Agent (judge):   three areas covered. Compose.

Four retrievals, two planning steps, one re-query when a result came back weak, versus the single shot a pipeline would have taken. That’s the whole trade in one trace. The agent caught the thin vesting result and went back for it, which is the failure the static pipeline shipped silently. It also cost roughly three-to-five times the model calls of a one-shot answer, because every plan, judge, and re-query is another round-trip to the model. Agentic RAG buys reliability on hard questions and pays for it in latency and tokens. That sentence is the whole article, and the discipline is knowing when the trade is worth it.

Where it shines

Two situations make the extra cost pay for itself.

The first is multi-hop and multi-part questions: anything that needs more than one fact, retrieved separately and combined. The relocation question is one; “which of our vendors fail the new compliance rule, and what’s the remediation for each” is another. A single retrieval can’t serve these because the sub-answers live in different documents and a one-shot top-*k* will over-weight whichever sub-question embeds strongest. The agent’s ability to decompose, retrieve per part, and notice a gap is the entire value.

The second is mixed-source corpora where the right source depends on the question. When some answers live in unstructured docs, some in a SQL table, some behind an API, and some in a knowledge graph, a static pipeline that always hits the vector store is wrong most of the time. An agent can route: read the question, pick the tool, and fall back to another source when the first comes up empty. This is common in enterprise settings where “the knowledge” is genuinely spread across systems, and it’s where retrieval-as-a-decision earns its keep over retrieval-as-a-step. The pattern has gone product-shaped, too: Microsoft’s Azure AI Search agentic retrieval (public preview in 2025; the core moved to GA in the April 2026 API, with the LLM query-planning step itself still in preview) ships exactly this loop as a managed service, with an LLM planning subqueries, running them in parallel, and merging the results. Microsoft has since wrapped the same engine into Foundry IQ (announced November 2025, in preview), pitched, in Microsoft’s own words, as a managed “knowledge layer” for agents.

Where it breaks

Be honest about the failure modes, because they’re nastier than a pipeline’s.

The cost and latency multiply. Three-to-five times the model calls is the rough rule, and it lands on every hard question, not just occasionally. A question that took 2 seconds and one model call now takes 6–10 seconds and five, and the bill scales with it. On a high-traffic assistant that delta is not a rounding error: it’s the difference between a viable unit economic and an unviable one.
Loops that don’t terminate. The signature agentic failure: the agent judges the results insufficient, re-queries, judges again, and never converges, burning budget and latency on a question it’s never going to answer. Every production loop needs a hard iteration cap and a “give up gracefully” branch, or it will eventually find the input that spins forever.
Confident wrong routing. When the agent decides not to retrieve (Self-RAG’s whole premise is that sometimes it shouldn’t), it can decide wrong and answer a factual question from parametric memory, hallucinating where a retrieval would have saved it. The decision to skip retrieval is itself a new failure surface that a static pipeline simply doesn’t have.
Harder to debug. A pipeline has one retrieval trace. An agent has a branching tree of plans, tool calls, and judgements, and when the answer is wrong you have to reconstruct which decision went wrong. The observability burden is real, and teams underestimate it.

When not to use it

If your questions are overwhelmingly single-fact lookups over one corpus (the FAQ-shaped workload from the naive-RAG article), agentic RAG is pure overhead. You pay 3–5× the cost and the added latency to wrap a decision around a question that only ever needed one retrieval, and a good advanced-RAG stack will beat it on both cost and speed. The loop earns its place only when a meaningful share of real questions are multi-hop, ambiguous, or span sources. Reach for it when you can point to the specific questions a pipeline keeps getting wrong; don’t reach for it because “agentic” is the word of the season.

Cost / latency / setup effort:

Combinations and hybrids

Agentic RAG isn’t a replacement for the earlier techniques: it’s a controller that sits on top of them. The tools the agent calls are naive RAG, advanced RAG, and GraphRAG; the agent just decides which one to use and whether to go again. The most effective real systems pair an agentic loop with a complexity gate in the Adaptive-RAG spirit: cheap questions bypass the loop entirely and get a one-shot answer, and only the hard ones pay for the full plan-retrieve-judge cycle. That gate is what keeps the cost honest: without it, you pay the 3–5× tax on every trivial lookup. Pair it the other way and agentic RAG also feeds the next article: an agent that decides what to remember across turns is the bridge from retrieval into memory.

Production checklist

Cap the loop. A hard maximum on retrieval iterations plus a graceful “I couldn’t find enough” branch. Non-negotiable, or you will ship the infinite loop.
Budget per query. Track and cap model calls and tokens per answer; alert when a single question blows past the expected envelope.
Log the full decision tree (every plan, tool call, observation, and judgement), not just the final retrieval. This is the only way to debug a wrong answer in a branching system.
Instrument the retrieve/skip decision separately; measure how often the agent chooses not to retrieve and how often that choice was wrong.
Gate by complexity so trivial questions never enter the loop. Measure what fraction of traffic actually needs it.
Watch p99 latency and tail cost, not the averages; the agentic tail is where both the budget and the user patience go.
Evaluate end-to-end, not per-retrieval: a loop can make three good retrievals and still compose a bad answer, so score the final result against a golden set.

The take

Agentic RAG turns retrieval from a fixed step into a decision the model makes, and that’s genuinely more powerful on the questions that need it: multi-hop, ambiguous, multi-source. But power you pay for on every query is a liability, not a feature. Wrap the loop in a complexity gate, cap it hard, instrument every decision, and spend the 3–5× only where the reliability gain is real. The teams that win with agentic RAG are the ones who know exactly which questions justify the loop, and route everything else around it.

I take on a small number of advisory engagements each year for teams hitting exactly these problems. Reach out if that’s you.

The Knowledge Layer

Discussion about this post

Ready for more?