Naive RAG: the baseline you'll always benchmark against

The simplest retrieval pipeline there is — chunk, embed, match, stuff into the prompt. It's also the thing every fancier technique has to prove it beats.

Jun 16, 2026

The problem

A team builds a support assistant that answers questions over about three hundred internal policy pages. It works for weeks. Then someone asks whether the company covers relocation costs for a transfer between two offices, and the assistant confidently quotes the parental relocation benefit — wrong policy, right neighborhood. The page with the correct answer exists. It just never makes it into the model’s context.

That is the naive-RAG failure in one sentence: the answer is in the corpus, the model is fine, and the retrieval step quietly hands over the wrong paragraph. Before you can fix that — with reranking, hybrid search, graphs, any of the machinery the rest of this series covers — you have to understand the pipeline that produced it. Naive RAG is the first and simplest thing you build in the knowledge layer, the part of an agent system that decides reliability in production and is an engineering discipline in its own right. Everything else in this series is a response to where naive RAG breaks.

How naive RAG works

The spine is four steps: split the documents into chunks, turn each chunk into a vector, store the vectors, and at query time pull the few chunks closest to the question and paste them into the prompt.

Article content — Figure 1 — Naive RAG end to end. No reranking, no query rewriting, no filtering — every later technique article adds a box to this diagram.

Each step is a single choice. Chunking is usually fixed-size — say 500 tokens with a 50-token overlap so a sentence straddling a boundary survives in at least one piece. Embedding is one model call per chunk; the common picks are OpenAI’s text-embedding-3-large, Cohere’s Embed 4 (shipped April 2025, with a 128k-token context window that lets you embed a 200-page document in one pass), or Voyage’s voyage-4-large (January 2026, which Voyage bills as the first production embedding model built on a mixture-of-experts architecture). Storage, for most teams starting out, is pgvector on the Postgres they already run — no new system to operate — or Chroma if they want something embedding-native and local. Retrieval is a cosine-similarity search for the top k chunks, k usually between 3 and 8. That’s the whole thing. You can stand it up in an afternoon.

In practice

Here’s the shape of it, stripped to pseudo-code:

# index time — runs once, then on updates
for doc in corpus:
    for chunk in split(doc, size=500, overlap=50):
        vec = embed(chunk, model="text-embedding-3-large")
        db.insert(text=chunk, vector=vec, source=doc.id)

# query time — runs per question
q_vec   = embed(user_question, model="text-embedding-3-large")
hits    = db.search(q_vec, top_k=5)            # cosine similarity
context = "\n\n".join(h.text for h in hits)
answer  = llm(f"Answer using only this context:\n{context}\n\nQ: {user_question}")

The economics are why this is everyone’s starting point. Embedding a 300-page handbook is roughly 200k tokens; at text-embedding-3-large prices — about $0.13 per million tokens — indexing the whole corpus costs you a few cents, once. Per query you pay one tiny embedding call plus one model call, and a cosine search over a few thousand chunks on pgvector returns in well under 50 ms. The latency a user feels is almost entirely the model generating the answer, not the retrieval. For a corpus this size, naive RAG is close to free and close to instant.

Where it shines

Two situations where I’d push back on anyone who wants something fancier.

The first is FAQ-shaped corpora — a body of mostly self-contained question-and-answer pairs or short policy statements, where the answer to any given question lives in one place. An internal IT helpdesk over a couple of hundred wiki pages, a product-support bot over a structured knowledge base: the unit of retrieval and the unit of answer line up, so pulling the single closest chunk usually pulls the right one. Adding a graph or an agentic loop here buys you latency and nothing else.

The second is low-volume, low-stakes internal tooling — the onboarding assistant, the “where’s the form for X” bot, the thing fifty employees hit a few times a week. The corpus is small, the cost of an occasional wrong answer is a follow-up Slack message rather than a compliance incident, and the engineering budget is better spent elsewhere. Naive RAG is the bicycle here: unglamorous, occasionally exactly enough, and the thing you measure every proposed upgrade against before you pay for the upgrade.

Where it breaks

The failure modes are specific, and they’re worth naming because every later technique exists to fix one of them.

Chunk-boundary loss. A fixed-size splitter has no idea what a sentence or a clause means; it cuts at a token count. If the answer to a question spans the seam between chunk 7 and chunk 8, similarity search may surface only one half. Counterintuitively, fancier chunking doesn’t reliably fix this — Renyi Qu and co-authors, in Is Semantic Chunking Worth the Computational Cost? (NAACL 2025 Findings, [arXiv 2410.13070](https://arxiv.org/abs/2410.13070)), found that semantic chunking’s extra compute often didn’t beat plain fixed-size chunks on real documents. The baseline is stubborn, which is exactly why it’s the baseline.

Semantically-close-but-wrong matches. Embeddings encode topical similarity, not factual identity. Ask about paternity leave and the closest vectors may be the maternity-leave chunks, because the model considers them near-synonymous — which is how the support assistant in the opening example quotes the wrong relocation policy. The retrieved chunk looks relevant, scores well, and is wrong, and the model has no way to know.

No sense of recency or version. Naive RAG ranks on similarity alone. If your corpus contains last year’s fee schedule and this year’s, both embed to nearly the same vector, and the pipeline has no notion that one supersedes the other — so it may hand the model the stale one, or both at once with no signal about which to trust.

No multi-hop, no relationships. Any question that requires joining two facts — “which of our vendors are subsidiaries of companies we’ve flagged?” — is dead on arrival, because each fact lives in a different chunk and similarity search retrieves them independently, with nothing to connect them.

When not to use it

The moment any of these is true, naive RAG stops being the right baseline and becomes a liability: the corpus updates frequently with facts that supersede each other, the questions require synthesis across multiple documents or reasoning over relationships, or a wrong answer is expensive — legal, medical, financial. If a confident wrong answer costs you a customer or a compliance finding, the few cents you saved on the retrieval pipeline is the wrong place to economize.

Cost / latency / setup effort:

Combinations and hybrids

Nobody runs naive RAG in production for long, but almost everyone runs something built on top of it. The upgrades are additive, and each targets one failure mode above. A reranker — Cohere Rerank, Voyage rerank — re-scores the top k so the genuinely-relevant chunk beats the merely-similar one; that’s the cheapest single fix and it’s the subject of the next article. Hybrid search bolts a keyword index (BM25) onto the vector search so exact terms like a product code or a statute number stop getting lost in semantic fog. Metadata filtering on a valid_from date is how you teach the pipeline about recency. And Anthropic’s Contextual Retrieval (September 2024) attacks chunk-boundary loss directly, by prepending a short LLM-generated description of each chunk’s context before embedding it. The point is that naive RAG isn’t a competitor to any of these — it’s the substrate they all assume.

Production checklist

Log every retrieval: the question, the k chunks returned, their similarity scores, and which document each came from. Without this you cannot debug a single wrong answer.
Hold out a golden set of 50–100 real questions with known-correct source chunks, and measure recall@k on it before and after any change.
Alert on low top-score queries — when the best chunk’s similarity is below a threshold, the model is probably about to improvise. Flag it rather than answer it.
Instrument the “no good chunk” path explicitly; decide whether the agent says “I don’t know” or falls back to the model’s parametric knowledge, and make that a deliberate choice, not an accident.
Track chunk size and overlap as config, not as buried constants — they are the parameters you’ll tune most.
Re-index on a schedule and log when each chunk was last refreshed, so a stale corpus is visible rather than silent.
Watch p99 latency as the corpus grows; the cliff comes later than you’d think on pgvector, but it comes.

The take

Naive RAG is not a strawman and it’s not a beginner’s mistake — it’s the control group. Build it first, measure it honestly, and make every fancier technique earn its place against those numbers. Most of the time the upgrade is worth it; some of the time the bicycle was already enough, and the only way to know is to have ridden it first.

New articles land here first. Subscribe for the full series. I take on a small number of advisory engagements each year for teams hitting exactly these problems. The fastest way to reach me is on LinkedIn; reach out if that’s you.

The Knowledge Layer

Discussion about this post

Ready for more?