Serbian Legal AI: a RAG architecture comparison
Five knowledge-layer architectures, one Serbian-law corpus, 300 graded answers. The agent led every quality axis among the arms you could ship, the typed knowledge graph didn’t earn its rung, and the caveats are printed next to the numbers they belong to.
The question nobody benchmarks honestly
Every team building a retrieval system hits the same fork, and almost nobody has numbers to settle it. Do you run plain vector search? Add reranking and hybrid search? Build a knowledge graph? Wrap the whole thing in an agent that decides what to fetch? Or skip retrieval and stuff the documents into a long-context window, now that the windows are big enough to hold them?
The honest answer is “it depends,” and the thing it depends on is almost never the cleverness of the technique. It is the shape of your corpus and the shape of the questions your users actually ask. I wanted that answer with numbers attached, so, together with Sasa Mirkovic, I built a system where the same question is answered five different ways over one real corpus, and we measured what each approach buys and what it costs: accuracy, citation quality, hallucination rate, temporal correctness, latency, and dollars per query, side by side.
If you have been following this series, you have met these architectures one at a time: naive RAG in article 2, the production-grade flat stack in article 3, GraphRAG in article 4, and agentic retrieval in article 5 (the fifth arm, long-context stuffing, is the one everyone argues about without needing an article). This is the post they were building toward, where the arguments stop and the measurement starts. The project is open source, linked at the bottom, and there is a screen-capture walkthrough of the running system further down.
The frame the whole thing rests on, and the through-line of everything I write: the knowledge layer, how an agent stores, updates, retrieves, and assembles what it knows, is the part that decides whether it is reliable in production. This is five architectures for that layer, measured on which parts actually pay.
One corpus, built deep
“One real corpus” should mean something, so here are the actual numbers. The corpus is Serbian statutory law from the official state legal-information system (PIS); the texts are public and excluded from copyright by statute, so there is no employer or client data anywhere near this. It holds 33 seed statutes plus a six-act shared backbone (the Constitution and the codes everything else leans on), spanning ten everyday-law clusters: employment and contributions, family and inheritance, real estate and construction, consumer law, traffic, personal tax and freelancing, data and digital, company life, civil disputes and enforcement, and health and welfare. From those laws the pipeline then walks the delegation graph and fetches every reachable implementing bylaw (about 475 materialize with article structure; roughly 550 more fetch as HTML but carry no article structure and stay graph-only stubs) plus 21 Constitutional-Court decisions. That is 536 acts with full text and 25,675 retrievable units: one chunk per article per version slice, including 3,257 dated version slices, around 910 extracted HTML-table annexes, and around 167 vision-OCR’d image annexes.
Every arm queries the same two stores:
A vector plus full-text index (LanceDB, gemini-embedding-2 embeddings, BM25 for the lexical leg), so a query can land on the exact text in force on a given date.
A typed temporal knowledge graph in plain SQLite: eleven edge types, including article-to-article cross-references, term-aware definition links, the law-to-bylaw delegation edge, and a constitutional-validity edge that models a court strike as a first-class temporal death.
Two invariants run through everything. First, no claim without provenance: every retrievable unit carries its law, article, version, and an ELI URL, and an answer with no citations is only legal when it abstains. Second, correctness is time-indexed: statutes amend, and a confidently cited repealed article is worse than no answer, so every unit carries validity dates.
Hold onto one set of numbers from that graph: the article-to-article cross-reference web has a mean out-degree of 0.57, 81% of article nodes have no outgoing reference at all, and only 264 of its edges (2.2%) cross from one law to another. That is most of the story of Finding 2.
The setup: a ladder, not a demo
The five arms are not five random systems. They are a ladder, where each rung adds exactly one class of capability over the one below it:
Context-Stuffing is the “why retrieve at all” arm, run as an oracle: it is handed every act the gold answer cites, a priori, and stuffs them whole into the window. That makes it a ceiling, not a deployable system (knowing which acts to read is precisely the problem retrieval exists to solve), so it is reported but kept out of the ranking. Think of it as an open-book exam where someone else has already found the right pages.
Naive RAG is the floor: embed the query, take the top-8 vector hits, generate. Deliberately dumb.
Advanced RAG is the production flat stack: hybrid search (dense vectors plus BM25, fused with Reciprocal Rank Fusion), a temporal filter that drops articles not in force on the asked-about date, a listwise LLM reranker, and a diversity cap so one chatty bylaw cannot crowd out the governing article.
Advanced + Graph takes advanced’s exact output as its seeds, then adds typed-graph traversal on top: cross-reference and definition hops, a delegation-gated law-to-bylaw hop, and the constitutional-validity path. It is a strict superset of advanced, pinned by a unit test: turn the structure off and it reproduces advanced exactly, so any difference between the two is attributable to structure and nothing else.
Agentic is a bounded ReAct loop (at most 10 steps) over seven typed tools that cover everything the other arms can reach: search, fetch an article, expand references, pull delegated bylaws, check constitutionality, read a law’s validity timeline.
Everything else is held constant: same corpus, same chunking, same generator (Gemini 3.6 Flash), same prompts, temperature zero. Retrieval strategy is the only treatment. Grading is a cross-family judge, Claude Opus 5 grading Gemini’s answers against gold, precisely because the first objection to any LLM-graded benchmark is “the model graded itself.” A second-family juror grades nearly every item too (293 of the 295 gradable pairs), and the agreement between them is reported.
Three honesty notes before the table, because they bound everything in it. This is a single-seed run, so every number is a point estimate on 60 items. The judge-juror agreement came out at α = 0.764, below the 0.80 bar I would want; a second juror pass made the disagreement diagnosable (81% of it sits on the correct-versus-partial boundary, so ordinal α reads 0.842, and the ranking with every engineered-versus-naive p-value survives re-derivation from the second judge’s labels), which bounds the caveat without dissolving it: margins of two or three items are still not safe to read; margins of sixteen-plus are. And the generator is a floating model alias with a run-to-run flip rate of about 6% of items (quantified on its predecessor alias), so any difference of four items or fewer sits inside the drift envelope. I will point at these caveats exactly where they bite.
The scoreboard
Sixty situational, first-person client questions (”I’m buying a flat and the contract goes to the notary...”), three difficulty tiers, five arms, 300 graded answers:
Accuracy counts only answers the judge marked fully correct; correctness gives half credit for partially correct ones. Provenance F1 is deterministic citation precision and recall against the gold articles, no judge involved. The intervals are Wilson 95% on the accuracy proportion, item-sampling only. Every engineered arm’s interval overlaps every other’s; only naive separates cleanly.
Deployable ranking: Agentic 0.87 > Advanced + Graph 0.77 ≈ Advanced 0.73 > Naive 0.48. Five things in that table are worth stopping on.
Finding 1: the retrieval ladder pays, and this one has receipts
Every engineered arm beats the naive floor decisively: paired exact McNemar p = 0.0003 or better on all three comparisons, with naive losing 16, 18, and 24 of the discordant items respectively. Naive’s confidence interval is the only one that does not overlap the rest of the board.
The cheapest step is still the biggest bargain: naive to advanced is +0.25 accuracy for about a cent and a half more per query. Hybrid search, a reranker, a temporal filter, and a diversity cap. No new model, no graph, no agent. If your RAG system is underperforming and you have not done these four things, that is almost certainly your problem, not the model. This is the least glamorous finding, and given the caveats above it is also the only one with margins wide enough that no plausible amount of judge disagreement or generator drift can touch it.
Finding 2: the typed graph added nothing, and the scope of that sentence is the finding
In the GraphRAG article I said a graph earns its keep when relationships are first-class in the questions your users ask. The corollary under test here is whether the corpus has the connection density to traverse. Here is what happened when I measured mine: Advanced + Graph came out at 0.77 against advanced’s 0.73, which sounds like a win until you run the paired test. Two items in one direction, four in the other, p = 0.69. In the previous campaign the graph arm sat nominally below advanced; scoring corrections landed in between, so the two campaigns are not strict replicates, but an ordering that will not stay put is exactly what a null effect looks like. This graph layer, on this corpus, neither beats nor loses to strong flat retrieval.
The instrumentation is what makes that sentence worth publishing. The superset design rules out “you built the graph arm wrong”: structure-off provably reproduces advanced, so the null delta is attributable to the structure itself. And the decomposition locates why it is null, stage by stage:
At the construction stage, the cross-reference web barely exists. Serbian statutes rarely cite each other by article number: the code of obligations has roughly 1,100 articles and only 12 of them carry a numbered cross-reference at all, none pointing outside the code. Corpus-wide, 81% of article nodes are isolated and 2.2% of reference edges cross laws. You cannot manufacture those edges without fabricating law. At the retrieval stage, the one dense structure the corpus does have, the law-to-bylaw delegation spine, hits a different wall: a single law can delegate to as many as 79 materialized bylaws, and the graph hop picks which ones to search in arbitrary stored order rather than by relevance. Ranking that fan-out by the query would fix it, but ranking by the query is retrieval, the thing the flat backbone already does over the whole corpus. That is the sharpest form of the result: on this corpus, the typed edges add no retrieval power that good flat retrieval doesn’t already have.
Two scope notes, both load-bearing. This is a claim about this graph implementation on this corpus, not about graph structure in general; the fan-out problem is a fixable engineering property, and fixing it is deliberately left to a held-out test so the negative result stays clean. And on the three constitutional-validity questions, where the validity edges genuinely exist, the graph arm was the only fixed-pipeline arm to go 3 for 3 (the agentic loop managed it too). Three questions proves nothing, but it points where structure’s remaining home turf is: validity- and definition-aware traversal, not cross-reference multi-hop.
The durable lesson survives all the caveats: whether a graph helps is a property of your corpus topology, not of the technique. Count your cross-references before you build one.
Finding 3: the agent is the headline, for the quality profile rather than the accuracy lead
Agentic tops the accuracy column at 0.87, a 13-point lead over advanced. I am deliberately not headlining that number, because it does not survive multiple-comparison correction: p = 0.0386 uncorrected, against a Bonferroni threshold of 0.0083 across the six pairwise tests, and against the graph arm it is nowhere near significance. On accuracy, the honest verdict is “promising, unproven at this sample size.”
What the data does support is the quality profile, and it is consistent across every axis I measured. Best deployable correctness (0.93). Best deployable provenance F1 (0.61). Misgrounding and hallucination rates as low as any arm’s on the board, the oracle included (0.07 and 0.07, versus naive’s 0.22 and 0.20). Zero over-abstentions: it never refused an answerable question, while naive refused seven. It is the only deployable arm that holds up on the hard tier (16 of 20, level with the oracle, though the oracle’s sixteenth hard item rests on a re-run cell the report discloses). And the price of all that is about 2.3× advanced’s cost ($0.068 vs $0.030 per query) and under twice the median latency, 26 seconds against 15.
That is a much better trade than the agentic folklore suggests. Retrieval-as-a-decision is a trustworthiness purchase: for a legal, medical, or financial assistant where the expensive failure is a confident wrong answer with a plausible citation, the safety delta is worth double the spend. For an internal FAQ bot it is not. That is the actual decision, and now it has a price tag.
Finding 4: the ceiling closed, and what that actually means
The oracle, handed every act the gold answer cites, scores 0.92. Two readings of that number are wrong and one is useful.
Wrong reading one: “long context killed RAG.” The oracle’s input assumption is the problem statement: someone has to know which acts the answer rests on, and that act selection is what retrieval does. The previous campaign measured its worth directly, as a disclosed same-day correction: an oracle stuffing only the single primary act scored 0.42, and handing over the full gold act set moved it to 0.93. Knowing which documents to read was worth more than any retrieval-stage refinement measured in this project. Wrong reading two: “so use the oracle’s setup anyway.” It costs $0.197 per query, 12.8× naive and 2.9× agentic, and it is temporally blind by construction: it stuffs current consolidated text, so on questions where the in-force answer is a superseded version it structurally cannot cite the right law.
The useful reading is narrower. The gap between the best deployable arm and the ceiling has closed to within what this benchmark can measure: agentic versus oracle is three items against six, p = 0.51, and that is a statement about statistical power, not proof of parity. Still, a bounded agent over typed tools, paying a third of the oracle’s price and keeping temporal awareness, is at least keeping pace with perfect act selection at this sample size. If that holds up at more seeds, it is the best argument I know for spending your budget on the knowledge layer rather than on a bigger window.
Finding 5: temporal correctness turned out to be a property of the knowledge base, not the model
The most transferable result is also the one I have to report most carefully. On questions dated into the past, where the correct answer is a superseded version of an article, what decides the outcome is whether the arm’s retrieval layer can reach and prefer the right version slice. Naive and the oracle index only current text, so where the in-force text at the asked date is a superseded slice, they cannot cite it, structurally. The three arms with a validity filter cannot cite an out-of-force one. Neither outcome could have come out otherwise, which is why I am calling this a mechanism demonstration, not a measurement: my dated tier held three items, and after two of them were passed by all five arms, exactly one item did the discriminating.
Full disclosure, because it belongs next to this number: that one discriminating item was itself corrected after arm behaviour had been analysed, an exception to the benchmark’s blind-authoring rule that the report states rather than hides, and it is a second reason no temporal effect is claimed here. There is also a methodological trap inside this that anyone building a temporal benchmark will hit: a per-citation validity metric ends up measuring the arm’s retrieval configuration, not its temporal reasoning, and an arm can pass it by citing the wrong law entirely. In my view this methodological finding will outlive the scoreboard. If your corpus has versions, effective dates, or amendment history, temporal correctness has to be built into storage and retrieval as validity intervals and an as-of filter. No prompt and no bigger model fixes it downstream, and measuring it takes a tier of materially amended articles I have not built yet.
Watch it work
Numbers are one kind of evidence; the system showing its work is another. I recorded a short screen-capture walkthrough of the running app, and it is embedded here on the canonical version of this post:
What it walks through, in order: a real question asked with an as-of date and the arm selector; the answer with citation chips and validity badges (green “in force since”, red “ceased to be valid”); the Runs drill-down for that exact answer, with per-stage latency, cost, and, for the agentic arm, the full tool trace of what the agent searched, fetched, and expanded before answering; the Benchmark tab, where the whole five-arm matrix in this post can be re-run with any capability toggled off; and the Corpus topology panel, the sparsity numbers behind Finding 2. Every number on those screens comes from the same telemetry rows the published report is computed from. There is no demo mode.
What this actually costs to run
Generating the full 300-answer campaign cost $20.91 in API calls across the five arms; the cross-family grading runs on top of that.
What this is, and what it isn’t
This is a build-in-public launch of a real open-source project, built in cooperation with Sasa Mirkovic (https://github.com/SasaMirkovic), who co-designed the ingestion pipeline and the knowledge base every arm retrieves over and advised on the retrieval architecture and the evaluation design. The evidential position is stated rather than implied. The three caveats from the top (single seed, below-threshold judge agreement, generator drift) ride every number above; the 16-to-24-item margins behind Finding 1 survive them, the small margins do not. The gold answers are source-authored and mechanically checked for grounding in the statute text they cite, but not yet reviewed by practising lawyers. No external system (LightRAG, Microsoft GraphRAG) has been run on this corpus yet, so the results say what each architecture buys relative to the others here, not how any of them compares to a published system. The hardened tier, with three-plus seeds, expert-validated gold, and external baselines, is the next phase, and it feeds the benchmark paper this project exists to support. The full report, including the sections titled “what to distrust” and “what this evaluation does not establish,” ships in the repo.
For an enterprise the translation is direct. Swap Serbian statutes for your regulated, amendment-heavy corpus (policies, contracts, compliance manuals, anything with versions and effective dates) and the decision rules transfer as-is: do the boring retrieval engineering first, count your cross-references before you build a graph, budget the agentic loop for the queries where being confidently wrong actually costs you, and put temporal validity in the knowledge base, because nothing downstream can recover it.
Run it yourself
The repo is public at https://github.com/RatkoNikolic/srb-legal-rag and small (~6 MB): the five instrumented arms, the eval harness, the 60-question golden set (CC BY 4.0, reusable in published work with attribution), 224 tests, the web UI, and six design docs that explain every decision with the literature behind it. The code is Apache-2.0. Setup instructions are in the README, and building your own knowledge base is three commands: a rate-limited crawl of the official source (about 7 minutes), a local graph build (seconds), and embedding (about $1.50 of API calls). Re-running the full benchmark needs a Gemini key plus an Anthropic key for the cross-family judge; generation alone ran me about $21, with grading on top.
One thing to know before you build, because it is unusual and deliberate: the repository ships the machinery, not the corpus. No statute text is distributed, and your rebuilt knowledge base will not byte-match the one these numbers were measured on. That is a measured fact, not a guess: a fresh rebuild produced 24,652 units against the published 25,675, with hundreds of article chunks existing only on one side. Serbian law amends continuously (the source moves under you), the annex layer needs its own paid extraction step whose vision-OCR portion is model output that varies even on identical inputs, and borderline documents flip between parseable and not. Comparing arms within your own build stays valid; comparing your numbers to mine does not. The exact knowledge base behind the published campaign, the full chunk set, the typed graph, and the vector index, is available on request (email in the repo README): it is too large to ship in the repo, and given the probabilistic construction and the drift in the source materials, handing over the measured artifact is the only honest way to let someone run the same comparison these numbers came from.
I take on a small number of advisory engagements each year for teams building retrieval systems that have to hold up in production. Reach out if that’s you.
What to read next
This series:
Article 1 — Your agent’s problem isn’t the model, it’s the knowledge layer
Article 2 — Naive RAG: the baseline you’ll always benchmark against
Article 3 — Advanced RAG: what you actually run in production
Article 4 — GraphRAG and friends: when entities and relationships beat similarity
Article 5 — Agentic RAG: when retrieval becomes a decision, not a pipeline
Article 6 — Vector vs graph vs episodic: a tour of agent memory systems (coming soon)
External:
Xiang et al. (2025), When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation. [https://arxiv.org/abs/2506.05690](https://arxiv.org/abs/2506.05690). Root-causes graph underperformance at the construction stage, the same decomposition Finding 2 runs.
Xiao et al. (2025), GraphRAG-Bench: Challenging Domain-Specific Reasoning for Evaluating Graph Retrieval-Augmented Generation. [https://arxiv.org/abs/2506.02404](https://arxiv.org/abs/2506.02404). The construction, retrieval, generation stage split and the structural stats (degree, isolated share) this benchmark reports.
Han et al. (2025), RAG vs. GraphRAG: A Systematic Evaluation and Key Insights. [https://arxiv.org/abs/2502.11371](https://arxiv.org/abs/2502.11371). The flat-versus-graph comparison precedent: measured evidence that graph RAG does not dominate flat RAG.
Fan et al. (2026), Can LLMs Time Travel? Enhancing Temporal Consistency in Legal Agentic Search through Reinforcement Learning. [https://arxiv.org/abs/2605.25920](https://arxiv.org/abs/2605.25920). Point-in-time legal retrieval as a first-class problem, the axis this benchmark could only demonstrate, not measure.
Yang et al. (EMNLP 2018), HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. [https://arxiv.org/abs/1809.09600](https://arxiv.org/abs/1809.09600). The origin of gold-context-as-ceiling: its gold-supporting-fact setting is “an upper bound on performance”, which is exactly how the oracle arm is used here.
srb-legal-rag: [https://github.com/RatkoNikolic/srb-legal-rag](https://github.com/RatkoNikolic/srb-legal-rag). The code, the question sets, and the full campaign report behind every number in this post.