Retrieval-augmented generation arrived in legal AI as the answer to hallucination. Give the model authoritative sources at query time, the story goes, and it stops making things up. The story is half true — and the missing half is where legal systems break.
The promise rests on a quiet assumption: that the right sources are actually retrieved. In the legal domain, that assumption fails more often than the generation does. The model writes fluently from whatever it is handed; the failure is in what it is handed, and in what it does at the citation boundary when it is handed nothing useful. Legal RAG does not fail at writing. It fails at finding — and at admitting when it has not found.
The hallucination floor RAG was meant to raise
Start with the problem RAG was brought in to solve. When unaided models are asked specific, verifiable questions about US case law, they fabricate between fifty-eight and eighty-eight percent of the time, and frequently cannot tell when they are doing it [1]. That is the floor. Grounding is supposed to raise it.
But grounding only raises the floor if the retrieval beneath it holds. And in law, it often does not. The realistic US legal-research benchmarks show that legal retrieval is an unsolved problem, not a deployed solution — retriever pipelines struggle on tasks built to mirror how lawyers actually research [3]. The work that isolates the retrieval step specifically, rather than scoring only the final answer, finds that step to be the weak link, the component most systems leave unmeasured [4]. RAG did not remove the failure. It moved it upstream, where it is harder to see.
The retriever pulls the wrong document — confidently
The failure has a concrete, recurring shape. In large legal corpora full of structurally similar documents, the retriever selects passages from entirely the wrong source — a failure mode named and quantified as Document-Level Retrieval Mismatch [5]. The model then builds a fluent, well-cited-looking answer on a document that does not support it. The output reads as grounded. It is not. Nothing in the generation step flags the error, because the generation step did its job — on the wrong input.
This is why “we added RAG” is not an assurance claim. A retrieval layer that confidently returns the wrong authority produces hallucinations that are harder to catch than the unaided model’s, because they arrive dressed in citations.
Closed-book, the model fabricates the authority
The citation boundary is where the failure becomes acute. Asked to supply case authorities without external grounding, legal models do not decline — they invent. On a benchmark built from a thousand real US judicial opinions, exact citation recovery sits below seven out of a hundred even for the strongest models, and Misleading Answer Rates — concrete but incorrect authorities offered with confidence — exceed ninety-four percent for twenty of twenty-one models tested [2]. Scale and legal-domain pretraining barely move it. And a prompt-only instruction to express uncertainty reduces some confident fabrication without improving citation correctness [2].
The pattern generalizes to argument generation, where models manufacture arguments even when no factual basis exists, and fail the abstention test most directly: told to stop when there is no shared basis, they keep going [6]. The throughline is the same as the retrieval failure — the system will not signal the absence of support. It fills the gap.
The failure is upstream and at the boundary, not in the prose
Put the evidence together and the diagnosis is precise. Legal AI’s fluent generation is not the problem. The problems are upstream — retrieval that returns the wrong authority — and at the boundary — a model that fabricates or misattributes a citation rather than declining. Both are invisible to anyone evaluating the polished answer, which is exactly why they are dangerous in a domain where a wrong citation is professional malpractice.
This reframes what “legal RAG” has to earn. It is not enough to retrieve something and generate over it. The system has to retrieve the right authority, cite it faithfully, and decline when it cannot — three guarantees the fluent answer does not provide on its own.
Legal RAG fails at retrieval and at the citation boundary, not at writing. The fluent, well-formatted answer is the trap: it hides a wrong-document retrieval or a fabricated authority behind prose that reads as grounded.
Adding RAG does not, by itself, make a legal system defensible. It relocates the failure to a layer most evaluations never look at — which is precisely why the next post is about the fixes that do hold.
