How to Evaluate RAG Without Fooling Yourself

November 01, 2026

4 min read

How to Evaluate RAG Without Fooling Yourself

A search system finds the right page. An AI model writes a fluent answer with a citation. Then someone opens the page and notices that one of the answer’s claims is not there.

Search succeeded. The answer failed. I want an evaluation that can tell those two things apart.

RAG, or retrieval-augmented generation, means finding source material and giving it to a model before it writes an answer. I check the search, the answer and the behaviour around them separately. LumenTrail is a small lab for the first part. It retrieves passages; it does not write answers.

Get a result you can inspect

You need Python 3.11 or newer, about five minutes and no account. Estimated service cost: £0 on your laptop. These starting passages and questions are synthetic.

git clone https://github.com/mrjhonyvidal/lumentrail.git
cd lumentrail
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e .
lumentrail init
lumentrail eval --method keyword --limit 2
lumentrail eval --method hybrid --limit 2

Look at cases, which lists the expected and retrieved passage IDs. Both methods find the labelled passage within two results on these three questions. That checks the wiring. The default dense vectors hash words and are not semantic embeddings. A perfect score here says nothing about clinical quality or a real search service.

The README covers manual setup and optional models. The search article explains the retrieval methods.

Write the questions before tuning

I start with questions people might actually ask, including awkward wording, acronyms, exact names, dates and negation. I also keep questions that the approved material cannot answer.

For each answerable question, a reviewer records the passage IDs that would supply the evidence. Several passages might be alternatives, or two might both be required. Those are different cases, so I document which judgement I mean.

The lab’s small record looks like this:

case = {
    "question": "What is AF?",
    "relevant_ids": ["demo-af"],
}

A real evaluation also needs the source version, review date and a way to resolve disagreement. Keep development and holdout cases separate. Nearly identical questions on both sides make an apparent improvement less convincing.

Public datasets are starting material, not automatic truth. MedQuAD preserves historical question and answer pairs and source links; some subsets intentionally lack answer text. HealthSearchQA supplies consumer-worded questions, but it does not identify the correct passages in your current index. Those require review.

Measure what the search found

Three measures in the lab answer different questions:

Measure	Plain meaning	What it misses
Recall at k	How much of the labelled evidence was in the first k results?	Whether the model used it
Precision at k	How much of those k places held relevant evidence?	Whether one missing passage was essential
Mean reciprocal rank	How early was the first relevant passage?	Whether all required evidence appeared

Here, k is the requested shortlist length. The lab divides precision by that length, even if fewer hits come back. That makes the convention explicit rather than quietly changing the denominator.

The recall calculation is small enough to read directly:

from contextlib import closing
from pathlib import Path
from lumentrail.embedding import HashEmbedder
from lumentrail.evaluation import evaluate
from lumentrail.storage import connect

with closing(connect(Path("data/lumentrail.db"))) as database:
    report = evaluate(database, [case], HashEmbedder(), method="keyword", limit=2)
    print(report["recall_at_k"])

For the synthetic case above, this prints 1.0. It does not test a generated answer. Measures called “context precision” in other tools may use model judgements rather than reviewed IDs, so record the definition before comparing numbers.

Keep no-answer cases in their own check. There is no relevant passage to divide by. LumenTrail requires at least one reviewed relevant ID for its retrieval report; it does not invent a recall score for an unanswerable question.

Read the misses before changing the model

When a question fails, I look at the full path. Was the evidence present in the source? Did cleaning remove it? Did chunking separate a warning from the statement it qualified? Did a filter exclude the right passage? Did the result fall below the context limit?

Each failure suggests different work. A page that was never ingested will not be repaired by a new embedding model. A rare exact name missed by dense search might benefit from keywords. A bad relevance label needs review, not model tuning.

Change one part and rerun the same cases. Keep corpus, filters and shortlist length fixed so the comparison means something. Add latency and cost when you move beyond the local baseline.

Review the claims in the answer

Once a model writes answers, take the same questions and read each claim against the supplied sources. I would use four outcomes to start: supported, unsupported, incomplete and correctly declined because the evidence was missing.

A citation beside a sentence does not establish support. If a passage says only that AF is an irregular rhythm, it cannot support an added treatment recommendation. Health examples need qualified domain review; retrieval scores cannot establish medical safety.

Check whether the model cites the passage that supports the specific claim, omits an important qualification or answers a question absent from the approved sources. Record those failures separately from search recall.

Tools such as Ragas can help organise answer and context checks. If a model judges the answers, compare its decisions with human reviews on your own cases first. Fluent wording can fool a judge too. LumenTrail does not calculate answer groundedness because it has neither generated answers nor reviewed answer claims.

Test the system around the answer

I keep a few checks outside the quality averages: one user’s material never reaches another user, deleted sources stop appearing, changed permissions take effect before context is built, and untrusted source text cannot authorise a tool action.

A stale cache or an overbroad permission filter can defeat a strong retrieval score. Test those paths end to end. LumenTrail’s public-data index has no tenant permission model. Its optional dated notes use synthetic data and remain separate from the shared corpus.

For a release comparison, save the corpus and model versions, prompt, filters, shortlist size and evaluation date. Show the number of cases and the failures beside each average. Keep private text out of shared traces unless there is a deliberate access and retention policy.

A useful first session is ten questions, reviewed passage IDs and a list of misses. Add answer review only when you add generation. That gives the team a specific next change and a way to tell whether it helped.