How to Evaluate RAG Without Fooling Yourself

September 27, 2026

6 min read

How to Evaluate RAG Without Fooling Yourself

A reader asks a health assistant, “What is AF?” The system retrieves a page about atrial fibrillation and writes a fluent answer with a citation. It looks convincing. Then someone follows the citation and notices that the answer includes a claim the page never made.

The search found a relevant page, but the answer still failed. If I score only the final response, I may miss why. If I score only search, I may declare success too early. I need to see the question, the passages returned, the passages sent to the model, the claims it wrote, and the sources it cited.

The search route is explained in the vector database guide. Here I use LumenTrail, a small retrieval lab with a runnable notebook, to examine the results. Its first five passages are synthetic, so the mechanics are easy to inspect. They are not a medical benchmark.

Write questions before tuning the system

I start by collecting questions people might actually type. A set made only of tidy questions copied from article headings will flatter the search system. I include everyday wording, acronyms, spelling errors, exact names, negation, dates, and questions that require more than one passage. I also add questions for which the indexed material has no answer.

Each question needs a judgement about what evidence would count. Sometimes several passages are acceptable; sometimes two passages are both required. I keep those cases separate. A reviewer should be able to open the source and explain why a passage is relevant. If reviewers disagree, I record the disagreement rather than pretending there is one obvious label.

Public datasets help, but I use them carefully. MedQuAD contains historical question and answer pairs gathered from NIH sites. Its source also notes that three MedlinePlus subsets omit answers for copyright reasons. I do not assume that an old answer is current advice. HealthSearchQA offers 3,173 consumer worded health questions. It does not give gold passages for my index, so I must have people review the evidence before using those questions to calculate retrieval scores.

LumenTrail keeps a reviewed retrieval question in a small JSONL record:

case = {
    "question": "What is AF?",
    "relevant_ids": ["demo-af"],
}

That ID refers to a synthetic teaching passage. In a real dataset I also keep the source version, review date and reviewer decision outside the small teaching format. I split development and holdout questions by source or entity where possible. If nearly identical MedQuAD pairs appear on both sides, the holdout can tell me little about new questions.

Find out whether the evidence was retrieved

Before asking a language model to write anything, I compare the ranked search results with the reviewed passage IDs. Three measures answer different questions:

Measure	What I learn	A limit to remember
Recall at k	How much of the relevant evidence appeared in the first k results?	A passage can be present and still be ignored later.
Precision at k	How much of that shortlist was relevant?	It does not tell me whether one missing passage was critical.
Mean reciprocal rank	How early did the first relevant passage appear?	It rewards the first hit, not complete evidence.

When there are several required passages, recall matters more than a single early hit. For questions where one passage is enough, reciprocal rank can show whether the useful page is near the top. If relevance has grades such as “partly useful” and “fully answers”, I can add a graded ranking measure such as nDCG. I keep the judgement rules fixed before comparing systems.

Precision at k here counts passages with reviewed relevant IDs in the first k results. A framework’s context precision may instead ask a model to judge whether the supplied passages are useful and well ranked. Those are related questions with different evidence and different failure modes. I name the metric, labels and judge in every report so that a score from one method is not quietly compared with another.

The core of LumenTrail’s recall calculation is deliberately plain:

retrieved = [hit["id"] for hit in hits]
relevant = set(case["relevant_ids"])
found = relevant.intersection(retrieved)
recall = len(found) / len(relevant)

Run the offline example from the project root:

lumentrail init
lumentrail eval --method keyword --limit 2
lumentrail eval --method dense --limit 2
lumentrail eval --method hybrid --limit 2

All three methods find the labelled passage within two results on this tiny set. That is useful as a smoke test. The offline vectors are hashes of words, not semantic embeddings, and the questions are known in advance. A perfect score here gives me no reason to claim that a real system is accurate or safe.

For a stronger comparison, I import a public corpus, review a holdout set, build a real embedding index, and run every retrieval method over the same source version, filters and k. I record latency with recall. Otherwise a change in the corpus or context size may look like a model improvement.

I keep no-answer questions in a separate set. They have no relevant passage ID, so dropping them into a recall calculation would hide the behaviour I care about. I test whether the final system admits that the approved sources do not answer them, and whether a tempting but unrelated hit makes it invent an answer. LumenTrail’s retrieval report requires at least one reviewed relevant ID; answer abstention needs an additional answer-level check.

Read the misses, not just the average

A single average hides the cases I most need to fix. I keep a small report of failed questions and inspect each search trace. I ask:

Was the answer present in the source collection at all?
Did cleaning or chunking remove the key sentence or its warning?
Did keyword search miss a paraphrase, or did dense search miss an acronym?
Did a tenant or date filter remove the right passage?
Was the right passage retrieved but pushed below the model’s context limit?
Was the review label wrong or incomplete?

The fix depends on the cause. If a passage was never ingested, changing HNSW settings will not help. If a rare drug name is missing from dense results, keyword search or a better candidate pool may help. If a chunk contains the right answer but lacks the exception beside it, I fix the chunk boundary. I rerun the same labelled questions after each change and keep the misses visible.

Check what the answer actually says

Once retrieval is good enough to supply evidence, I review the answer claim by claim. A claim is grounded when the cited context supports that specific statement. A citation is precise when the linked passage supports the sentence beside it. A citation to a related page is not enough.

The distinction is easy to see with a synthetic example. Suppose the only passage says, “AF is an irregular heart rhythm.” An answer that repeats that statement is supported by the passage. If it adds a claim about how long AF lasts or what treatment a person needs, that extra claim is unsupported by this passage, even if the citation points to the right page.

I track at least four answer outcomes: supported, unsupported, incomplete, and correctly abstained. I also count an answer as a failure when it confidently responds to a question with no evidence in the approved sources. For health uses, I add expert review for dose, interaction, allergy, pregnancy and urgent symptom cases. A model judged groundedness score cannot approve clinical advice.

Ragas and DeepEval include measures for retrieved context and answer faithfulness. Evidently can help organise evaluations and monitoring. They can make review easier to scale, but I first compare judge results with human reviews on my own examples. A judge model may accept a plausible claim that the source does not support, especially when the wording is subtle. LumenTrail’s offline runner measures labelled retrieval only; it does not pretend to calculate groundedness without generated answers and reviewed claims.

Test the boundaries around search

The answer is only one part of the service. A system can have high retrieval recall and still leak another person’s note or keep a deleted page in its index. I keep separate tests for the surrounding behaviour:

Access: a user must not retrieve a passage from another tenant, however close its vector is.
Freshness: when a source changes or is withdrawn, the old version must stop appearing within the promised update window.
Memory: a dated journal note should appear only for its owner and only while it is valid. LumenTrail’s memory example uses synthetic local notes, not patient records.
Graph expansion: following an entity link can find a missing passage, but it can also add unrelated neighbours. I compare the expanded result with ordinary retrieval.
Prompt injection: a retrieved page might tell Codex, Claude Code or the answer model to ignore instructions. Retrieved text remains data, not authority.
Operations: I watch query latency, model cost, index build time, deletion time and the effect of changing embedding models.

I test permission changes and deletions end to end. It is not enough to hide a forbidden passage after retrieval if it was already placed in a model prompt or a log.

Keep a scorecard that can survive a release

My release report separates retrieval, answer quality and system behaviour. It shows the question count and corpus version beside each score, then breaks failures into slices such as acronyms, no-answer questions, stale sources and access boundaries. I include p95 latency and cost alongside quality. A single blended score makes trade-offs harder to see.

For each run I save the index and embedding versions, retrieval method, reranker, filters, k, prompt version and evaluation date. That lets me reproduce a change and see whether a better answer came from improved search, a larger context window or a changed model. I keep private text out of shared reports and set a retention period for traces.

I agree on the release thresholds with the people who own the product risk. I keep a stable holdout set for comparison and add new cases from real failures to a growing regression set. I do not rewrite the holdout after every experiment to make a new model look better. For sensitive questions, I review traces with the appropriate domain experts. Logs keep source IDs and versions; private question text needs its own retention and access policy.

The next useful step is small: take ten questions your users care about, label the passages that would answer them, and compare keyword, dense and hybrid results. Then inspect the answer claims and citations for those same questions. The resulting failure list will tell you what to build next far more clearly than a single impressive demo.