How to Evaluate Grounded RAG Answers with an Evidence Sheet

A RAG evaluation groundedness checklist is useful only when it leaves an inspectable record. For each test answer, split the material claims, connect every claim to an exact source passage, and write a bounded verdict with a reason. Then save the query, retrieved passages, answer, citations, source versions, and system version so the same case can be replayed later. The result is not an accuracy score or a promise that an answer is true. It is a claim-level evidence sheet showing what the visible evidence supports, where a citation is merely attached, and where the system should record an honest miss.

This article provides that record-and-replay protocol. It deliberately does not replace the broader explanation of grounded AI search, citations, and abstention. Its narrower job is to make one test answer auditable without inventing a benchmark or hiding judgment inside a pass rate.

Who made this protocol, how it was prepared, and why

Michael Shamanoff, founder of Achla AI, is the named author. The protocol was prepared from the owner-approved editorial brief, the current product artifact, and official documentation from Microsoft, Ragas, and AWS. Demand for this topic remains an experimental hypothesis because no GSC, Keyword Planner, licensed-volume, or stable autocomplete evidence was available.

The method turns a response into small evidence decisions a reviewer can reproduce. In this grounded answer evaluation, a linked source may be relevant to the topic yet fail to support the sentence beside it. The purpose remains bounded: diagnose visible retrieval and support problems in a fixed test case. Readers who need a compact definition before using the sheet can consult the AI search glossary.

Start with a fixed evidence fixture

Do not begin by assigning a score. Begin by freezing a case. A RAG evaluation test set should record the exact query, the source collection version, the retrieved passages in returned order, the generated answer, and the citations rendered to the user. Also state the expected evidence state: which source passages a careful answer would need and which requested point the collection cannot answer.

Here is a synthetic fixture used throughout this article. It describes an imaginary store; it is not an Achla AI result or a customer result.

Query Q-017: “Can I return a final-sale item after 30 days, and when does refund processing begin?”

Source D-01, `/returns`, version `returns-v4`, passage `p04`: “Items bought directly from the example store may be returned within 30 calendar days of delivery. Final-sale items are excluded.”

Source D-02, `/refunds`, version `refunds-v2`, passage `p07`: “Refund processing begins after the returned item reaches the warehouse.”

Expected evidence state: D-01 supports a 30-day window for eligible direct purchases and explicitly excludes final-sale items. D-02 supports the start event for refund processing. Neither passage states how long processing takes.

Recorded answer A-017: “You can return any item within 30 days, including final-sale items.[D-01] Your refund starts when you submit the request.[D-02] It is normally completed within five days.”

The fixture is intentionally imperfect. Its value is that every later verdict can point back to fixed text. If a team is testing vocabulary mismatch before this step, the separate guide to semantic retrieval across different wording explains why retrieval and answer generation should be examined as different stages.

Split the answer into material claims

A material claim could change a reader’s decision if it were wrong or missing. Split conjunctions, quantities, exclusions, dates, causal statements, and procedural steps when they depend on different evidence. Do not grade a whole paragraph as one unit.

For A-017, the sheet records four claims:

1. C-017-1: Any item can be returned.
2. C-017-2: The return window is within 30 days.
3. C-017-3: Refund processing starts when the request is submitted.
4. C-017-4: Refund processing is normally completed within five days.

“Including final-sale items” belongs in C-017-1 because it expands “any item” and conflicts with an explicit exclusion. The 30-day statement is separate because the passage does support that time window, subject to scope. The final duration claim is separate because neither retrieved passage discusses duration.

This decomposition prevents one supported fragment from laundering an unsupported one. It also makes disagreement visible. A reviewer can challenge the boundary of C-017-1 without silently changing the verdict for C-017-2. For scientific or technical collections, the same discipline is central to a citation-first research workflow, although domain experts may need to decide what counts as material.

Build the claim-level evidence sheet

Use one row per material claim. A workable evidence sheet needs these fields:

• case ID, query ID, answer ID, and claim ID;
• the claim text copied from the recorded answer;
• attached citation IDs exactly as rendered;
• retrieved source ID, source version, passage ID, and returned rank;
• the exact passage text used for review;
• citation attachment verdict and reason;
• semantic support verdict and written reason;
• visible-evidence diagnostic;
• reviewer, review time, and unresolved note.

The sheet for the fixture looks like this:

Keep the exact passage in the test record, not only the page URL. Pages can contain several relevant-looking sections, and a replay cannot distinguish them if the locator is vague. If a source has stable section anchors, store the URL and anchor as well as a captured passage ID. If it does not, store a content hash and enough neighboring text to relocate the passage after an edit.

Official frameworks use related separations. Microsoft’s RAG evaluator documentation distinguishes retrieval-focused evaluation from response-focused evaluation. Ragas documents faithfulness in terms of claims that can be supported by retrieved context. AWS documents separate retrieve-only and retrieve-and-generate evaluation modes. Those references help keep retrieval relevance evaluation distinct from RAG faithfulness evaluation; they do not supply a universal threshold for this sheet.

Separate citation attachment from semantic support

“Attached” answers a mechanical question: did the recorded response associate a source ID with the claim or sentence? “Supported” answers a semantic question: does the exact passage justify the claim with its important scope, exclusions, quantities, and timing intact?

Never collapse those fields. A citation support check asks both questions independently. C-017-1 has an attached citation, but the cited passage contradicts the claim. C-017-4 has no attached citation and no retrieved support. A different case might have no rendered citation even though a retrieved passage supports the claim; that would be an attachment problem, not proof of a retrieval failure.

Use four bounded semantic verdicts:

• supported: the visible passage supports the material claim and its important qualifiers;
• partially supported: it supports a separable part but leaves a material qualifier, scope, or relationship unresolved;
• unsupported: the visible passages do not justify the claim or they conflict with it;
• not assessable: the record is incomplete or the decision requires expertise or evidence outside the fixture.

Every verdict needs a written reason. The label is an index, not the analysis. “Not assessable” should identify what is missing; it must not become a convenient substitute for review.

Diagnose only what the visible record permits

The evidence sheet can expose a symptom without proving its hidden cause. Use cautious diagnostic language and keep observation separate from hypothesis.

The matrix is not an automated evaluator and does not claim which component caused an error. It narrows the next inspection. This distinction is especially useful when moving from page-finding to an answer interface; the architectural difference is described in why site search finds pages while visitors need answers.

Record an honest miss as a first-class outcome

Honest miss testing adds one case whose requested fact is absent. For Q-017, the five-day duration has no support. A bounded response could say: “The available sources say processing begins after the returned item reaches the warehouse, but they do not state a completion time.” The evidence sheet should record the supported start event and the unavailable duration separately.

For the current Achla AI artifact, the primary grounded path requires answer text together with citations before that path is treated as grounded; the widget also has a visible no-answer state and can render source links. This is a description of the inspected local implementation, not a performance or reliability claim. A miss still needs replay metadata, because a later source version may add the missing fact.

Save replay and version metadata

A replay is useful only if the team knows what stayed fixed and what changed. Store the following with each case:

• query text and query ID;
• answer text and answer ID;
• ordered retrieved passages, their scores if the system exposes them, and any filters;
• rendered citation IDs and their claim or sentence attachment;
• collection snapshot ID, source IDs, source versions, passage IDs, and content hashes;
• retrieval configuration identifier and generation configuration identifier;
• prompt or instruction version, model identifier as recorded by the system, and application revision;
• locale, test time, reviewer, and evidence-sheet schema version.

On RAG regression replay, create a new run ID. Do not overwrite the old answer or retrofit new passages into the old record. Compare claim rows only when their fixtures are compatible. If the query changed, the source snapshot changed, or a locator now resolves to different text, mark that dimension explicitly. A changed outcome may be worth investigating, but the sheet alone does not establish why it changed.

Avoid turning the protocol into a leaderboard. Do not average verdict labels into an invented score, set an arbitrary pass threshold, or publish a success rate from an unrepresentative test set. A team may aggregate results under a separately approved evaluation design, but this record is meant to preserve evidence and judgment before aggregation.

Assemble the audit packet

The smallest useful audit packet contains the frozen fixture, raw retrieval output, recorded answer, rendered citations, completed claim rows, reviewer notes, version metadata, and replay comparison. Include the evidence-sheet schema and a manifest with file hashes. If any item is unavailable, list it as missing instead of reconstructing it from memory.

For a bounded documentation corpus, keep this packet beside the operational owner of the documentation search workflow, rather than separating the evidence record from the source-version history.

Keep reviewer decisions append-only or versioned. A later reviewer may disagree with a claim split or verdict; retain both the original row and the documented revision. That history is more useful than a clean-looking sheet whose decisions cannot be traced.

Limitations

This protocol evaluates support against captured, visible evidence. It does not prove that a source is factually correct, current, complete, or appropriate for a high-stakes decision. It does not detect every relevant source that retrieval might have missed. It does not replace subject-matter review, security testing, privacy review, accessibility review, or evaluation of latency and user experience.

Claim boundaries and semantic verdicts require judgment. Different reviewers may reasonably disagree, particularly when a passage implies rather than states a relationship. Record that disagreement and the reason. Do not convert this protocol into an automated judge or universal RAG benchmark without a separately validated method.

Use the sheet on a bounded site corpus

Choose a small, versioned set of real visitor questions, include at least one known evidence gap, and preserve the audit packet for every replay. That produces an inspectable RAG evaluation groundedness checklist without claiming more than the record shows.

If you want to test the workflow on content you control, Connect your site and keep factual, security, and publication decisions with a human reviewer.