Evaluating RAG: retrieval, groundedness and completeness are different tests

A working demo is not an evaluation plan

Retrieval-augmented generation links document search with response generation. A successful demonstration can show that the pieces connect, but it does not establish how reliably the application answers the questions its readers actually ask. Microsoft describes several evaluation dimensions rather than one universal measure: retrieval, document retrieval, groundedness, relevance and response completeness. Their separation matters because a system can succeed at one stage and fail at another. A fluent answer is especially easy to overvalue when nobody records which evidence was found and which requested details were omitted.[1]

Here the question is how to evaluate that workflow, rather than repeat the journal's introductory architecture overview. Its focus is how to diagnose outcomes, not how to introduce retrieval itself. The suggested practices are an evaluation framework, not measured results or a claim that one product is best. Start by defining the application task and the mistakes that would make its answers unhelpful. Then preserve enough information to inspect the search and response independently. If an evaluation reports only a single average score, ask which behaviors contributed to it and whether important failures can disappear inside that summary.[1]

Retrieval quality comes before answer quality

Microsoft treats retrieval as an upstream dependency: missing evidence makes satisfactory corpus-specific answers less likely. Retrieval evaluation therefore asks whether the selected context is relevant to the question and whether useful material appears prominently. This is not identical to asking whether the response sounds good. A system may retrieve an obsolete document, an unrelated section or several near-duplicates, and then produce a plausible paragraph from the inadequate context. The answer stage cannot be evaluated fairly without knowing what it was given.[1]

Imagine an assistant answering a question about the current reimbursement policy. If the search returns a superseded policy, the failure begins before generation. A suggested diagnostic record includes the query, retrieved document identifiers, document versions and the actual context passed to the model. Where access restrictions apply, those records also require appropriate protection. This recommendation is not a statement that Microsoft's evaluator automatically verifies policy freshness or authorization. It is a way to make the application's search behavior inspectable so that a bad result can be traced to a specific stage instead of vaguely blamed on AI.[1]

Ground truth gives document retrieval a clearer target

Microsoft separates labeled document-retrieval evaluation from model-assisted assessments that do not use those relevance labels. A labeled dataset identifies which documents are useful for particular questions, enabling information-retrieval metrics that compare returned material with a known reference. Such labels still require careful construction. A label represents an evaluation judgment about a task, not an unquestionable universal fact. If the corpus changes or the question is ambiguous, a previously useful document can stop being the right target. The dataset's scope and update process belong beside the score.[1]

Build an initial query set with ordinary questions, difficult wording variants and cases requiring several documents. For each case, record why a document is considered relevant and whether other documents are also acceptable. Do not label only the result your current retriever already returns; that would make the test circular. Microsoft lists metrics such as ranking quality and retrieval coverage, but a team still needs to decide which trade-off matters for its application. A narrow factual lookup and a broad comparison may need different evidence patterns. Keep those task differences visible rather than forcing every query into one success definition.[1]

Groundedness asks whether the answer stays with its evidence

In Microsoft's framework, groundedness measures alignment with the context supplied to the response generator. This catches an important failure: the model adds claims that the provided evidence does not support. However, support within retrieved text is not identical to independent verification that the source is correct, current or suitable. A response can faithfully repeat a mistaken document. The evaluator's object is the relationship between response and context. It should therefore be read alongside source quality and retrieval checks rather than treated as a comprehensive truth detector.[1]

A hypothetical support answer might wrongly claim a feature exists on every plan when the evidence discusses only one. That extra generalization can be a groundedness problem even if the prose is confident. A suggested human review breaks important responses into claims and checks each against the actual supplied passage. Record where the answer qualifies uncertainty or invents a missing detail. This is not a requirement that every word be copied from the source. The useful distinction is between a reasonable paraphrase supported by evidence and an unsupported extension that changes the meaning.[1]

Completeness asks what the answer left out

Microsoft presents completeness as a recall-oriented counterpart to groundedness. Given an expected response or ground truth, it examines whether the generated answer includes critical information. An answer can contain nothing unsupported and still fail the user's task because it omits a crucial condition. This is why simply instructing a model to be brief and evidence-bound does not establish useful coverage. A response that says too little may avoid fabrication while remaining practically misleading. Evaluate the necessary content separately from whether each included claim is supported.[1]

Suppose an otherwise accurate setup answer names a supported device but omits the adapter required by the evidence. The answer may be grounded, yet incomplete for the requested setup. A suggested rubric identifies indispensable facts before reviewing the model's wording. Allow different valid phrasing, but do not let superficial similarity stand in for coverage. Ground-truth answers also need review: an incomplete reference can reward an incomplete generated response. This article does not supply a universal completeness threshold. Define expectations in the application's context and inspect the cases that fall near any chosen acceptance boundary.[1]

Relevance is about the user's actual request

Microsoft's relevance evaluator asks how effectively the response addresses the question. This dimension is distinct from the response's relationship to context. A system can retrieve accurate material and produce a well-supported answer to a nearby question instead of the one asked. For example, a hypothetical question requesting a comparison might receive a general explanation of only one option. The content can be true and supported while failing the requested decision. Relevance brings the user's intent back into the evaluation rather than treating any factual paragraph as success.[1]

Keep the original question and the expected answer form, whether that is an explanation, comparison, procedure or admission of insufficient evidence. Include queries with constraints, such as a particular version or operating condition. During review, ask whether the response respected those constraints and whether it silently substituted an easier task. These are task-oriented recommendations, not guarantees provided by an evaluator API. Automated scoring is most useful when its definition is explicit. If reviewers disagree on what relevance means, resolve the rubric before interpreting the resulting numbers as a stable benchmark.[1]

Model-based judges require calibration

Microsoft documents model-based evaluators that return scores, explanations and threshold-based outcomes. Those outputs are useful diagnostics, but the score is a judgment produced under a particular configuration. A suggested evaluation record therefore includes the judge model, rubric, threshold and the relevant test inputs. Do not present a model-based score as though it were a direct physical measurement. If the judge or prompt changes between runs, investigate whether observed changes reflect the application or the evaluation procedure itself.[1]

Check automated judgments against a manageable set of carefully reviewed examples. Inspect false approvals and false rejections, especially for ambiguous questions, conflicting documents and incomplete responses. A suggested practice is to read the reason field rather than looking only at the numeric result. The explanation may reveal that the judge considered the wrong evidence or rewarded surface fluency. This does not make model-assisted evaluation useless; it makes its limits visible. Keep human review proportionate to the application's consequences, and do not use an arbitrary threshold as a substitute for deciding which errors are acceptable.[1]

Keep the benchmark reproducible

Version the evaluation definition, including questions, corpus, relevance labels, answer rubric and judge configuration. Microsoft distinguishes labeled document-retrieval metrics from model-assisted judgments, so those methods should not be mixed without explanation. A release note records what changed between runs and which comparisons remain valid. Protect any private source passages within the evaluation records. Reproducibility does not require making sensitive material public; it requires a controlled way for authorized reviewers to reconstruct why an outcome received its judgment.[1]

Compare changes without losing the failure cases

Microsoft's retrieval guidance supports comparing settings such as chunking, retrieval and ranking behavior. Change one important factor at a time where practical and retain the per-question outcomes. Averages help summarize results, but they can conceal regressions for a small yet important task category. A suggested release review includes representative failures alongside aggregate measures and notes whether the corpus, generation model or judge changed. That makes improvements easier to interpret and avoids attributing every difference to the one setting the team hoped would help.[1]

Test questions the corpus cannot answer, as well as questions with conflicting document versions. The application should have a defined response to those situations rather than filling gaps with plausible invention. Evaluate that behavior using the same task-specific care as ordinary answers. The final lesson is to separate questions: did retrieval find suitable evidence, did the response stay supported, did it address the request and did it include what mattered? No single dimension settles all four. A disciplined RAG evaluation makes these distinctions visible, so an apparently successful answer can be improved for the right reason rather than accepted because it reads smoothly.[1]