RAG Evaluation Metrics: Faithfulness, Relevance, Recall, and RAGAS

RAG evaluation metrics answer a question that a demo cannot: did the retrieval-augmented generation system produce a useful answer for the right reasons? A response can sound fluent and still cite the wrong document, miss the key evidence, or invent a claim that never appeared in the retrieved context.

A reliable evaluation process separates those failures. It measures retrieval quality, answer grounding, relevance, latency, token usage, and cost on the same evaluation dataset. That makes it possible to compare a prompt change, a retriever change, or a model change without relying on a handful of subjective examples.

This guide explains the four metrics most teams start with: faithfulness, answer relevancy, context precision, and context recall. It then builds a runnable Python evaluation loop that calls multiple models through an OpenAI-compatible API, calculates deterministic checks, records latency and tokens, and produces a comparison table. Ragas is included as an evaluation framework for LLM-judged metrics, but the core dataset and API loop remain understandable without hiding everything behind a library.

Use one OpenAI-compatible client for the API layer and compare the three model routes with identical cases. The quality scores are workload-specific, so the useful result is a measured route choice for your data.

Verification status: The article follows the current Ragas stable documentation and public OurToken model pages checked on 2026-08-05. The API examples are documentation-verified and require your own API key for live execution. Model prices and provider behavior can change, so re-check the live model pages before budgeting.

What Are RAG Evaluation Metrics?

A RAG pipeline has at least two quality surfaces:

Question
  -> retriever
  -> ranked contexts
  -> prompt builder
  -> language model
  -> answer with optional citations

The retriever can fail before the model ever sees the question. It might return a relevant document at rank 8 but fill the top three slots with unrelated chunks. The model can then fail even with good context by ignoring the evidence, answering a different question, or adding unsupported details.

That is why one score is not enough. A production evaluation should connect each metric to a failure mode:

The best RAG evaluation dataset contains real user questions, expected answers or reference claims where available, and the contexts your retriever actually returned. Do not evaluate only the final answer. Save the intermediate retrieval result so you can distinguish a retrieval failure from a generation failure.

Design the evaluation dataset

Start with a small, labeled dataset rather than a huge collection of random prompts. Fifty to 200 representative questions is enough to identify major regressions. Include easy lookups, ambiguous questions, multi-hop questions, questions with no answer in the corpus, and questions that should trigger a safe refusal.

A practical JSON Lines file named rag_eval_cases.jsonl looks like this. The sample includes retrieved contexts so the harness can run immediately:

{"id":"billing-001","question":"How long can an invoice remain unpaid?","reference_answer":"Invoices remain open for 30 days before escalation.","reference_claims":["Invoices remain open for 30 days before escalation."],"expected_source_ids":["billing-policy-v3"],"contexts":[{"source_id":"billing-policy-v3","text":"Invoices remain open for 30 days before escalation.","is_relevant":true}]}
{"id":"security-002","question":"What should I do after losing a security key?","reference_answer":"Report it to security and revoke the key immediately.","reference_claims":["The key should be revoked immediately.","The loss should be reported to security."],"expected_source_ids":["security-runbook-v2"],"contexts":[{"source_id":"security-runbook-v2","text":"Report a lost security key to security and revoke it immediately.","is_relevant":true}]}

Keep the original retrieved contexts with each case or in a joined result file. The evaluation set describes what should happen; the run output describes what actually happened. That separation lets you compare model routes without rewriting the test cases.

Separate retrieval and generation checks

For retrieval evaluation, inspect the ranked context IDs before calling the model. If the expected source is absent, the generation model cannot be judged fairly for that example. Mark the case as a retrieval miss, then report generation metrics on the cases that contain sufficient evidence as well as on the full set.

For generation evaluation, pass the retrieved context and the question to each model with the same prompt template. Keep model temperature, output limit, citation format, and system instructions fixed during a comparison. Otherwise, you are measuring several changes at once.

A good run record contains:

case_id, model_id, retrieved_source_ids, answer, citations,
faithfulness, answer_relevancy, context_precision, context_recall,
uncached_input_tokens, cached_input_tokens, cache_write_tokens, output_tokens, latency_ms, error, prompt_version

The Four Core RAGAS Metrics

Ragas documents a set of metrics for evaluating LLM applications, including Faithfulness, Context Precision, Context Recall, and Response Relevancy. The names describe different parts of the pipeline, but the exact score should always be interpreted alongside the examples that produced it.

Faithfulness and the RAG faithfulness score

Faithfulness measures whether the claims in the generated answer are supported by the retrieved context. A high faithfulness score does not mean the answer is useful. The model can produce a perfectly grounded answer that fails to address the question. It means the answer is less likely to contain claims that the supplied context cannot support.

Conceptually:

faithfulness = supported_answer_claims / answer_claims

The denominator is not always trivial. A single sentence may contain multiple claims, and an LLM judge may disagree about how to split them. Use the score as a trend and investigate representative failures rather than treating 0.83 as a universal truth.

Example: If the context says “invoices remain open for 30 days” and the answer says “invoices remain open for 30 days and late fees are always waived,” the second claim lowers faithfulness unless the context also supports the fee policy.

Ragas provides a dedicated Faithfulness metric reference. It is especially useful for policy, support, compliance, and enterprise search systems where unsupported claims are more dangerous than short answers.

Answer relevancy

Answer relevancy measures whether the answer addresses the user’s question. It is different from correctness and different from faithfulness. A response can be relevant but wrong, or correct but irrelevant to what the user asked.

Example: For “How do I rotate an API key?”, a response explaining token pricing may be factually accurate but has poor answer relevancy. For “Which documents are required for onboarding?”, a response listing unrelated account settings is a relevance failure even if those settings exist in the retrieved context.

Use answer relevancy to detect prompt templates that produce long generic summaries instead of direct answers. It is also useful when comparing models with different verbosity: a longer response is not automatically more relevant.

In current Ragas documentation, the metric is presented as Response Relevancy. Searchers may still use rag answer relevancy and answer relevancy evaluation, so cover both terms in the article and map them to the current library terminology when implementing.

Context precision

Context precision measures whether relevant context appears near the top of the retrieved ranking. It focuses on ordering, not only on whether the right document appears somewhere in the result set.

A retriever that returns the correct source at rank 10 may have acceptable recall but poor precision for a top-3 prompt. The model sees irrelevant chunks first, consumes more tokens, and has to work harder to find the evidence. That can lower answer quality and increase cost.

The Ragas Context Precision reference covers the metric definition and implementation context. In your own evaluator, make the cutoff explicit: precision@3, precision@5, or the exact number of chunks injected into the prompt.

Context recall

Context recall measures whether the retrieved context contains the information required to answer the question, usually against a reference answer or reference claims. It is a coverage metric.

A low context recall score often points to chunking, metadata filters, query rewriting, embedding quality, or index freshness. Increasing the number of retrieved chunks can improve recall while damaging precision and increasing prompt cost, so report both metrics together.

The Ragas Context Recall reference is the source for the metric-specific behavior. In practice, store the expected source IDs or claims in your test set so a retrieval change can be explained rather than only observed as a score movement.

Interpreting metrics together

The four metrics become useful when read as a matrix:

Do not optimize one metric in isolation. A higher top-k may increase context recall while lowering context precision and increasing token cost. The goal is the best quality-cost tradeoff for the product, not the highest score on a single chart.

RAG Evaluation Metrics Python Example

The following evaluator is deliberately self-contained. It calls an OpenAI-compatible Chat Completions endpoint, runs the same cases through multiple model IDs, applies deterministic claim and retrieval checks, and records tokens, latency, and cost. It is a baseline harness, not a replacement for human review or an LLM-judge framework.

Install the dependencies:

pip install openai

Create an API key through the OurToken API Keys page, then set the environment variables:

export OURTOKEN_API_KEY="paste-your-key-locally"
export OURTOKEN_BASE_URL="https://api.ourtoken.ai/v1"

On Windows PowerShell:

$env:OURTOKEN_API_KEY = "paste-your-key-locally"
$env:OURTOKEN_BASE_URL = "https://api.ourtoken.ai/v1"

Multi-model API evaluation harness

The script below assumes each evaluation case already contains contexts, reference_claims, and expected_source_ids. Your retriever should populate contexts before this function runs.

import json
import os
import time
from dataclasses import dataclass
from typing import Any

from openai import OpenAI


@dataclass(frozen=True)
class ModelRoute:
    model_id: str
    input_per_million: float
    output_per_million: float
    cached_input_per_million: float = 0.0
    cache_write_per_million: float = 0.0


ROUTES = {
    "gpt-5.6-terra": ModelRoute("gpt-5.6-terra", 0.40, 2.40, cached_input_per_million=0.04, cache_write_per_million=0.50),
    "deepseek-v4-pro": ModelRoute("deepseek-v4-pro", 0.3480, 0.6960, cached_input_per_million=0.0030),
    "glm-5.2": ModelRoute("glm-5.2", 0.8400, 2.6400, cached_input_per_million=0.1560),
}

client = OpenAI(
    api_key=os.environ["OURTOKEN_API_KEY"],
    base_url=os.getenv("OURTOKEN_BASE_URL", "https://api.ourtoken.ai/v1"),
)


def usage_breakdown(usage: Any) -> dict[str, int]:
    data = usage.model_dump() if hasattr(usage, "model_dump") else dict(usage)
    prompt_tokens = int(data.get("prompt_tokens", data.get("input_tokens", 0)) or 0)
    completion_tokens = int(
        data.get("completion_tokens", data.get("output_tokens", 0)) or 0
    )
    details = data.get("prompt_tokens_details") or data.get("input_token_details") or {}
    cached_tokens = int(details.get("cached_tokens", 0) or 0)
    cache_write_tokens = int(details.get("cache_write_tokens", 0) or 0)
    uncached_tokens = max(prompt_tokens - cached_tokens - cache_write_tokens, 0)
    return {
        "uncached_input_tokens": uncached_tokens,
        "cached_input_tokens": cached_tokens,
        "cache_write_tokens": cache_write_tokens,
        "output_tokens": completion_tokens,
    }


def token_cost(usage: Any, route: ModelRoute) -> float:
    tokens = usage_breakdown(usage)
    return (
        tokens["uncached_input_tokens"] / 1_000_000 * route.input_per_million
        + tokens["cached_input_tokens"] / 1_000_000 * route.cached_input_per_million
        + tokens["cache_write_tokens"] / 1_000_000 * route.cache_write_per_million
        + tokens["output_tokens"] / 1_000_000 * route.output_per_million
    )


def normalize_claim(text: str) -> set[str]:
    words = ''.join(char.lower() if char.isalnum() else ' ' for char in text)
    return {word for word in words.split() if len(word) > 3}


def lexical_claim_support(answer: str, contexts: list[str]) -> float:
    answer_words = normalize_claim(answer)
    context_words = normalize_claim(' '.join(contexts))
    if not answer_words:
        return 0.0
    return len(answer_words & context_words) / len(answer_words)


def source_recall(expected_ids: list[str], retrieved_ids: list[str]) -> float:
    expected = set(expected_ids)
    if not expected:
        return 1.0
    return len(expected & set(retrieved_ids)) / len(expected)


def answer_relevancy(answer: str, question: str) -> float:
    answer_words = normalize_claim(answer)
    question_words = normalize_claim(question)
    if not question_words:
        return 0.0
    return min(len(answer_words & question_words) / len(question_words), 1.0)


def context_precision(contexts: list[dict[str, Any]], top_k: int) -> float:
    ranked = contexts[:top_k]
    if not ranked:
        return 0.0
    relevant_seen = 0
    precision_sum = 0.0
    for rank, item in enumerate(ranked, start=1):
        if item.get("is_relevant", False):
            relevant_seen += 1
            precision_sum += relevant_seen / rank
    total_relevant = sum(item.get("is_relevant", False) for item in contexts)
    if total_relevant == 0:
        return 0.0
    return precision_sum / min(total_relevant, top_k)


def run_case(case: dict[str, Any], route: ModelRoute) -> dict[str, Any]:
    context_text = '\n\n'.join(
        f"[{item['source_id']}] {item['text']}" for item in case["contexts"]
    )
    prompt = (
        "Answer the question only from the supplied context. "
        "If the context is insufficient, say so. Keep the answer concise.\n\n"
        f"Question: {case['question']}\n\nContext:\n{context_text}"
    )

    started = time.perf_counter()
    try:
        response = client.chat.completions.create(
            model=route.model_id,
            messages=[
                {
                    "role": "system",
                    "content": "You are a grounded RAG evaluation assistant.",
                },
                {"role": "user", "content": prompt},
            ],
            max_tokens=350,
        )
        latency_ms = round((time.perf_counter() - started) * 1000, 2)
        answer = response.choices[0].message.content or ""
        retrieved_ids = [item["source_id"] for item in case["contexts"]]
        usage = response.usage

        tokens = usage_breakdown(usage)
        return {
            "case_id": case["id"],
            "model_id": route.model_id,
            "answer": answer,
            "faithfulness_proxy": round(
                lexical_claim_support(answer, [item["text"] for item in case["contexts"]]),
                4,
            ),
            "answer_relevancy_proxy": round(
                answer_relevancy(answer, case["question"]), 4
            ),
            "context_recall": round(
                source_recall(case["expected_source_ids"], retrieved_ids), 4
            ),
            "context_precision": round(
                context_precision(case["contexts"], top_k=min(3, len(case["contexts"]))), 4
            ),
            "uncached_input_tokens": tokens["uncached_input_tokens"],
            "cached_input_tokens": tokens["cached_input_tokens"],
            "cache_write_tokens": tokens["cache_write_tokens"],
            "output_tokens": tokens["output_tokens"],
            "cost_usd": round(token_cost(usage, route), 8),
            "latency_ms": latency_ms,
            "error": None,
        }
    except Exception as error:
        return {
            "case_id": case["id"],
            "model_id": route.model_id,
            "answer": "",
            "error": str(error),
            "latency_ms": round((time.perf_counter() - started) * 1000, 2),
        }


with open("rag_eval_cases.jsonl", encoding="utf-8") as file:
    cases = [json.loads(line) for line in file if line.strip()]

results = [run_case(case, route) for route in ROUTES.values() for case in cases]

with open("rag_eval_results.json", "w", encoding="utf-8") as file:
    json.dump(results, file, ensure_ascii=False, indent=2)

print(json.dumps(results, ensure_ascii=False, indent=2))

The script labels lexical scores as _proxy on purpose. Word overlap is a useful smoke signal, but it is not a complete semantic evaluation. A high lexical overlap can still be wrong, and a correct paraphrase can have low overlap. Use these proxies to catch obvious regressions, then add Ragas metrics and human review for decisions that matter.

Add context precision

Context precision needs relevance labels for retrieved chunks. Add a boolean such as is_relevant to each context after annotation or a separate relevance judge. Then compute precision at the number of chunks your prompt actually sends:

def context_precision(contexts: list[dict], top_k: int) -> float:
    ranked = contexts[:top_k]
    if not ranked:
        return 0.0
    relevant_seen = 0
    precision_sum = 0.0
    for rank, item in enumerate(ranked, start=1):
        if item.get("is_relevant", False):
            relevant_seen += 1
            precision_sum += relevant_seen / rank
    total_relevant = sum(item.get("is_relevant", False) for item in contexts)
    if total_relevant == 0:
        return 0.0
    return precision_sum / min(total_relevant, top_k)

Use the same top_k for every model comparison. If one run injects three chunks and another injects eight, you changed both retrieval and model context cost.

Compare Models by Quality, Latency, and Cost

A multi-model RAG evaluation should compare the same cases, prompt template, retrieval output, output limit, and validation rules. Only then can you attribute changes to the model route.

The three OurToken routes in the sample harness have different price profiles:

Prices are current page values checked for this draft and are not permanent quotes. Re-check the live model pages before using them in a budget.

Aggregate results by model

After the script writes rag_eval_results.json, aggregate by model and report both means and distributions. A mean faithfulness score alone can hide ten catastrophic failures behind ninety good answers.

from collections import defaultdict
import json

with open("rag_eval_results.json", encoding="utf-8") as file:
    rows = json.load(file)

by_model = defaultdict(list)
for row in rows:
    if not row.get("error"):
        by_model[row["model_id"]].append(row)

for model_id, model_rows in by_model.items():
    count = len(model_rows)
    print(
        model_id,
        {
            "cases": count,
            "avg_faithfulness_proxy": round(
                sum(row["faithfulness_proxy"] for row in model_rows) / count, 4
            ),
            "avg_relevancy_proxy": round(
                sum(row["answer_relevancy_proxy"] for row in model_rows) / count, 4
            ),
            "avg_context_recall": round(
                sum(row["context_recall"] for row in model_rows) / count, 4
            ),
            "avg_latency_ms": round(
                sum(row["latency_ms"] for row in model_rows) / count, 2
            ),
            "total_cost_usd": round(
                sum(row["cost_usd"] for row in model_rows), 6
            ),
        },
    )

A useful decision table includes quality, reliability, and spend:

Do not choose a model solely because it has the highest average score. A route that costs three times as much for a one-point improvement may be wrong for a high-volume FAQ system and right for a compliance assistant. Report cost per successful answer and segment by task type.

RAG evaluation token cost

Token cost belongs inside the evaluation report because an evaluator can become expensive. If every test case sends ten retrieved chunks and a long rubric prompt to three models, the evaluation run may cost more than the feature you are trying to optimize.

Use a cost formula that matches the provider’s usage fields:

uncached_input_cost = uncached_input_tokens / 1,000,000 * input_rate
output_cost = output_tokens / 1,000,000 * output_rate
cached_input_cost = cached_input_tokens / 1,000,000 * cached_input_rate
cache_write_cost = cache_write_tokens / 1,000,000 * cache_write_rate
total_cost = uncached_input_cost + cached_input_cost + cache_write_cost + output_cost

Cached and cache-write tokens are input-token breakdowns, not extra tokens. When prompt_tokens includes both fields, calculate uncached_input_tokens = prompt_tokens - cached_tokens - cache_write_tokens, then apply each applicable rate once.

For repeated evaluation prompts, stable instructions and rubric text may benefit from prompt caching. The OpenAI-compatible prompt caching guide covers the engineering pattern. In an evaluation pipeline, log cache hits separately so a lower bill is not mistaken for a smaller dataset.

RAG Evaluation for Production

A production RAG evaluation pipeline needs more than a notebook score. It needs versioned datasets, repeatable prompts, threshold policies, human review, and a deployment gate.

Source documents
  -> chunker version
  -> index version
  -> evaluation questions
  -> retriever + top-k contexts
  -> model route
  -> metric calculation
  -> cost and latency logging
  -> regression report
  -> release decision

Version every part that can change the result: embedding model, chunking strategy, metadata filters, reranker, prompt template, model ID, output limit, and metric prompt. If a score moves, you need to know which component changed.

Thresholds and human review

Thresholds should be tied to risk. A support FAQ might tolerate a small number of incomplete answers if it offers a clear escalation path. A compliance workflow may require high faithfulness and citation validation before an answer reaches a user.

Use a release gate that combines automated and human checks:

Do not set thresholds before looking at failure examples. An arbitrary 0.8 threshold may reject good paraphrases or accept confidently wrong responses. First inspect your dataset, establish a baseline, then choose thresholds that match the business risk.

When evaluation shows that one model is best for easy prompts and another for difficult cases, use the results to inform routing policy rather than merely producing a leaderboard.

Evaluation tools and framework choices

Ragas is useful for LLM-judged metrics and dataset abstractions. LangChain provides evaluation integrations and application plumbing. OpenAI’s Evals guide describes evaluation as a way to test model and application behavior. You do not need all three in every project.

Conclusion and FAQ

RAG evaluation metrics are useful when they explain failure, not when they decorate a dashboard. Faithfulness checks grounding, answer relevancy checks whether the response addresses the question, context precision checks ranking quality, and context recall checks evidence coverage. Together with latency, token usage, cost, and human review, they provide a practical view of system quality.

A strong implementation path is: build a representative dataset, save retrieved contexts, run the same cases across GPT-5.6 Terra, DeepSeek V4 Pro, and GLM 5.2, record tokens and latency, add Ragas metrics where needed, and choose a model based on cost per successful answer. When you are ready to run the evaluation against live routes, create an OurToken API key for the evaluation run and use the OurToken configuration docs for endpoint setup.

FAQ

What are the most important RAG evaluation metrics?

Start with faithfulness, answer relevancy, context precision, and context recall. Add latency, token cost, failure rate, citation validity, and human review for production decisions.

What is a RAG faithfulness score?

It estimates how much of an answer is supported by the retrieved context. A low score often indicates unsupported claims, weak grounding instructions, poor context, or a model that fills gaps with guesses.

What is the difference between context precision and context recall?

Context precision measures whether relevant chunks appear near the top of the ranking. Context recall measures whether the retrieved set contains the evidence required to answer the question. A system can have high recall and poor precision when the right document appears only at a low rank.

How do I compare RAG models?

Use the same questions, retrieved contexts, prompt template, output limit, and scoring rules for each model. Compare quality, tail failures, latency, retries, and cost per successful answer rather than average quality alone.