Most RAG tutorials show you how to build one. Almost none teach you how to measure whether it's working. I built three RAG systems before this clicked for me. The gap between demo and production is real, and it is mostly an evaluation problem.
The core problem: RAG evaluation is genuinely hard. You're evaluating a pipeline that chains retrieval, passage selection, and generation. A failure at any stage compounds. A metric that looks good on one stage might be terrible for the whole system.
This guide cuts through the noise. I cover the metrics that matter, why they matter, and how to actually use them.
The RAG Evaluation Stack
RAG systems have three distinct failure modes. Your evaluation strategy needs to catch all three.
Stage 1: Retrieval Quality
Retrieval quality determines the ceiling for everything downstream. If the retrieved passages don't contain the answer, no amount of clever prompting fixes it.
Recall@K measures what fraction of all relevant passages appear in the top-K results. For a question with multiple supporting facts spread across documents, recall tells you whether the system even has access to the information it needs.
recall@k = (number of relevant documents in top-k) / (total number of relevant documents)
Recall@K is a lower bound. A system can have perfect recall and still retrieve passages full of noise. You need more.
Precision@K measures how many of the top-K results are actually relevant. High precision means the model wastes less context budget on irrelevant material. Low precision dilutes the generation signal.
The tradeoff between recall and precision depends on your context window. If you're context-rich, optimize recall. If you're context-constrained, precision matters more.
NDCG@K (Normalized Discounted Cumulative Gain) weights ranked results by relevance. A relevant document at rank 1 scores higher than the same document at rank 5. Use NDCG when rank order matters for your pipeline.
Stage 2: Passage Selection
Retrieval finds candidate passages. Selection picks which ones to include in the context.
Context Precision measures whether relevant content ranks higher within a passage. A retrieved passage might be 80% relevant and 20% noise. Context precision penalizes passages with relevant content buried deep inside.
Context Recall compares the retrieved context against a ground-truth answer. It checks whether all facts needed to answer are present in the retrieved context, not just in the raw document store.
This is where many tutorials cut corners. They test retrieval against a document store, not against the actual context fed to the generator. These are different things. I learned this by building eval pipelines that tested both and watching them disagree.
Stage 3: Response Quality
The generator produces the final answer. You need to evaluate what comes out.
RAGAs (Retrieval-Augmented Generation Assessment) is a framework developed specifically for this. It decomposes response evaluation into three scores:
Faithfulness measures whether the generated answer stays consistent with the retrieved context. A faithful answer doesn't hallucinate facts that aren't in the context.
Answer Relevancy measures whether the response actually addresses the question. A relevant answer provides information the user was asking for, not a plausible-sounding tangent.
Context Relevancy measures whether the retrieved context is useful for answering the question. High context relevancy means the passages selected actually contain information needed to construct the answer.
RAGAs scores range from 0 to 1. Most production systems I've seen score between 0.4 and 0.7 on faithfulness. Scores above 0.6 typically indicate a system worth deploying. Above 0.75 is strong.
The Hallucination Problem
Hallucination is not a retrieval problem. It's a generation problem.
Retrieval metrics can be perfect and the model still hallucinates. This is why you need answer-level evaluation that goes beyond context faithfulness.
Self-RAG and similar approaches try to solve this by having the model critique its own responses against retrieved context. The model learns to flag when it's generating from knowledge not present in retrieved passages.
These approaches add latency and cost. The tradeoff often isn't worth it for factual Q&A where retrieval is already strong. It matters more for open-ended generation where the model is tempted to extrapolate.
Metrics That Look Good But Mislead
BLEU and ROUGE score surface-level overlap between generated and reference answers. They're popular because they're automated and easy. They're also nearly useless for RAG evaluation.
Two answers can be semantically equivalent and score 0.3 on BLEU. Two answers can be factually opposite and score 0.7 on ROUGE. These metrics measure text similarity, not answer quality.
The trap: teams optimize for BLEU/ROUGE because they're easy to measure. The system learns to produce text that tricks the metric without improving actual answer quality. I have watched this happen twice. It is not fun to debug.
Perplexity on the generated text is similarly misleading. Low perplexity means the model is confident in its text. A confidently wrong answer still fools perplexity-based metrics.
Building a Practical Evaluation Pipeline
The right evaluation pipeline combines automated metrics with human evaluation.
Automated Metrics You Can Run in CI
Set up automated scoring on every commit:
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
def evaluate_rag_response(question, answer, contexts, ground_truth):
metrics = [
faithfulness,
answer_relevancy,
context_precision,
]
result = evaluate(
question=question,
answer=answer,
contexts=contexts,
ground_truth=ground_truth,
metrics=metrics
)
return {
"faithfulness": result["faithfulness"],
"answer_relevancy": result["answer_relevancy"],
"context_precision": result["context_precision"],
}
RAGAs supports this out of the box. The framework accepts a dataset of question/answer/context triples and returns scored results.
The cost: RAGAs itself calls an LLM to evaluate responses. Using GPT-4 for evaluation at scale gets expensive fast. Use smaller models (GPT-3.5, Claude Haiku) for triage, flag borderline cases for human review.
Human Evaluation Traps
Human evaluation is necessary but expensive. The trap is asking humans the wrong questions.
Bad question: "Rate this answer from 1-5 on quality."
This produces inconsistent, subjective scores that don't generalize. A 4 from one evaluator means something different from a 4 from another.
Good question: "Does this answer contain any factual errors compared to the retrieved context? (Yes/No)"
Binary factual checks are fast, consistent, and actionable. When I find errors, I escalate to detailed review.
Evaluating Retrieval in Isolation
Test retrieval separately before touching generation:
def evaluate_retrieval(test_dataset, k_values=[1, 3, 5, 10]):
"""Measure retrieval quality independent of generation."""
results = []
for item in test_dataset:
query = item["query"]
relevant_docs = set(item["relevant_doc_ids"])
retrieved = retrieval_model.query(query, k=max(k_values))
retrieved_ids = set(retrieved["doc_ids"])
for k in k_values:
retrieved_k = set(retrieved_ids[:k])
precision = len(retrieved_k & relevant_docs) / k
recall = len(retrieved_k & relevant_docs) / len(relevant_docs)
results.append({
"k": k,
"precision": precision,
"recall": recall,
"query": query,
})
return pd.DataFrame(results)
This runs fast, costs nothing, and gives you reliable signal about retrieval quality before you invest in generation improvements.
The Most Common Failure Mode
The failure mode I see most often: evaluation looks great in testing, collapses in production.
The reason is distribution shift. Test datasets are static. Production queries drift over time as users discover new question patterns, as documents get stale, as the document corpus grows and changes structure.
Your evaluation pipeline needs to detect distribution shift. Monitor retrieval recall quarterly. Re-run human evaluation on production queries monthly. When scores drop, investigate before users complain.
What to Actually Optimize
If you're starting from scratch, optimize in this order:
-
Retrieval recall first. No generation improvement fixes missing context. Measure what percentage of your knowledge base the system can actually surface for any given query.
-
Context precision second. Once recall is solid, reduce noise. High precision means the generator spends its context budget on signal, not noise.
-
Faithfulness third. Faithful generation keeps users trusting the system. Measure it, set a threshold, reject generations that fall below it.
-
Answer relevancy fourth. Everything above this point measures parts of the system. Answer relevancy measures the whole thing working together.
The mistake most teams make is starting at step 4 without the foundation. They tune prompts, try different models, add retrieval augmentation, and waste time on generation tweaks while retrieval quietly fails.
Thresholds That Work
Based on production deployments I've seen:
- Retrieval Recall@5 < 0.7: Retrieval needs work before anything else matters.
- Context Precision@5 < 0.5: Too much noise in retrieved passages for the generator to reliably extract signal.
- Faithfulness < 0.5: The model is hallucinating frequently. This is a generation problem, not a retrieval problem.
- Answer Relevancy < 0.6: Either retrieval is missing context, or generation isn't using what it has.
These thresholds aren't universal. Adjust based on your domain. A medical RAG system needs higher faithfulness than a creative writing assistant. A legal RAG system needs higher recall than a product FAQ bot.
The Human Feedback Loop
Metrics don't capture everything. A system that scores well on RAGAs can still produce answers that feel cold, unhelpful, or misaligned with user intent in ways the metrics don't measure.
Set up a way to capture user feedback at scale. Thumbs up/down on answers. "This answer was helpful" buttons. Follow-up questions that indicate whether the initial answer actually resolved the user's need.
Use this feedback to build a human-labeled eval set. Rotate it quarterly. This labeled data is the foundation for fine-tuning the generator, choosing between models, and catching regressions that automated metrics miss.
The Real Takeaway
RAG evaluation is not solved by any single metric. The metrics that are easy to automate (BLEU, ROUGE, perplexity) are the ones that don't measure what matters. The metrics that measure what matters (faithfulness, answer relevancy) require LLM-based evaluation that's expensive and imperfect.
Build a multi-layered evaluation strategy. Automated metrics catch regressions. Human evaluation catches misalignment. Production monitoring catches distribution shift.
Measure retrieval, generation, and end-to-end quality separately. A failure in retrieval compounds through generation. Fix the foundation before optimizing the surface.
The teams that ship reliable RAG systems treat evaluation as a first-class concern, not an afterthought. They build eval pipelines before they build the retrieval system. They set thresholds and reject systems that don't meet them. They monitor in production and iterate when metrics drift.
I wish I had learned this two systems ago. Most RAG tutorials never get to this point. Now you know what actually matters.