Why Your RAG Metrics Are Lying to You

Introduction

If you're building with RAG, you've likely done the responsible thing: assembled a golden dataset, run your retrieval evaluation, and checked the numbers before shipping. And the numbers probably looked fine — maybe even great. MRR near 1.0, solid recall, respectable nDCG. By every standard measure, your retriever is doing its job.

So why are your users still getting confidently wrong answers?

The issue isn't that you skipped evaluation — it's that the metrics you used were designed for a different world. nDCG, MRR, and Recall all emerged from decades of web search research, built on a foundational assumption: the end consumer is a human who scans results, picks what looks right, and ignores the rest. That assumption breaks completely when the consumer is an LLM that reads everything you give it and tries to synthesize it all into a single answer.

This creates a dangerous blind spot. Documents that a human would glance at and dismiss — a training log about squats when you asked about deadlifts, a YC talk about pivoting when you asked about B2B sales — become active liabilities in an LLM's context window. They're not just unhelpful; they're distractors that inject plausible-sounding but wrong information into the model's reasoning. And traditional metrics score them the same as any other irrelevant result: zero. Neutral. Harmless.

They're anything but harmless.

This post lays out the problem in detail, then introduces UDCG (Utility and Distraction-aware Cumulative Gain) — a metric built specifically for evaluating retrieval in RAG systems. The core idea is straightforward: distractors don't have zero utility, they have negative utility. By extending DCG to allow negative scores, we can finally quantify the hidden harm that traditional metrics miss, compare ranking strategies on what actually matters for LLM performance, and make informed decisions about how many documents to retrieve.

The Problem: Perfect Metrics, Broken RAG

If you've built a RAG system, you've probably evaluated it properly before shipping. You created a golden dataset with labeled queries and relevant documents, ran your retrieval evaluation, and got results like this:

These numbers tell a compelling story: your retriever consistently places relevant documents at the top, finds most of what's needed, and ranks results well. Your evaluation passed. You ship to production.

But then something puzzling happens. Despite passing your retrieval evaluation, users start complaining:

“The AI keeps giving confident but wrong answers.”

“It mixes up information from different topics.”

“Sometimes it gives me details I never asked for.”

Here's the uncomfortable truth: your evaluation was measuring the wrong thing.

Traditional retrieval metrics (MRR, nDCG, Recall) answer the question: “Did we find the relevant documents?”

But for RAG, the real question is: “Will this context help or hurt the LLM's answer?”

These are not the same question. A retrieval result can find all the relevant documents (high recall) while simultaneously including documents that will confuse the LLM (distractors). Traditional metrics don't penalize distractors — they treat them the same as any other non-relevant document.

The gap isn't in whether you evaluate — it's in WHAT you evaluate.

The Hidden Danger: Distractors

Let's make this concrete with an example. Imagine you're building a personal knowledge assistant, and a user asks:

Query: “What was my 500 lb deadlift PR?”

Your retriever returns 5 documents:

From a traditional IR perspective, this looks great — MRR = 1.0, the correct answer is at position 1. The system found what the user needed.

But here's what actually happens when this context reaches the LLM. The model reads ALL 5 documents and sees: “500 lbs” (correct), “405 lbs” (squat), “315 lbs” (form practice), “455 lbs” (mock meet opener). The LLM might now confidently answer with any of the wrong numbers.

These aren't random irrelevant documents — they're worse. They're plausible distractors: documents that look related, contain similar terminology, but provide wrong or confusing information.

Why Traditional Metrics Fail

To understand why traditional metrics fail for RAG, we need to understand what they were designed for and the assumptions they make.

The History: Metrics Built for Human Search

Traditional IR metrics like nDCG, MRR, and MAP were developed in the 1990s–2000s for evaluating web search engines. They were designed to answer one question: “How well does this ranked list serve a human user?”

These metrics are built on three key assumptions:

Assumption 1: Users examine results sequentially and stop early. When you Google something, you don't read all 10 results on the first page. You scan from top to bottom, click on something promising, and stop. This is why MRR focuses on the position of the FIRST relevant result.

Assumption 2: Irrelevant results are simply ignored. If result #4 is off-topic, a human just skips it. It doesn't harm them. This is why nDCG assigns zero utility to irrelevant results — they're neutral, not harmful.

Assumption 3: Value decays predictably by position. A relevant result at position 1 is more valuable than at position 5, because humans are less likely to scroll down. The logarithmic discount in DCG models this “patience decay.”

Why These Assumptions Break for LLMs

Now consider how an LLM agent uses retrieved documents. Every single assumption breaks:

Assumption 1 breaks: LLMs don't stop early. An LLM doesn't “click” on one result and stop. It receives ALL retrieved documents concatenated into its context window. It processes everything you give it.

Assumption 2 breaks: Irrelevant results are NOT ignored. This is the critical failure. When a human sees an off-topic result, they recognize it and skip it. When an LLM sees plausible-but-wrong information in its context, it often incorporates it into its reasoning. LLMs are trained to synthesize information from their context — they're designed to find connections and use all available information.

Assumption 3 partially breaks: Position matters less. While some LLMs show slight primacy/recency biases, they generally attend to all context. A distractor at position 5 can be just as harmful as one at position 2.

The Core Problem: Distractors Have Negative Value

Here's the key insight that traditional metrics miss entirely:

Traditional metrics treat that last category as zero. But for RAG, a plausible distractor is worse than having no result at all. It actively injects false premises into the LLM's reasoning.

This is why the article “Mutually Assured Distraction” makes such a powerful point:

“You cannot reason your way out of bad context.”

The research shows something counterintuitive: Chain-of-thought prompting actually DEGRADES performance when distractors are present. The more the model “thinks,” the more it incorporates the misleading information. Even worse, in agent loops where the LLM makes multiple decisions: 90% per-step accuracy → 53% accuracy after just 6 steps. The errors compound.

The Evaluation Gap

Most teams already do the right thing — they evaluate with golden datasets before shipping. The problem isn't lack of evaluation, it's that standard retrieval metrics have a blind spot.

We need metrics designed for RAG, not for human searchers.

The Solution: UDCG (Utility and Distraction-aware DCG)

If traditional metrics don't capture the harm of distractors, we need a new metric that does. Enter UDCG: Utility and Distraction-aware Cumulative Gain.

The key insight is simple but powerful: distractors don't have zero utility — they have NEGATIVE utility.

Utility Scoring

Traditional DCG assigns utility scores where irrelevant documents get zero. Both “completely off-topic” and “plausible but wrong” get the same score.

UDCG introduces negative utilities:

This simple change has profound implications. A retrieval system that returns 3 relevant docs and 2 distractors now scores worse than one that returns 3 relevant docs and 2 off-topic docs.

The Math: How UDCG Works

The formula is identical to standard DCG — the only difference is that utilities can be negative:

Each document's utility is divided by a logarithmic discount based on its position. UDCG uses the exact same formula, but because utilities can now be negative, distractors subtract from the cumulative score. A distractor at rank 1 hurts more than one at rank 5 due to the position discount.

Worked example — 5 documents:

Architecture: The Evaluation Pipeline

Now that we understand the concept, let's see how to build an evaluation system around it. The pipeline has four main steps:

Distractor Detection: How It Works

One of the trickiest parts of UDCG evaluation is identifying which documents are distractors. Here's the decision flow:

Why these heuristics?

1. Known distractors (-1.0): If you've manually labeled documents as distractors from previous error analysis, they get the maximum penalty.
2. High relative score (-0.5): If a document scores within 70% of the top result but isn't relevant, it's suspicious. The retriever thinks it's good, but it's not — classic hard negative.
3. Top 3 position (-0.5): Documents in the top 3 positions are almost always included in LLM context. A non-relevant document here is dangerous regardless of its score.
4. Everything else (0.0): Low-ranked, low-scoring irrelevant documents are neutral. They're easy for the LLM to ignore because they're clearly off-topic.

These thresholds are configurable. What counts as a “distractor” and how severely to penalize it depends on your domain:

Real Results: Before vs After

Let's see how this works in practice. We evaluated a RAG system with 20 queries and 100 documents.

Before (with a bug in distractor detection)

Our initial implementation had a bug — it wasn't properly detecting distractors because we used an absolute score threshold (0.5) when our ranking profile returned scores around 0.03.

After (with fixed distractor detection)

After switching to relative score thresholds, the true picture emerged:

What changed? The traditional metrics (MRR, Precision, Recall) stayed the same. But UDCG revealed that 67% of our top-5 results were distractors — documents that would confuse the LLM.

Example of detected distractors

These distractors are insidious because they're semantically similar to the query. They all relate to YC, startups, and business growth. A traditional retriever sees them as “close matches.” But for RAG, they're poison.

Comparing Ranking Profiles

One practical application of UDCG is comparing different ranking strategies. We tested multiple Vespa ranking profiles:

The counterintuitive insight: Look at simple-hybrid: it has the best recall at 83% — by traditional metrics, it's a great choice. But it also has the highest distractor rate at 62%. For every 5 documents retrieved, about 3 are distractors. For RAG, this is the WORST choice, not the best.

Meanwhile, rrf-hybrid has lower recall (72.8%) but zero distractors. For RAG applications, this is far superior.

Applying This at Runtime

UDCG is an offline evaluation metric — you run it on test queries to understand your retrieval quality. But the insights directly inform runtime decisions:

The general pattern: use the optimal k discovered during evaluation, apply a relative score threshold to filter candidates, and cap the final result set. This treats UDCG evaluation as a calibration step — you run it offline to discover the right parameters, then apply those parameters at query time.

Key Takeaways

References

• Mutually Assured Distraction