Layered Chunking

Introduction

Most RAG systems look like they work — until you inspect what actually reaches the LLM. Embedding similarity retrieves chunks that are topically adjacent but informationally irrelevant, and you can't see it in your eval metrics. Layered chunking fixes this by scoring every chunk on two independent signals — semantic similarity and keyword overlap — and dropping any chunk that doesn't pass both. The result: a 40% reduction in false positive chunks, lower token costs, and an LLM that synthesizes from clean context instead of noise. This post walks through the problem, the mechanism, the benchmarks, and why it matters even more in agentic architectures.

Why Your RAG Pipeline Is Quietly Failing

Here's something that doesn't show up in your demo but ruins production: most RAG systems retrieve chunks that are semantically adjacent to the answer but not actually the answer.

You've tuned your embeddings. You've picked a chunking strategy. You've wired up a vector database. Your retrieval “works” — the top-K chunks come back, they look vaguely related, and the LLM generates something plausible. Ship it.

Except the LLM is hallucinating details. Or hedging with “based on the provided context, it appears that...” Or confidently synthesizing an answer from a data table that has nothing to do with the user's question. The retrieval step looked fine. The chunks were semantically similar. But similar isn't the same as relevant.

This is the central failure mode of embedding-only retrieval: cosine similarity captures topical proximity, not informational relevance. A chunk that lives in the same embedding neighborhood as your query — because it's from the same paper section, uses similar vocabulary, or discusses adjacent concepts — will score highly even if it contains zero useful information for the actual question asked.

And it's not an edge case. In our testing across information retrieval research papers, roughly 1 in 5 chunks selected by embedding similarity alone were false positives — topically adjacent but informationally irrelevant.

The Real Cost of “Good Enough” Retrieval

Why does this matter? Three reasons that compound on each other:

1. LLM output quality is gated on input quality. This is the most important and least discussed constraint in RAG system design. A frontier model with three precisely relevant chunks will outperform the same model with five chunks where two are noise. The irrelevant chunks don't just waste tokens — they actively confuse the model. They introduce contradictory signals, dilute the relevant context, and give the model material to hallucinate from. Garbage in, confident garbage out.

2. Token costs scale with retrieval sloppiness. Every irrelevant chunk you send to the LLM costs real money. At scale, the difference between sending 3 relevant chunks and 5 chunks (with 2 being noise) is a 40% increase in input tokens per query — with no quality benefit. Across millions of queries, that's significant spend for worse results.

3. User trust erodes silently. When a RAG system returns a plausible-sounding but subtly wrong answer because the LLM synthesized from irrelevant context, the user doesn't file a bug report. They just stop trusting the system. By the time you notice in your metrics, the damage is done.

The fix isn't a better embedding model. It's not a smarter chunking strategy. It's requiring chunks to prove their relevance through multiple independent signals before they ever reach the LLM.

That's what layered chunking does.

The Problem: Semantic Similarity Lies

Let's make this concrete. Consider a query against a corpus of IR research papers: “why is colbert effective?”

A standard retrieval pipeline computes cosine similarity between the query embedding and each chunk embedding, then picks the top-K. Here's what actually happens with a real 4-chunk document from the ColBERTv2 paper:

Chunk 1 scores higher than chunk 0 on semantic similarity. It lives in the same embedding neighborhood because it's from the same paper section. But it's a table of query examples — it says nothing about why ColBERT is effective.

A similarity-only retriever selects chunks 1, 2, and 3. The LLM gets a table of random queries where it should have gotten analysis content. This is the topical proximity trap in action — chunk 1 is about ColBERT, it's from the ColBERT paper, but it answers a completely different question.

Now multiply this across every query your system handles. One in five chunks is a false positive like this. Your LLM is working with polluted context on 20% of its inputs, and you can't see it because the retrieval metrics look fine.

The Fix: Require Both Signals

Layered chunking scores each chunk on two independent axes:

1. Semantic score: How close is the chunk embedding to the query embedding?
2. Lexical score: Does the chunk contain the actual query terms (via BM25)?

The key insight is in how these scores are combined. Rather than adding them for all chunks, an inner join computes the combined score only for chunks that have both signals. Chunks with high embedding similarity but zero keyword overlap are dropped entirely.

Here's the same document under layered chunking:

Chunk 1 has no BM25 score because it doesn't contain “colbert” or “effective.” The join operation excludes it entirely. The LLM now gets three chunks that are both semantically and lexically relevant.

Document Retrieval vs. Chunk-Level Retrieval

To understand where layered chunking fits, it helps to see the full spectrum of retrieval architectures — from returning whole documents down to returning individual filtered chunks.

The Three Retrieval Architectures

1. Document-Level Retrieval

The simplest approach: match queries to whole documents. This is how traditional search engines work.

The problem is obvious: the LLM gets entire documents when it needs specific passages. You pay for 45K tokens when 3K would suffice.

2. Standard Chunk-Level Retrieval

Split documents into chunks, embed each chunk, retrieve the top-K most similar chunks.

Better — you've cut tokens by 94%. But ~20% of chunks are still false positives (topically adjacent, informationally irrelevant).

3. Layered Chunk-Level Retrieval (Dual-Criteria Filtering)

Score every chunk on both semantic similarity AND keyword overlap. Only chunks that pass both filters survive.

Where Layered Chunking Sits

The evolution from document to chunk to layered retrieval trades recall for precision at each step — and for RAG, that's the right trade:

How It Works: The Five-Stage Pipeline

Each document stores its text as an array of chunks alongside a 2D embedding tensor — one vector per chunk. When a query arrives, the ranking pipeline processes it through five stages:

The critical stage is Stage 4 — the inner join. It operates like a database inner join: only chunks present in both the semantic tensor and the lexical tensor get a combined score. This is what makes the approach fundamentally different from hybrid search, which adds scores for all chunks regardless.

Here's a concrete trace:

Document-Level Ranking

After chunk-level filtering, documents are ranked by the sum of all qualifying chunk scores. This is a deliberate design choice for RAG:

A document with three good chunks outranks a document with one great chunk. For RAG, you want the document with the most relevant content overall — not just the best single passage. This is the opposite of traditional hybrid search, which ranks by the single best chunk.

Measuring What Matters: Document vs. Chunk Metrics

A common mistake in RAG evaluation is measuring at the wrong granularity. Document-level and chunk-level metrics answer fundamentally different questions, and using the wrong one hides the exact problems layered chunking solves.

Document-Level Metrics

These answer: “Did we find the right documents?”

Document-level metrics are useful for corpus coverage — did the retrieval system surface the right papers? But they tell you nothing about whether the LLM received the right content from those papers.

Chunk-Level Metrics

These answer: “Did we feed the LLM the right content?”

Chunk-level metrics expose the problem that document-level metrics hide:

The document-level metrics can't distinguish between these two systems. Both retrieve the ColBERTv2 paper. But one sends the LLM a random query table, and the other doesn't. Only chunk-level metrics reveal the difference.

Why Chunk Metrics Are Harder

Chunk-level evaluation requires chunk-level relevance judgments — a human (or strong LLM judge) must label each individual chunk as relevant or irrelevant for a given query. This is significantly more work than document-level labeling:

This labeling cost is why most RAG benchmarks report document-level metrics — and why the chunk precision problem goes undetected. If you're serious about RAG quality, invest in chunk-level ground truth for your top 50-100 queries. It's tedious but it's the only way to measure what your LLM actually sees.

Advanced Ranking Strategies

The base layered approach can be extended with additional signals. Each strategy adds a second-phase re-ranking step on top of the dual-criteria filter:

The two-phase approach means you get the precision benefits of dual-criteria filtering on all documents (cheap, fast), then apply more expensive scoring to only the survivors.

Benchmarks

We tested across 100 queries on a corpus of information retrieval research papers (5 PDFs, ~150 pages, ~600 chunks total):

Key takeaways:

• Precision jumped from 0.67 to 0.85 — fewer irrelevant chunks in the top 3
• Latency dropped 6% — server-side tensor operations are faster than Python post-processing
• App server CPU dropped 73% — scoring and filtering moved to the search engine
• Recall traded off slightly (0.85 to 0.79) — the dual-criteria filter is stricter, which can exclude borderline-relevant chunks that lack exact keywords

The recall tradeoff is real but manageable. For RAG applications where precision matters more than recall (you'd rather send 3 great chunks than 5 mediocre ones to the LLM), layered chunking wins convincingly.

Ranking Profile Comparison

Different profiles suit different use cases:

Practical Considerations

1. Parameter alignment matters. The query parameter name must be consistent across your rank profile definition, your query DSL, and your request body. A mismatch produces confusing errors because the search engine treats the unresolved embedding expression as a literal string.

2. The join can be too strict for sparse content. If your documents have very short chunks or your queries use uncommon terms, the keyword side of the join may match nothing, dropping all chunks from a document. Monitor for zero-result queries and consider falling back to similarity-only retrieval when this happens.

3. Network transfer includes unfiltered chunks. The filtering happens server-side for scoring purposes, but the full chunk array currently transfers to the client. The workaround is client-side filtering by the returned chunk indices. Native schema configurations can eliminate this overhead, but not all client libraries support it yet.

4. Dimensionality matters for the distance computation. When your query embedding is 1D and your chunk embeddings are 2D (one vector per chunk), you must specify which dimension to reduce over. Otherwise you'll get a dimension mismatch error.

When to Use Layered Chunking

Use it when:

• You're building a production RAG pipeline and precision matters
• Your documents have multiple chunks per page (the dual-criteria filter shines with 3+ chunks)
• You want to reduce LLM token costs by sending fewer, better chunks
• Your queries contain specific terms (not just vibes-based semantic queries)

Skip it when:

• You're prototyping and iteration speed matters more than quality
• Your documents are single-chunk (nothing to filter)
• Your queries are purely semantic (“find me something about machine learning feelings”)
• You need maximum recall and can't afford to miss borderline-relevant chunks

In the Era of Agents: Why Retrieval Precision Matters More Than Ever

The shift from single-shot RAG to agentic architectures doesn't reduce the importance of retrieval precision — it amplifies it. When agents make autonomous decisions based on retrieved context, every false positive chunk doesn't just degrade one answer. It corrupts a decision that cascades through a chain of subsequent actions.

Agents Are Retrieval Consumers at Scale

A traditional RAG pipeline handles one query at a time: user asks, system retrieves, LLM answers. An agent makes dozens of retrieval calls per task, each one informing the next action:

Five retrieval calls, each feeding into the next stage. If retrieval call 1 returns a chunk about ColBERT's tokenization implementation details instead of its effectiveness, the analysis agent draws wrong conclusions that propagate through the entire chain.

The Agent Retrieval Loop

Agents don't just consume retrieval results — they reason over them and generate follow-up queries. This creates a feedback loop where retrieval quality compounds:

The math is stark. With standard retrieval, an agent's context degrades below 50% signal after just three retrieval calls. With layered chunking, it stays above 66% even after five calls. For agents, precision isn't a nice-to-have — it's a prerequisite for multi-step reasoning.

Multi-Agent Architectures Amplify Retrieval Errors

In multi-agent systems where specialized agents communicate and share context, the impact of retrieval quality multiplies further:

When agents hand off context to each other, a false positive in one agent's retrieval becomes an unverifiable assumption in the next agent's input. The downstream agent has no way to distinguish a “fact” that came from a relevant chunk versus one hallucinated from a noisy chunk. Clean retrieval is the trust anchor for multi-agent coordination.

What Agents Need from Retrieval

The requirements shift when retrieval serves agents instead of single-turn RAG:

The key insight: agents can compensate for lower recall by making follow-up queries, but they cannot compensate for low precision. If a chunk is in the context, the agent trusts it. There's no “are you sure?” step in an automated pipeline. This makes the dual-criteria filter of layered chunking — which specifically targets precision — a natural fit for agentic architectures.

Layered chunking aligns with this shift. By aggressively filtering out false positives at the retrieval layer, it gives agents a clean foundation for each reasoning step — and lets them make the recall trade-off themselves by issuing more targeted follow-up queries when needed.

The Bigger Picture

RAG systems have a quality ceiling, and it's not the LLM — it's what you feed it. We've spent enormous effort on better embedding models, smarter chunking strategies, and more capable language models. But the weakest link in most pipelines is the gap between “semantically similar” and “actually relevant.”

Layered chunking closes that gap by treating retrieval as an evidence problem: a chunk should only reach the LLM if it can demonstrate relevance through multiple independent signals. The inner join is the mechanism, but the principle is what matters. Don't trust a single similarity score. Require corroboration.

The 40% reduction in false positives isn't just a number — it's the difference between an LLM that confidently synthesizes from relevant context and one that hallucinates from noise. Your users can tell the difference, even if your eval metrics can't.

References

• Introducing Layered Ranking for RAG Applications