The Same Intuition, Two Different Bets

Introduction

There is a pattern that shows up when you build search systems seriously enough: one retrieval pass is rarely sufficient. The first query comes back too sparse, or scores everything weakly, or surfaces documents that are adjacent to the question but not quite answering it. Something has to decide what to do next.

Two systems have independently arrived at this same diagnosis and built very different solutions around it. The first is RLM — Recursive Language Models — a research proposal from MIT CSAIL that gives the language model itself the tools to manage its own reading: peek at context subsets, grep for specific patterns, partition and recurse. The model decides when it has enough and what to do when it doesn't. The second is a deterministic retrieval pipeline — a production search system with no language model inside the retrieval loop, where every escalation, reformulation, and gap-fill decision is governed by rules written in advance.

This post works through the comparison: where the iteration lives, who decides when to stop, how each system detects what is missing, and what each approach is optimised for. The goal is not to pick a winner but to understand why two systems built around the same observation look so different — and what that tells you about a design choice your own search system will eventually have to make.

Background: The Two Systems

Before comparing them, it helps to understand what each system actually is.

Recursive Language Models (RLM) are a research proposal from MIT CSAIL. The idea is to wrap any language model with a thin scaffolding layer that gives it tools: the ability to peek at subsets of a large context, run code against it, and spawn isolated sub-calls on smaller pieces. The model itself decides when to use these tools and how. It reads a portion of the context, judges whether that is enough to answer the question, and if not, chooses its next action — search for something specific, partition the context further, summarise a section, recurse into a sub-problem. There are no pre-defined steps. The model plans its own path through the context. The result is a system where the language model manages its own reading process rather than passively consuming whatever is placed in front of it.

What makes this interesting is that when models are given these tools and a hard problem, they independently develop recognisable strategies — not because they were told to, but because the strategies are the natural response to the constraint. They peek at a small sample of the context first to understand its structure before committing to a reading approach. They grep with regex patterns when looking for a specific entity or keyword, turning a 200K-token scan into a targeted 2K-token read. They partition and map — splitting the context into segments, spawning a sub-call on each, then synthesising the results — when the question requires a global picture rather than a local lookup. They summarise sections into compressed representations so the outer loop reasons over digests rather than raw text. On structured tasks, they write and execute Python directly. These strategies are not mutually exclusive. On complex tasks, models chain them.

The deterministic retrieval pipeline compared here is a different kind of system entirely. It is a multi-phase retrieval engine — built on top of a hybrid search index that combines keyword and semantic matching — with no language model inside the retrieval loop. When a query arrives, the pipeline analyses its intent and likely information needs, selects an appropriate retrieval strategy, fires a search query, and then checks the results against a set of pre-defined conditions. If the results are too sparse or score too weakly, it escalates. If the query itself appears to be the problem, it reformulates. If specific information needs are not covered by what came back, it fires targeted follow-up queries to fill those gaps. Everything is governed by rules written in advance.

The contrast is sharp: RLM is a generation-time system where the model manages its own reading. The deterministic pipeline is a retrieval-time system that manages everything before the model is involved.

The Shared Observation

Neither system believes that one pass is enough.

The RLM paper arrives at this through the lens of context length. When you give a model too much text, its ability to recall specific details degrades — not catastrophically, but quietly, in ways that accumulate. The fix proposed is decomposition: instead of one large call, break the context into pieces, process each piece, and synthesise the results.

The deterministic retrieval pipeline arrives at the same conclusion through the lens of retrieval quality. A single query may return too few results, may score everything weakly, may surface documents that are adjacent to the question but not answering it. The fix built here is iteration: check whether what came back is sufficient, escalate if not, reformulate if the query itself was the problem, fill gaps where specific information is missing.

Same diagnosis — one pass isn't enough — two completely different treatments. The interesting question is not which one is right. It is why they look so different, and what each approach is optimised for.

Where the Iteration Lives

This is the most important structural difference, and everything else follows from it.

RLM places the iteration inside the model's reasoning process. The model reads what it receives, decides it's insufficient, and chooses its next action. The loop is: receive context → read a piece → judge whether this is enough → decide what to read next → repeat until an answer is possible. The model is the agent of the loop.

This is not a minor implementation difference. It is a fundamental architectural choice about which layer of the system carries the adaptive intelligence. RLM trusts the model to manage its own reading. The deterministic pipeline manages the reading for the model before the model is involved.

Who Decides Whether the First Pass Was Enough

When the first attempt returns something, both systems face the same question: is this good enough, or should I try again?

The distinction matters because the two signals diverge on exactly the cases you care about most. If the retriever returns ten results that all score confidently but are all about a related-but-wrong topic, proxy signals say “sufficient” and the pipeline stops. The model in RLM would read those results, recognise the mismatch, and try again. Conversely, if the retriever returns three results that score below a threshold but are precisely the right three documents, proxy signals say “escalate.” RLM would read those three documents, recognise they are sufficient, and answer.

The pipeline optimises for the common case. The vast majority of the time, result count and score distribution are reasonable proxies for sufficiency. RLM optimises for correctness on hard cases, at the cost of latency and unpredictability on all cases.

Who Decides What to Try Next

When both systems determine that the first pass was not enough, they have to choose a follow-up strategy.

Every strategy the pipeline can try was written by an engineer in advance. There is no mechanism for the system to discover a useful strategy that wasn't anticipated at design time.

This is a real limitation. But it is also why the pipeline's behaviour is fully auditable. For any query, you can trace exactly which strategy fired and why. Nothing happens that wasn't explicitly designed to happen. In RLM, the model's meta-decisions are harder to inspect — you can see what actions it took, but the reasoning that produced them lives inside the model's generation process.

How Each System Detects What Is Missing

Both systems try to identify specific things that the first pass failed to surface. This is where the conceptual gap is most visible.

One implication worth being explicit about: RLM does not fix bad retrieval. If the retriever returns the wrong documents, the model will navigate through them carefully and still arrive at the wrong answer. The gap detection problem and the retrieval quality problem are distinct.

RLM earns its place when retrieval is working and the bottleneck is reading. When retrieval is the problem, adding a reading layer on top does not help.

How Each System Measures Success

The evaluation frameworks the two systems use reveal what each considers the primary failure mode.

RLM measures recall under long context. The benchmark task is: a specific fact is buried in 132,000 tokens of surrounding content. Can the model retrieve and correctly use that fact? The failure mode is the model missing something that is present but obscured by context volume. The metric is accuracy: did the model get the right answer from the right content?

The deterministic pipeline measures distractor harm. The evaluation framework — UDCG — assigns negative scores to retrieved documents that are semantically plausible but factually misleading. A result that looks relevant but injects false premises scores worse than no result at all. The failure mode is not missing something that is present — it is including something that is wrong.

The insight from the pipeline's evaluation work is that chain-of-thought reasoning degrades when distractors are present — the more carefully the model reasons, the worse it performs, because it reasons carefully using bad premises. RLM's recursive decomposition strategy, applied on a context full of plausible distractors, might navigate more carefully through them — but it would still be navigating toward wrong information.

Both failure modes are real. In practice, systems face both: the retrieved set may be too large to process well and may contain misleading content. The two approaches address complementary halves of the same problem.

The Trust Question

Underlying all the specific comparisons is a deeper question about where trust is placed in the system's design.

Neither stance is obviously correct. Trusting the model means the system is only as good as the model's meta-reasoning capabilities, which vary significantly across model families and sizes. Trusting the rules means the system is only as good as the engineer's ability to anticipate what situations will arise and design the right response for each.

There is also a practical dimension to this. A rule-based system is debuggable — you can inspect every decision, understand why it was made, and change a specific threshold if it is wrong. A model-based system is observable — you can see what actions the model took — but the reasons for those actions live inside the model's generation process and are harder to diagnose or correct.

For a production search system, debuggability often wins by default. When something goes wrong at scale, you want to be able to point to the specific rule that produced the bad outcome and change it. When the bad outcome comes from a model's autonomous decision-making process, the fix is less obvious.

Two Failure Modes, One Root Cause

Despite the different failure modes each system measures, they share the same root cause: a single retrieval or reading pass is not enough for complex queries.

The two systems represent different theories about where the intervention should happen.

When Each Bet Pays Off

Where they should be combined

The cleanest deployment of both is as sequential layers, not alternatives. The pipeline runs first and produces a curated set of retrieved content. If that content is still too large or structurally complex for reliable single-pass reading, the caller's model applies RLM-style decomposition to read it. Each layer solves what it is good at: the pipeline removes distractors and fills obvious gaps; RLM handles reading complexity that survives the retrieval filter.

Conclusion

Reading the RLM paper against this system's design is clarifying not because one approach is right, but because both approaches are internally coherent and designed around real constraints.

RLM starts from the observation that models degrade on long contexts, decides that the model itself is best placed to manage this, and builds a system that gives the model the tools to decompose its own reading. The result is a system that handles hard queries better, at the cost of unpredictable latency and cost.

The deterministic pipeline starts from the observation that distractors in context actively harm model output, decides that distractor prevention is better than distractor navigation, and builds a system that manages retrieval quality before the model is involved. The result is a system with predictable performance characteristics and an explicit, auditable decision trail — but with pre-programmed rules that cap the ceiling on what the system can adapt to.

The conceptual frame that makes both coherent: they are solving the same problem at different layers. The problem is that naive retrieval — one query, take the top results, hand them to the model — is not good enough for complex queries. The solution is iteration. The question is whether that iteration happens at the retrieval layer, driven by rules, or at the reading layer, driven by the model.

That question does not have a universal answer. It has an answer that depends on what your failure modes are, what your latency requirements are, how much you trust your rules, and how much you trust your model.

The clearest way to frame it: RLM solves a reading problem, not a finding problem. If the right content cannot be located, fix the retrieval. If the right content is found but is too large or complex to read reliably in one pass, that is where RLM earns its place.

The best systems will probably do both.

References

• Recursive Language Models — Alex L. Zhang and Omar Khattab, MIT CSAIL — arXiv:2512.24601
• Vespa — The open-source big data serving engine used for the deterministic retrieval pipeline