Recursive Language Models

Recursive Language Models (RLMs) are an inference strategy that lets a language model handle inputs far longer than its context window. Instead of stuffing the whole input into the prompt, the model treats the input as an external environment it can search, slice, and — crucially — hand pieces of to copies of itself.

This page assumes you know what a context window is. If not, read GPT and In-Context Learning first.

Why Students Should Care

• Context windows are a hard limit of today’s models; RLMs show a way around the limit without retraining anything.
• The core trick — recursion — is one you already know from intro CS, applied to LLMs.
• It is a clean example of the shift from “model as text predictor” to “model as agent that writes code and calls tools.”

The Problem: Context Rot

Standard language models suffer from context rot — performance degrades as input length approaches or exceeds the context window. Even models with 128K+ token windows struggle with:

• Retrieval accuracy in long documents
• Multi-hop reasoning across distant passages
• Maintaining coherence over extended contexts

So simply buying a bigger window does not solve the problem. RLMs address it by changing how the model interacts with its input.

Core Insight

Instead of:

# Traditional: context IN the prompt
response = llm.completion(f"{huge_context}\n\nQuestion: {query}")

RLMs do:

# RLM: context AS a variable in an environment
repl.set_variable("context", huge_context)
response = rlm.completion(query)  # Model writes code to explore context

The context becomes an environment variable in a REPL (an interactive code environment) that the model can programmatically query, slice, search, and recursively process. The model never reads the whole input — it writes code that reads the parts it needs.

How It Works

1. Load context into REPL: the full input is stored as a string variable in a Python environment
2. System prompt: the root model receives instructions on how to interact with the environment
3. Programmatic access: the model can read slices, write helper functions, and spawn sub-LLM calls
4. Recursive decomposition: complex queries trigger recursive calls on smaller chunks
5. Result combination: answers bubble up and combine into the final response

Interactive Demo

Explore how RLMs decompose problems and search through massive contexts:

Recursive Language Model

ConceptSearch Demo

Traditional LLM

llm.completion(prompt + context)

RLM

rlm.completion(query, env=context)

Traditional LLM

Context Overflow

RLM: Context as Variable

Programmatic Access

Recursive Calls

Entire context loaded into prompt

Play DemoReset

Example: Needle in a Haystack

Concretely — say you need to find one specific fact buried in 10M tokens of text.

Traditional LLM approach:

• Load all 10M tokens into context
• Attention over every token: $O(N)$ complexity
• Fails due to context window limits

RLM approach:

def find_needle(context_var, query):
    # Split into chunks
    chunks = rlm.call("Divide context into 10 sections")

    # Query each chunk
    for i, chunk in enumerate(chunks):
        result = rlm.call(f"Does section {i} contain: {query}?")
        if result.found:
            # Recursive drill-down
            return find_needle(chunk, query)

    # Base case: small enough to read directly
    return rlm.call(f"Extract answer from: {context_var}")

Complexity: $O(\log N)$ — exponentially faster than a linear scan. The takeaway: divide-and-conquer works for reading, not just for sorting.

Architecture

The RLM system consists of:

Component	Role
Root LLM	Orchestrates the search, never sees raw context
REPL Environment	Holds context as variable, executes model-generated code
Sub-LLM Calls	Recursive invocations on context slices
Sandbox	Secure execution (Docker, Modal, or local)

Results

From the paper’s benchmarks:

Task	Vanilla LLM	RLM	Improvement
Needle-in-Haystack (1M tokens)	23%	94%	+71%
Multi-hop QA	31%	78%	+47%
Long Document Summarization	45%	82%	+37%

Key findings:

• Processes inputs 100x beyond context windows
• No degradation at 10M+ tokens
• RLM-Qwen3-8B outperforms base model by 28.3% on average
• Approaches GPT-5 quality on long-context tasks

Code Example

Using the official RLM library:

from rlm import RLM

# Initialize with any backend
rlm = RLM(
    backend="openai",
    backend_kwargs={"model_name": "gpt-5-nano"},
    verbose=True,
)

# Process arbitrarily long context
with open("giant_document.txt") as f:
    context = f.read()  # 10M+ characters

result = rlm.completion(
    query="What are the key findings about climate change?",
    context=context
)
print(result.response)

Why “Recursive”?

The model calls itself on sub-problems — the classic definition of recursion:

rlm(query, full_context)
├── rlm(query, chunk_1)
│   ├── rlm(query, chunk_1a)
│   └── rlm(query, chunk_1b)
├── rlm(query, chunk_2)
└── combine(results)

Each sub-call can spawn its own sub-calls until reaching a base case small enough to answer directly.

Limitations

• Latency overhead: synchronous sub-calls increase end-to-end time
• Simple tasks: overkill for short contexts where direct inference is faster
• Cost: multiple LLM calls per query
• Complexity: requires REPL environment setup

Future Directions

• Asynchronous sub-calls: parallel recursive queries
• Native training: models trained end-to-end for recursive reasoning
• Long-horizon agents: tasks spanning weeks with persistent context management

Common Confusion

• RLMs vs. long-context models: a long-context model reads more tokens in one forward pass; an RLM keeps the model’s window small and navigates a large input with code. They attack the same problem from opposite directions.
• RLMs vs. RAG (retrieval-augmented generation): RAG retrieves chunks with a separate search index chosen ahead of time; an RLM decides at inference time, in code it writes itself, which parts of the input to read and can recurse on them.
• “Recursive” does not mean recurrent: this has nothing to do with recurrent neural networks. The recursion is at the level of whole model calls, not hidden states.

Where To Go Next

• Read In-Context Learning to understand what a model can do with the context it does see.
• Read Chain-of-Thought — RLMs extend the same “spend more compute by generating more steps” idea to code and sub-calls.
• Read Transformer to see why attention cost and context limits exist in the first place.
• Read Scaling Laws for the background on model capability vs. compute.

Key Resources

• Paper: Recursive Language Models (arXiv:2512.24601) https://arxiv.org/abs/2512.24601

• Code: Official implementation https://github.com/alexzhang13/rlm

• Blog: Prime Intellect’s RLM overview https://www.primeintellect.ai/blog/rlm