Relational Recurrent Neural Networks

Relational Recurrent Neural Networks combine the step-by-step processing of RNNs with the relational reasoning of attention. Instead of one big memory vector, the model keeps several memory slots — and the slots can talk to each other using attention.

Prerequisites: Understanding LSTMs for gated memory, and the attention mechanism from Attention Is All You Need. This page combines the two.

Why Students Should Care

• It is a bridge architecture: half LSTM, half Transformer. Seeing both in one model clarifies what each ingredient contributes.
• It shows why attention helps reasoning: some tasks need stored facts to be compared, not just stored.
• The multi-slot memory idea connects to Neural Turing Machines and to how we think about memory in agents today.

The Motivating Problem

Suppose a model reads: “Ball A is 3m away. Ball B is 7m away. Ball C is 5m away. Which ball is second farthest?”

A standard LSTM squeezes everything into one fixed memory cell. It can store the three distances (roughly), but answering requires comparing stored facts against each other — an operation the LSTM cell simply does not have. Complex reasoning needs:

• Multiple pieces of information stored simultaneously
• Interactions between stored memories
• Dynamic retrieval based on relationships

Relational Memory Core (RMC)

The fix: keep a set of memory slots $M = [m_1, m_2, ..., m_N]$, and at every timestep let the slots update each other using attention:

$$M^{t+1} = \text{MHDPA}(M^t) + \text{MLP}(\text{MHDPA}(M^t))$$

where MHDPA is Multi-Head Dot Product Attention — the same attention used in Transformers.

You do not need to memorize the equation. The important idea is: each memory slot looks at all the other slots and updates itself based on what it finds — exactly like tokens attending to each other in a Transformer, but applied to memories instead.

Key Innovation

Memories attend to each other, not just to inputs:

$$A = \text{softmax}\left(\frac{M W_Q (M W_K)^T}{\sqrt{d_k}}\right) M W_V$$

This is standard query-key-value attention where the queries, keys, and values all come from the memory matrix itself. It is what lets the model compare “ball A’s distance” against “ball B’s distance” internally.

Interactive Demo

Watch memory slots interact via attention:

Relational Memory Core

←t = 0→

Memory Slots (M)

m₁

m₂

m₃

m₄

Attention between memories

Multi-Head Attention

Memory slots attend to each other via MHDPA (Multi-Head Dot Product Attention)

Gated Update

Attended memories combined via gates, similar to LSTM

Key Equation

M^t+1 = MHDPA(M^t) + MLP(MHDPA(M^t))

Memories update by attending to each other—enabling relational reasoning over time

Gating Mechanism

Attention alone would happily overwrite everything each step. Like LSTMs, the RMC uses gates to blend the new attended memory with the old one:

$$\tilde{M} = \sigma(W_g[\tilde{A}; M]) \odot \tilde{A} + (1 - \sigma(W_g[\tilde{A}; M])) \odot M$$

The takeaway: a learned gate decides, per element, how much of the freshly-computed memory to accept and how much of the old memory to keep — preventing catastrophic forgetting of important facts.

Architecture

Input → Linear projection → Concatenate with memories
     → Multi-head self-attention over all slots
     → MLP (residual)
     → Gated update
     → Output from attended memories

Results

Language Modeling (WikiText-103)

Model	Perplexity
LSTM	48.7
Transformer	44.1
Relational Memory	31.6

Program Evaluation (Nth Farthest)

Task: given N objects, find the Nth farthest from a query — a pure relational-comparison task.

Model	Accuracy
LSTM	17%
DNC	37%
RMC	91%

The gap on Nth Farthest is the headline: when the task is all comparison, memory interaction is not a nice-to-have — it is the difference between failing and solving it.

Why It Works

1. Multiple memories: can store several facts side by side
2. Memory interaction: facts can “talk” to each other via attention
3. Attention routing: retrieval is dynamic, based on relevance
4. Temporal integration: still processes sequences step by step like an RNN

Connection to Transformers

RMC uses key Transformer ingredients:

• Multi-head attention
• Residual connections
• Layer normalization

The main difference: RMC processes sequences recurrently (one step at a time, with a fixed number of memory slots), while Transformers process all positions in parallel.

Common Confusion

• Relational RNN vs. Relation Networks: same lab, related idea, different mechanism. Relation Networks compare all pairs of objects in a single input; the RMC lets memory slots interact across time.
• Not a Transformer with extra steps: the RMC is still recurrent — it has a bounded memory that persists across timesteps, whereas a Transformer re-reads the whole sequence.
• Slots vs. cell state: an LSTM has one memory vector updated by gates; the RMC has $N$ memory vectors updated by attention and gates.

Where To Go Next

• Read Relational Reasoning for the pairwise-comparison idea in its simplest form.
• Read Understanding LSTMs for the gating machinery the RMC borrows.
• Read Attention Is All You Need for the attention machinery it borrows.
• Read Neural Turing Machines for an earlier take on slot-based external memory.
• Read Transformer for the architecture that ultimately won by dropping recurrence entirely.

Key Paper

• Relational Recurrent Neural Networks — Santoro et al. (2018)
  https://arxiv.org/abs/1806.01822