Recurrent Neural Network Regularization

Recurrent Neural Network Regularization is the paper by Zaremba, Sutskever, and Vinyals that solved an annoying practical problem: standard dropout, the go-to trick for preventing overfitting, breaks RNNs. Their fix — drop only the “vertical” connections, never the “horizontal” memory path — became the standard recipe.

Prerequisites: know what dropout is and how an LSTM carries memory across timesteps. Both pages are short.

Why Students Should Care

• LSTMs of this era overfit badly; without a working regularizer they had to be kept small. This paper let people train much larger recurrent models.
• It teaches a transferable lesson: where you apply a regularizer matters as much as whether you apply it.
• The “protect the memory path” insight echoes through later designs — residual streams in Transformers are similarly kept clean.

The Problem: Dropout Noise Compounds Over Time

Dropout randomly zeroes activations during training. In a feedforward net, each activation gets zapped at most once per forward pass — no big deal. But an RNN reuses its hidden state at every timestep. Naively dropping the recurrent connection looks like this:

$$h_t = f(W_h \cdot \text{dropout}(h_{t-1}) + W_x \cdot x_t)$$

Now the memory gets corrupted by fresh noise at every single step. Over a 100-step sequence, that is 100 rounds of damage — the network can no longer retain long-term information, which was the whole point of using an LSTM.

The Solution: Only Drop the Non-Recurrent Connections

Apply dropout on the way into and out of each layer, but leave the timestep-to-timestep path untouched:

$$h_t = f(W_h \cdot h_{t-1} + W_x \cdot \text{dropout}(x_t))$$ $$y_t = \text{dropout}(h_t)$$

The recurrent path $h_{t-1} \rightarrow h_t$ remains clean, preserving memory. Information still gets regularized — but only once per layer it passes through, not once per timestep.

Where to Apply Dropout

Connection	Apply Dropout?
Input → Hidden	✓ Yes
Hidden → Hidden (recurrent)	✗ No
Hidden → Output	✓ Yes
Between LSTM layers	✓ Yes

One rule of thumb captures the whole paper: drop vertical arrows, never horizontal ones.

Interactive Demo

Visualize how dropout randomly masks neurons during training:

Dropout Regularization

TrainingInference

Dropout Rate: 50%Resample

Training

Randomly zero neurons with probability p. Scale remaining by 1/(1-p).

Inference

Use all neurons. No scaling needed (inverted dropout).

Why Dropout Works

• Prevents co-adaptation: neurons can't rely on specific others

• Implicit ensemble: trains exponentially many sub-networks

• Noise injection: adds regularization similar to data augmentation

LSTM-Specific Details

For multi-layer LSTMs, dropout is applied between layers:

Layer 1: dropout(x) → LSTM → h1
Layer 2: dropout(h1) → LSTM → h2
Layer 3: dropout(h2) → LSTM → h3
Output:  dropout(h3) → Linear → y

Within each LSTM cell, the recurrent connections remain untouched.

Results

On Penn Treebank word-level language modeling (lower perplexity = better):

Model	Perplexity
LSTM (no dropout)	120.7
LSTM (naive dropout)	Diverges
LSTM (this paper)	78.4

Proper dropout reduces perplexity by 35% — and note that naive dropout is not just worse, it fails to train at all.

Mathematical Formulation

During training with dropout rate $p$, kept activations are rescaled:

$$\tilde{h}_t^l = \text{dropout}(h_t^l, p) \cdot \frac{1}{1-p}$$

The scaling factor ensures expected values match at test time, so you can turn dropout off for inference without recalibrating anything.

Key Insights

1. Dropout placement matters: the recurrent path needs to stay clean so memory and gradients survive
2. Same mask per sequence: use a consistent dropout mask across timesteps rather than resampling each step
3. Higher dropout for deeper networks: more layers tolerate — and benefit from — more regularization

Common Confusion

• This is not “no dropout in RNNs”: dropout works great in RNNs — just on the input, output, and between-layer connections, not on the hidden-to-hidden loop.
• Regularization vs. architecture: this paper changes training, not the LSTM cell itself. The model at test time is a plain LSTM.
• Later variants exist: “variational dropout” (Gal and Ghahramani) later showed a principled way to drop recurrent connections using one fixed mask per sequence; this paper’s simpler recipe was the established baseline.

Where To Go Next

• Read Dropout for the original technique and why randomly deleting neurons prevents overfitting.
• Read Understanding LSTMs for the memory dynamics this paper is careful not to break.
• Read RNN Effectiveness for the kind of language models this recipe made trainable at scale.
• Read Layer Normalization for another technique adapted specially for recurrent networks.

Key Paper

• Recurrent Neural Network Regularization — Zaremba, Sutskever, Vinyals (2014)
  https://arxiv.org/abs/1409.2329