# Recurrent Neural Network Regularization

> How to apply dropout to LSTMs without disrupting memory dynamics

- Type: Paper
- By: Zaremba, Sutskever & Vinyals
- Year: 2014
- Topics: Deep learning, RNNs, LSTMs, Regularization
- Page: https://aipedia.org/rnn-regularization

## Resources

- [Recurrent Neural Network Regularization](https://arxiv.org/abs/1409.2329): Paper, Zaremba, Sutskever & Vinyals, 2014

**Recurrent Neural Network Regularization** is the paper by Zaremba, Sutskever, and Vinyals that solved an annoying practical problem: standard dropout, the go-to trick for preventing overfitting, breaks RNNs. Their fix — drop only the "vertical" connections, never the "horizontal" memory path — became the standard recipe.

> Prerequisites: know what [dropout](https://aipedia.org/dropout) is and how an [LSTM](https://aipedia.org/understanding-lstms) carries memory across timesteps. Both pages are short.

## Why Students Should Care

- LSTMs of this era overfit badly; without a working regularizer they had to be kept small. This paper let people train much larger recurrent models.
- It teaches a transferable lesson: **where** you apply a regularizer matters as much as whether you apply it.
- The "protect the memory path" insight echoes through later designs — residual streams in [Transformers](https://aipedia.org/transformer) are similarly kept clean.

## The Problem: Dropout Noise Compounds Over Time

Dropout randomly zeroes activations during training. In a feedforward net, each activation gets zapped at most once per forward pass — no big deal. But an RNN reuses its hidden state at *every* timestep. Naively dropping the recurrent connection looks like this:

$$
h_t = f(W_h \cdot \text{dropout}(h_{t-1}) + W_x \cdot x_t)
$$

Now the memory gets corrupted by fresh noise at every single step. Over a 100-step sequence, that is 100 rounds of damage — the network can no longer retain long-term information, which was the whole point of using an LSTM.

## The Solution: Only Drop the Non-Recurrent Connections

Apply dropout on the way *into* and *out of* each layer, but leave the timestep-to-timestep path untouched:

$$
h_t = f(W_h \cdot h_{t-1} + W_x \cdot \text{dropout}(x_t))
$$

$$
y_t = \text{dropout}(h_t)
$$

The recurrent path $h_{t-1} \rightarrow h_t$ remains clean, preserving memory. Information still gets regularized — but only once per layer it passes through, not once per timestep.

## Where to Apply Dropout

| Connection | Apply Dropout? |
|-----------|---------------|
| Input → Hidden | ✓ Yes |
| Hidden → Hidden (recurrent) | ✗ No |
| Hidden → Output | ✓ Yes |
| Between LSTM layers | ✓ Yes |

One rule of thumb captures the whole paper: **drop vertical arrows, never horizontal ones**.

## Interactive Demo

Visualize how dropout randomly masks neurons during training:

*Interactive demo: https://aipedia.org/rnn-regularization*

## LSTM-Specific Details

For multi-layer LSTMs, dropout is applied between layers:

```
Layer 1: dropout(x) → LSTM → h1
Layer 2: dropout(h1) → LSTM → h2
Layer 3: dropout(h2) → LSTM → h3
Output:  dropout(h3) → Linear → y
```

Within each LSTM cell, the recurrent connections remain untouched.

## Results

On Penn Treebank word-level language modeling (lower perplexity = better):

| Model | Perplexity |
|-------|------------|
| LSTM (no dropout) | 120.7 |
| LSTM (naive dropout) | Diverges |
| LSTM (this paper) | **78.4** |

Proper dropout reduces perplexity by 35% — and note that naive dropout is not just worse, it fails to train at all.

## Mathematical Formulation

During training with dropout rate $p$, kept activations are rescaled:

$$
\tilde{h}_t^l = \text{dropout}(h_t^l, p) \cdot \frac{1}{1-p}
$$

The scaling factor ensures expected values match at test time, so you can turn dropout off for inference without recalibrating anything.

## Key Insights

1. **Dropout placement matters**: the recurrent path needs to stay clean so memory and gradients survive
2. **Same mask per sequence**: use a consistent dropout mask across timesteps rather than resampling each step
3. **Higher dropout for deeper networks**: more layers tolerate — and benefit from — more regularization

## Common Confusion

- **This is not "no dropout in RNNs"**: dropout works great in RNNs — just on the input, output, and between-layer connections, not on the hidden-to-hidden loop.
- **Regularization vs. architecture**: this paper changes *training*, not the LSTM cell itself. The model at test time is a plain LSTM.
- **Later variants exist**: "variational dropout" (Gal and Ghahramani) later showed a principled way to drop recurrent connections using one fixed mask per sequence; this paper's simpler recipe was the established baseline.

## Where To Go Next

- Read [Dropout](https://aipedia.org/dropout) for the original technique and why randomly deleting neurons prevents overfitting.
- Read [Understanding LSTMs](https://aipedia.org/understanding-lstms) for the memory dynamics this paper is careful not to break.
- Read [RNN Effectiveness](https://aipedia.org/rnn-effectiveness) for the kind of language models this recipe made trainable at scale.
- Read [Layer Normalization](https://aipedia.org/layer-normalization) for another technique adapted specially for recurrent networks.