# Batch Normalization

> Normalizing layer inputs to accelerate deep network training

- Type: Concept
- By: Ioffe & Szegedy
- Year: 2015
- Topics: Deep learning, Training, Optimization, Architecture
- Page: https://aipedia.org/batch-normalization

## Resources

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://arxiv.org/abs/1502.03167): Paper, Ioffe & Szegedy, 2015
- [Layer Normalization](https://arxiv.org/abs/1607.06450): Paper, Ba, Kiros & Hinton, 2016
- [Group Normalization](https://arxiv.org/abs/1803.08494): Paper, Wu & He, 2018
- [How Does Batch Normalization Help Optimization?](https://arxiv.org/abs/1805.11604): Paper, Santurkar et al., 2018

**Batch Normalization** (BatchNorm), introduced by Ioffe and Szegedy in 2015, is one of the most important techniques in deep learning. The idea: before passing a layer's outputs onward, rescale them so each feature has roughly zero mean and unit variance across the current mini-batch. This small change enables faster convergence, higher learning rates, and more stable training of deep networks.

> This page covers normalization across the **batch**. Its sibling, [Layer Normalization](https://aipedia.org/layer-normalization), normalizes across the features of a single example and is the default in Transformers. Reading both makes the contrast click.

## Why Students Should Care

- BatchNorm (or a variant of it) appears in nearly every modern architecture; it helped make very deep networks like [ResNet](https://aipedia.org/resnet) trainable.
- It is a classic example of a simple idea with a huge practical payoff — and a debated explanation for why it works.
- The train-vs-inference distinction in BatchNorm is a common source of real-world bugs (`model.eval()` exists largely because of it).

## The Problem: Shifting Inputs

Picture layer 5 of a deep network. Its input is the output of layers 1 through 4 — and those layers' weights change with every training step. So layer 5's input distribution keeps drifting: the "ground moves under its feet" while it tries to learn. The original paper named this **internal covariate shift**.

Networks must constantly adapt to new input distributions, slowing training.

## The Solution: Normalize Each Mini-Batch

BatchNorm's fix: standardize each feature using the statistics of the current mini-batch. For each feature in a layer:

$$
\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}
$$

where:
- $\mu_B = \frac{1}{m}\sum_{i=1}^m x_i$ (batch mean)
- $\sigma_B^2 = \frac{1}{m}\sum_{i=1}^m (x_i - \mu_B)^2$ (batch variance)
- $\epsilon$ is a small constant for numerical stability

This is the familiar "subtract the mean, divide by the standard deviation" from statistics — applied per feature, per mini-batch, inside the network.

## Learnable Parameters

Forcing every feature to have zero mean and unit variance would limit what the layer can represent. So BatchNorm adds a learnable scale ($\gamma$) and shift ($\beta$):

$$
y_i = \gamma \hat{x}_i + \beta
$$

The takeaway: **the network normalizes first, then learns whatever scale and offset it actually wants** — including undoing the normalization entirely if that helps.

## Interactive Demo

Visualize how BatchNorm stabilizes activations during training:

*Interactive demo: https://aipedia.org/batch-normalization*

## Where to Apply BatchNorm

```
Without BatchNorm:    Input → Linear → Activation → Linear → ...
With BatchNorm:       Input → Linear → BatchNorm → Activation → Linear → ...
```

Typically applied after the linear transformation but before the activation function.

## Training vs. Inference

Here is the subtle part. At inference time you might process a single example — there is no batch to compute statistics from. So BatchNorm behaves differently in the two modes:

**During training**: Use mini-batch statistics ($\mu_B$, $\sigma_B^2$)

**During inference**: Use running averages computed during training:

$$
\mu_{running} = \alpha \cdot \mu_{running} + (1-\alpha) \cdot \mu_B
$$

This ensures deterministic outputs at inference time. Forgetting to switch modes (in PyTorch, `model.eval()`) is a classic bug.

## Why It Works

The original paper attributed success to reducing internal covariate shift, but later research suggests other factors:

1. **Smoother loss landscape**: BatchNorm makes the optimization surface smoother, allowing larger learning rates
2. **Gradient flow**: Normalization prevents gradients from vanishing or exploding
3. **Regularization**: Mini-batch noise acts as a regularizer

The honest summary: BatchNorm clearly helps, and the *why* is still partly an open research question.

## Benefits

| Benefit | Explanation |
|---------|-------------|
| **Faster training** | 10-14x fewer training steps |
| **Higher learning rates** | Stable training with larger steps |
| **Reduced initialization sensitivity** | Less dependent on weight initialization |
| **Regularization effect** | Reduces need for dropout |

## BatchNorm Variants

The family differs mainly in *which dimension* gets normalized:

| Variant | Normalization Dimension | Use Case |
|---------|------------------------|----------|
| **BatchNorm** | Across batch | CNNs (requires large batches) |
| **LayerNorm** | Across features | Transformers, RNNs |
| **InstanceNorm** | Per sample, per channel | Style transfer |
| **GroupNorm** | Groups of channels | Small batch sizes |

## The Forward Pass

The core computation is only a few lines:

```python
def batch_norm(x, gamma, beta, eps=1e-5):
    # x shape: (batch_size, features)
    mu = x.mean(dim=0)
    var = x.var(dim=0)

    x_norm = (x - mu) / torch.sqrt(var + eps)

    return gamma * x_norm + beta
```

## Limitations

1. **Batch size dependency**: Small batches give noisy statistics
2. **Not suited for RNNs**: Variable sequence lengths complicate batch statistics
3. **Train/test discrepancy**: Different behavior in train vs. inference modes

For transformers and RNNs, [Layer Normalization](https://aipedia.org/layer-normalization) is preferred.

## The Math: Backward Pass

BatchNorm is differentiable, so gradients flow through the normalization during [backpropagation](https://aipedia.org/backpropagation). The gradients for the learnable parameters are simple:

$$
\frac{\partial L}{\partial \gamma} = \sum_i \frac{\partial L}{\partial y_i} \cdot \hat{x}_i
$$

$$
\frac{\partial L}{\partial \beta} = \sum_i \frac{\partial L}{\partial y_i}
$$

The gradient through $\hat{x}$ involves the chain rule through the mean and variance — messier, but your framework handles it automatically.

## Historical Impact

BatchNorm enabled:
- Training of much deeper networks (ResNet's 152 layers)
- Higher learning rates (faster experimentation)
- Reduced hyperparameter sensitivity
- Standard component in nearly all modern architectures

## Common Confusion

- **BatchNorm vs. LayerNorm**: BatchNorm normalizes each feature *across examples in the batch*; LayerNorm normalizes *across the features of one example*. This is the single most important distinction to remember.
- **BatchNorm is not just preprocessing.** Input standardization happens once to the data; BatchNorm happens inside the network, at every normalized layer, on every step, with learnable parameters.
- **Training and inference behave differently.** BatchNorm is one of the few layers where the same input can produce different outputs depending on the mode.
- **"Internal covariate shift" is the historical motivation, not the settled explanation** — later work (Santurkar et al., 2018) argues the benefit comes mainly from a smoother loss landscape.

## Where To Go Next

- Read [Layer Normalization](https://aipedia.org/layer-normalization) for the batch-independent variant used in Transformers.
- Read [ResNet](https://aipedia.org/resnet) for the architecture whose extreme depth BatchNorm helped make trainable.
- Read [Dropout](https://aipedia.org/dropout) for the other classic regularizer, and how the two interact.
- Read [Backpropagation](https://aipedia.org/backpropagation) if the backward-pass section felt too fast.