Identity Mappings in Deep Residual Networks

Identity Mappings in Deep Residual Networks is the follow-up paper to ResNet. It asks a sharper question — why exactly do skip connections work, and can we make them work better? — and answers it with an improved design, the pre-activation ResNet, which enables training networks with over 1000 layers.

Read ResNet first. This page assumes you know what a residual block and a skip connection are.

Why Students Should Care

• This paper shows how a tiny architectural change — reordering ReLU and batch norm — can be the difference between a network that trains and one that does not.
• It gives the cleanest mathematical picture of why skip connections help, via a direct gradient formula.
• The lesson generalizes: keep the shortcut path as clean as possible. Modern architectures follow this principle.

The Problem with Original ResNet

Think of a residual network as having two paths: a shortcut path (the identity) and a residual path (the layers). The whole benefit of ResNet comes from the shortcut being a clean, unobstructed highway.

But in the original ResNet, the block applies ReLU after the addition:

$$h_{l+1} = \text{ReLU}(h_l + \mathcal{F}(h_l, W_l))$$

That means the shortcut path is not a pure identity — the signal passes through a nonlinearity at every block. Over hundreds of blocks, these small obstructions add up and can impede gradient flow in very deep networks.

Pure Identity Shortcuts

The key insight: for optimal gradient propagation, the shortcut should be a clean identity mapping — nothing on the highway at all:

$$h_{l+1} = h_l + \mathcal{F}(h_l, W_l)$$

Why does this matter? Apply the formula repeatedly and something nice happens — any deep layer is just the sum of a shallow layer plus all the residuals in between:

$$h_L = h_l + \sum_{i=l}^{L-1} \mathcal{F}(h_i, W_i)$$

In plain English: layer 1000 can see layer 1 directly, with no nonlinearities in the way.

Pre-activation Design

How do you get pure identity shortcuts while keeping batch norm and ReLU? Move them before the convolution instead of after (hence “pre-activation”):

$$y = x + \text{Conv}(\text{ReLU}(\text{BN}(x)))$$

Same components, different order. This subtle rearrangement leaves the shortcut path completely clean and enables training of 1001-layer networks.

Interactive Demo

Compare the original post-activation design with the improved pre-activation version:

Identity Mappings in ResNets

Original (Post-activation)

x

Conv

BN

ReLU

Conv

BN

+ add

ReLU

out

ReLU after addition blocks gradient

Pre-activation

x

BN

ReLU

Conv

BN

ReLU

Conv

+ add

out

Clean identity path for gradients

Gradient Flow Comparison

Original: gradient passes through ReLU after addition

Pre-activation: gradient flows directly via identity

Original:

h_l+1 = ReLU(h_l + F(h_l))

Pre-activation:

h_l+1 = h_l + F(BN(ReLU(h_l)))

1001

Layers trainable

4.92%

CIFAR-10 error

~0.5%

Improvement

Why Pre-activation Works

The gradient of the loss with respect to any layer $l$:

$$\frac{\partial \mathcal{L}}{\partial h_l} = \frac{\partial \mathcal{L}}{\partial h_L} \left(1 + \frac{\partial}{\partial h_l}\sum_{i=l}^{L-1}\mathcal{F}_i\right)$$

You do not need to memorize this. The important part is the “1”: it means the gradient at any layer includes a direct, undiminished copy of the gradient from the top of the network, unimpeded by nonlinearities. Vanishing gradients cannot happen along the shortcut path.

Results

Architecture	CIFAR-10 Error	CIFAR-100 Error
ResNet-110 (original)	6.61%	-
ResNet-110 (pre-act)	6.37%	-
ResNet-1001 (pre-act)	4.92%	22.71%

Pre-activation enables training of networks 10× deeper with better performance.

Common Confusion

• This is a different paper from ResNet. ResNet (2015) introduced residual learning; this paper (2016, same authors) analyzed it and improved the block design.
• Post-activation vs. pre-activation refers to where ReLU and BN sit relative to the addition: after the addition in the original design, before the convolution in the improved one.
• Both designs use skip connections. The difference is whether the shortcut path is a pure identity (pre-activation) or passes through a ReLU (original).

Where To Go Next

• Read ResNet if you have not already — this paper builds directly on it
• Read Batch Normalization to understand the BN component being reordered here
• Read Transformer to see clean residual paths reused in modern architectures

Key Papers

• Identity Mappings in Deep Residual Networks — He, Zhang, Ren, Sun (2016)
  https://arxiv.org/abs/1603.05027