ResNet

ResNet (Residual Network) introduced skip connections: shortcuts that let a layer’s input bypass the layer and be added back to its output. This simple change made it possible to train networks with over 150 layers, and ResNet won the 2015 ImageNet challenge with a 3.57% error rate — surpassing human-level performance.

If CNNs are new to you, read AlexNet or CS231n first. This page covers the original ResNet. For the follow-up paper that refined the design, see Identity Mappings in Deep Residual Networks.

Why Students Should Care

• Skip connections are now everywhere: nearly every modern vision model and every Transformer uses residual connections.
• ResNet explains a core lesson of deep learning: the obstacle to depth was not overfitting but optimization — and a small architectural change fixed it.
• ResNet-style backbones remained the default in computer vision for years and are still widely used.

The Degradation Problem

Here is the puzzle that motivated ResNet. You take a 20-layer network that works well, add more layers, and expect it to do at least as well — the extra layers could simply copy their input. Instead, the deeper network gets worse:

Crucially, this was not overfitting — training error also increased. The deeper network was not memorizing too much; it was failing to optimize at all. Apparently, plain stacked layers find it surprisingly hard to learn even the identity function (“just pass the input through unchanged”).

Residual Learning

ResNet’s fix: stop asking layers to learn the full mapping. Instead of learning a desired mapping $H(x)$ directly, each block learns only the residual — the difference between the output and the input:

The block’s output then becomes:

The key insight: if the best thing a block can do is nothing (identity), it is much easier to push $F(x) \rightarrow 0$ (just shrink some weights toward zero) than to make a stack of nonlinear layers learn $H(x) = x$ exactly.

You do not need to memorize the equations. The important idea is: each block learns a small correction to its input, not a whole new representation.

Skip Connections

The identity shortcut $x$ bypasses the layers and is added to the output:

These shortcuts:

• Add no extra parameters — the shortcut is just addition
• Enable gradient flow through hundreds of layers
• Allow each block to refine features rather than transform them completely

Block Architectures

• Basic Block (ResNet-18/34): two 3×3 convolutions
• Bottleneck Block (ResNet-50/101/152): 1×1 → 3×3 → 1×1 convolutions, where the 1×1 layers shrink and then restore the channel count for efficiency

Interactive Demo

Explore residual blocks and toggle skip connections to see their effect:

Why It Works

Skip connections create gradient highways. Look at how the gradient flows backward through a block:

The “1” term is the shortcut. Even if the gradient through the layers ($\partial F / \partial x$) becomes tiny, the gradient can still flow directly backward through the addition. That prevents vanishing gradients even in very deep networks.

One-line takeaway: the shortcut guarantees every layer receives a usable gradient, no matter how deep the network is.

Impact

ResNet’s influence extends far beyond image classification:

• Foundation for most modern vision architectures
• Inspired the residual connections used in Transformers
• Enabled training of networks with 1000+ layers (see Identity Mappings)

Common Confusion

• The degradation problem is not overfitting. Overfitting means low training error but high test error. Degradation means even training error got worse with depth — an optimization failure, not a generalization failure.
• ResNet is the architecture; a skip (residual) connection is the technique. Skip connections now appear in many models that are not ResNets, including Transformers.
• ResNet vs. Identity Mappings: the original 2015 paper introduced residual learning; the 2016 follow-up analyzed the shortcuts more carefully and proposed the improved pre-activation design.

Where To Go Next

• Read Identity Mappings in Deep Residual Networks for the refined pre-activation design
• Read AlexNet for the network that started the deep CNN era
• Read Batch Normalization for the other key ingredient in training deep networks
• Read Transformer to see residual connections outside of vision

Key Papers

• Deep Residual Learning for Image Recognition — He, Zhang, Ren, Sun (2015)
  https://arxiv.org/abs/1512.03385