Residual Connections

A residual block computes

$y=x+F(x).$

Instead of forcing a stack to learn the entire desired mapping $H(x)$, it can learn a residual

$F(x)=H(x)-x.$

1. Identity becomes easy

If the best transformation is “do almost nothing,” the block can set $F(x)\approx0$, leaving $y\approx x$.

Without a shortcut, many nonlinear layers would have to learn identity themselves.

2. Gradient has a direct path

Differentiate:

$\frac{\partial y}{\partial x}=I+\frac{\partial F}{\partial x}.$

The identity term creates a path for signal and gradient that does not depend entirely on the product of many learned Jacobians.

x ───────────────┐
│                + → y
└→ F(x) ─────────┘

3. Shapes must match

Addition requires compatible shapes. If channel/feature dimensions change, a projection such as a linear or 1×1 convolution can transform the shortcut.

4. Beyond ResNet

Residual connections became foundational not only in vision but in Transformers, where attention and MLP sublayers sit inside residual streams.

What to Remember

Residual connections create easy identity paths and let layers focus on learning corrections.

Next → Gradient Clipping: residual paths help gradients travel, but what if a particular update still becomes dangerously huge?