Vanishing and Exploding Gradients

Backprop repeatedly multiplies local Jacobians. In a scalar simplification,

$\frac{\partial L}{\partial x_0}=\frac{\partial L}{\partial x_n}\prod_{k=1}^{n}\frac{\partial x_k}{\partial x_{k-1}}.$

If each factor is about 0.5, then after 20 layers:

$0.5^{20}\approx9.5\times10^{-7}.$

If each factor is 1.5:

$1.5^{20}\approx3325.$

1. Vanishing

Early layers receive almost no learning signal. Saturating sigmoid/tanh derivatives can contribute strongly to this.

2. Exploding

Huge gradients create unstable parameter jumps, NaNs and divergence.

3. Remedies are connected

Careful initialization controls early scale. ReLU-like activations reduce saturation. Normalization stabilizes distributions. Residual connections shorten effective gradient paths. Gradient clipping limits extreme updates.

These are not disconnected tricks; they attack signal propagation from different angles.

4. Diagnose, do not guess

Log per-layer gradient norms:

for n,p in model.named_parameters():
    if p.grad is not None:
        print(n, p.grad.norm().item())

A healthy training loop observes gradients rather than assuming they are fine.

What to Remember

Depth multiplies derivative effects. Stable deep learning requires controlling the scale of forward activations and backward gradients.

Next → Batch Normalization: can we explicitly stabilize intermediate activation statistics during training?