Gradient Clipping

A common method clips the global gradient norm. Let all parameter gradients form one conceptual vector $g$ with norm $\lVert g\rVert$.

If the norm exceeds threshold $c$:

$g\leftarrow g\frac{c}{\lVert g\rVert}.$

The direction stays the same; the magnitude is capped.

1. Example

If gradient norm is 100 and threshold is 5, multiply every gradient by 0.05. The new global norm is 5.

2. Why clipping helps

Recurrent networks and unstable deep models can occasionally produce exploding gradients. One extreme step may destroy useful parameters or create NaNs. Clipping is a safety rail.

3. Clipping is not a cure for every instability

If gradients are always enormous, inspect learning rate, initialization, normalization, loss scaling and data. Constant clipping can hide a deeper problem.

4. PyTorch

loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()

Clip after gradients are computed and before the optimizer update.

What to Remember

Norm clipping caps dangerous gradient magnitude while preserving direction, but persistent clipping is a diagnostic signal.

Next → Mixed Precision: once training is numerically stable, can we use lower-precision arithmetic to make it faster and smaller without losing useful accuracy?