Momentum

Plain SGD uses only the current gradient. Momentum maintains a velocity:

$v_t=\beta v_{t-1}+g_t,$

$\theta_t=\theta_{t-1}-\eta v_t.$

1. Ball rolling downhill

Imagine a ball in a narrow valley. Gradients may alternate left/right across the valley while consistently pointing forward along it.

SGD:      /\/\/\/\/→
momentum: ─────────→

Repeated forward components accumulate. Oscillating components partially cancel.

2. Exponential memory

Expanding the recurrence shows

$v_t=g_t+\beta g_{t-1}+\beta^2g_{t-2}+\cdots.$

Recent gradients matter most; older ones fade geometrically. With $\beta=0.9$, the optimizer remembers a substantial history without storing every gradient.

3. Why it can overshoot

Velocity has inertia. Near a minimum, momentum may carry parameters past the optimum. Learning rate and momentum must be tuned together.

4. PyTorch

opt = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

The optimizer stores one velocity tensor per parameter.

What to Remember

Momentum smooths noisy directions and accelerates persistent ones by accumulating an exponentially weighted gradient history.

Next → RMSProp: momentum remembers direction; can we also adapt step sizes to the scale of each parameter’s gradients?