RMSProp

RMSProp tracks an exponential moving average of squared gradients:

$s_t=\rho s_{t-1}+(1-\rho)g_t^2.$

Then

$\theta_t=\theta_{t-1}-\eta\frac{g_t}{\sqrt{s_t}+\epsilon}.$

Operations are elementwise.

1. The intuition

If one parameter repeatedly receives large gradients, its $s_t$ grows and the denominator shrinks its effective step. A parameter with consistently smaller gradients gets less reduction.

large-gradient coordinate → larger denominator → smaller effective move
small-gradient coordinate → smaller denominator → relatively larger move

This can reduce zig-zagging on differently scaled directions.

2. Why square gradients?

Squaring removes sign and measures magnitude. The moving average remembers recent gradient scale without storing full history.

3. Epsilon

$\epsilon$ prevents division by zero and improves numerical stability. It looks tiny but is part of the algorithm.

4. Relation to momentum

Momentum adapts direction through first-moment history. RMSProp adapts scale through second-moment-like history. Adam will combine both ideas.

What to Remember

RMSProp normalizes each coordinate’s gradient by a running estimate of its recent magnitude.

Next → Adam: what happens when we combine momentum-like direction memory with RMSProp-like scaling?