Learning-Rate Schedules

A fixed learning rate is simple. Schedules deliberately change $\eta_t$ over training.

1. Warmup

Large early updates can be unstable when activations, optimizer moments and representations are not yet settled. Warmup gradually raises the learning rate from a small value.

2. Decay

Later in training, smaller steps help refine the solution. Common schedules include step decay, exponential decay, cosine decay and polynomial decay.

Cosine decay has the form

$\eta_t=\eta_{min}+\frac12(\eta_{max}-\eta_{min})\left(1+\cos\frac{\pi t}{T}\right).$

3. One-cycle intuition

Some schedules intentionally raise learning rate and then lower it, using large steps for exploration and small steps for convergence.

4. Count the right unit

Schedules may be defined per optimizer step, epoch or token. For large language models, “number of tokens processed” is often more meaningful than epochs.

5. Schedule and optimizer are coupled

A learning rate that works for Adam may not work for SGD. Changing batch size can also change a useful learning-rate range.

What to Remember

The learning rate is a trajectory, not necessarily one constant. Warmup manages early instability; decay supports later refinement.

Next → Weight Initialization: before the first optimizer step, how should millions of weights be chosen so signals can even travel through the network?