TensorTau PathToAGI Training Deep Networks Level 07

Level 07 · Training Deep Networks · Chapter 095

Adam

Can we combine momentum’s directional memory with adaptive per-parameter scaling?

Adam maintains two moving averages:

mt=β1mt−1+(1−β1)gtm_t=\beta_1m_{t-1}+(1-\beta_1)g_t

vt=β2vt−1+(1−β2)gt2.v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2.

mtm_t tracks gradient direction; vtv_t tracks squared magnitude.

1. Bias correction

Both averages start at zero, so early values are biased toward zero. Adam corrects them:

\hat v_t=\frac{v_t}{1-\beta_2^t}.$$ Then $$\theta_t=\theta_{t-1}-\eta\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}.$$ ## 2. Why Adam is popular It often works well with little tuning, handles sparse/noisy gradients reasonably, and adapts scales per parameter. This made it a common default, especially for Transformers. ## 3. Adam is not “always better” Optimizer quality depends on task, model, schedule and regularization. SGD with momentum remains competitive in many vision settings. AdamW decouples weight decay from Adam's adaptive gradient update and is widely used in modern deep learning. ## 4. State cost Adam stores first- and second-moment tensors for each parameter, adding optimizer memory beyond weights and gradients. ## What to Remember Adam = momentum-like first moment + RMSProp-like second moment + bias correction. **Next → Learning-Rate Schedules: even with a smart optimizer, should step size stay constant from the first update to the last?**

Six ways to work the chapter, climbing from explaining it in your own words to investigating it like a researcher. Reading is not enough — this is where the chapter lands.

  1. Explain: Roles of Adam’s m and v.
  2. Calculate: Compute first-step bias-corrected moments for a scalar gradient.
  3. Predict: Why are uncorrected early moments too small?
  4. Break: Compare an absurdly large Adam learning rate with a sensible one.
  5. Build: Implement scalar/vector Adam from scratch.
  6. Investigate: Compare SGD+momentum, Adam and AdamW under matched training budgets.