091 — Stochastic Gradient Descent

Where We Are

Backpropagation gives gradients. We now need a rule for turning them into learning.

Full-batch gradient descent computes

$g=\frac1N\sum_{i=1}^{N}\nabla_\theta L_i.$

Then

$\theta\leftarrow\theta-\eta g.$

For modern datasets, computing that exact average before every update is expensive.

1. Use a sample instead

Stochastic gradient descent uses one example—or, in practice, a mini-batch—to estimate the full gradient.

$g_B=\frac1{|B|}\sum_{i\in B}\nabla_\theta L_i.$

If the batch is sampled fairly, this is a noisy but useful estimate of the population direction.

full gradient:       ─────────▶
SGD step 1:        ↗
SGD step 2:           ↘
SGD step 3:        →
long-run trend:       ────────▶

2. Noise is not only a cost

Noisy updates are cheaper, let us update more frequently, and can help optimization avoid some sharp or fragile regions. But too much noise makes training unstable.

3. Learning rate controls step size

$\Delta\theta=-\eta g.$

Too small: learning crawls. Too large: loss oscillates or diverges. The optimizer and learning-rate schedule therefore belong together.

4. PyTorch skeleton

optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
optimizer.zero_grad()
loss.backward()
optimizer.step()

backward() computes gradients. step() updates parameters. Keep those jobs conceptually separate.

What to Remember

SGD trades exact gradients for cheap, frequent, noisy estimates. Deep learning became practical partly because this approximation works remarkably well.

Next → Mini-Batches: how much data should each noisy gradient estimate use?