TensorTau PathToAGI Training Deep Networks Level 07

Level 07 · Training Deep Networks · Chapter 092

Mini-Batches

Why do deep-learning systems usually train on 32, 128, or thousands of examples at a time instead of exactly one or the whole dataset?

1. A compromise

A mini-batch of size BB estimates the gradient:

gB=1B∑i=1B∇Li.g_B=\frac1B\sum_{i=1}^{B}\nabla L_i.

Small batches are noisy but cheap per update. Large batches are smoother and exploit hardware well, but consume more memory and may require learning-rate adjustments.

2. Shape becomes operational

For images:

single image: [3,224,224]
batch of 64:  [64,3,224,224]

The batch axis allows GPUs to perform large matrix/tensor operations efficiently.

3. Epoch, step, batch

If a dataset has 10,000 examples and batch size is 100:

1 epoch = 10,000 examples seen
steps per epoch = 100

An epoch is a pass through data. A step is one optimizer update.

4. Gradient noise shrinks with batch size

Averaging more independent examples generally reduces gradient variance. But doubling batch size does not automatically double learning quality. Hardware utilization, optimizer settings, data correlation and generalization all matter.

5. Last batch and shuffling

Training loaders usually shuffle data each epoch and may produce a smaller final batch unless drop_last=True. For sequence models, batches may also involve padding or token-count constraints.

What to Remember

Batch size is simultaneously a statistical choice, memory choice and hardware choice.

Next → Momentum: if gradients zig-zag from batch noise and curved valleys, can the optimizer remember where it has been moving?

Six ways to work the chapter, climbing from explaining it in your own words to investigating it like a researcher. Reading is not enough — this is where the chapter lands.

  1. Explain: Batch, step and epoch.
  2. Calculate: Steps/epoch for N=50,000 and B=128.
  3. Predict: Which has noisier gradients: B=8 or B=2048?
  4. Break: Train with batch size 1 and the largest batch that fits; compare behaviour.
  5. Build: Write a mini-batch iterator from NumPy arrays.
  6. Investigate: Measure throughput, memory and validation performance across batch sizes.