Batch Normalization

For one feature across a mini-batch, BatchNorm computes

$\mu_B=\frac1B\sum_i x_i,$

$\sigma_B^2=\frac1B\sum_i(x_i-\mu_B)^2,$

$\hat x_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}},$

then restores learnable freedom:

$y_i=\gamma\hat x_i+\beta.$

1. Why gamma and beta?

Normalization should not permanently force every representation to mean 0 and variance 1. Learned $\gamma$ and $\beta$ let the network choose an appropriate scale and offset.

2. Training and inference differ

During training, BatchNorm uses batch statistics. During inference, it typically uses running estimates accumulated during training. This is why model.train() and model.eval() matter.

3. Batch dependence

Very small or non-i.i.d. batches can make statistics noisy. BatchNorm also couples examples through shared batch statistics.

4. CNN shapes

For PyTorch image tensors [B,C,H,W], BatchNorm2d normally learns one gamma/beta per channel and aggregates statistics across batch and spatial positions.

What to Remember

BatchNorm normalizes using batch-dependent statistics, then applies learned scale/shift. It often stabilizes optimization but behaves differently at train and inference time.

Next → Layer Normalization: can we normalize each example independently so sequence models do not depend on other batch members?