Layer Normalization

LayerNorm normalizes features within each example rather than a feature across batch examples.

For hidden vector $x\in\mathbb R^D$:

$\mu=\frac1D\sum_jx_j,$

$\sigma^2=\frac1D\sum_j(x_j-\mu)^2,$

$y_j=\gamma_j\frac{x_j-\mu}{\sqrt{\sigma^2+\epsilon}}+\beta_j.$

1. BatchNorm versus LayerNorm

BatchNorm: normalize a feature using many examples
LayerNorm: normalize many features inside one example

Therefore LayerNorm behaves consistently for batch size 1 and does not require running batch statistics at inference.

2. Why Transformers use it

Sequence batches can have variable lengths and batch composition. LayerNorm’s per-token/per-example hidden-feature normalization fits naturally with Transformer representations.

For a tensor [B,T,D], LayerNorm(D) normalizes along the last D dimension while preserving shape.

Shape Check

[B,T,D] → LayerNorm(D) → [B,T,D]

3. Pre-norm and post-norm

Transformer blocks differ in where LayerNorm is placed relative to attention/MLP and residual connections. Pre-norm architectures often improve optimization stability in deep Transformers.

What to Remember

LayerNorm stabilizes feature scale within each example/token and is independent of batch statistics.

Next → Dropout: instead of normalizing activations, can random temporary removal of features discourage brittle co-adaptation?