Weight Initialization

If every weight were zero, symmetric neurons would receive identical gradients and remain identical. We need randomness—but not arbitrary scale.

1. Variance through a layer

For

$z_j=\sum_{i=1}^{n}w_{ij}x_i,$

if terms are roughly independent and zero-mean,

$\operatorname{Var}(z)\approx n\operatorname{Var}(w)\operatorname{Var}(x).$

If weight variance is too large, activations grow layer after layer. Too small, they shrink.

2. Xavier/Glorot

For tanh-like activations, Xavier initialization chooses a scale based on fan-in/fan-out, roughly preserving variance across forward and backward flow.

A common variance is

$\operatorname{Var}(w)\approx\frac{2}{fan_{in}+fan_{out}}.$

3. He/Kaiming

ReLU zeros roughly half of symmetric inputs. He initialization compensates with

$\operatorname{Var}(w)\approx\frac{2}{fan_{in}}.$

4. Initialization is architecture-aware

Residual networks, Transformers and very deep models may use specialized scaling. Initialization interacts with normalization, activation and residual paths.

Shape Check

For a linear layer Linear(512,2048), fan-in is 512 and fan-out is 2048. The dimensions describe both connectivity and sensible initialization scale.

What to Remember

Good initialization breaks symmetry while keeping signal scales usable long enough for learning to begin.

Next → Vanishing and Exploding Gradients: why do deep chains make small scaling mistakes multiply dramatically?