KL Divergence

1. Two Distributions

Suppose reality is

$P=[0.9,0.1]$

but our model says

$Q=[0.5,0.5].$

Both are valid probability distributions. Entropy tells us uncertainty inside one distribution. It does not directly tell us how badly $Q$ approximates $P$.

KL divergence does:

Interpretation: expected extra log-surprise when data comes from $P$ but we encode or model it using $Q$.

2. Why P Weights the Sum

The world generates outcomes according to $P$, so mismatch matters most where $P$ places mass. A model can waste probability on impossible regions, but it is especially costly when it assigns tiny probability to outcomes reality frequently produces.

If $P(x)>0$ but $Q(x)=0$, KL becomes infinite: the model declared a possible real event impossible.

3. KL Is Not a Distance

KL satisfies

$D_{KL}(P\|Q)\ge0$

and equals zero only when the distributions match (almost everywhere).

But generally

$D_{KL}(P\|Q)\ne D_{KL}(Q\|P).$

So KL is asymmetric and is not a metric distance.

The direction matters.

4. Connection to Cross-Entropy

Expand:

Since

$H(P)=-\sum P\log P$

and

$H(P,Q)=-\sum P\log Q,$

we get

If $P$ is fixed, minimizing cross-entropy with respect to $Q$ is equivalent to minimizing $D_{KL}(P\|Q)$.

This is why classification training can be understood as pushing the model distribution toward the target distribution.

5. Tiny Example

Let target be one-hot:

$P=[1,0,0].$

Model predicts

$Q=[0.7,0.2,0.1].$

Cross-entropy is

Only the true class survives because all other $P_i$ are zero.

For a one-hot target, $H(P)=0$, so cross-entropy equals KL divergence from target to model.

6. Code

import numpy as np

def kl(p, q):
    p = np.asarray(p, float)
    q = np.asarray(q, float)
    mask = p > 0
    return np.sum(p[mask] * np.log(p[mask] / q[mask]))

print(kl([0.9, 0.1], [0.5, 0.5]))
print(kl([0.5, 0.5], [0.9, 0.1]))

Predict whether the two outputs match before running.

7. Shape Check

For distributions over 10 classes:

P: [64, 10]
Q: [64, 10]

Compute KL across the class axis and obtain

KL per example: [64]

Average across the batch to get a scalar training objective.

8. Real-World Connection

KL divergence appears in variational inference, VAEs, knowledge distillation, RL policy constraints, distribution shift analysis and information theory. Different KL directions can encourage different approximation behavior, which becomes important in generative modeling.

Mastery Check

1. Why is KL weighted by $P$?
2. Why can KL be infinite?
3. Why is it not a metric distance?
4. Derive $H(P,Q)=H(P)+D_{KL}(P\|Q)$.
5. Why does minimizing cross-entropy minimize KL when $P$ is fixed?

Bridge to Chapter 037

KL exposed cross-entropy inside its algebra. Now we can study the loss directly and answer the practical question:

Why does classification training use cross-entropy instead of squared error?