Generalization

1. Training data is a sample, not the world

Imagine an unknown data-generating distribution $P(x,y)$. We observe a finite training set sampled from it. The real goal is low population risk:

$R(f)=\mathbb E_{(x,y)\sim P}[L(f(x),y)].$

But we cannot usually compute that expectation. We minimize empirical risk instead:

$\hat R(f)=\frac1N\sum_{i=1}^{N}L(f(x_i),y_i).$

Generalization asks when $\hat R$ is informative about $R$.

2. Independence matters

If train and test examples come from the same stable distribution and are sufficiently independent, held-out data estimates future performance. Leakage, duplicated entities and distribution shift break this reasoning.

3. Capacity matters

If we test one fixed coin, a sample estimates its probability reasonably well. If we search millions of rules and report only the luckiest one, some rule may fit by chance.

The richer the hypothesis class, the more data or regularization we generally need to distinguish signal from accidental fit.

4. Generalization is the central promise of ML

Optimization explains how we fit examples. Generalization explains why fitting examples can be useful beyond them.

Deep learning succeeds not because empirical loss alone guarantees truth, but because architecture, data scale, optimization, regularization and task structure together create solutions that often transfer.

What to Remember

• Population risk is what we ultimately care about.
• Empirical risk is what finite training data lets us measure.
• Generalization is the gap between those worlds.
• Independence, distribution match, data size and model capacity all matter.

Next → Sample Complexity Intuition: how does the amount of data needed grow as the learning problem becomes more demanding?