Overfitting and Underfitting

1. Three possible fits

Imagine noisy points following a gentle curve.

underfit:   --------
good fit:     ~~~~~
overfit:    /\/\/\/\

Underfitting means the model cannot capture the real structure. Overfitting means it captures the structure plus accidents of the training sample.

2. Error patterns

These are diagnostic patterns, not rigid laws.

3. Capacity

Model capacity describes how many kinds of functions a model can represent. A straight line has less capacity than a degree-20 polynomial; a shallow tree less than an unrestricted tree.

As capacity rises, training error usually falls. Validation error often falls first, reaches a useful region, then rises.

4. Noise cannot be learned usefully

Suppose

$y=f(x)+\epsilon.$

The target contains true signal $f(x)$ and noise $\epsilon$. A model should learn the stable signal. Memorizing the particular noise in the training sample hurts future predictions.

5. More data changes the game

A high-capacity model trained on ten examples can memorize easily. The same model trained on ten million varied examples has much less freedom to memorize every accidental detail while remaining wrong everywhere else.

Capacity is therefore meaningful only relative to data, noise, task and training procedure.

What to Remember

• Underfitting: model too constrained or insufficiently trained to capture signal.
• Overfitting: model adapts to training-specific detail that does not generalize.
• Validation behaviour helps distinguish them.
• More capacity is neither automatically good nor bad.

Next → Bias–Variance Tradeoff: can we give these two failure modes a more precise statistical language?