1. Two kinds of instability
Imagine repeatedly collecting new training sets from the same world and fitting a model each time.
High bias: all fitted models miss in a similar way.
High variance: the fitted model changes dramatically from sample to sample.
high bias: ××× far from target
high variance: × × × scattered around target2. Squared-error decomposition
For regression, expected prediction error can be decomposed conceptually as
Bias asks whether the average prediction is systematically wrong. Variance asks how sensitive the learned prediction is to the sampled training data.
3. Complexity moves the balance
A rigid model tends toward high bias and low variance. A very flexible model tends toward lower training bias but can have high variance.
This is why model selection is not “choose the most powerful model.” We want the model/training setup with the best unseen-data behaviour.
4. Ensembles revisit the idea
Random Forests average many unstable trees. Averaging can reduce variance while retaining useful nonlinear structure. Boosting often attacks bias by adding corrections. Earlier algorithms now fit into a larger statistical story.
5. Modern deep learning nuance
The classical U-shaped story is useful, but modern overparameterized networks can show more complicated phenomena such as double descent. Treat bias–variance as a diagnostic lens, not an absolute law about parameter count.
What to Remember
Bias is systematic error; variance is sensitivity to sampled data. Good generalization requires managing both.
Next → Regularization: can we deliberately discourage a flexible model from using its freedom in fragile ways?