TensorTau PathToAGI Doing Machine Learning Honestly Level 05

Level 05 · Doing Machine Learning Honestly · Chapter 064

Error Analysis

Instead of blindly tuning the model, can we study its mistakes to discover what improvement is actually needed?

1. “20% error” is not one problem

Suppose an image classifier makes 200 mistakes:

80  blurry images
55  wrong labels
35  unusual viewpoints
20  tiny objects
10  other

A new architecture may not fix mislabeled data. Error categories turn a vague score into an engineering plan.

2. Inspect examples

For classification, study false positives and false negatives separately. Slice by meaningful groups: device type, geography, object size, sentence length, class, confidence, time period.

For regression, inspect the largest residuals and systematic patterns in residual plots.

3. Estimate the ceiling

If 27% of errors come from wrong labels, fixing modeling alone cannot recover all of them. If 60% come from low-resolution inputs, better data collection may beat another month of tuning.

4. Avoid post-hoc storytelling

Error analysis itself can overfit the validation set. Form hypotheses, make a change, and verify with untouched evidence or repeated experiments.

What to Remember

The aggregate metric tells you that the model fails. Error analysis helps tell you how and why.

Next → Hyperparameter Search: after identifying a real modeling problem, how should we search choices such as depth, learning rate and regularization?

Six ways to work the chapter, climbing from explaining it in your own words to investigating it like a researcher. Reading is not enough — this is where the chapter lands.

  1. Explain: Metric versus error analysis.
  2. Calculate: Convert error-category counts into percentages and prioritize them.
  3. Predict: Which errors are most likely fixed by collecting more data versus changing architecture?
  4. Break: Optimize only the easiest slice and show aggregate or other-slice harm.
  5. Build: Produce a dataframe of model errors sorted by confidence/residual.
  6. Investigate: Create a taxonomy for 100 real errors and propose three testable interventions.