TensorTau PathToAGI Doing Machine Learning Honestly Level 05

Level 05 · Doing Machine Learning Honestly · Chapter 061

Evaluation Metrics

What does “good model” mean when different applications care about different kinds of error?

1. Metrics encode priorities

For classification, accuracy answers “what fraction were correct?” Precision asks “when we predicted positive, how often were we right?” Recall asks “of all true positives, how many did we find?”

The harmonic mean combines precision and recall:

F1=2PRP+R.F_1=2\frac{PR}{P+R}.

But F1 is not automatically the best metric; it silently weights precision and recall in a particular way.

2. ROC and PR curves

A single threshold hides model behaviour. Curves sweep many thresholds.

ROC plots true-positive rate against false-positive rate. Precision–Recall curves are often more revealing for rare positive classes because they focus attention on positive predictions.

3. Regression metrics

Mean absolute error:

MAE=1N∑∣yi−y^i∣.MAE=\frac1N\sum|y_i-\hat y_i|.

Root mean squared error:

RMSE=1N∑(yi−y^i)2.RMSE=\sqrt{\frac1N\sum(y_i-\hat y_i)^2}.

RMSE punishes large errors more strongly. R2R^2 compares squared error against predicting the target mean, but can be misinterpreted if used without context.

4. Always connect metric to consequence

A 5-minute ETA error and a $500,000 price error are not equivalent merely because both are “one error.” Business or scientific costs should guide the metric and threshold.

What to Remember

There is no universally best metric. A metric is a mathematical statement about which mistakes matter.

Next → Calibration and Uncertainty: even if a model ranks examples correctly, do its reported probabilities mean what they claim?

Six ways to work the chapter, climbing from explaining it in your own words to investigating it like a researcher. Reading is not enough — this is where the chapter lands.

  1. Explain: Precision vs recall with a search-and-rescue example.
  2. Calculate: Compute accuracy, precision, recall and F1 from a confusion matrix.
  3. Predict: Which curve is often more informative for 0.5% positives: ROC or PR?
  4. Break: Find two classifiers with equal accuracy but very different minority behaviour.
  5. Build: Implement common classification/regression metrics from raw arrays.
  6. Investigate: Design a metric/threshold policy for a real decision with asymmetric costs.