TensorTau PathToAGI Doing Machine Learning Honestly Level 05

Level 05 · Doing Machine Learning Honestly · Chapter 060

Class Imbalance

If 99 out of 100 examples belong to one class, can 99% accuracy describe a useless model?

1. The accuracy trap

Suppose 1% of transactions are fraud. A model that always predicts “not fraud” gets

accuracy=99%.\text{accuracy}=99\%.

It catches zero frauds.

The arithmetic is correct; the conclusion is wrong.

2. Confusion matrix first

                 predicted +   predicted -
actual +              TP            FN
actual -              FP            TN

From these four counts we can ask meaningful questions:

precision=TPTP+FP\text{precision}=\frac{TP}{TP+FP}

recall=TPTP+FN\text{recall}=\frac{TP}{TP+FN}

Fraud detection may value recall because missed fraud is expensive. An alerting system may also care deeply about precision because too many false alarms make humans ignore it.

3. The threshold is a decision

A classifier often outputs a score or probability. Changing the decision threshold trades false positives against false negatives.

Do not treat 0.5 as a law of nature.

4. Training strategies

Possible responses include class-weighted losses, over/under-sampling, focal loss in some deep-learning settings, collecting more minority examples and threshold tuning based on deployment costs.

Never balance the test set merely to make the metric look nicer; evaluation should reflect the question you actually care about.

What to Remember

  • Accuracy can hide minority-class failure.
  • Inspect the confusion matrix.
  • Precision and recall answer different operational questions.
  • Thresholds and costs belong to the problem definition.

Next → Evaluation Metrics: which number should we use when different mistakes have different meanings?

Six ways to work the chapter, climbing from explaining it in your own words to investigating it like a researcher. Reading is not enough — this is where the chapter lands.

  1. Explain: Why is 99% accuracy sometimes worthless?
  2. Calculate: For TP=40, FP=10, FN=60, TN=9890 compute accuracy, precision and recall.
  3. Predict: What happens to recall as the positive threshold is lowered?
  4. Break: Build an always-majority classifier on a 99:1 dataset.
  5. Build: Plot precision and recall across thresholds.
  6. Investigate: Choose a threshold from an explicit false-positive/false-negative cost model.