TensorTau PathToAGI Doing Machine Learning Honestly Level 05

Level 05 · Doing Machine Learning Honestly · Chapter 074

Statistical Significance for ML Experiments

Model B is 0.6 points better than A. Is that convincing evidence or ordinary experimental noise?

1. Start with a null model

A statistical test often asks how surprising our observed difference would be under a null hypothesis such as “the methods have equal expected performance.”

A small p-value means the data would be unusual under that null model. It is not the probability that the null hypothesis is true.

2. Significance is not importance

With enough data, a tiny practically irrelevant difference can be statistically detectable. Always report effect size and uncertainty alongside significance.

statistical question: is the difference distinguishable from noise?
practical question:   is the difference worth caring about?

3. Use paired evidence when appropriate

If A and B are evaluated on the same examples or same seeds, tests that use pairing can remove much irrelevant variation. The exact test depends on metric, sampling design and assumptions.

4. Multiple comparisons

Try enough architectures, datasets and metrics and some “significant” result may occur by chance. Pre-specified hypotheses, correction methods and transparent reporting reduce this danger.

What to Remember

Statistical significance is one tool for quantifying evidence—not a magic stamp of truth. Effect size, uncertainty, design quality and practical consequences matter too.

Next → How to Avoid Cherry-Picking: how can an entire experimental workflow accidentally manufacture a success story?

Six ways to work the chapter, climbing from explaining it in your own words to investigating it like a researcher. Reading is not enough — this is where the chapter lands.

  1. Explain: Why is p<0.05 not “95% probability the idea is true”?
  2. Calculate: Compare effect sizes to their run-to-run SDs.
  3. Predict: What happens to false positives as the number of uncorrected tests grows?
  4. Break: Simulate many comparisons where all methods are actually equal and find apparent winners.
  5. Build: Implement a paired permutation test for per-example correctness differences.
  6. Investigate: Report effect size, CI and a suitable significance analysis for A vs B.