TensorTau PathToAGI Doing Machine Learning Honestly Level 05

Level 05 · Doing Machine Learning Honestly · Chapter 072

Confidence Intervals

Our experiment produced a mean score—but how uncertain is that estimate?

A standard deviation describes variation among observations/runs. A confidence interval targets uncertainty in an estimated quantity such as a mean.

For independent observations and suitable assumptions, a familiar interval is

xˉ±t∗sn.\bar x \pm t^*\frac{s}{\sqrt n}.

The s/ns/\sqrt n term is the standard error of the mean.

1. Interpretation matters

A frequentist 95% confidence interval does not mean there is a 95% probability that this already-computed fixed interval contains the fixed true mean. Its guarantee concerns the long-run behaviour of the interval-building procedure under its assumptions.

2. ML data can violate assumptions

Seeds, folds and examples may not be truly independent. Test-set metrics such as accuracy are often computed on finite shared examples. Bootstrap procedures can be useful, but resampling must respect groups/time dependence.

3. Practical meaning

Intervals force us to distinguish “92.0” from “92.0 ± substantial uncertainty.” If intervals are wide, collecting more evidence may be more valuable than debating tiny score differences.

What to Remember

Confidence intervals communicate estimation uncertainty, but their validity depends on the sampling procedure and assumptions.

Next → Ablation Studies: once a system has many components, how do we learn which ingredient actually caused the improvement?

Six ways to work the chapter, climbing from explaining it in your own words to investigating it like a researcher. Reading is not enough — this is where the chapter lands.

  1. Explain: Standard deviation versus standard error.
  2. Calculate: Compute SE for SD=2.0 with n=4,16,100.
  3. Predict: How does interval width scale as n quadruples?
  4. Break: Treat highly correlated repeated measurements as independent and inspect falsely narrow intervals.
  5. Build: Implement a bootstrap confidence interval for accuracy.
  6. Investigate: Compare normal/t-based and bootstrap intervals across different sample sizes.