Bayes' Rule: What Evidence Does to Belief

Where We Are

Chapter 028 taught us $P(A\mid B)$. But evidence often arrives in the opposite direction from the question we care about. A test manufacturer reports $P(positive\mid disease)$. A patient asks for $P(disease\mid positive)$.

Those are not the same probability.

1. The Medical-Test Surprise

Imagine 10,000 people.

• 1% have a disease → 100 people.
• The test catches 99% of diseased people → 99 true positives.
• It falsely marks 5% of healthy people positive → 495 false positives among 9,900 healthy people.

So 594 people test positive, but only 99 are actually ill.

A 99%-sensitive test produced only about a 17% disease probability after a positive result.

Why? Because the disease was rare before testing. The base rate matters.

2. Derive Bayes Instead of Memorizing It

From conditional probability:

$P(A\mid B)=\frac{P(A\cap B)}{P(B)}.$

Also:

$P(B\mid A)=\frac{P(A\cap B)}{P(A)}.$

Therefore

$P(A\cap B)=P(B\mid A)P(A).$

Substitute:

That is Bayes’ rule. It is not magic. It is the same intersection counted from two directions.

3. Prior, Likelihood, Evidence, Posterior

For a hypothesis $H$ and evidence $E$:

Names:

• $P(H)$ — prior: belief before evidence,
• $P(E\mid H)$ — likelihood: how compatible evidence is with the hypothesis,
• $P(E)$ — evidence or marginal likelihood,
• $P(H\mid E)$ — posterior: updated belief.

Mental model:

prior belief
    ×
how well hypothesis predicts evidence
    ↓
renormalize
    ↓
posterior belief

4. Where Does P(E) Come From?

If hypotheses $H_1,\dots,H_k$ cover all possibilities,

For disease vs no disease:

This denominator ensures posterior probabilities sum to one.

5. Odds Form

Bayes can also be written as

For evidence $E$ comparing $H_1$ with $H_0$:

This makes the update interpretation especially clear: evidence multiplies existing odds.

6. Repeated Evidence

Suppose evidence arrives sequentially. Today we observe $E_1$, tomorrow $E_2$.

Today’s posterior becomes tomorrow’s prior.

prior → evidence 1 → posterior 1
                     ↓
                   new prior → evidence 2 → posterior 2

Bayesian inference is therefore a disciplined way to keep updating beliefs as information arrives.

7. Code From Scratch

def bayes(prior, sensitivity, false_positive_rate):
    p_positive = (
        sensitivity * prior
        + false_positive_rate * (1 - prior)
    )
    return sensitivity * prior / p_positive

print(bayes(0.01, 0.99, 0.05))
# about 0.167

Now vary only the prior from 0.01 to 0.10. The same test gives a very different posterior. That experiment teaches more than memorizing the equation.

8. Bayes in Machine Learning

Bayesian reasoning appears in:

• Naive Bayes classifiers,
• Bayesian linear regression,
• probabilistic graphical models,
• uncertainty estimation,
• model comparison,
• filtering and tracking,
• scientific inference.

Even when a neural network is not explicitly Bayesian, the distinction between prior information, evidence and updated belief is a powerful way to reason about predictions.

9. Break It: Base-Rate Neglect

A tempting mistake is:

“The test is 99% accurate, so a positive result means 99% chance of disease.”

This silently swaps $P(+\mid D)$ with $P(D\mid +)$ and ignores the prior prevalence.

Another mistake is to treat the likelihood as a probability distribution over hypotheses. $P(E\mid H)$ is a function of $H$ for fixed evidence, but it does not automatically sum or integrate to one over $H$.

10. Shape Check

Suppose there are 4 hypotheses and 20 observations. A table of likelihoods $P(E_i\mid H_j)$ has shape

[20, 4]

A prior over the 4 hypotheses has shape [4]. After multiplying each row by the prior and normalizing across hypotheses, the posterior table remains [20, 4].

11. History

Thomas Bayes’ posthumously published work introduced a form of inverse probability; Pierre-Simon Laplace developed the idea much further. The modern Bayesian view is not merely a formula but a framework: represent uncertainty, combine it with evidence, update coherently.

Mastery Check

1. Why can a highly sensitive medical test still have a modest $P(disease\mid positive)$?
2. Derive Bayes’ rule from conditional probability.
3. Explain prior, likelihood and posterior in ordinary language.
4. What does the evidence denominator do?
5. What is base-rate neglect?

Bridge to Chapter 030

So far our events were statements such as “disease” or “positive.” Machine learning works with quantities: price, token count, temperature, class index, reward.

How do we attach probabilities to values a quantity might take?

We need random variables and expectation.