Conditional Probability

Where We Are

Chapter 027 gave us probability as a language for uncertainty. But most real decisions are not made with zero context. A doctor learns a test result. A spam filter sees a word. A vision model sees an edge. We need probabilities after we know something.

1. The Problem: The Denominator Changed

Suppose a class has 100 students:

• 60 play cricket,
• 40 do not,
• 30 of the cricket players also play badminton.

If we choose a student from the whole class, the probability of badminton-and-cricket is 30/100.

But now someone tells us:

The chosen student plays cricket.

We are no longer choosing from 100 possible students. Our new world contains only the 60 cricket players. Of those 60, 30 play badminton.

So

The vertical bar means given that.

Conditional probability is ordinary probability after shrinking the universe to cases consistent with what we know.

2. Deriving the Formula

The cases satisfying both $A$ and $B$ have probability $P(A\cap B)$. Once we know $B$, the surviving universe has probability mass $P(B)$.

Therefore

provided $P(B)>0$.

Rearrange:

This multiplication rule will become the engine behind Bayes’ rule and probabilistic models.

3. A 2×2 Table Makes It Visible

Suppose 1,000 emails contain:

Before seeing the word:

$P(Spam)=300/1000=0.30.$

After learning the email contains “prize”:

$P(Spam\mid prize)=180/200=0.90.$

The evidence changed the reference population from 1,000 emails to 200 emails.

4. Direction Matters

A common mistake is to assume

$P(A\mid B)=P(B\mid A).$

Usually false.

From the table:

$P(Spam\mid prize)=180/200=0.90$

but

$P(prize\mid Spam)=180/300=0.60.$

Same intersection. Different denominator.

This asymmetry is exactly why medical-test reasoning is often surprising.

5. Independence

If learning $B$ does not change our belief about $A$, then

$P(A\mid B)=P(A).$

We call $A$ and $B$ independent.

Substitute into the multiplication rule:

Independence does not mean the events cannot happen together. It means knowing one gives no information about the other.

Mutually exclusive events are different: if one happens, the other cannot.

6. The Chain Rule of Probability

For three events,

More generally,

This identity is profound for modern AI. An autoregressive language model uses the same structure:

Predicting the next token is conditional probability repeated through a sequence.

7. Code From Scratch

spam = 300
prize = 200
spam_and_prize = 180

p_spam = spam / 1000
p_spam_given_prize = spam_and_prize / prize
p_prize_given_spam = spam_and_prize / spam

print(p_spam)              # 0.30
print(p_spam_given_prize)  # 0.90
print(p_prize_given_spam)  # 0.60

Do not memorize the division. Ask first: what is my new universe? That tells you the denominator.

8. Shape Check

Suppose a classifier predicts probabilities for a batch of 32 examples and 10 classes.

logits        [32, 10]
softmax       [32, 10]

Each row is a conditional distribution

$P(class\mid input).$

The input is the evidence; the row contains beliefs conditioned on that evidence.

9. Break It

Mistake 1 — swapping the condition

$P(A\mid B)$ and $P(B\mid A)$ answer different questions.

Mistake 2 — using the old denominator

Once evidence is known, count only cases compatible with it.

Mistake 3 — confusing independence with exclusivity

Independent events may occur together. Mutually exclusive nonzero-probability events cannot be independent.

Mistake 4 — conditioning on an impossible event

The elementary formula requires $P(B)>0$.

10. Real-World Connection

Conditional probability appears everywhere:

• diagnosis: $P(disease\mid symptoms)$,
• classification: $P(label\mid x)$,
• language models: $P(next\ token\mid context)$,
• recommendation: $P(click\mid user,item,context)$,
• forecasting: $P(future\mid history)$.

Machine learning is often the problem of learning a useful conditional distribution from data.

11. History / Paper Trail

The idea of conditional probability predates machine learning by centuries. Its importance exploded in statistics because evidence changes what should be believed. Thomas Bayes and Pierre-Simon Laplace pushed this logic into systematic inference. The next chapter derives the rule that reverses a condition.

Mastery Check

Without looking back, explain:

1. Why does conditioning change the denominator?
2. Why is $P(A\mid B)$ generally different from $P(B\mid A)$?
3. What does independence mean in plain English?
4. Derive $P(A\cap B)=P(A\mid B)P(B)$.
5. How does the probability chain rule connect to language models?

Bridge to Chapter 029

We can compute $P(test\ positive\mid disease)$ from clinical studies. But the patient wants the reverse:

Given a positive test, what is the probability I actually have the disease?

That reversal creates Bayes’ rule.