Entropy and Information

1. Surprise First

If an event has probability 1, learning that it happened tells you nothing surprising. If its probability is 1 in a million, observing it carries much more information.

A useful measure of surprise is

Rare event → large surprise. Common event → small surprise.

Why logarithm? Independent probabilities multiply, while information should add:

$-\log[p(a)p(b)]=-\log p(a)-\log p(b).$

2. Entropy = Expected Surprise

For distribution $P$:

Entropy is the average information obtained when observing an outcome drawn from $P$.

A fair coin has more entropy than a coin that lands heads 99.9% of the time, because the fair coin is harder to predict.

3. Fair Coin vs Certain Coin

Using base-2 logs, information is measured in bits.

Fair coin:

$H=-2(0.5\log_2 0.5)=1\ bit.$

Certain coin:

$H=-(1\log_2 1)=0.$

Maximum uncertainty for $K$ equally likely classes is

$H=\log K.$

4. Entropy and Compression

If common symbols get short codes and rare symbols get long codes, the average optimal code length is closely connected to entropy. Shannon showed that entropy sets a fundamental limit on lossless compression.

This gives information a concrete meaning: uncertainty is tied to how many bits we need to describe outcomes.

5. Entropy in Classification

Suppose a 3-class model predicts

[0.98, 0.01, 0.01]

Low entropy: the model is concentrated.

Another predicts

[0.34, 0.33, 0.33]

High entropy: the model is uncertain.

But low entropy does not guarantee correctness. A model can be confidently wrong.

6. Code

import numpy as np

def entropy(p):
    p = np.asarray(p, dtype=float)
    p = p[p > 0]
    return -(p * np.log2(p)).sum()

print(entropy([0.5, 0.5]))
print(entropy([0.99, 0.01]))
print(entropy([1/3, 1/3, 1/3]))

Predict the ordering before running.

7. Shape Check

For classifier probabilities

P: [batch, classes] = [64, 10]

entropy computed across the class axis gives

H: [64]

one uncertainty value per example.

8. Break It

Entropy measures uncertainty of a distribution, not whether the distribution matches reality. A confidently wrong model has low predictive entropy.

Also, the log base changes units—bits for base 2, nats for natural log—but not the underlying ordering.

9. Real-World Connection

Entropy appears in decision trees, active learning, information gain, language modeling, uncertainty measures, compression, reinforcement learning and generative modeling.

Mastery Check

1. Why should rare events carry more information?
2. Why is logarithm useful for information?
3. What is entropy in plain English?
4. Why does a fair coin have more entropy than a nearly certain coin?
5. Why can low entropy still be wrong?

Bridge to Chapter 036

Entropy measures one distribution’s uncertainty. Machine learning often needs to compare two distributions: reality $P$ and model $Q$.

How much information do we lose when Q pretends to be P?

That leads to KL divergence.