Maximum A Posteriori Estimation

1. From MLE to MAP

Maximum likelihood chooses

$\hat\theta_{MLE}=\arg\max_\theta p(D\mid\theta).$

But sometimes parameter values are not equally plausible before seeing data. Bayes gives

$p(\theta\mid D)\propto p(D\mid\theta)p(\theta).$

MAP chooses the most probable parameter after combining data with the prior:

Taking logs:

Or minimizing negatives:

That second term will look very familiar: it becomes regularization.

2. Gaussian Prior → L2 Regularization

Assume weights have prior

$w\sim\mathcal N(0,\tau^2).$

Then

For many weights:

Therefore MAP with a zero-centered Gaussian prior is equivalent to optimizing data loss plus an L2 penalty.

Regularization can be interpreted as a prior belief that very large weights are less plausible.

3. Laplace Prior → L1 Regularization

A Laplace prior has density proportional to

$e^{-\lambda|w|}.$

Its negative log prior contributes

$\lambda|w|.$

So MAP with a Laplace prior produces L1 regularization, which tends to encourage sparse parameters.

4. Tiny Coin Example

Observe 3 heads and 1 tail. MLE for head probability is

$\hat p=3/4=0.75.$

With very little data, that estimate can be unstable. A prior centered near 0.5 pulls the MAP estimate toward 0.5. As data grows, likelihood usually dominates and prior influence weakens.

This is the core intuition:

little data  → prior matters more
lots of data → evidence dominates

5. MAP Is Still a Point Estimate

Bayesian inference gives a full posterior distribution $p(\theta\mid D)$. MAP throws most of that information away and keeps only the highest point.

So MAP is not “fully Bayesian uncertainty.” It is a convenient point estimate informed by a prior.

6. Code Intuition

import torch

w = torch.tensor([3.0, -4.0], requires_grad=True)
data_loss = torch.tensor(2.0)
lambda_ = 0.1

negative_log_prior = lambda_ * (w**2).sum()
map_objective = data_loss + negative_log_prior
print(map_objective)

The penalty is not an arbitrary extra punishment; under a Gaussian prior it is the negative log prior.

7. Break It

A prior is not a license to force the answer you want. Strong priors can overwhelm small datasets. Bad priors can bias estimates. MAP also depends on parameterization and hides posterior uncertainty.

8. Real-World Connection

MAP connects Bayesian reasoning to regularized regression, weight decay intuition, sparse models and probabilistic estimation. It explains why “fit the data” and “prefer simpler/plausible parameters” can live in one objective.

Mastery Check

1. How does MAP differ from MLE?
2. Why does a Gaussian prior produce an L2 penalty?
3. Why does a Laplace prior produce L1?
4. When does a prior matter most?
5. Why is MAP not the same as retaining a full posterior?

Bridge to Chapter 035

Likelihood and posterior tell us which outcomes or parameters are plausible. But we still need a number describing how uncertain an entire distribution is.

That number is entropy.