Gaussian Mixture Models

Where We Are

K-Means draws a hard line: a point belongs to cluster 0 or cluster 1. Real data is often ambiguous.

Gaussian Mixture Models (GMMs) replace hard membership with probabilities.

1. Imagine several invisible generators

Suppose exam scores come from two overlapping populations. We do not know which student came from which population, but imagine each population generates scores according to a Gaussian.

A mixture model says:

1. choose component $k$ with probability $\pi_k$;
2. sample $x$ from that component’s Gaussian.

The resulting density is

$p(x)=\sum_{k=1}^{K}\pi_k\,\mathcal N(x\mid\mu_k,\Sigma_k).$

The weights satisfy $\pi_k\ge0$ and $\sum_k\pi_k=1$.

2. Soft membership

For a particular point we can ask

$p(z=k\mid x).$

These are called responsibilities. A point near the overlap might have

cluster A: 0.65
cluster B: 0.35

K-Means cannot express that uncertainty.

3. Geometry improves too

A Gaussian has a covariance matrix $\Sigma$. Therefore a GMM component can represent stretched and rotated elliptical clusters.

K-Means:        GMM:
   ○○○          //////
  ○ • ○        ///•///
   ○○○          //////
 spherical      elliptical

4. Likelihood is the objective

Given data $x_1,\dots,x_N$, we want parameters that make the observed data likely:

$\log p(X)=\sum_i\log\left(\sum_k\pi_k\mathcal N(x_i\mid\mu_k,\Sigma_k)\right).$

There is a difficulty: the logarithm surrounds a sum. We do not know the hidden component $z_i$ that generated each point.

This is why the next chapter exists.

5. Connection to K-Means

K-Means can be viewed roughly as a limiting case where components are spherical with similar variance and assignments become almost completely certain. GMMs are therefore not an unrelated algorithm; they soften and generalize the clustering story.

6. Shape Check

For N data points, D features and K components:

X               [N,D]
means           [K,D]
covariances     [K,D,D]
responsibility  [N,K]

Each row of the responsibility matrix sums to 1.

Break It

Fit too many Gaussian components to too little data. A component may collapse around a point with tiny covariance, creating numerical trouble and overfitting. Regularization still matters in unsupervised learning.

What to Remember

• A GMM models data as a weighted sum of Gaussian generators.
• Membership is probabilistic, not hard.
• Covariance lets clusters be elliptical.
• Hidden component identities make direct maximum-likelihood optimization awkward.

Next → Expectation Maximization: how can we learn model parameters when the variable we need—the component identity—is hidden?