Gaussian Distribution

1. The Pattern

Measure heights, sensor noise, repeated experimental errors, or averages of many small effects. Often values cluster near a center and become rarer farther away.

The Gaussian distribution models that pattern:

Do not start by memorizing it. Read its pieces.

• $\mu$ moves the bell left or right.
• $\sigma$ controls its width.
• $(x-\mu)^2$ measures squared distance from the center.
• the negative exponential makes distant values rapidly less likely.
• the front constant makes total area equal one.

2. Mean and Standard Deviation

For

$X\sim\mathcal N(\mu,\sigma^2),$

$E[X]=\mu,\qquad Var(X)=\sigma^2.$

Roughly:

• 68% of mass lies within $\mu\pm\sigma$,
• 95% within $\mu\pm2\sigma$,
• 99.7% within $\mu\pm3\sigma$.

These are useful intuition, not substitutes for exact calculations.

3. Why Squared Error Appears Again

Take the log density:

If a regression model assumes target noise is Gaussian with fixed variance,

$y=f_\theta(x)+\epsilon,\qquad \epsilon\sim\mathcal N(0,\sigma^2),$

then maximizing likelihood is equivalent to minimizing

So mean squared error is not merely a convenient smooth penalty. It corresponds to a probabilistic assumption: errors are Gaussian with constant variance.

That is the bridge between probability and loss functions.

4. Standardization

Transform

Then a Gaussian variable becomes standard normal:

$Z\sim\mathcal N(0,1).$

Interpretation: $z$ tells us how many standard deviations a value lies from the mean.

This same idea motivates feature standardization before many ML algorithms.

5. Multivariate Gaussian

For vector data, the Gaussian becomes

Now:

• $\boldsymbol\mu$ is the center vector,
• $\Sigma$ is the covariance matrix,
• covariance controls the orientation and width of the probability cloud.

In 2D, equal-density contours are ellipses. Eigenvectors of $\Sigma$ give principal directions; eigenvalues give spread along those directions.

Linear algebra and probability have now met directly.

6. Why Gaussians Appear So Often

The central limit theorem says, under broad conditions, sums or averages of many independent-ish small contributions tend toward a Gaussian distribution after suitable normalization.

That does not mean all data is Gaussian. Income, word frequencies, waiting times and many other phenomena can be strongly non-Gaussian.

Use the Gaussian because its assumptions fit—not because bell curves are fashionable.

7. Code

import numpy as np

mu = 10.0
sigma = 2.0
x = np.random.normal(mu, sigma, size=100_000)

print(x.mean())
print(x.std())
print(np.mean(np.abs(x-mu) <= sigma))  # near 0.68

For a 2D Gaussian:

mean = np.array([0., 0.])
cov = np.array([[2., 1.5],
                [1.5, 2.]])
X = np.random.multivariate_normal(mean, cov, 5000)

Plot the points and compare the cloud’s direction with covariance eigenvectors.

8. Shape Check

For $d=20$ features:

mean μ:        [20]
covariance Σ: [20, 20]
sample x:      [20]

For a batch of 64 samples:

X: [64, 20]

9. Break It

Heavy tails

A Gaussian can underestimate extreme events.

Multiple groups

A single Gaussian cannot represent two separated clusters well; mixtures will help later.

Wrong variance assumption

MSE corresponds to a particular Gaussian-noise story. If noise variance changes with $x$, the simple model is misspecified.

10. Real-World Connection

Gaussian ideas appear in initialization, noise models, Bayesian inference, Kalman filters, Gaussian mixtures, VAEs, diffusion models and statistical error analysis.

Mastery Check

1. What do $\mu$ and $\sigma$ control geometrically?
2. Why does the log Gaussian contain squared error?
3. Under what noise assumption does MSE arise naturally?
4. What does a z-score mean?
5. How does a covariance matrix shape a multivariate Gaussian?

Bridge to Chapter 033

A distribution gives probabilities once its parameters are known. Real learning reverses the problem:

Given observed data, which parameter values make those observations most plausible?

That is maximum likelihood.