Covariance and Correlation

1. The Problem

Suppose we record study hours $X$ and exam score $Y$. The mean tells us each variable’s center, but not whether students above the mean in study hours also tend to be above the mean in score.

We need a measure of co-movement.

2. Center Both Variables

For one observation, compare deviations from the means:

$X-\mu_X,\qquad Y-\mu_Y.$

Multiply them.

• both above their means → positive product,
• both below → positive product,
• one above and one below → negative product.

Average these products:

Equivalent form:

$Cov(X,Y)=E[XY]-E[X]E[Y].$

3. Tiny Example

Let

X = [1, 2, 3]
Y = [2, 4, 6]

Means are 2 and 4. Centered values:

Xc = [-1, 0, 1]
Yc = [-2, 0, 2]

Products:

[2, 0, 2]

Their average is positive, so the variables move together.

If $Y=[6,4,2]$, the products become negative and covariance is negative.

4. The Scale Problem

Covariance depends on units. Change meters to centimeters and covariance changes dramatically even though the relationship is identical.

Normalize by standard deviations:

This is correlation. It lies between -1 and 1.

• +1: perfect positive linear relation,
• 0: no linear correlation,
• -1: perfect negative linear relation.

5. Correlation Is Not Causation

Ice-cream sales and sunburn cases may be correlated because both rise with hot weather. Buying ice cream does not cause sunburn.

A correlation describes a statistical relationship, not a causal mechanism.

Also, zero correlation does not necessarily mean independence. A nonlinear relationship can have zero linear correlation.

Example: if $X$ is symmetric around zero and $Y=X^2$, knowing $X$ tells you $Y$ exactly, yet their correlation can be zero.

6. Covariance Matrix

For a vector $\mathbf{x}\in\mathbb{R}^d$:

Entry $(i,j)$ is the covariance between features $i$ and $j$.

Diagonal entries are variances.

For 3 features:

This is the same object that appeared in PCA. Now we understand its probabilistic meaning.

7. Code

import numpy as np

X = np.array([1., 2., 3., 4.])
Y = np.array([2., 3., 5., 8.])

xc = X - X.mean()
yc = Y - Y.mean()

cov = (xc * yc).mean()
corr = cov / (X.std() * Y.std())

print(cov, corr)

Compare with np.cov carefully: libraries may use $n-1$ instead of $n$ in sample covariance.

8. Shape Check

Data matrix:

X: [1000, 50]

1000 examples, 50 features.

After centering, $X_c$ stays [1000, 50].

Then

$X_c^TX_c$

has shape

[50, 50]

That is why a feature covariance matrix is square in the number of features.

9. Real-World Connection

Covariance and correlation appear in PCA, Gaussian models, portfolio risk, feature analysis, whitening, uncertainty propagation and diagnostics. In deep learning, correlated activations can make optimization geometry awkward; normalization and representation learning often change these relationships.

Mastery Check

1. Why multiply centered deviations?
2. What does the sign of covariance mean?
3. Why normalize covariance to obtain correlation?
4. Why does correlation not prove causation?
5. Can dependent variables have zero correlation?
6. Why is a 50-feature covariance matrix 50×50?

Bridge to Chapter 032

We can now describe center, spread and relationships. One distribution appears constantly because many noisy effects combine into a bell-shaped pattern.

Why does the Gaussian distribution appear everywhere?