PCA from Eigenvectors

Where We Are

High dimensions can make learning harder. But many dimensions are redundant: height in centimetres and height in inches are almost the same information.

Principal Component Analysis (PCA) searches for better axes.

1. Rotate before you throw anything away

Imagine points forming a long diagonal cloud.

y ↑           •
  |        •
  |     •
  |  •
  +------------→ x

The x and y axes are not aligned with the cloud. Rotate the coordinate system so axis 1 follows the long direction and axis 2 follows the thin direction.

If almost all variation lies along axis 1, we can keep one number instead of two.

2. Centre the data

Let $X$ have shape [N,D]. Subtract each feature mean:

$X_c=X-\mu.$

Then compute covariance

$C=\frac{1}{N-1}X_c^TX_c.$

C has shape [D,D] and tells us how features vary together.

3. Eigenvectors reveal special directions

For an eigenvector $v$,

$Cv=\lambda v.$

The eigenvector gives a direction; its eigenvalue tells us the variance along that direction.

Sort eigenvectors by descending eigenvalue:

$\lambda_1\ge\lambda_2\ge\cdots.$

The first principal component is the direction with maximum variance. The next is the highest-variance direction perpendicular to the first, and so on.

4. Project onto fewer directions

Choose the first $k$ eigenvectors as columns of $W$:

X_c  [N,D]
W    [D,k]
Z=X_cW [N,k]

If $k<D$, Z is compressed.

Shape Check

For 10,000 examples with 100 features compressed to 8 components:

[10000,100] @ [100,8] → [10000,8]

5. Explained variance

The fraction explained by component $j$ is

$\frac{\lambda_j}{\sum_i\lambda_i}.$

This lets us choose $k$ based on retained information rather than guesswork.

6. Important caution

PCA maximizes variance, not usefulness for your label. A low-variance direction could still be crucial for classification. PCA is unsupervised.

Feature scaling also matters: a feature measured in huge units can dominate covariance.

What to Remember

PCA is best understood geometrically as find the important directions, rotate into them, then optionally discard weak directions.

Next → PCA from SVD: can we reach the same principal directions directly from the data matrix, without explicitly forming covariance?