PCA from SVD

Where We Are

Eigenvector PCA forms $X_c^TX_c$. But Chapter 040 introduced a decomposition that acts directly on a matrix:

$X_c=U\Sigma V^T.$

The connection is exact.

1. Square the data matrix conceptually

Starting with

$X_c=U\Sigma V^T,$

consider

$X_c^TX_c.$

Substitute the SVD:

$X_c^TX_c=(V\Sigma U^T)(U\Sigma V^T).$

Because $U^TU=I$,

$X_c^TX_c=V\Sigma^2V^T.$

But covariance is proportional to $X_c^TX_c$:

$C=\frac{1}{N-1}V\Sigma^2V^T.$

Therefore the columns of $V$ are the covariance eigenvectors—the PCA directions.

The covariance eigenvalues are

$\lambda_i=\frac{\sigma_i^2}{N-1}.$

2. One idea, two viewpoints

Eigen route:
X → covariance XᵀX → eigenvectors

SVD route:
X → UΣVᵀ → right singular vectors V

Both produce the same principal subspace (up to sign and degeneracies).

3. Why SVD is often preferred

Explicitly forming $X^TX$ can worsen numerical conditioning and may be inefficient. Practical PCA implementations often use SVD or related solvers directly.

4. Compression and reconstruction

Keep the first $k$ right singular vectors $V_k$.

$Z=X_cV_k$

and approximately reconstruct

$\hat X=ZV_k^T+\mu.$

This is not arbitrary compression. Among rank-$k$ linear approximations, truncated SVD gives the smallest Frobenius reconstruction error (Eckart–Young theorem).

5. Shape Check

If X is [N,D] and we keep k components:

V_k       [D,k]
Z=X V_k   [N,k]
X_hat     [N,D]

6. Small PyTorch connection

Xc = X - X.mean(0, keepdim=True)
U, S, Vh = torch.linalg.svd(Xc, full_matrices=False)
Vk = Vh[:k].T
Z = Xc @ Vk

Before running, trace every shape.

What to Remember

• PCA covariance eigenvectors are the right singular vectors of centered data.
• PCA eigenvalues equal squared singular values divided by $N-1$.
• SVD gives a numerically natural path to PCA.
• Truncation gives optimal linear low-rank reconstruction.

Next → t-SNE and UMAP: PCA preserves linear directions. What if the interesting structure bends and curves?