Dimensionality Reduction: t-SNE and UMAP Intuition

Where We Are

PCA finds a linear projection that preserves variance. It is excellent for compression and often useful for visualization. But curved manifolds can be flattened badly by one global linear projection.

Methods such as t-SNE and UMAP focus more strongly on neighbourhood structure.

1. The visualization goal is different from compression

When we map 100 dimensions to 2, information must be lost. There is no perfect map.

So first ask: what relationships do we want to preserve?

PCA emphasizes global linear variance. t-SNE emphasizes local neighbour probabilities. UMAP builds a weighted neighbourhood graph and tries to preserve much of its topology in a low-dimensional embedding.

2. t-SNE intuition

t-SNE converts pairwise neighbourhood relationships in the original space into probabilities. It creates another probability distribution in 2-D and moves points to make the two distributions similar, using a KL-divergence objective.

Chapter 107 has returned.

high-D neighbours → probabilities P
2-D neighbours    → probabilities Q
                 minimize KL(P || Q)

A heavy-tailed Student-t distribution in the low-dimensional space helps prevent many points from crowding together.

3. UMAP intuition

UMAP starts by building a local neighbour graph: strong edges mean points are locally close. It then searches for a low-dimensional arrangement whose fuzzy graph resembles the original one.

The mathematics uses ideas from topology and manifold learning, but the first-principles picture is:

preserve the pattern of local connections rather than every raw coordinate.

4. The biggest visualization trap

A beautiful 2-D plot can lie.

Distances between far-apart t-SNE clusters are often not globally meaningful. Cluster sizes can be misleading. Different random seeds and hyperparameters can create different pictures.

Never conclude “there are exactly six real groups” merely because a t-SNE picture shows six islands.

5. Perplexity and neighbourhood scale

For t-SNE, perplexity roughly controls the neighbourhood scale considered. For UMAP, n_neighbors plays a related intuitive role. Small neighbourhoods emphasize very local structure; larger settings consider broader relationships.

6. A responsible workflow

1. Standardize or otherwise prepare features appropriately.
2. Often reduce extreme dimensionality with PCA first.
3. Run multiple seeds/settings.
4. Colour points only with metadata not used to create the embedding.
5. Verify any visual claim with quantitative analysis in the original space.

What to Remember

• PCA is linear and useful for compression as well as visualization.
• t-SNE and UMAP are primarily powerful visualization/manifold tools.
• Their 2-D geometry is an approximation, not ground truth.
• Hyperparameters and random seed can change the picture.

This closes Level 4. We now know many ways to fit a model—but a new danger appears.

Next → Train, Validation and Test Sets: how do we know whether any of these models learned a reusable rule rather than memorized the examples we showed them?