Dimensionality Reduction
Projecting high-dimensional data into fewer dimensions while preserving structure — the role of PCA, t-SNE, and UMAP.
Rotate the axis — PC₁ captures 98.3% of variance at this angle. Find the maximum!
Dimensionality reduction transforms high-dimensional data into a lower-dimensional representation while preserving as much information as possible.
Why reduce dimensions?
- Visualization: can't plot 50D data, but can plot 2D or 3D
- Noise reduction: remove dimensions with little signal
- Computational efficiency: many algorithms are slow in high dimensions
- Curse of dimensionality: nearest neighbors, density estimates, and models that rely on distance become unreliable in high dimensions
Methods:
- Linear: PCA, LDA, Random Projections
- Nonlinear (manifold learning): t-SNE, UMAP, Isomap, autoencoders
- Always trades some information loss for a more compact, often more interpretable representation
- Linear methods (PCA) are fast, deterministic, and preserve global structure
- Nonlinear methods (t-SNE, UMAP) better preserve local neighborhood structure, at the cost of speed and determinism
- Effective dimensionality reduction depends on data actually having lower intrinsic dimension than its raw representation
- Over-interpreting t-SNE/UMAP plots: cluster sizes, shapes, and inter-cluster distances in these embeddings are not meaningful — only "these points are close" within a plot is reliable
- Applying PCA to unscaled features: a feature with a much larger numeric range will dominate the principal components unless features are standardized first
10,000 news articles, each represented as a 50,000-word count vector. After PCA to 2D, articles about sports cluster together, political articles form another cluster, science a third. The 2D visualization reveals structure invisible in the original 50,000D space.
The "curse of dimensionality" says that high-dimensional space is mostly empty. Explain intuitively why a k-NN classifier becomes unreliable in high dimensions.
Solution
In dimensions, the fraction of volume of a unit cube covered by a ball of radius goes to 0 as . Equivalently: to fill a fixed fraction of a high-dimensional space, you need exponentially more points.
For k-NN: with limited data, the "nearest neighbor" in 100D might be very far away — so far that it's no longer meaningfully similar to the query point. All points become roughly equidistant (the ratio of max to min distance ), making "nearest" meaningless. The algorithm degrades to random guessing.
Dimensionality reduction helps by projecting into a space where the data is dense enough for distances to be meaningful.
Related concepts
Needs first