Principal Component Analysis
Finding the directions of maximum variance in data by computing eigenvectors of the covariance matrix — the workhorse of dimensionality reduction.
Rotate the candidate axis. PCA chooses the direction where the projected points spread out the most.
Principal Component Analysis (PCA) reduces the dimensionality of data by finding a smaller set of uncorrelated variables (principal components) that capture most of the variance.
The idea: high-dimensional data often lies near a lower-dimensional subspace. PCA finds the directions of maximum variance and projects the data onto them.
First principal component (PC1): the direction along which the data varies most. Second (PC2): the direction of next-most variance, orthogonal to PC1. And so on.
Images of faces are 100×100 pixels = 10,000 dimensions. PCA finds that most variation between faces can be captured in the top 50 principal components ("eigenfaces"). Storing these 50 numbers per image instead of 10,000 saves space and speeds up similarity comparisons.
You have 2D data that looks like a diagonal ellipse. After PCA, what do the first and second principal components represent geometrically?
Solution
PC1 aligns with the long axis of the ellipse — the direction of maximum variance. PC2 is orthogonal to PC1, aligning with the short axis. After projecting onto just PC1, you retain most of the variance (the long dimension of the ellipse) and discard the smaller cross-axis variation. The data is effectively 1D after this projection.
Related concepts
Needs first