Linear Discriminant Analysis

A classification method that finds the linear combination of features maximising between-class separation relative to within-class scatter.

LDA — shared covariance ellipses, boundary shifts with class priors

separation 1.2

P(C₁) 0.50

Dashed ellipses share the same shape (shared Σ). When P(C₁) > 0.5, the boundary shifts toward class 0 — class 1 gets a wider decision region.

Definition

Linear Discriminant Analysis (LDA) is a generative classifier that assumes each class is drawn from a multivariate Gaussian distribution with the same covariance matrix.

The model:

Class $k$ has prior $\pi_k = P(Y=k)$
Features $\mathbf{x}$ given class $k$ follow $\mathcal{N}(\boldsymbol{\mu}_k, \Sigma)$ (shared covariance $\Sigma$ )

By Bayes' theorem, the posterior:

$P(Y=k \mid \mathbf{x}) \propto \pi_k \cdot \exp\left(-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu}_k)^T \Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu}_k)\right)$

The decision boundary between classes $k$ and $l$ is linear in $\mathbf{x}$ (the quadratic terms cancel because $\Sigma$ is shared).

Key properties

Decision boundaries are always linear (hyperplanes), thanks to the shared-covariance assumption
A generative model: estimates the full class-conditional distributions, not just the boundary
Doubles as a dimensionality reduction technique: $K$ classes need at most $K-1$ discriminant directions
More data-efficient than logistic regression when the Gaussian assumption genuinely holds

Common mistakes

Using LDA when classes have clearly different shapes/spreads: the shared-covariance assumption is violated, biasing the boundary — QDA is the natural fallback
Applying LDA directly when $p > n$ : the estimated covariance matrix becomes singular; shrinkage or dimensionality reduction is needed first

Gender classification

Features: height ( $\mu_{\text{male}}=177$ , $\mu_{\text{female}}=164$ ), weight. LDA estimates class means and a pooled covariance, then classifies by which class has higher posterior probability. The boundary is a line in (height, weight) space.

Try it

LDA is a generative model. What does it model that logistic regression (a discriminative model) doesn't? What's a potential advantage of the generative approach?

Solution

LDA models the joint distribution $P(\mathbf{x}, Y) = P(Y)P(\mathbf{x}|Y)$ — the distribution of features within each class. Logistic regression models only $P(Y|\mathbf{x})$ directly.

Advantage of generative approach: you can generate synthetic data from the fitted model; you can compute $P(\mathbf{x})$ (for outlier detection); estimates can be more efficient when the Gaussian assumption holds, because the generative model extracts more signal from the data.

Disadvantage: if the Gaussian assumption is violated, LDA is misspecified — logistic regression, which doesn't assume a distribution on $\mathbf{x}$ , is more robust.

Related concepts

Needs first

Eigenvalues and Eigenvectors Variance

Logistic Regression Naive Bayes Principal Component Analysis Quadratic Discriminant Analysis

View in full concept graph →