Geometry of Decision Boundaries
Comparing classifiers by the shape of the regions they carve out â linear hyperplanes, Voronoi cells, axis-aligned boxes, and curved kernel boundaries.
Every classifier, no matter how it's trained, ultimately draws some boundary through feature space separating one predicted class from another. Comparing classifiers by the shape of boundary they can draw â rather than just their accuracy â reveals a lot about when each one is the right tool:
- Logistic regression, LDA, linear SVM: a single straight line (hyperplane in higher dimensions)
- k-Nearest Neighbors: a jagged, piecewise-linear boundary following Voronoi cell edges
- Decision trees: axis-aligned rectangular regions, like a staircase
- Kernel SVM, neural networks: smooth or highly flexible curved boundaries
If the true boundary between two classes is a circle (one class inside, one outside), a linear classifier (logistic regression, linear SVM) can never represent it well â no straight line separates a disk from its surroundings. A kernel SVM with an RBF kernel, or a sufficiently deep decision tree, can.
A decision tree's boundary is always made of axis-aligned rectangles. Why would a decision tree struggle to cleanly separate two classes divided by a diagonal line ?
Solution
A decision tree can only split on one feature at a time with a threshold (e.g. "" or "") â it can never directly express a diagonal condition like "." To approximate a diagonal boundary, it must build a staircase of many small axis-aligned rectangles, which is wasteful (needs many splits) and never perfectly smooth, no matter how deep the tree grows.