Decision Trees
Flowchart-like models that recursively partition the feature space by asking yes/no questions â interpretable but prone to overfitting.
Current path
age = 35, income = 60k
Medium Risk
Each highlighted node narrows the region until the example lands in a leaf.
A decision tree classifies data by asking a sequence of yes/no questions about features. Each question splits the data into two groups; the process repeats until the leaf nodes are pure (one class only) or stopping criteria are met.
Structure:
- Internal nodes: feature-based decisions (e.g., "Is age < 30?")
- Branches: outcomes of decisions
- Leaf nodes: predicted class (or value, for regression)
Decision trees are interpretable â you can follow the path from root to leaf and explain exactly why a prediction was made.
- Fully interpretable: every prediction traces back to an explicit sequence of feature thresholds
- Naturally handle both numeric and categorical features without preprocessing
- Make no assumption about the underlying data distribution
- Prone to high variance: small changes in training data can produce a very different tree structure
- Letting a tree grow unconstrained: an unpruned tree can memorize training data perfectly (one leaf per example) while generalizing poorly
- Trusting feature importance from a single tree: importances from one tree can be unstable; ensemble-based importance (e.g. from a random forest) is far more reliable
A bank decides whether to approve a loan based on credit score and income:
- Credit score âĨ 700? Yes â Approve. No â Is income > $60k? Yes â Approve with higher rate. No â Reject.
Each rule is transparent and auditable. This is why decision trees are used in finance, medicine, and law.
A decision tree memorizes training data perfectly (one leaf per training example). What is its training error? Why is this bad?
Solution
Training error = 0% â it perfectly memorizes every training example. This is bad because the tree is just a lookup table; it has learned the training data's noise, not the underlying pattern.
On new data, it will perform no better than random guessing for examples it hasn't seen before. This is extreme overfitting. In practice, we limit tree depth, require minimum samples per leaf, or prune the tree after building it.
Related concepts
Needs first