∑Richard Hummel Math

Model Evaluation

Confusion matrices, accuracy, precision, recall, F1 score, ROC curves, and AUC — the toolkit for measuring classifier and regressor performance.

Confusion matrix — what the model actually predicted
Predicted Positive
Predicted Negative
Actual
Positive
80
TP
20
FN
Actual
Negative
15
FP
85
TN
82.5%
Accuracy
84.2%
Precision
80.0%
Recall
82.1%
F1 Score
Precision = TP/(TP+FP) — of predicted positives, how many are correct?
Recall = TP/(TP+FN) — of actual positives, how many did we catch?
F1 = harmonic mean of precision and recall — balances both
Accuracy is misleading with class imbalance — prefer F1
Definition

Model evaluation measures how well a trained model performs on data it hasn't seen. Choosing the right metrics depends on the task.

For classification:

  • Accuracy: fraction of correct predictions = (TP+TN)/(TP+FP+FN+TN)(TP + TN)/(TP+FP+FN+TN)
  • Precision: TP/(TP+FP)TP/(TP+FP) — of predicted positives, how many are right?
  • Recall (Sensitivity): TP/(TP+FN)TP/(TP+FN) — of actual positives, how many did we catch?
  • F1 score: 2⋅Precision⋅Recall/(Precision+Recall)2 \cdot \text{Precision} \cdot \text{Recall} / (\text{Precision} + \text{Recall})

Confusion matrix: shows all four outcomes (TP, TN, FP, FN) for a binary classifier.

Key properties
  • Precision and recall trade off against each other as the decision threshold moves — improving one typically costs the other
  • Accuracy alone is uninformative (even misleading) on imbalanced datasets
  • F1 score is the harmonic mean of precision and recall, so it's dominated by whichever of the two is smaller
  • AUC-ROC is threshold-independent — it summarizes performance across every possible decision threshold at once
Common mistakes
  • Reporting accuracy alone on an imbalanced dataset: a model that always predicts the majority class can score deceptively high accuracy while being useless
  • Choosing a metric after seeing results, to make a model look good: the metric should reflect the real-world cost of false positives vs. false negatives, decided before evaluation
Medical diagnosis

A cancer screening test: 90 patients have cancer (positive), 910 don't (negative). The model predicts 80 TP, 10 FN, 15 FP, 895 TN.

Accuracy = 975/1000 = 97.5%. But recall = 80/90 = 89%. The model misses 10% of cancer cases — arguably unacceptable for a screening tool, even though overall accuracy looks excellent.

Try it

A fraud detection model has 99.5% accuracy on a dataset where 0.5% of transactions are fraudulent. Is this impressive? What metric is more appropriate?

Solution

Not impressive at all — a model that always predicts "not fraud" achieves 99.5% accuracy without detecting a single fraudulent transaction!

For imbalanced classes, use precision, recall, F1, or AUC-ROC. With fraud as the positive class:

  • Recall matters most (catching fraud cases)
  • Precision matters for avoiding false positives (annoying legitimate customers)
  • F1 balances these; AUC measures tradeoff across all thresholds

Never use accuracy alone for imbalanced datasets.

Related concepts

View in full concept graph →