Cross-Validation

Estimating model generalisation by repeatedly training on subsets and evaluating on the held-out remainder — k-fold, leave-one-out.

K-fold cross-validation — each fold takes a turn as test set

k=5Click a row to highlight it

Definition

Cross-validation is a technique for estimating how well a model will perform on unseen data, by testing it on different subsets of the available data.

The most common version is k-fold cross-validation:

Split data into $k$ equal folds
For each fold: train on the other $k-1$ folds, evaluate on this fold
Report the average evaluation score across all $k$ folds

Common values: $k = 5$ or $k = 10$ . Leave-one-out (LOO): $k = n$ .

Cross-validation answers: "If I train on most of my data and test on the rest, how accurate am I — reliably, not just by luck?"

Key properties

Every data point is used for both training and validation exactly once across all folds
Averaging across folds reduces the variance of the performance estimate compared to a single train/test split
The choice of $k$ trades estimator bias (smaller $k$ , less training data per fold) against variance and compute cost (larger $k$ )
Results in $k$ independently trained models, not one — the final deployed model is typically retrained on all the data

Common mistakes

Letting any information from the validation fold leak into training: feature scaling, imputation, or feature selection must be fit only on the training folds, never on the full dataset before splitting
Using ordinary k-fold CV on time series: random folds let the model "see the future" — always use time-respecting splits (walk-forward validation) for temporal data
Tuning hyperparameters using the final test set: this silently turns the test set into part of training, biasing the reported performance optimistically

Tuning k in k-NN

You have 100 training examples. You try $k \in \{1, 3, 5, 7, 9\}$ for a k-NN classifier, evaluating each with 5-fold CV.

$k$	CV accuracy
1	68%
3	78%
5	82%
7	80%
9	76%

Choose $k=5$ . Now train on all 100 examples and deploy.

Try it

Why shouldn't you use your test set to tune hyperparameters, even if you never use it during training?

Solution

If you tune hyperparameters based on test set performance, the test set becomes effectively a training set — your model is adapted to it. The test score no longer measures generalization to truly unseen data; it measures performance on "the test set specifically." This overfitting is subtle and hard to detect, but the resulting performance estimate is optimistically biased. Rule: the test set is touched exactly once, after all decisions are made.

Related concepts

Needs first

Sampling

Decision Trees Ensemble Methods K-Nearest Neighbors Model Evaluation Regularization

View in full concept graph →