What is cross-validation?

Why one split isn’t always enough

With a single 80/20 split, the score depends on which rows happen to land in the test set. On a large dataset that hardly matters. On a few hundred or a few thousand rows, a lucky or unlucky split can move the score by several points, enough to pick the wrong model.

How k-fold cross-validation works

1. Shuffle the data and divide it into k equal parts, called folds. Five is a common choice.
2. Train on four folds and test on the fifth.
3. Repeat five times, so each fold is the test set exactly once.
4. Average the five scores, and look at how much they vary.

Every row is used for testing exactly once and for training four times. The average is a fairer estimate, and the spread between folds tells you how much to trust it: if fold scores range from 0.70 to 0.90, small differences between models mean little.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
print("Fold scores:", scores.round(3))
print(f"Mean {scores.mean():.3f}, spread ±{scores.std():.3f}")

Putting the scaler inside the pipeline matters: it’s fitted on the training folds only, every time. Scaling the whole dataset first would let information from the test fold leak into training.

When to use something else

• Data over time, such as sales or transactions: never shuffle. Train on the past and test on the period after, moving forward step by step. Scikit-learn’s TimeSeriesSplit does this, and forecasting calls it backtesting.
• Several rows per person or item, such as many orders per customer or many photos per part: keep each group in one fold, with GroupKFold. Otherwise the model recognizes the customer rather than learning the pattern.
• A rare class: use StratifiedKFold, which keeps the same share of each class in every fold. Scikit-learn does this by default for classifiers.
• Very large datasets, or slow models: one well-chosen split is usually enough, and five trainings may cost too much.

Cross-validation and the test set

Use cross-validation on your training data to compare models and choose settings. Keep a separate test set aside and use it once, at the end, for the score you report. If you choose between many models using cross-validation, the winner’s cross-validation score is a little too good, because you picked it for being high. The untouched test set corrects for that.