What is overfitting, and how do you prevent it?

An everyday example

Imagine a student who prepares for an exam by memorizing last year’s answers. On last year’s paper they score 100%. On this year’s paper, with new questions, they struggle, because they learned the answers, not the subject. A model can do exactly the same: learn the quirks and noise of its training examples so well that it can’t handle new ones.

How to spot it

Compare the score on the training data with the score on data the model hasn’t seen, such as a validation set:

• Both scores are poor: the model is too simple, or the features don’t carry the signal. That’s underfitting.
• Training score far better than validation score: the model is memorizing. That’s overfitting.
• Both good and close together: the model has learned something that carries over to new data. That’s the goal.

A small gap is normal. A large one, such as 99% on training data and 75% on validation data, means the model won’t perform in real use the way its training score suggests.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = make_classification(n_samples=1000, n_features=20, n_informative=5, flip_y=0.1, random_state=0)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.25, random_state=0)

for depth in [2, 5, None]:   # None = grow until every training example is memorized
    tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_train, y_train)
    print(f"max_depth={depth}: train {tree.score(X_train, y_train):.2f}, validation {tree.score(X_val, y_val):.2f}")

The unlimited tree scores a perfect 1.00 on the data it trained on and noticeably less on validation data: it has memorized the noise.

Why it happens

• The model is too flexible for the amount of data, such as a deep tree or a large neural network trained on a few hundred rows.
• There are many features and few examples, so some features match the labels by pure chance.
• Training runs too long, especially for neural networks, which keep improving on training data long after they stop improving on new data.
• You test many ideas against the same validation set, and slowly tune the model to that particular set.

How to prevent it

1. Get more data. It’s the most reliable fix: it’s much harder to memorize 100,000 examples than 1,000.
2. Use a simpler model, or limit its flexibility: a shallower tree, fewer trees, fewer layers, or a regularization penalty such as Ridge regression.
3. Stop training early when the validation score stops improving. Most libraries have an early-stopping option.
4. Remove features that don’t help, especially IDs, free-text codes and anything with a different value for almost every row.
5. Use cross-validation on small datasets, so your estimate doesn’t depend on one lucky split.
6. Keep a test set you never tune on, and look at it once, at the end.