A simple example
Say you want to predict which customers will cancel their subscription. Your table has a column called cancel_reason. A model trained with it scores almost 100%: every customer with a reason filled in has, of course, canceled. But when you use the model on today’s customers, nobody has a cancel reason yet, because they haven’t canceled. The model has learned to read the answer, not to predict it.
That’s data leakage: the training data contains information that wouldn’t exist at the moment the model makes its prediction.
The three most common kinds
- Columns that contain the answer. Anything filled in after the outcome: a cancel reason, a refund date, “was this card reported stolen.” Ask of every column: would I know this at the moment I make the prediction?
- Information from the future. Features computed over a period that runs past the prediction date, such as “total orders this year” computed today for a prediction made in March. Time-based data needs features built as of a cutoff date.
- The same example in training and test. Duplicate rows, or several photos of the same item split across both sets. The model recognizes what it has already seen, and the test score is inflated.
How to spot it
- The score is too good to be true. If a first model reaches 99% on a hard problem, suspect leakage before celebrating.
- One feature dominates. Check feature importance. If a single column explains almost everything, look at what it really means and when it gets filled in.
- The model does much worse in real use than in testing. That gap is the classic sign that the test didn’t match reality.
How to prevent it
- Split your data before exploring it closely, and keep the test set aside until the end.
- For anything that happens over time, split by date: train on the past, test on a later period.
- Build features as of a cutoff date, using only what was known at that moment.
- Remove duplicates, and keep related rows (the same customer, the same product, the same patient) on the same side of the split.
- Fit preprocessing, such as scaling or filling in missing values, on the training data only. A scikit-learn pipeline does this for you.
Why it matters
Leakage doesn’t cause an error message. Everything runs, the numbers look great, and the problem only shows up after the model is in use, when decisions have already been made on its predictions. That’s why it’s one of the most expensive mistakes in machine learning, and why every guide on this site builds its test set to match how the model will really be used.