What is data drift?

Why it happens

A model learns the world as it was when its training data was collected. The world keeps moving: prices change, new products launch, customers behave differently in a recession, a new phone camera changes how photos look, fraudsters try new tricks. None of this causes an error message. The model simply applies yesterday’s patterns to today’s data.

Two kinds of change

• Data drift (the inputs change): the mix of what the model sees shifts. More orders come from mobile, the average basket grows, a new region opens. The rule linking inputs to outcomes may still hold, but the model is now working in areas where it saw little training data.
• Concept drift (the meaning changes): the same inputs now lead to different outcomes. A discount that used to keep customers no longer does, or words that meant a complaint now mean praise. This is usually more damaging, because the model’s learned rules are now wrong.

How to monitor it

Watch three things, from the cheapest to the most reliable:

1. The inputs. Compare each feature’s recent distribution with the training data: averages, missing-value rates, the share of each category. Sudden changes often come from a broken data pipeline rather than the world, and they’re worth catching quickly.
2. The predictions. Track the share of each predicted class or the average predicted value. If the model suddenly flags twice as many transactions, something changed.
3. The real outcomes. When the true answers arrive, such as whether a customer really left, measure the model’s real score over time. This is the only direct measure, but labels often arrive weeks later.

import numpy as np
from scipy.stats import ks_2samp

rng = np.random.default_rng(0)
training = rng.normal(50, 10, 5000)    # e.g. basket size when the model was trained
this_week = rng.normal(56, 12, 800)    # the same feature in recent data

result = ks_2samp(training, this_week)
print(f"Training mean {training.mean():.1f}, this week {this_week.mean():.1f}, KS statistic {result.statistic:.2f}")
# A larger KS statistic means the two distributions differ more; track it every week

What to do about it

• Retrain on a schedule with the most recent data: monthly is common, weekly for fast-changing problems such as fraud or demand forecasting.
• Retrain when monitoring says so, rather than only on the calendar, if your outcome labels arrive quickly.
• Before switching, compare the retrained model with the current one on the latest data, and switch only if it’s better.
• Fix the pipeline first if the change came from a bug, such as a renamed column or a new unit. Retraining on broken data makes things worse.