What is a baseline model?

Why you need one

A score on its own means nothing. Is 85% accuracy good? If 85% of your emails aren’t spam, a filter that never flags anything scores 85% too. Is a forecast error of 12% good? Not if “same as last week” is off by 10%. A baseline gives your score something to be compared with, and it often turns out to be harder to beat than expected.

Baselines also protect you from wasted work. If a simple rule gets you most of the way, you can ship the rule now and spend your time on better data instead of a bigger model.

Good baselines for each kind of problem

• Classification: always predict the most common class. Then try one obvious rule, such as “customers who haven’t ordered in 60 days will churn,” and a logistic regression.
• Number prediction: predict the median of the training data. Then a simple rule, such as an average per category, and a linear regression.
• Forecasting: “same as last period,” or “same as the same day last week” for weekly patterns, called a seasonal naive forecast.
• Recommendations: recommend the most popular items. It’s surprisingly hard to beat.
• Text: TF-IDF with logistic regression, which trains in seconds and is often good enough.

How to use a baseline

1. Measure the baseline on the same test set, with the same metric, as every model you try.
2. Write the number down before training anything else.
3. Report every model as a comparison: “The model catches 72% of churners in the top 10% of the list; the recency rule catches 55%.”
4. If a model doesn’t beat the baseline clearly, don’t ship the model. Check your features and labels, or ship the baseline.

from sklearn.dummy import DummyClassifier, DummyRegressor

most_common = DummyClassifier(strategy="most_frequent")   # classification
median = DummyRegressor(strategy="median")                # number prediction
# Fit and score them exactly like any other model