Tabular classification

How to predict which employees might leave

Use the HR records you already keep to see where people are likely to resign in the next year, and why, so you can fix the causes team by team before good people leave.

  • 11 steps
  • 8 min read
  • Runs in Google Colab

What you’ll build

  • A model that gives each employee a chance of resigning in the next 12 months, tested on a later year
  • Checks that those chances are honest, and that the model works as well for different groups of people
  • A team-by-team report of expected leavers and the conditions behind them

Tools

Python, pandas, scikit-learn

Step 1: Decide what the predictions are for, and what they’re not for

A prediction that someone might resign is sensitive. Used well, it shows where pay, workload or management need attention. Used badly, it gets people passed over for projects or promotions because a model guessed they might go, which makes them more likely to leave. Agree on the uses before writing any code:

  • Yes: finding the teams and conditions linked to people leaving, planning hiring, and offering stay conversations to everyone in a group at risk.
  • No: decisions about one person’s pay, promotion, projects or performance. Keep individual scores to a small HR group, and tell employees what the model uses and why.

Then define what the model predicts: resigned in the 12 months after a snapshot date. Leave out retirements, layoffs and contracts that ended; they have different causes and different fixes. Before you start, check with your HR and legal teams which records you may use: privacy law in many places requires telling employees how their data is used.

Step 2: Get one row per employee as of a date

Ask HR for a snapshot of everyone on the payroll on one date, with what was known about them that day: team, level, years in the company, months since their last promotion, pay compared with the middle of their pay band, last rating, overtime, manager changes, commute and their latest engagement survey answer. Then add the date each person resigned, if they did. Two snapshots a year apart let you learn from one year and test on the next.

No data to hand? This script makes two sample snapshots, on January 1, 2024 and January 1, 2025, of a company of about 3,000 people in five departments. Hidden in it are common reasons people leave: no promotion for a long time, pay below the band (worst for strong performers), long hours, new managers, low engagement and the first two years in a job.

CodeOptional: make sample employees.csv
pythonRun in Colab
# pip install pandas numpy scikit-learn
import numpy as np
import pandas as pd

rng = np.random.default_rng(5)
DEPARTMENTS = ["engineering", "sales", "support", "operations", "finance"]

def snapshot(date, n, start_id):
    """Everyone on the payroll on one date, as HR records them, and when anyone resigned afterwards."""
    e = pd.DataFrame({
        "employee_id": np.arange(start_id, start_id + n),
        "snapshot_date": pd.Timestamp(date),
        "department": rng.choice(DEPARTMENTS, n, p=[0.30, 0.25, 0.20, 0.15, 0.10]),
        "gender": rng.choice(["woman", "man", "not stated"], n, p=[0.46, 0.50, 0.04]),
        "level": rng.choice([1, 2, 3, 4], n, p=[0.40, 0.32, 0.20, 0.08]),
        "tenure_years": rng.exponential(3.5, n).round(1),
        "months_since_promotion": rng.integers(0, 60, n),
        "pay_vs_band": rng.normal(1.0, 0.08, n),                # salary divided by the middle of the role's pay band
        "rating": rng.choice([2, 3, 4, 5], n, p=[0.08, 0.52, 0.30, 0.10]),
        "overtime_hours": rng.gamma(1.5, 5, n),                 # a month, on average
        "manager_changes_2y": rng.poisson(0.6, n),
        "commute_km": rng.gamma(2, 8, n).round(0),
        "engagement": rng.choice([1, 2, 3, 4, 5], n, p=[0.06, 0.14, 0.32, 0.34, 0.14]).astype(float),
    })
    # Teams differ: support works the most overtime and is paid lowest in its bands
    e["overtime_hours"] = (e.overtime_hours + e.department.map({"support": 6, "operations": 3}).fillna(0)).round(0)
    e["pay_vs_band"] = (e.pay_vs_band - 0.04 * (e.department == "support")).round(2)
    e["months_since_promotion"] = np.minimum(e.months_since_promotion, (e.tenure_years * 12).astype(int))
    # Hidden reasons people leave: stuck without a promotion, paid below the band (worst for strong performers),
    # long hours, new managers, low engagement, and the first two years in a job.
    logit = (-3.0 + 0.9 * (e.months_since_promotion > 30) + 6 * (1 - e.pay_vs_band).clip(lower=0)
             + 1.1 * ((e.rating >= 4) & (e.pay_vs_band < 0.95)) + 0.05 * (e.overtime_hours - 8)
             + 0.45 * e.manager_changes_2y - 0.45 * (e.engagement - 3) + 0.6 * (e.tenure_years < 2)
             + 0.012 * e.commute_km + 0.35 * (e.department == "sales"))
    skipped = rng.random(n) < 0.12 + 0.10 * (logit > -1.5)       # people about to leave skip the survey more often
    e.loc[skipped, "engagement"] = np.nan
    leaves = rng.random(n) < 1 / (1 + np.exp(-logit))
    e["resignation_date"] = pd.NaT
    e.loc[leaves, "resignation_date"] = e.snapshot_date[leaves] + pd.to_timedelta(rng.integers(1, 366, leaves.sum()), unit="D")
    return e

employees = pd.concat([snapshot("2024-01-01", 3_000, 1), snapshot("2025-01-01", 3_100, 5_001)])
employees.to_csv("employees.csv", index=False)
print(employees.groupby("snapshot_date").resignation_date.count())
What you’ll see
snapshot_date
2024-01-01    541
2025-01-01    546
Name: resignation_date, dtype: int64
CodeLoad the snapshots and mark who resigned
pythonRun in Colab
import numpy as np
import pandas as pd
pd.set_option("display.width", 120, "display.max_columns", 20)

df = pd.read_csv("employees.csv", parse_dates=["snapshot_date", "resignation_date"])
# Left = resigned in the 12 months after the snapshot
df["left"] = ((df.resignation_date > df.snapshot_date) &
              (df.resignation_date <= df.snapshot_date + pd.DateOffset(years=1))).astype(int)
print(df.groupby("snapshot_date").left.agg(people="size", share_left="mean").round(3))
What you’ll see
               people  share_left
snapshot_date                    
2024-01-01       3000       0.180
2025-01-01       3100       0.176

About 18% of people resigned within a year of each snapshot. That’s the rate any list has to beat.

Step 3: Choose the columns the model may use

Two columns stay out. Gender isn’t a reason anyone resigns, and a model that uses it, or something that stands in for it, can treat people differently for who they are; keep it only to check the model in Step 9. The resignation date is the answer itself. Leave out anything else recorded after the snapshot too, such as notice periods or exit interviews: that’s data leakage, and it makes a model look excellent in testing and useless in real use.

CodeThe model’s columns
pythonRun in Colab
FEATURES = ["department", "level", "tenure_years", "months_since_promotion", "pay_vs_band", "rating",
            "overtime_hours", "manager_changes_2y", "commute_km", "engagement"]
# Not features: gender (kept only to check fairness in Step 9) and resignation_date (it's the answer)
df["department"] = df.department.astype("category")
print(df[FEATURES].isna().mean().loc[lambda s: s > 0].round(2))   # who skipped the engagement survey
What you’ll see
engagement    0.15
dtype: float64

15% of people have no engagement answer because they skipped the survey. Don’t drop them: people who are thinking of leaving are often the ones who don’t answer. Step 6 fills the gap and adds a column that says the answer was missing, so the model can learn whether skipping matters.

Step 4: Learn from one year, test on the next

Train on the 2024 snapshot and who resigned during 2024, and test on the 2025 snapshot and who resigned during 2025. That’s how the model will be used: trained on the past, then applied to the people on the payroll today. A random split would mix the two years and hide what changes from one year to the next.

CodeSplit by snapshot
pythonRun in Colab
train = df[df.snapshot_date == "2024-01-01"]   # learn from who left during 2024
test = df[df.snapshot_date == "2025-01-01"]    # test on who left during 2025
X_train, y_train, X_test, y_test = train[FEATURES], train.left, test[FEATURES], test.left
print(len(train), "people to learn from,", len(test), "to test on")
What you’ll see
3000 people to learn from, 3100 to test on

Step 5: Measure two simple rules first

Measure two things for every approach. The ROC-AUC says how well it ranks people overall, from 0.5 (no better than chance) to 1 (perfect). The top 10% list is what HR can act on: of the 10% of people an approach ranks riskiest, how many really resigned? Then try two rules HR teams often use as a baseline: the longest without a promotion, and the lowest paid compared with their band.

CodeTwo simple rules
pythonRun in Colab
from sklearn.metrics import roc_auc_score

def report(name, risk, share=0.1):
    """ROC-AUC, and who a list of the riskiest 10% would hold: the share of it who left, and the share of all leavers on it."""
    top = np.argsort(-np.asarray(risk))[: int(len(risk) * share)]
    left = y_test.to_numpy()[top]
    print(f"{name:<24} ROC-AUC {roc_auc_score(y_test, risk):.3f}   "
          f"{left.mean():.0%} of the list left   it holds {left.sum() / y_test.sum():.0%} of all leavers")

print(f"Left within the year: {y_test.mean():.0%} of everyone")
report("Months since promotion", X_test.months_since_promotion)
report("Paid below the band", -X_test.pay_vs_band)
What you’ll see
Left within the year: 18% of everyone
Months since promotion   ROC-AUC 0.527   24% of the list left   it holds 14% of all leavers
Paid below the band      ROC-AUC 0.623   32% of the list left   it holds 18% of all leavers

The longest-without-a-promotion rule barely beats chance: a ROC-AUC of 0.527, and 24% of its list resigned. Ranking by pay does better, at 0.623, and 32% of its list resigned.

Step 6: Train two models and compare

logistic regression adds up each column’s effect, so its results are easy to explain. Gradient boosting builds many small decision trees, and can find combinations such as strong performers who are paid below their band. Train both on 2024 and test both on 2025.

CodeLogistic regression and gradient boosting
pythonRun in Colab
from sklearn.compose import make_column_transformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numbers = [f for f in FEATURES if f != "department"]
logreg = make_pipeline(
    make_column_transformer(
        (OneHotEncoder(handle_unknown="ignore"), ["department"]),
        # a missing survey answer gets the median, plus a column that says it was missing
        (make_pipeline(SimpleImputer(strategy="median", add_indicator=True), StandardScaler()), numbers),
        verbose_feature_names_out=False),
    LogisticRegression(max_iter=1000))
logreg.fit(X_train, y_train)
report("Logistic regression", logreg.predict_proba(X_test)[:, 1])

boost = HistGradientBoostingClassifier(max_iter=300, learning_rate=0.03, max_leaf_nodes=15, min_samples_leaf=40,
                                       l2_regularization=1.0, categorical_features=["department"], random_state=0)
boost.fit(X_train, y_train)    # handles the missing answers on its own
report("Gradient boosting", boost.predict_proba(X_test)[:, 1])
What you’ll see
Logistic regression      ROC-AUC 0.728   45% of the list left   it holds 26% of all leavers
Gradient boosting        ROC-AUC 0.722   43% of the list left   it holds 24% of all leavers
Share of each top 10% list who resigned within the yearAnyone, at random: 18%; Longest without a promotion: 24%; Lowest paid for the band: 32%; Logistic regression: 45%; Gradient boosting: 43%.Anyone, at random18%Longest without a promotion24%Lowest paid for the band32%Logistic regression45%Gradient boosting43%
Of the 310 people each approach ranks riskiest in the 2025 snapshot, the share who resigned in the next 12 months (higher is better). The average is 18%. Logistic regression’s list is two and a half times that; gradient boosting is close behind.

Logistic regression ranks best, with a ROC-AUC of 0.728: 45% of its top 10% resigned, two and a half times the average, and the list holds 26% of everyone who left. Gradient boosting comes close, at 0.722 with 43% of its list resigning, but it isn’t better. When a simple model ties a complex one, use the simple one: it’s easier to explain to HR and to the people it’s about. The rest of this guide uses logistic regression.

The numbers also say something important: three in four people who resign aren’t on the list. People leave for reasons no HR record holds, such as a partner’s new job or an offer out of the blue. That’s why this model is for finding patterns across teams, not for judging one person.

Step 7: Check that the chances are honest

HR will read a 30% chance as “3 in 10 people like this resign”. Check that it’s true: group the test employees by their predicted chance and compare it with how many actually resigned.

CodePredicted against actual, by group
pythonRun in Colab
test = test.assign(risk=logreg.predict_proba(X_test)[:, 1])
band = pd.cut(test.risk, [0, 0.1, 0.2, 0.3, 0.5, 1], labels=["under 10%", "10–20%", "20–30%", "30–50%", "over 50%"])
print(test.groupby(band, observed=True).agg(people=("left", "size"), predicted=("risk", "mean"), actually_left=("left", "mean")).round(2))
What you’ll see
           people  predicted  actually_left
risk                                       
under 10%    1020       0.06           0.06
10–20%       1033       0.14           0.15
20–30%        515       0.25           0.22
30–50%        431       0.38           0.39
over 50%      101       0.60           0.50

The two columns match closely: 6% predicted and 6% resigned in the lowest group, 38% predicted and 39% resigned in the 30–50% group. Only the small top group is a little high, at 60% predicted and 50% resigned. Honest chances matter because Step 10 adds them up into a count of expected leavers per team.

Step 8: Find what drives the risk

Shuffle one column at a time in the test year and see how much worse the ranking gets: the bigger the drop, the more the model relies on that column. Then the model’s weights show which way each column pushes.

CodeWhat the model relies on
pythonRun in Colab
from sklearn.inspection import permutation_importance

# How much worse the ranking gets when one column is shuffled: the bigger the drop, the more the model leans on it
drop = permutation_importance(logreg, X_test, y_test, scoring="roc_auc", n_repeats=10, random_state=0)
print(pd.Series(drop.importances_mean, index=FEATURES).sort_values(ascending=False).round(3))
coefs = pd.Series(logreg[-1].coef_[0], index=logreg[0].get_feature_names_out())
print(coefs.sort_values().round(2))
What you’ll see
pay_vs_band               0.070
engagement                0.063
manager_changes_2y        0.033
overtime_hours            0.026
months_since_promotion    0.015
rating                    0.013
commute_km                0.005
tenure_years              0.004
department                0.003
level                     0.001
dtype: float64
engagement                    -0.56
pay_vs_band                   -0.51
tenure_years                  -0.14
department_finance            -0.09
department_operations         -0.02
department_engineering        -0.01
missingindicator_engagement    0.01
department_support             0.02
level                          0.04
department_sales               0.12
commute_km                     0.13
rating                         0.22
months_since_promotion         0.22
overtime_hours                 0.30
manager_changes_2y             0.35
dtype: float64

Pay compared with the band and engagement matter most, then manager changes and overtime. The weights point the way you’d expect: higher pay and engagement go with staying; more manager changes, more overtime and longer without a promotion go with leaving. Higher ratings go with leaving too, because strong performers have options. These are all things a company can change, which makes them more useful than any one person’s score.

Step 9: Check that the model works as well for every group

Use the gender column you kept out of the model to compare groups: the share of each group on the top 10% list, how many of those resigned, and how much of each group’s leavers the list holds. Large gaps mean the model works better for some people than for others, often because of a column that stands in for the group.

CodeCompare groups
pythonRun in Colab
flagged = test.risk >= test.risk.quantile(0.9)
fair = test.assign(flagged=flagged).groupby("gender").apply(lambda g: pd.Series({
    "people": len(g), "share_flagged": g.flagged.mean(), "share_left": g.left.mean(),
    "flagged_who_left": g.left[g.flagged].mean(), "leavers_flagged": g.flagged[g.left == 1].mean()}), include_groups=False)
print(fair.round(2))
What you’ll see
            people  share_flagged  share_left  flagged_who_left  leavers_flagged
gender                                                                          
man         1554.0           0.10        0.18              0.50             0.27
not stated   128.0           0.12        0.21              0.40             0.22
woman       1418.0           0.10        0.17              0.41             0.24

Men and women are on the list at the same rate (10%), and it holds a similar share of each group’s leavers (27% and 24%). Of the people on the list, more of the men resigned (50% against 41%). With about 150 people of each on the list, part of that gap is chance: check it again every time you retrain, and if it lasts, look for the column behind it. The “not stated” group, at 128 people, is too small to read much into.

Step 10: Report by team, not by name

Before using the model, retrain it on both years. Then, each quarter, score the latest snapshot and add up the chances by team. Because the chances are honest (Step 7), the sum is the number of people each team can expect to lose in the next 12 months. Here the 2025 snapshot stands in for today’s.

CodeExpected leavers by team
pythonRun in Colab
final = logreg.fit(pd.concat([X_train, X_test]), pd.concat([y_train, y_test]))   # learn from both years

def team_report(snapshot):
    """Expected leavers per team over the next 12 months, with the conditions the model leans on most."""
    s = snapshot.assign(risk=final.predict_proba(snapshot[FEATURES])[:, 1])
    return s.groupby("department", observed=True).agg(
        people=("risk", "size"), expected_leavers=("risk", "sum"), average_risk=("risk", "mean"),
        overtime_hours=("overtime_hours", "mean"), pay_vs_band=("pay_vs_band", "mean"),
        skipped_survey=("engagement", lambda x: x.isna().mean())).sort_values("average_risk", ascending=False)

print(team_report(df[df.snapshot_date == "2025-01-01"]).round(2))
What you’ll see
             people  expected_leavers  average_risk  overtime_hours  pay_vs_band  skipped_survey
department                                                                                      
support         619            138.60          0.22           13.48         0.96            0.16
sales           781            143.30          0.18            7.62         1.00            0.16
operations      469             84.90          0.18           10.69         1.00            0.17
finance         304             47.32          0.16            7.45         1.00            0.13
engineering     927            139.46          0.15            7.04         1.00            0.14

Support stands out: the highest average risk (22%), the most overtime (13.5 hours a month, against 7 to 11 elsewhere) and the lowest pay in its bands. That points to a fix for the team, such as more staff or a pay review, rather than a list of names. Run the same report every quarter: if a change works, the team’s expected leavers should fall, and then its actual ones.

  • Retrain once a year, when the next year’s resignations are complete, and repeat Steps 7 and 9 each time.
  • Watch for change. A reorganization, a new pay policy or a move to a new office changes why people leave. That’s data drift: if expected and actual leavers per team drift apart, retrain.
  • Measure the fixes, not the model. Compare teams that got a change with similar teams that haven’t yet, over the following year.

Common pitfalls

  • Using the score for decisions about one person, which can push out the very people it flags.
  • Training on columns recorded after the snapshot, such as notice periods or exit interviews.
  • Dropping people who skipped the survey, who are often the ones most likely to leave.
  • Reading the chances as exact without checking them against what really happened.
  • Checking the overall score but never comparing how the model does for different groups of people.

Check what you learned

4 quick questions. Choose an answer to see whether it’s right, and why.

  1. 1. Why are gender and the resignation date kept out of the model’s columns?

  2. 2. Logistic regression and gradient boosting rank people almost equally well. Which should you use?

  3. 3. Why check that the predicted chances match how many people actually resigned?

  4. 4. Support has the highest risk, the most overtime and the lowest pay in its bands. What’s the best use of that?

Want this tailored to your data, team and budget? Create a plan for your project

More on Tabular classification · Next guide: How to forecast inventory demand and set reorder points

Create my plan