How to route support tickets to the right team

Step 1: Count what misrouting costs

Before any modeling, measure the problem. Look at last month’s tickets: how long did they wait before reaching the right team, and how many were moved from one team to another? A ticket that bounces twice can wait a day longer than it needs to.

That number shapes the goal. You don’t need every ticket routed automatically. Routing 70% of tickets correctly and instantly, and sending the other 30% to a person, can already halve the waiting time. Write the target down, for example: at least 95% of automatically routed tickets go to the right team.

Step 2: Learn from tickets you’ve already solved

Your help desk already holds the labels: every closed ticket records which team solved it. Use the final team, not the first one it was sent to, because the first assignment is exactly the guess you want to improve on. Export the ticket text, when it was created and the final team.

# pip install pandas scikit-learn
import numpy as np
import pandas as pd

tickets = pd.read_csv("tickets.csv", parse_dates=["created_at"])   # created_at, subject, body, team
tickets["text"] = tickets.subject.fillna("") + " " + tickets.body.fillna("")
print(tickets.team.value_counts())

Expect some teams to get far more tickets than others; that’s class imbalance. Teams with only a handful of tickets can’t be learned well, so group them under “other” and let a person handle those for now.

counts = tickets.team.value_counts()
rare = counts[counts < 50].index
tickets["team"] = tickets.team.where(~tickets.team.isin(rare), "other")

Step 3: Test on the most recent tickets

Products, features and common problems change over time, so test the model the way it will be used: train on older tickets and test on the latest two months. A random split would let the model learn about issues that only appear in the test period.

cutoff = tickets.created_at.max() - pd.DateOffset(months=2)
train, test = tickets[tickets.created_at <= cutoff], tickets[tickets.created_at > cutoff]
print(len(train), "training tickets,", len(test), "test tickets")

Step 4: Beat the busiest-team baseline

The simplest router sends every ticket to the busiest team. Its accuracy is that team’s share of tickets, and it’s the baseline to beat. Then train a real model: TF-IDF turns each ticket into word weights, and logistic regression learns which words point to which team. It trains in seconds and handles dozens of teams.

from sklearn.dummy import DummyClassifier
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report
from sklearn.pipeline import make_pipeline

baseline = DummyClassifier(strategy="most_frequent").fit(train.text, train.team)
print("Busiest-team baseline:", round(accuracy_score(test.team, baseline.predict(test.text)), 3))

model = make_pipeline(
    TfidfVectorizer(ngram_range=(1, 2), min_df=2, sublinear_tf=True),
    LogisticRegression(max_iter=2000, C=5),
)
model.fit(train.text, train.team)
print(classification_report(test.team, model.predict(test.text), digits=3, zero_division=0))

Read the report team by team. recall shows how many of a team’s tickets reach it; precision shows how many tickets sent to a team really belong there. A team with low precision is receiving other teams’ work.

Step 5: See which teams get mixed up

List the pairs of teams the model confuses most often. They usually point to overlapping responsibilities rather than a weak model.

pred = model.predict(test.text)
mix = pd.crosstab(test.team.values, pred, rownames=["true"], colnames=["predicted"]).stack()
off = mix[[a != b for a, b in mix.index]]
print(off[off > 0].sort_values(ascending=False).head(5))

If billing and accounts are often swapped, read some of those tickets. You may find they really need both teams, or that the boundary between the teams isn’t clear to people either. Fixing the process can help more than any model change.

Step 6: Route only the confident predictions

The model gives each team a probability. When the top one is high, routing automatically is safe; when it’s low, a person should decide. Try a few thresholds and see the trade-off between how many tickets route themselves and how many of those land in the right place.

proba = model.predict_proba(test.text)
confidence = proba.max(axis=1)
right = model.classes_[proba.argmax(axis=1)] == test.team.values

for t in [0.3, 0.4, 0.5, 0.6, 0.7, 0.8]:
    auto = confidence >= t
    if auto.any():
        print(f"threshold {t:.1f}: {auto.mean():.0%} routed automatically, {right[auto].mean():.1%} of them correct")

Pick the lowest threshold that meets your target from Step 1. For the tickets below it, show the person doing triage the model’s top two suggestions; picking from two is much faster than reading the whole team list.

top2 = model.classes_[np.argsort(proba, axis=1)[:, -2:][:, ::-1]]
in_top2 = [team in row for team, row in zip(test.team, top2)]
print(f"Right team among the top two: {np.mean(in_top2):.0%}")

Step 7: Route new tickets

Retrain on all your tickets, then save the model with the threshold you chose, so the two always travel together. Your help desk can call the routing function when a ticket arrives, or a small scheduled job can route new tickets every minute.

import joblib

THRESHOLD = 0.6   # the value you chose in Step 6
model.fit(tickets.text, tickets.team)
joblib.dump({"model": model, "threshold": THRESHOLD}, "router.joblib")

def route(subject, body):
    router = joblib.load("router.joblib")
    proba = router["model"].predict_proba([f"{subject} {body}"])[0]
    best = proba.argmax()
    if proba[best] < router["threshold"]:
        return "triage"
    return router["model"].classes_[best]

print(route("Charged twice", "I was billed two times for my October invoice."))

No labeled tickets yet? A language model can sort tickets with no training, given only the team names and a sentence about each. That’s zero-shot classification. It’s a good way to start, and the tickets it routes become the labels for the model above.

Step 8: Learn from re-routes

Every time an agent moves an automatically routed ticket to another team, you get a free, up-to-date label. Track the re-route rate every week: it’s the live version of your test score. If it rises, look at which teams are involved. Often a new product or a reorganized team is the cause; that’s data drift, and retraining on the latest tickets fixes most of it.

When a new team is created, the model can’t send anything there until it has examples. Route its tickets by hand, or with a simple keyword rule, for a few weeks, then retrain.

Common pitfalls

• Training on the first team a ticket was assigned to, instead of the team that solved it.
• Routing every ticket automatically, including the ones the model is unsure about.
• Testing on a random split, which hides how the model copes with new products and new problems.
• Keeping teams with a handful of tickets as separate classes the model can’t learn.
• Never retraining after teams change, so tickets keep going to a queue that no longer exists.