How to label data for machine learning

Step 1: Look for labels you already have

Before paying anyone to label, check your own systems. Many labels are recorded as a side effect of normal work: the team that finally solved a support ticket, a refund that was approved, a part that failed inspection, an email a user moved to spam. They’re free, and there are usually thousands of them.

Existing labels have flaws: they reflect old processes, busy people and shortcuts. But they’re a good start, and the checks later in this guide work on them too. Label by hand when no such record exists, or when the records answer a different question from the one your model must answer.

Step 2: Design the label set

This guide labels support tickets by team, but the same steps work for photos, reviews, transactions or any other examples. A good label set has a few properties:

• The labels don’t overlap. If a ticket can honestly be both “billing” and “account,” people will split those tickets at random. Either merge the two labels, or add a rule for which wins.
• There’s a way out. Add “other” or “not enough information,” so nobody is forced to guess. Guessed labels look like real ones and are hard to find later.
• It matches what you’ll do with it. Ten fine-grained labels are only useful if the system acts differently on each one. Fewer labels are faster to apply and easier to agree on.
• Every label is common enough. A label that appears in one example out of a thousand needs a lot of labeling before the model sees it often. Group rare cases under “other” at first.

Decide also whether an example gets exactly one label or can have several. One label is simpler for people and models. Choose several only when an example really can belong to several groups, such as a photo with both a dent and a scratch.

Step 3: Write labeling guidelines

Guidelines turn your idea of each label into rules other people can follow. Keep them short enough to read in five minutes: a definition and a real example for each label, and a list of tricky cases with the agreed answer. The tricky cases are the most valuable part, and the list grows every time labelers disagree.

# Labeling guidelines: support tickets (version 3)

## The task
Read the ticket and choose the ONE team that should handle it first.

## Labels
billing: charges, refunds, invoices, payment methods.
  Example: "I was charged twice this month."
account access: logins, passwords, two-factor codes, locked accounts.
  Example: "The password reset email never arrives."
technical problem: part of the product doesn't work as it should.
  Example: "Export to PDF fails with an error."
feature request: asks for something the product doesn't do yet.
  Example: "Could you add a dark mode?"
other: none of the above, or not enough information. Don't guess.

## Tricky cases (add one each time labelers disagree)
- Can't log in to update a card: account access. They must log in first.
- Wants a refund because a feature broke: billing. They're asking for money.
- Several problems in one ticket: label the first one the customer mentions.

Give the guidelines a version number, and record which version each example was labeled with. When a rule changes, you’ll know which labels may need a second look.

Step 4: Set up a labeling tool

Spreadsheets work for a hundred examples, but they don’t record who labeled what, and they make it easy to edit the wrong row. A labeling tool shows one example at a time with the labels as buttons, and keeps a full history. Label Studio is free, open source and handles text, images and audio; CVAT is a common choice for images and video.

pip install label-studio
label-studio start

It opens in your browser. Create a project, choose the text classification template, enter your labels and import a CSV file of examples. When a round is done, export the results as CSV. The code below expects one row per label given, with the columns item_id, text, annotator and label; rename the exported columns to match.

Step 5: Run a pilot and measure agreement

Before labeling thousands of examples, have two people label the same 100 to 200, independently, using the guidelines. Then compare. How often they agree, beyond what chance alone would give, is inter-annotator agreement. The usual measure is Cohen’s kappa: 1 means they always agree, 0 means they agree no more than chance would.

# pip install pandas scikit-learn
import numpy as np
import pandas as pd
from sklearn.metrics import cohen_kappa_score

pilot = pd.read_csv("pilot_labels.csv")   # item_id, text, annotator, label
both = pilot.pivot_table(index="item_id", columns="annotator", values="label", aggfunc="first").dropna()
a, b = both.columns[:2]
print(f"{len(both)} items labeled by both {a} and {b}")
print(f"Raw agreement: {(both[a] == both[b]).mean():.0%}")
print(f"Cohen's kappa: {cohen_kappa_score(both[a], both[b]):.2f}")

As a rough guide, a kappa above 0.8 is very good, 0.6 to 0.8 is workable, and below 0.6 means the guidelines need work before you label more. Kappa matters most when one label is common: if 95% of tickets are “technical problem,” two people who almost always say so agree about 90% of the time by chance alone, and the raw agreement looks better than it is.

The score tells you whether there’s a problem; the disagreements tell you what it is. Print them and read them together with the labelers.

print(pd.crosstab(both[a], both[b]))

texts = pilot.drop_duplicates("item_id").set_index("item_id").text
disagree = both[both[a] != both[b]].join(texts)
print(disagree.head(20).to_string())

Most disagreements fall into a few patterns: two labels that overlap, a kind of example the guidelines don’t mention, or one person reading a rule differently. Fix each pattern with a new rule or a merged label, then run another small pilot. Two or three rounds are usually enough.

Step 6: Keep quality up with a gold set

Once labeling is under way, quality can drift: people get tired, new labelers join, and rules are forgotten. A gold set catches this. Ask your most experienced people to agree on the right answer for 100 or so examples, then mix them in among the normal work, so nobody knows which ones they are. Each labeler’s accuracy on the gold examples shows who needs help, and with which labels.

gold = pd.read_csv("gold.csv")       # item_id, gold_label
labels = pd.read_csv("labels.csv")   # item_id, text, annotator, label
checked = labels.merge(gold, on="item_id")
checked["correct"] = checked.label == checked.gold_label
print(checked.groupby("annotator").correct.agg(gold_items="count", accuracy="mean").round(2))

missed = checked[~checked.correct]
print(missed.groupby(["gold_label", "label"]).size().sort_values(ascending=False).head(5))

A labeler well below the others usually needs a short conversation, not replacing: check the mistakes together against the guidelines. If everyone misses the same gold examples, the gold answer or the rule may be wrong. Recheck it.

For the examples that matter most, such as your test set, have two people label each one and a third settle the disagreements. It costs more, but your test score is only as reliable as the test labels.

Step 7: Label the most useful examples next

Random examples are often easy: the model already gets most of them right, so labeling them teaches it little. active learning picks the examples the model is least sure about instead. Train a model on what you’ve labeled so far, score the unlabeled examples, and send the ones where its top two labels are closest to the labelers.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

labeled = pd.read_csv("labeled.csv")    # item_id, text, label: final, agreed labels
pool = pd.read_csv("unlabeled.csv")     # item_id, text: not labeled yet

def new_model():
    return make_pipeline(
        TfidfVectorizer(ngram_range=(1, 2), min_df=2, sublinear_tf=True),
        LogisticRegression(max_iter=2000, C=5),
    )

model = new_model().fit(labeled.text, labeled.label)
top2 = np.sort(model.predict_proba(pool.text), axis=1)[:, -2:]
pool["margin"] = top2[:, 1] - top2[:, 0]          # small: torn between two labels

unsure = pool.nsmallest(160, "margin")
also_random = pool.drop(unsure.index).sample(40, random_state=0)
batch = pd.concat([unsure, also_random])
batch[["item_id", "text"]].to_csv("next_batch.csv", index=False)
print(f"Next batch: {len(batch)} tickets, median margin {unsure.margin.median():.2f}")

The batch mixes in some random examples on purpose. Uncertain examples alone can all come from one confusing corner of your data, and the model also needs to keep seeing ordinary examples. Retrain after each batch and repeat.

Active learning helps most when the labels are expensive and the interesting cases are rare. On some data it barely beats random picking. With a few thousand labels already, you can check before relying on it: replay the labeling on data you’ve labeled, once picking at random and once picking the least sure, and compare the scores after each round.

from sklearn.metrics import f1_score
from sklearn.model_selection import train_test_split

def simulate(strategy, start=100, step=100, rounds=8, seed=0):
    rng = np.random.default_rng(seed)
    train, test = train_test_split(labeled, test_size=0.3, stratify=labeled.label, random_state=seed)
    picked = list(rng.choice(len(train), start, replace=False))
    scores = []
    for _ in range(rounds):
        part = train.iloc[picked]
        m = new_model().fit(part.text, part.label)
        scores.append(round(f1_score(test.label, m.predict(test.text), average="macro"), 3))
        rest = np.setdiff1d(np.arange(len(train)), picked)
        if strategy == "least sure":
            p = np.sort(m.predict_proba(train.text.iloc[rest]), axis=1)
            order = np.argsort(p[:, -1] - p[:, -2])
        else:
            order = rng.permutation(len(rest))
        picked += list(rest[order[:step]])
    return scores

for strategy in ["random", "least sure"]:
    print(f"{strategy:>10}:", simulate(strategy))

This needs about 1,500 labeled examples. If “least sure” reaches the same score with fewer labels, use it. If the two lines are close, pick at random: it’s simpler, and the labeled set stays representative.

Step 8: Let the model draft labels

Checking a suggested label is much faster than choosing one from scratch. Once the model is reasonably good, use it to draft labels for the examples it’s confident about, and have people accept or correct them. Most labeling tools, Label Studio included, can show these drafts as pre-filled answers.

rest = pool.drop(batch.index)
proba = model.predict_proba(rest.text)
rest = rest.assign(suggested=model.classes_[proba.argmax(axis=1)], confidence=proba.max(axis=1).round(2))
drafts = rest[rest.confidence >= 0.9]
print(f"{len(drafts)} of {len(rest)} tickets get a draft label with confidence 0.9 or more")
drafts[["item_id", "text", "suggested"]].to_csv("drafts_to_check.csv", index=False)

Drafts have a cost: people tend to accept what they’re shown, even when it’s wrong. Keep some gold examples in every batch with deliberately wrong drafts, and check how many of them get corrected. If most slip through, people are clicking “accept” without reading.

No labels at all yet? A large language model can draft the first labels with no training, given the label names and the guidelines. That’s zero-shot classification, shown in the ticket routing guide. Check its drafts the same way, and never use them as test labels without a person checking each one.

Step 9: Find labeling mistakes

Every labeled dataset has some mistakes, often a few percent, and they hurt twice: the model learns from the wrong answers, and the test score is measured against them. That’s label noise. A model can help find it. Train on part of the data, predict the rest, and look at the examples where the model confidently disagrees with the label it was given. Using cross-validation, every example is predicted by a model that never saw its label.

from sklearn.model_selection import cross_val_predict

proba = cross_val_predict(new_model(), labeled.text, labeled.label, cv=5, method="predict_proba")
classes = np.sort(labeled.label.unique())   # the column order of proba
given = proba[np.arange(len(labeled)), np.searchsorted(classes, labeled.label)]
check = labeled.assign(
    model_says=classes[proba.argmax(axis=1)],
    model_prob=proba.max(axis=1).round(2),
    given_prob=given.round(2),
)
suspects = check[check.model_says != check.label].sort_values("given_prob")
print(f"{len(suspects)} labels the model disagrees with; the 20 most doubtful:")
print(suspects.head(20)[["text", "label", "model_says", "model_prob"]].to_string())

Have a person review the top of the list. Many will be real mistakes; others are hard examples that the guidelines don’t cover, which become new rules. Don’t let the model relabel them automatically: it would only learn its own opinion back. The open-source library cleanlab does the same kind of check with more care, if you want to go further.

Step 10: Decide when you have enough

More labels help less and less as you go. A learning curve shows where you are: train on growing parts of your labeled data and watch the score. If it’s still climbing, keep labeling. If it has flattened, more of the same labels won’t help much; better guidelines, fixing mistakes or labeling the rare cases will.

from sklearn.model_selection import learning_curve

sizes, _, scores = learning_curve(
    new_model(), labeled.text, labeled.label, cv=5, scoring="f1_macro",
    train_sizes=[0.1, 0.25, 0.5, 0.75, 1.0], shuffle=True, random_state=0,
)
for n, s in zip(sizes, scores.mean(axis=1)):
    print(f"{n:>6} labeled tickets: macro F1 {s:.3f}")

Look at the score for each label too. A curve that’s flat overall can hide one rare label that would still improve with more examples, and that’s where the next round of labeling should go.

Common pitfalls

• Labeling thousands of examples before running a pilot, then finding out the labels overlap.
• Guidelines that live in one person’s head, so every new labeler learns different rules.
• Forcing a choice when the example is unclear, instead of offering “other” or “not enough information.”
• Judging quality by raw agreement when one label is far more common than the others.
• Using model-drafted labels in the test set without a person checking each one.
• Labeling only the examples the model is unsure about, so the labeled set no longer looks like real data.
• Never measuring label quality, then blaming the model for mistakes that are in the labels.