What is inter-annotator agreement and Cohen’s kappa?

Why it matters

A model learns the pattern in its labels. If two careful people, reading the same guidelines, label the same ticket differently, the labels hold two patterns, and the model learns a blur of both. Agreement between labelers is the quickest way to find out, before you’ve labeled thousands of examples, whether the task is clear enough to learn.

It also sets expectations. If people agree on 85% of examples, part of the test set is labeled one way when another person would have labeled it differently. A model scoring far above that agreement on such a test set is a reason to look for a bug, not to celebrate.

Why raw agreement misleads

The simplest measure is the share of examples on which two people give the same label. It looks reassuring when one label is common. Say 95% of transactions are normal and two reviewers each flag 5% as fraud. Even if they flag completely different transactions, they agree on 90% of them, because both say “normal” most of the time.

How Cohen’s kappa works

Kappa compares the agreement you see with the agreement you’d expect if each person kept their own label mix but chose at random:

• Observed agreement: the share of examples both people labeled the same way.
• Chance agreement: worked out from how often each person uses each label. In the fraud example, it’s 0.95 × 0.95 + 0.05 × 0.05 ≈ 90%.
• Kappa is observed minus chance, divided by 1 minus chance. It’s 1 when they always agree, 0 when they agree only as often as chance, and below 0 when they agree less than that.

# pip install scikit-learn
from sklearn.metrics import cohen_kappa_score

ana = ["ok", "ok", "ok", "ok", "ok", "ok", "ok", "ok", "ok", "fraud"]
ben = ["ok", "ok", "ok", "ok", "ok", "ok", "ok", "ok", "fraud", "ok"]
agree = sum(x == y for x, y in zip(ana, ben)) / len(ana)
print(f"Raw agreement: {agree:.0%}")                       # 80%
print(f"Cohen's kappa: {cohen_kappa_score(ana, ben):.2f}")  # below 0: worse than chance

They agree on 80% of transactions, but never on which one is fraud, and that’s the only question that matters. Kappa shows it.

What a good score is

• Above 0.8: very good. The guidelines are clear enough to label at scale.
• 0.6 to 0.8: workable for most tasks. Read the disagreements to find rules worth adding.
• Below 0.6: the guidelines, or the label set, need work before you label more.

Treat these as rough guides. Subjective tasks, such as rating how angry a review is, rarely reach 0.8 however good the guidelines are. Then use fewer, broader labels, or have several people label each example and take the majority.

More than two labelers, and other measures

Cohen’s kappa compares two people. With more labelers, compute it for each pair and look at the lowest pairs, or use Fleiss’ kappa or Krippendorff’s alpha, which handle any number of labelers and missing labels. For labels with an order, such as 1 to 5 stars, a weighted kappa counts a 4 against a 5 as a smaller disagreement than a 1 against a 5; scikit-learn’s cohen_kappa_score takes weights="linear" for that.

How to raise agreement

1. Print the disagreements and read them with the labelers.
2. Group them into patterns: overlapping labels, a missing rule, an unclear example.
3. Add a rule with a real example for each pattern, or merge labels that people can’t tell apart.
4. Run another small round on new examples and measure again.