Lesson 4 of 11
Checking label quality
In plain words: Have two people label the same examples. How often they agree, beyond chance, tells you whether the task is clear enough to learn.
- Measure inter-annotator agreement with Cohen’s kappa: above 0.8 is very good, below 0.6 means the guidelines need work.
- Don’t trust raw agreement when one label is common: if 95% of examples get the same label, two people agree about 90% of the time by chance alone.
- Read the disagreements, not just the score. Most come from a few unclear cases, and each one becomes a new rule.
- Mix a gold set of examples with agreed answers into the normal work, to check each labeler’s accuracy as you go.
- After training, look for label noise: examples where a model that never saw them confidently predicts a different label. Many are mistakes.
- Have the test set labeled twice, with a third person settling disagreements: the test score is only as good as the test labels.
Tip If people can’t agree on a label, a model can’t learn it either. Fix the rules before labeling more.
See it in a real project
- How to label data for machine learning: agreement between two labelers, a gold set and finding wrong labels
- How to build a spam filter, start to finish: reading the mistakes, wrong labels included
- How to sort customer reviews by sentiment: reviews whose star rating doesn’t match the text