Lesson 5 of 11
Label less: active learning and drafts
In plain words: You rarely need to label everything. Label the examples that teach the model most, and let a model draft labels for people to check.
- Start small: a few hundred labeled examples per label is often enough for a first model and a first score.
- active learning: label next the examples the model is least sure about, mixed with some random ones so the data stays realistic.
- Drafts: let a model, or a language model with zero-shot prompting, suggest labels for people to accept or fix. Checking is faster than labeling.
- People tend to accept suggestions. Hide some wrong drafts among gold examples and check that they get corrected.
- Rules and existing signals, such as keywords or a user’s “report spam” click, can label a lot of data roughly. Check a sample by hand.
- Never put unchecked model labels in the test set.
Tip Plot a learning curve. When doubling the labels barely moves the score, spend the time on guidelines, mistakes and rare cases instead.
See it in a real project
- How to label data for machine learning: picking the next batch, model drafts and a learning curve
- How to route support tickets to the right team: a zero-shot language model before you have labels
- How to flag unusual expense claims: turning every review into a label