The idea
Training a large model from scratch takes enormous amounts of data and computing power. But a model trained on millions of images already knows edges, shapes and textures, and a language model already knows grammar and a lot about the world. Fine-tuning reuses that knowledge: you start from the pretrained model and continue training on a few hundred or a few thousand examples of your own task. It learns the specifics of your problem much faster, and from far less data, than starting from zero.
Common examples
- Images: a model pretrained on everyday photos, fine-tuned on 500 photos of your product to spot defects.
- Text classification: a language model fine-tuned on labeled support tickets to route them to the right team.
- Language models: a chat model fine-tuned on examples of your company’s writing style, or to always answer in a specific format.
Fine-tuning, prompting or RAG?
For language models, fine-tuning is rarely the first step. Try the cheaper options first:
- Prompting: describe the task clearly, perhaps with a few examples in the prompt (few-shot). It costs nothing to try and often works.
- RAG: if the model needs facts from your documents, retrieve the right passages and include them in the prompt. Fine-tuning is a poor way to teach facts that change.
- Fine-tuning: when the model must reliably follow a style, format or narrow task that prompting can’t pin down, or when a small fine-tuned model can replace a large, expensive one.
For images, the order is different: fine-tuning a pretrained image model is usually the standard first approach, because there’s no prompting equivalent for most visual tasks.
How much data and compute?
- Image classification: often a few hundred labeled images per class. A small model fine-tunes in minutes on a single GPU, such as a free Colab one.
- Text classification: a few hundred to a few thousand labeled examples, with similar compute.
- Language model behavior: a few hundred to a few thousand high-quality examples. Many providers offer fine-tuning as a managed service, priced by the amount of training text.
Quality matters more than quantity: 300 clean, consistent examples beat 3,000 messy ones.
Doing it well
- Keep a test set the fine-tuned model never trains on, and compare it with the model before fine-tuning, so you know the gain is real.
- Watch for overfitting: with few examples, fine-tuning for too long makes the model memorize them. Use early stopping on a validation set.
- Start with a low learning rate. Large changes can make the model forget what it already knew.