What is transfer learning?

The idea

People don’t learn every skill from scratch. Someone who plays the piano picks up the organ faster, because much of what they know carries over. Models work the same way. A network trained to recognize 1,000 kinds of objects in millions of photos has learned, in its early layers, to detect edges, textures and shapes. Those are useful for almost any image task, from spotting cracks in parts to sorting X-rays.

Transfer learning keeps that general knowledge and only teaches the model the part that’s specific to your task.

Two common ways to do it

1. Use the model as a feature extractor. Run your images or texts through the pretrained model, take its internal representation (a list of numbers, an embedding), and train a small, simple model such as logistic regression on top. It’s fast and works with very little data.
2. Fine-tune it. Replace the model’s last layer with one for your classes, then continue training on your data, often with the early layers frozen at first. It takes a bit more data and compute, and usually gives the best results.

# pip install torch torchvision scikit-learn
import torch
from torchvision import models

weights = models.ResNet18_Weights.DEFAULT
backbone = models.resnet18(weights=weights)   # trained on ImageNet
backbone.fc = torch.nn.Identity()             # drop the 1,000-class layer, keep the features
backbone.eval()
preprocess = weights.transforms()

def features(pil_images):
    with torch.no_grad():
        return backbone(torch.stack([preprocess(im) for im in pil_images])).numpy()

# Then: LogisticRegression().fit(features(train_images), train_labels)

Why it matters

• Far less data: hundreds of examples instead of hundreds of thousands.
• Far less compute: minutes on one GPU, or even a CPU for feature extraction, instead of days on many.
• Better results on small datasets: the pretrained knowledge stops the model from memorizing your few examples.

When it works less well

Transfer learning works best when the original task resembles yours. A model pretrained on everyday photos transfers well to product photos, and less well to microscope slides or satellite radar. For very different data, look for a model pretrained on a similar domain (medical images, legal text, code), or fine-tune more layers with more data.