What is TF-IDF?

The two parts

• TF, term frequency: how often a word appears in this document. A support ticket that says “refund” three times is probably about refunds.
• IDF, inverse document frequency: how rare the word is across all documents. “The” and “please” appear everywhere, so they say little about any one ticket. “Chargeback” appears in a few, so it’s informative.

Multiply the two, and each word’s weight reflects how characteristic it is of this particular text. Every document becomes a long row of numbers, one column per word in the vocabulary, mostly zeros.

from sklearn.feature_extraction.text import TfidfVectorizer

docs = ["I want a refund for my order",
        "My order arrived late",
        "Refund not received, please refund my card"]
vec = TfidfVectorizer()
X = vec.fit_transform(docs)

words = vec.get_feature_names_out()
row = X[2].toarray()[0]
for i in row.argsort()[::-1][:4]:
    print(f"{words[i]:>10}: {row[i]:.2f}")   # "refund" scores highest in the third text

Why it’s still worth using

TF-IDF is old and simple, and it ignores word order and meaning. Yet TF-IDF with a linear model such as logistic regression remains one of the best first models for text classification:

• It trains in seconds on a laptop, even on hundreds of thousands of texts.
• It needs no GPU and no downloads.
• It’s easy to inspect: you can list the words that push a prediction each way.
• On many practical tasks, such as spam filtering, routing tickets or tagging documents, it comes close to much larger models.

That makes it the ideal baseline: build it first, then only move to a pretrained language model if it clearly does better on your test set.

Settings that matter

• ngram_range=(1, 2) adds word pairs, so “not good” is a separate feature from “good.” Usually worth it.
• min_df=2 drops words that appear in only one document, mostly typos and IDs.
• sublinear_tf=True dampens repeated words, so a word said ten times doesn’t count ten times as much.
• Fit the vectorizer on training data only, inside a pipeline, so the test set’s vocabulary doesn’t leak into training.

Where it falls short

TF-IDF doesn’t know that “cancel” and “terminate” mean the same thing, so it needs examples of both. It struggles with sarcasm, long-range meaning and very short texts. When those matter, embeddings or a fine-tuned language model usually do better.