How to sort customer reviews by sentiment

Step 1: Decide what you’ll do with the answer

Sentiment is only useful if someone acts on it. Pick the action first, because it decides which mistakes matter:

• Catch unhappy customers: route negative reviews to support the same day. Missing a negative review is the costly mistake, so you care most about recall on the negative class.
• Track the trend: report the share of negative reviews each week. Small mistakes average out, but the model must be equally good every week.
• Find what people complain about: read the negative reviews by product or topic. Here a mix-up between neutral and negative matters less than finding the patterns.

This guide does all three, with three classes: positive, neutral and negative. Two classes are easier, but real reviews often say “fine, nothing special,” and forcing those into positive or negative makes the trend noisy.

Step 2: Use star ratings as free labels

Labeling thousands of reviews by hand is slow. If your reviews come with a star rating, you already have labels: 1 or 2 stars is negative, 3 is neutral, and 4 or 5 is positive. Export your reviews as a CSV with two columns, text and stars.

# pip install pandas scikit-learn
import pandas as pd

reviews = pd.read_csv("reviews.csv")             # text, stars (1 to 5)
reviews = reviews.dropna(subset=["text"]).drop_duplicates("text")
reviews["sentiment"] = pd.cut(reviews.stars, bins=[0, 2, 3, 5],
                              labels=["negative", "neutral", "positive"]).astype(str)
print(reviews.sentiment.value_counts(normalize=True).round(2))

No reviews of your own yet? Practice on public data. The Yelp review dataset has 650,000 reviews with 1 to 5 stars, and a sample of 50,000 is plenty.

# pip install datasets
from datasets import load_dataset

yelp = load_dataset("yelp_review_full", split="train").shuffle(seed=0).select(range(50_000)).to_pandas()
reviews = pd.DataFrame({"text": yelp.text, "stars": yelp.label + 1})   # labels are 0 to 4

Step 3: Split, then beat a baseline

Keep 20% of the reviews aside for testing, with the same mix of classes in both parts. Removing duplicates first, as above, matters: a review copied into both sets is data leakage and inflates the score.

Then measure a baseline: a “model” that always answers the most common class. With 70% positive reviews it reaches 70% accuracy while catching no unhappy customer at all. The macro F1 score averages the score of each class equally, so it exposes this immediately: the baseline gets zero for negative and neutral.

from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyClassifier
from sklearn.metrics import f1_score

X_train, X_test, y_train, y_test = train_test_split(
    reviews.text, reviews.sentiment, test_size=0.2, stratify=reviews.sentiment, random_state=0)

baseline = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
print("Baseline macro F1:", round(f1_score(y_test, baseline.predict(X_test), average="macro"), 3))

Step 4: Train a fast text model

TF-IDF turns each review into word weights, and logistic regression learns which words point to which class. Two settings matter for sentiment. Word pairs (ngram_range=(1, 2)) let the model see “not good” as different from “good.” And class_weight="balanced" makes mistakes on the rare classes count more, so the model doesn’t simply learn to say positive.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import classification_report

model = make_pipeline(
    TfidfVectorizer(ngram_range=(1, 2), min_df=2, sublinear_tf=True),
    LogisticRegression(max_iter=1000, class_weight="balanced"),
)
model.fit(X_train, y_train)
print(classification_report(y_test, model.predict(X_test), digits=3))

It trains in seconds on a laptop. Read the report row by row: positive will score highest, negative close behind, and neutral lowest. That’s normal, because a 3-star review is mixed by nature (“great food, slow service”) and people use the middle rating differently.

Step 5: Read the mistakes

A confusion table shows which classes get mixed up. Most errors should sit next to neutral; negative reviews predicted as positive are the serious ones.

from sklearn.metrics import confusion_matrix

labels = ["negative", "neutral", "positive"]
pred = model.predict(X_test)
print(pd.DataFrame(confusion_matrix(y_test, pred, labels=labels), index=labels, columns=labels))

test = pd.DataFrame({"text": X_test, "true": y_test, "pred": pred})
mistakes = test[test.true != test.pred]
print(mistakes.sample(min(20, len(mistakes)), random_state=0).to_string())

Read twenty or so mistakes yourself. You’ll usually find three kinds: sarcasm (“great, it broke on day two”), mixed reviews, and labels that are simply wrong, such as a 5-star rating on an angry review. The first two are hard for any model; the third tells you the labels have noise, so don’t chase a perfect score.

Step 6: Check which words drive each class

Logistic regression is easy to inspect: each word has a weight per class. Looking at the top words is a fast sanity check.

vec = model.named_steps["tfidfvectorizer"]
clf = model.named_steps["logisticregression"]
words = vec.get_feature_names_out()
for i, label in enumerate(clf.classes_):
    top = clf.coef_[i].argsort()[-12:][::-1]
    print(f"{label:>8}: " + ", ".join(words[top]))

You should see words like “worst,” “refund” and “never again” for negative, and “love” and “highly recommend” for positive. If a product name or a store location tops a list, the model has learned that this product gets bad reviews, not what a bad review sounds like. Consider removing those names from the text before training.

Step 7: Try a pretrained model, and keep the better one

A pretrained model has already learned sentiment from millions of texts, so it handles sarcasm and unusual wording better, with no training at all. It’s slower and needs more memory, so compare it on the same test set and keep it only if the gain is worth the cost.

# pip install transformers torch
from transformers import pipeline

pretrained = pipeline("sentiment-analysis", model="cardiffnlp/twitter-roberta-base-sentiment-latest")
sample = X_test.sample(500, random_state=0)
guess = [r["label"] for r in pretrained(sample.str.slice(0, 1000).tolist(), batch_size=32)]
print("Pretrained macro F1:", round(f1_score(y_test[sample.index], guess, average="macro"), 3))
print("Our model macro F1: ", round(f1_score(y_test[sample.index], model.predict(sample), average="macro"), 3))

If the pretrained model wins by a wide margin, use it, or fine-tune it on your own labeled reviews for an extra gain. If the scores are close, keep the small model: it’s cheaper to run and easier to explain.

Step 8: Score new text and track the trend

Retrain on all your labeled reviews, save the model, and use it on new text. It works on support emails and survey answers too, but check first: those are written differently from reviews. Label about 200 of them by hand and measure the score before trusting the trend.

import joblib

model.fit(reviews.text, reviews.sentiment)        # learn from every labeled review
joblib.dump(model, "sentiment.joblib")

emails = pd.read_csv("support_emails.csv", parse_dates=["date"])   # date, text
proba = model.predict_proba(emails.text)
emails["sentiment"] = model.classes_[proba.argmax(axis=1)]
emails["confidence"] = proba.max(axis=1).round(2)

weekly = emails.groupby(emails.date.dt.to_period("W")).sentiment.value_counts(normalize=True).unstack()
print(weekly.round(2))

Low-confidence predictions are good candidates for a person to read. Route those, plus every negative one, to the team, and let the rest feed the dashboard.

Step 9: Keep it honest over time

Language changes: new products, new slang, a new competitor everyone compares you to. That’s data drift, and it slowly lowers the score without any error message. Once a month, score the newest reviews that have stars and compare the predictions with the stars. If the macro F1 drops, retrain on recent reviews. Also watch the share of each class: a sudden jump in negative predictions is either real news or a broken model, and it’s worth reading a sample to find out which.

Common pitfalls

• Judging the model by accuracy. With mostly positive reviews, a useless model scores high.
• Forgetting negations. Single words can’t tell “good” from “not good”; word pairs or a pretrained model can.
• Trusting the model on a new kind of text, such as emails or tweets, without labeling a sample first.
• Letting product names or store locations become the signal instead of the words people use.
• Treating a sentiment score as a reason. It tells you who is unhappy, not why; read the reviews too.