Logo
Machine Learning (2026-2027) - Sentiment Analysis with Binary Classification

In the print dialog, choose "Save as PDF" as the destination.

Machine Learning, Week 10

Sentiment Analysis with Binary Classification

Turn a review into word counts, then into a score between 0 and 1 with logistic regression.

Objectives

  • Explain sentiment analysis as binary classification: the score is the probability of the positive class
  • Describe what CountVectorizer does: lower case, tokens, stop words, unigrams and bigrams, the vocabulary
  • Compute a review's score by hand from the bias, the weights and the sigmoid
  • Train and evaluate the notebook's model: confusion matrix, accuracy, precision, recall, ROC AUC
  • Score new reviews with predict_proba, and explain where a bag of words goes wrong

Week 10 of the plan

Where This Sits in the Course

  • The notebook: Sentiment Analysis, logistic regression on 50,000 movie reviews labelled 0 or 1
  • Real-world scenario: give a text a score from 0 to 1, from negative to positive
  • Project milestone this week: Project Teams Presentations and Discussions

The Scenario: Score a Restaurant Review

ReviewThe plan expects
Best meal I've ever had and awesome service, too!0.9 or higher
Long lines and poor customer servicecloser to 0

Plan for the Two Hours

PartWhat we doTime
1 and 2Sentiment as a probability; text to numbers with CountVectorizer40 min
3Scoring a review by hand20 min
4The notebook in Colab: train, evaluate, score20 min
5Limits of the model, and your project10 min
6Practice with answers30 min

Part 1

Sentiment as a Probability

Binary classification on text

What Is Sentiment Analysis?

Sentiment analysis
Assigning a text a score from 0 to 1, where 0 is negative and 1 is positive sentiment.
TermIn this week's data
Sampleone review, a string of text
Label y0 negative, 1 positive
Features xthe word counts of the review
Score pthe probability that the label is 1

The Pipeline

Flow diagram: Review then Vectorize then (counts) Score z then Squash then Decide.

Review

a string

Vectorize

CountVectorizer

counts

Score z

b + Σ w·x

Squash

p = σ(z)

Decide

p ≥ 0.5: positive

Part 2

Text to Numbers

CountVectorizer and the bag of words

The Bag of Words

Bag of words
Keep a list of known words, the vocabulary, and describe each text by how many times each word of the list appears in it.
  • A model cannot work with text: it needs numbers
  • One row per text, one column per vocabulary entry, cells are counts
  • The order of the words is thrown away: that is why it is a bag

CountVectorizer in the Notebook

python
vectorizer = CountVectorizer(
    ngram_range=(1, 2),
    stop_words='english',
    min_df=20)
x = vectorizer.fit_transform(
    df['Text'])
y = df['Sentiment']
From sklearn.feature_extraction.text
  1. Lower case: pOOr becomes poor
  2. Tokens: runs of 2 or more letters, digits or _
  3. Remove stop words: 318 words such as the, and, even not
  4. Add bigrams: neighbouring pairs such as customer service
  5. Keep the vocabulary: entries found in at least 20 reviews (min_df=20)
  6. Count each entry: one row per review

From the notebook

Worked Example: What the Vectorizer Keeps

python
text = vectorizer.transform(['The long l3ines   and; pOOr customer# service really turned me off...123.'])
text = vectorizer.inverse_transform(text)
print(text)

Step 1: Lower Case and Tokens

text
the, long, l3ines, and, poor, customer, service,
really, turned, me, off, 123

Step 2: Remove the Stop Words

text
long, l3ines, poor, customer, service, really, turned, 123

Step 3: Add the Bigrams

text
long l3ines, l3ines poor, poor customer, customer service,
service really, really turned, turned 123

Step 4: Keep the Vocabulary

text
[array(['customer', 'long', 'poor', 'really', 'service', 'turned'],
      dtype='<U25')]

Worked Example: Three Reviews as a Count Matrix

Unigrams only. Predict the columns, and the cell of review 2 under awesome: the vectorizer counts

python
vec = CountVectorizer(stop_words='english')
x = vec.fit_transform(['Long lines and poor customer service',
    "Best meal I've ever had and awesome service, too!",
    'Awesome meal, awesome service'])
text
   awesome  best  customer  lines  long  meal  poor  service  ve
0        0     0         1      1     1     0     1        1   0
1        1     1         0      0     0     1     0        1   1
2        2     0         0      0     0     1     0        1   0

Part 3

Scoring a Review by Hand

The logistic regression of week 4, on word counts

The Formula of Week 4

z = b + w1 x1 + w2 x2 + ... + wn xn
p = σ(z) = 11 + e-z
SymbolMeaning
bthe bias, model.intercept_: here -0.0488
w_ithe weight of vocabulary entry i, from model.coef_
x_ihow many times entry i appears: mostly 0
σ, ethe sigmoid; e = 2.71828, and σ(0) = 0.5

Which Words Push Hardest?

Bar chart of the 12 most negative weights, led by disappointment, waste and worst, and the 12 most positive, led by funniest, 10 10 and excellent
  • A weight above 0 pushes to positive, below 0 to negative
  • Most negative: disappointment -2.1646, waste -2.1606, worst -2.1519
  • Most positive: funniest 1.6471, 10 10 1.4037, excellent 1.3094
  • No known word at all: p = σ(-0.0488) = 0.4878

By hand, step 1

Worked Example: Long Lines and Poor Customer Service

Which features does the model see?

Featurexww·x
customer1-0.1980-0.1980
lines1-0.1184-0.1184
long1-0.0284-0.0284
poor1-1.4874-1.4874
service1-0.4832-0.4832

Step 2: Compute z

b = -0.0488, weights rounded to 4 decimals

Σ w x = -0.1980 - 0.1184 - 0.0284 - 1.4874 - 0.4832
Σ w x = -2.3154, z = -0.0488 + (-2.3154) = -2.3642

Step 3: The Sigmoid and the Decision

e2.3642 = 10.6355
p = 11 + 10.6355 = 0.0859

By hand, step 1

Worked Example: Best Meal I've Ever Had

Best meal I've ever had and awesome service, too!

Featurexww·x
awesome10.73100.7310
best10.70900.7090
meal10.00620.0062
service1-0.4832-0.4832
ve1-0.0820-0.0820

Step 2: z and p

The plan says 0.9 or higher

z = -0.0488 + 0.7310 + 0.7090 + 0.0062 - 0.4832 - 0.0820 = 0.8322
p = 11 + e-0.8322 = 11 + 0.4351 = 0.6968

Why Not 0.9?

Positive, but predict_proba gives 0.69682

  • The model learned from movie reviews: there service is a negative word (-0.4832)
  • staff is -0.4587 too, while rude is only -0.0364
  • A short text has few features, so Σ w·x stays small and p stays near the middle
  • The number depends on the training data; the same code trained on restaurant reviews would learn restaurant weights

Try It: The Sentiment Meter

Part 4

The Notebook in Colab

Train, evaluate, score

Open the Notebook in Colab

text
https://colab.research.google.com/github/
jeffprosise/Machine-Learning/blob/master/
Sentiment%20Analysis.ipynb
Join the three lines into one address, or click "Open in Colab" on the lesson page.
In the notebookChange it to
read_csv('Data/reviews.csv', ...)read the file from the repository (next slide)
plot_confusion_matrix(...)ConfusionMatrixDisplay.from_estimator(...)
pickle.dump(..., open('Data/sentiment.pkl', 'wb'))save to 'sentiment.pkl'

Load the Data, Remove Duplicates

python
url = ("https://raw.githubusercontent.com/"
       "jeffprosise/Machine-Learning/master/Data/reviews.csv")
df = pd.read_csv(url, encoding="ISO-8859-1")
df.groupby('Sentiment').describe()
df = df.drop_duplicates()

Vectorize and Split

python
x = vectorizer.fit_transform(df['Text'])
y = df['Sentiment']

x_train, x_test, y_train, y_test = train_test_split(
    x, y, test_size=0.5, random_state=0)
  • x.shape is (49581, 33752): 17,528 unigrams and 16,224 bigrams
  • Only 0.30% of the 1.67 billion cells are not 0: a sparse matrix
  • 50/50 split: 24,790 reviews to train, 24,791 to test

Train the Model

python
from sklearn.linear_model import LogisticRegression

model = LogisticRegression(max_iter=1000, random_state=0)
model.fit(x_train, y_train)
  • fit learns 33,752 weights, one per vocabulary entry, plus the bias b
  • max_iter=1000 gives the solver room to converge on so many features; our run needed 78
  • Nothing new: the same LogisticRegression as week 4

The Confusion Matrix

On the 24,791 test reviews

Confusion matrix of the sentiment model: 10792 and 1577 in the Negative row, 1456 and 10966 in the Positive row
python
ConfusionMatrixDisplay.from_estimator(
    model, x_test, y_test,
    display_labels=['Negative', 'Positive'],
    cmap='Blues',
    xticks_rotation='vertical')

Worked Example: Score the Model

TN 10792, FP 1577, FN 1456, TP 10966. Predict A accuracy, and P precision and R recall of positive

A = 10792 + 1096624791 = 0.8777
P = 1096610966 + 1577 = 0.8743, R = 1096610966 + 1456 = 0.8828

ROC AUC: Judging the Scores

python
from sklearn.metrics import roc_auc_score

probabilities = model.predict_proba(x_test)
roc_auc_score(y_test, probabilities[:, 1])

Try It: Move the Threshold

Score New Reviews

You computed both by hand. Predict the output

python
reviews = ['Long lines and poor customer service',
           "Best meal I've ever had and awesome service, too!"]
model.predict_proba(vectorizer.transform(reviews))[:, 1]
text
[0.08594148 0.69681757]

Part 5

Where a Bag of Words Goes Wrong

88% accurate, and still easy to fool

Negation Is Thrown Away

ReviewFeaturesp
The food was goodfood, good0.5885
The food was not good??

Word Order Is Thrown Away

Reviewp
Great food but terrible service0.2591
Terrible food but great service0.2591

Before you practise

Common Mistakes

  • Reading Data/reviews.csv in Colab from a folder that does not exist there
  • Calling plot_confusion_matrix: use ConfusionMatrixDisplay.from_estimator
  • fit_transform on new reviews: new text needs vectorizer.transform([...])
  • Passing a plain string to transform: it needs a list of texts
  • Reading predict_proba(...)[0][0]: the sentiment score is column 1
  • Expecting not to flip a score while stop_words='english' removes it

Your team project

Project Milestone: Presentations and Discussions

  • This week: Project Teams Presentations and Discussions
  • Next week, final submission: code on a GitHub repository, a 5-minute video, the proposal, the presentation, the logo
  • Additionally: a project user interface (mobile, web or desktop)
  • Text in your project? Vectorize, train, score with predict_proba, and save the model and the vocabulary

Key Takeaways

  1. Sentiment analysis is binary classification: the score is p, the probability of positive
  2. CountVectorizer: lower case, tokens, stop words out, unigrams and bigrams, vocabulary only
  3. The score is σ(b + Σ w·x): every known word pushes by its weight
  4. The notebook's model: accuracy 0.8777, ROC AUC 0.9452 on 24,791 test reviews
  5. A bag of words ignores order and, without stop words, negation; the weights reflect the training data

Part 6

Practice: Your Turn

About 30 minutes, answers follow each task

About 5 minutes

Practice 1: Vectorize by Hand

The waiter was rude and the food was cold.

text
tokens:   the, waiter, was, rude, and, the, food, was, cold
kept:     waiter, rude, food, cold
n-grams:  waiter, rude, food, cold, waiter rude, rude food, food cold

About 5 minutes

Practice 2a: Score It by Hand

The waiter was rude and the food was cold. Only these four words are in the vocabulary; b = -0.0488. Compute z, p and the prediction.

Featurew
waiter-0.0937
rude-0.0364
food0.2302
cold-0.2052
z = -0.0488 + (-0.1051) = -0.1539
p = 11 + 1.1664 = 0.4616

About 5 minutes

Practice 2b: A Very Positive Review

Excellent food, friendly staff, highly recommend! b = -0.0488

Σ w x = 3.3489, z = 3.3001, p = 11 + 0.0369 = 0.9644

About 5 minutes

Practice 3: Choose a Threshold

Goal: find the most negative reviews. Compute TN / (TN + FP) at both. Which wins, at what cost?

tTN, FPFN, TP
0.310232, 21371028, 11394
0.711251, 11182054, 10368
t = 0.3: 1023212369 = 0.8272, t = 0.7: 1125112369 = 0.9096

About 10 minutes

Practice 4: In Colab

  1. Run the notebook with the three changes; score both scenario reviews in one call
  2. Score The food was good and The food was not good
  3. Refit with CountVectorizer(ngram_range=(1, 2), min_df=20): stop words kept
  4. Same split (random_state=0), retrain: number of features, accuracy, ROC AUC?
  5. Score the two food reviews again. What changed, and why?

Answers

Practice 4: Answer

QuestionOur run
Scenario reviews[0.08594148 0.69681757]
Food reviews, stop words removed0.5885 and 0.5885
Stop words kept: features, accuracy, AUC74,938, 0.8967, 0.9574
Food reviews, stop words keptgood 0.7082, not good 0.2912
Whynot good is now a feature, weight -0.7838

Open this lesson

Mahmoud AbasSentiment Analysis with Binary Classification