Mahmoud Abas, Full-Stack Software Engineer & AI

Machine Learning, Week 4

Logistic Regression

Predict whether a Titanic passenger survives, and how sure the model is.

Objectives

Explain how logistic regression turns a passenger into a score z, a probability p and a class
Compute the sigmoid, a probability and a prediction by hand from trained weights
Prepare the Titanic data: features, target, one-hot encoding, missing ages
Train LogisticRegression and read accuracy, the confusion matrix, precision, recall and F1
Move the threshold and explain the trade between precision and recall
Choose the features and the target of your own project

Week 4 of the plan

Where This Sits in the Course

Two notebooks: Titanic (Logistic Regression), a compact model on four columns, and Logistic_Regression_Classification, which explores, fills in and tunes
Real-world scenario: a binary-classification model that predicts whether a passenger will survive or not, using the Titanic dataset
Project milestone this week: Project Data Features Extraction and Target Variable Identification

Plan for the Two Hours

Part	What we do	Time
1	The idea, and a passenger by hand	25 min
2	Notebook 1: prepare, train, score	25 min
3	Evaluating: matrix, precision, recall, threshold	20 min
4	Notebook 2: explore, fill in, tune; your project	20 min
5	Takeaways, then practice with answers	30 min

Part 1

The Idea of Logistic Regression

A score, a squash, a decision

What Is Logistic Regression?

Logistic regression: A learning algorithm for binary classification: it predicts one of two classes, 0 or 1, and yields the probability of class 1 as well.

Score: weigh the features and add a bias, giving one number z
Squash: the sigmoid turns z into a probability p between 0 and 1
Decide: compare p with a threshold; predict uses 0.5

The Vocabulary, on Titanic

Term	In the Titanic data
Features `x`	input columns: age, sex, passenger class
Target `y`	`Survived`: `1` survived, `0` perished
Positive class	class `1`, whose probability the model reports
Probability `p`	the model's estimate that `y = 1`
Threshold `t`	the cut on `p`: `p ≥ t` gives `1`, else `0`

Step 1: The Linear Score

z = b + w₁ x₁ + w₂ x₂ + ... + w_n x_n

x_1 to x_n are the n features of one passenger
w_1 to w_n are the learned weights, in model.coef_
b is the bias, in model.intercept_

Step 2: The Sigmoid

p = σ(z) = 11 + e^-z

e is the constant 2.71828
For any z, the value σ(z) lies between 0 and 1: a probability
σ(0) = 0.5: a score of zero means no preference
Large positive z gives p near 1; large negative z gives p near 0

Worked Example: Three Values of the Sigmoid

σ(2) = 11 + e^-2 = 11 + 0.1353 = 0.8808

σ(-2) = 11 + e² = 11 + 7.3891 = 0.1192

Step 3: The Threshold

p ≥ t ⇒ ŷ = 1

p < t ⇒ ŷ = 0

ŷ is the predicted class and t the threshold
predict uses t = 0.5, and σ(0) = 0.5, so that is the same as asking whether z ≥ 0
Checked on the notebook model: predict and predict_proba ≥ 0.5 agree on all 143 test passengers

Try It: From a Passenger to a Probability

By hand

Worked Example: The Trained Weights

LogisticRegression(random_state=0) on six features, weights rounded to 3 decimals

Feature	Weight
bias `b`	1.212
`Age`	-0.039
`Sex_female`	1.149
`Sex_male`	-1.150

Feature	Weight
`Pclass_1`	1.200
`Pclass_2`	0.007
`Pclass_3`	-1.207

Step 1: The Feature Vector

A 30-year-old female traveling in first class

python

female = [[30, 1, 0, 1, 0, 0]]

Step 2: The Score

z = 1.212 + (-0.039)(30) + 1.149 + 1.200

z = 1.212 - 1.170 + 1.149 + 1.200 = 2.391

Step 3: The Probability

p = 11 + e^-2.391 = 11 + 0.0915 = 0.916

Step 4: The Class

p = 0.916 ≥ 0.5 ⇒ ŷ = 1

Worked Example: A 60-Year-Old Man in Third Class

male = [[60, 0, 1, 0, 0, 1]]

z = 1.212 + (-0.039)(60) + (-1.150) + (-1.207) = -3.485

p = 11 + e^3.485 = 11 + 32.622 = 0.030 ⇒ ŷ = 0

Part 2

Notebook 1: A Compact Titanic Model

Four columns, one model, real probabilities

Open Notebook 1 in Colab

text

https://colab.research.google.com/github/
jeffprosise/Machine-Learning/blob/master/
Titanic%20(Logistic%20Regression).ipynb

Join the three lines into one address, or use the Open in Colab link on the lesson page.

python

import pandas as pd

url = ("https://raw.githubusercontent.com/"
       "jeffprosise/Machine-Learning/master/Data/titanic.csv")
df = pd.read_csv(url)      # 891 rows, 12 columns

Choose the Features and Encode

python

df = df[['Survived', 'Age', 'Sex', 'Pclass']]
df = pd.get_dummies(df, columns=['Sex', 'Pclass'])
df.dropna(inplace=True)      # 714 rows remain

Keep the target Survived and three features: Age, Sex, Pclass
get_dummies one-hot encodes: one new 0/1 column per value, because a model needs numbers
dropna removes the 177 rows with no age: 714 remain, 424 perished and 290 survived

Worked Example: Encode Two Passengers

Sex	Sex_female	Sex_male
male (row 0)	0	1
female (row 1)	1	0

Pclass	Pclass_1	Pclass_2	Pclass_3
3 (row 0)	0	0	1
1 (row 1)	1	0	0

Split, Train, Score

python

x = df.drop('Survived', axis=1)
y = df['Survived']
x_train, x_test, y_train, y_test = train_test_split(
    x, y, test_size=0.2, stratify=y, random_state=0)

model = LogisticRegression(random_state=0)
model.fit(x_train, y_train)
model.score(x_test, y_test)          # 0.8322

571 rows to train, 143 to test; stratify=y keeps the survivor share at 0.406 in both
Accuracy 0.8322: 119 of 143 test passengers right

Part 3

Evaluating a Classifier

Which mistakes, and how many

The Confusion Matrix

Confusion matrix of the notebook model on 143 test passengers: 78 and 7 in the perished row, 17 and 41 in the survived row

Rows: the true class. Columns: the prediction
The diagonal, 78 and 41, is correct
7 perished but were called survivors (FP)
17 survived but were called perished (FN)

python

from sklearn.metrics import ConfusionMatrixDisplay
ConfusionMatrixDisplay.from_estimator(
    model, x_test, y_test,
    display_labels=['Perished', 'Survived'],
    cmap='Blues')

Worked Example: Accuracy

A: accuracy, N: all test passengers, TN and TP: the correct cells

	Predicted 0	Predicted 1
Actual 0	78 (TN)	7 (FP)
Actual 1	17 (FN)	41 (TP)

A = TN + TPN = 78 + 41143 = 0.832

Worked Example: Precision of Class 1

	Predicted 0	Predicted 1
Actual 0	78 (TN)	7 (FP)
Actual 1	17 (FN)	41 (TP)

P = TPTP + FP = 4141 + 7 = 0.854

Worked Example: Recall of Class 1

	Predicted 0	Predicted 1
Actual 0	78 (TN)	7 (FP)
Actual 1	17 (FN)	41 (TP)

R = TPTP + FN = 4141 + 17 = 0.707

Worked Example: The F1-Score

P = 0.854 and R = 0.707 from the last two slides
F1 combines them: it is high only when both are high

F₁ = 2 · P · RP + R = 2 · 0.854 · 0.7070.854 + 0.707 = 0.774

The Classification Report

text

              precision    recall  f1-score   support

           0       0.82      0.92      0.87        85
           1       0.85      0.71      0.77        58

    accuracy                           0.83       143
   macro avg       0.84      0.81      0.82       143
weighted avg       0.83      0.83      0.83       143

print(classification_report(y_test, y_predicted))

Row 1 holds our three values, rounded to two decimals
Row 0 treats perished as positive: precision 78/95 = 0.821, recall 78/85 = 0.918
support counts the test passengers of each class; notebook 2 reads the weighted avg F1

The ROC Curve

ROC curve of the notebook model on the test set, area under the curve 0.88, above the dashed diagonal

One point per possible threshold
Up: the share of survivors found (recall)
Right: the share of non-survivors wrongly called survivors
Dashed diagonal: guessing. Area under the curve 0.88; 1.0 is perfect

python

from sklearn.metrics import RocCurveDisplay
RocCurveDisplay.from_estimator(
    model, x_test, y_test)

Moving the Threshold

Threshold	TN, FP, FN, TP	Precision	Recall
0.3	68, 17, 7, 51	0.750	0.879
0.5	78, 7, 17, 41	0.854	0.707
0.7	83, 2, 30, 28	0.933	0.483

Try It: Move the Threshold

Part 4

Notebook 2: Explore, Fill In, Tune

The same 891 passengers, cleaned more carefully

Open Notebook 2 in Colab

text

https://colab.research.google.com/github/
tirthajyoti/Machine-Learning-with-Python/blob/master/
Classification/Logistic_Regression_Classification.ipynb

Join the three lines into one address, or use the link on the lesson page.

python

url = ("https://raw.githubusercontent.com/tirthajyoti/"
       "Machine-Learning-with-Python/master/Datasets/titanic_train.csv")
train = pd.read_csv(url)      # the same 891 rows

Explore Before You Model

Count plot of Survived by sex: 468 men and 81 women perished, 109 men and 233 women survived

Fraction of passengers who survived by class: 0.630 in first, 0.473 in second, 0.242 in third

Women: 233 of 314 survived (0.742). Men: 109 of 577 (0.189)
Survival falls with class: 0.630, 0.473, 0.242
These are the columns that got the largest weights in the model

Fill In Missing Ages by Class

python

a = list(train.groupby('Pclass')['Age'].mean())
def impute_age(cols):
    Age = cols['Age']
    Pclass = cols['Pclass']
    if pd.isnull(Age):
        if Pclass == 1:
            return a[0]
        elif Pclass == 2:
            return a[1]
        else:
            return a[2]
    else:
        return Age
train['Age'] = train[['Age', 'Pclass']].apply(impute_age, axis=1)

Worked Example: One Filled Age

Pclass	Mean age	Missing ages filled
1	38.23	30
2	29.88	11
3	25.14	136

Drop, Then Encode

python

train.drop('Cabin', axis=1, inplace=True)
train.dropna(inplace=True)                 # 889 rows
train.drop(['PassengerId', 'Name', 'Ticket'], axis=1, inplace=True)

sex = pd.get_dummies(train['Sex'], drop_first=True)
embark = pd.get_dummies(train['Embarked'], drop_first=True)
train.drop(['Sex', 'Embarked'], axis=1, inplace=True)
train = pd.concat([train, sex, embark], axis=1)

Cabin is missing in 687 rows: drop the column
Identifiers say nothing general about survival: not features
drop_first=True: Sex keeps only male; Embarked keeps Q and S, and port C is Q = 0, S = 0
Result: target Survived and 8 features

How F1 Depends on the Settings

A few hundredths between two models can come from the split alone

F1-score against the penalty 1/C on a log scale: 0.63 at a penalty of 1000, rising to between 0.83 and 0.85 near a penalty of 5

F1-score against the random seed of the split, from 101 to 198: values jump between 0.76 and 0.86

Left: small C is a strong penalty on the weights; at 1/C = 1000, F1 drops to 0.63
Right: only the random seed of the split changes, and F1 moves between 0.76 and 0.86

Running Notebook 2 Today

pandas 3.0.6 and scikit-learn 1.9.1

In the notebook	What happens now	Fix
`cols[0]`	`KeyError: 0`	`cols['Age']`
`n_jobs=4`	`FutureWarning`: no effect	remove it
`max_iter=100`	`ConvergenceWarning` in 183 of 200 fits	`max_iter=1000`
`l[len(l)-2]`	F1 is text: the axis is scrambled	`float(...)`

Before you practise

Common Mistakes

Reading the data file from a folder that does not exist in Colab
Feeding a text column such as Sex to the model without encoding it
Keeping identifiers such as PassengerId or Name as features
Forgetting that predict_proba returns two columns: [:, 1] is class 1
Judging a classifier by accuracy alone
Crowning a model on a difference that another split could erase

Your team project

Project Milestone: Features and Target

Name your target variable: a class, like Survived, or a number, like a price
List candidate features; drop identifier columns
Count missing values with df.info(): drop the column, drop the rows, or impute
One-hot encode every text column with pd.get_dummies
Write down the final feature list and the shape of your table

Key Takeaways

Logistic regression: a score z, the sigmoid, then a threshold
σ(0) = 0.5, so at threshold 0.5 the class is 1 exactly when z ≥ 0
Models need numbers: encode text columns, drop or impute missing values
Evaluate on test data: confusion matrix, accuracy, precision, recall, F1
Moving the threshold trades precision against recall
One split can mislead: cross-validate

Part 5

Practice: Your Turn

About 30 minutes, answers follow each task

About 10 minutes

Practice 1: Two Passengers by Hand

Weights: b = 1.212, Age -0.039, Sex_female 1.149, Sex_male -1.150, Pclass_1 1.200, Pclass_2 0.007, Pclass_3 -1.207
Passenger 1: an 8-year-old boy in second class
Passenger 2: a 30-year-old man in first class
For each: the feature vector, then z, p and the class at threshold 0.5

Answers

Practice 1: Answers

Passenger	Vector `x`	Score `z`	`p` and class
boy, 8, class 2	`[8, 0, 1, 0, 1, 0]`	1.212 - 0.312 - 1.150 + 0.007 = -0.243	1 / (1 + 1.2751) = 0.440, class `0`
man, 30, class 1	`[30, 0, 1, 1, 0, 0]`	1.212 - 1.170 - 1.150 + 1.200 = 0.092	1 / (1 + 0.9121) = 0.523, class `1`

About 5 minutes

Practice 2: The Matrix at Threshold 0.3

	Predicted 0	Predicted 1
Actual 0	68	17
Actual 1	7	51

A = 119143 = 0.832

P = 5168 = 0.750

R = 5158 = 0.879

About 5 minutes

Practice 3: Impute and Encode by Hand

Passenger	Pclass	Age	Embarked
A	1	40	C
B	1	missing	S
C	1	50	S
D	2	30	S

Passenger	Pclass	Age	Embarked
E	2	missing	Q
F	3	20	S
G	3	missing	S
H	3	28	Q

About 10 minutes

Practice 4: In Colab

Run notebook 1 up to model.score, reading the data by URL
Print the probability of survival for [[8, 0, 1, 0, 1, 0]]
Predict the test set with threshold 0.4 and print the confusion matrix and report
Which of precision and recall of class 1 went up?

Question	Answer
Probability for the boy	43.9%, class `0`
Matrix at 0.4	`[[75, 10], [10, 48]]`
Recall of class `1`	up, 0.71 to 0.83
Precision of class `1`	down, 0.85 to 0.83

Open this lesson

Mahmoud AbasLogistic Regression