Logistic Regression

This section introduces the first classifier of the course: logistic regression. Linear regression in week 3 predicted a number, a house price. Logistic regression predicts one of two classes, here whether a Titanic passenger survived or not, and it also gives the probability of that answer. You will first follow the model by hand for real passengers, then prepare the data, train, score and tune the model with the two notebooks of week 4.

Objectives

By the end of the section you should be able to:

• Explain how logistic regression turns the features of a passenger into a score z, a probability p and a class.
• Compute the sigmoid, a probability and a prediction by hand from the weights of a trained model.
• Prepare the Titanic data: pick the features and the target, one-hot encode text columns, and handle missing ages.
• Train LogisticRegression, then read the accuracy, the confusion matrix, precision, recall and the F1-score.
• Move the decision threshold and explain the trade between precision and recall.
• Choose the features and the target variable of your own team project.

Where this sits in the course

Week 4 of the plan has three parts:

• Two notebooks. Titanic (Logistic Regression) builds a compact model on four columns. Logistic_Regression_Classification explores the same data with plots, fills in missing ages, and studies how the F1-score changes with the settings.
• A real-world scenario. A binary-classification model that predicts whether a passenger will survive or not, using the Titanic dataset.
• A project milestone. Project Data Features Extraction and Target Variable Identification: this week your team decides which columns are the features and which column is the target.

Both notebooks use the same file: 891 passengers, one row each, with a Survived column that holds 1 for survived and 0 for perished.

Part 1: The idea of logistic regression

What logistic regression is

Logistic regression is a learning algorithm for binary classification: the target has exactly two values, 0 and 1. In the words of the notebook, logistic regression not only makes predictions, it yields probabilities as well. For every passenger it returns a probability of survival between 0 and 1, and the class follows from that probability.

The vocabulary, on Titanic

The model in three steps

1. Score. Multiply every feature by its weight and add a bias. The result is one number, z.
2. Squash. Pass z through the sigmoid function. The result p is always between 0 and 1.
3. Decide. Compare p with the threshold. LogisticRegression.predict uses 0.5.

During fit, the model learns the weights and the bias from the training rows. At prediction time it only runs these three steps.

Step 1: the linear score

z = b + w1·x1 + w2·x2 + ... + wn·xn

• x1 to xn are the n features of one passenger.
• w1 to wn are the weights the model learned, one per feature. scikit-learn stores them in model.coef_.
• b is the bias, stored in model.intercept_.

A positive weight pushes z up, towards survival. A negative weight pushes it down. This is the same weighted sum as linear regression; the next step is what makes it a classifier.

Step 2: the sigmoid

p = σ(z) = 1 / (1 + e^(-z))

e is the constant 2.71828. The sigmoid has three properties you should remember:

• For any z, the value σ(z) lies between 0 and 1, so it can be read as a probability.
• σ(0) = 0.5: a score of zero means "no preference".
• A large positive z gives p close to 1, a large negative z gives p close to 0.

Worked example: three values of the sigmoid

Compute σ(0), σ(2) and σ(-2).

σ(0)  = 1 / (1 + e^0)  = 1 / (1 + 1)      = 0.5
σ(2)  = 1 / (1 + e^-2) = 1 / (1 + 0.1353) = 0.8808
σ(-2) = 1 / (1 + e^2)  = 1 / (1 + 7.3891) = 0.1192

Notice that σ(2) + σ(-2) = 1: the curve is symmetric around the point (0, 0.5).

Step 3: the threshold

With the default threshold 0.5, the prediction is 1 when p ≥ 0.5 and 0 otherwise. Because σ(0) = 0.5, that is the same as asking whether z ≥ 0. We checked this on the notebook's model: model.predict gives exactly the same classes as predict_proba ≥ 0.5 for all 143 test passengers.

To play with the three steps, open the sigmoid explorer full screen. Choose a passenger, press Play to walk the age from 0 to 80, and switch the horizontal axis to z to see the whole S-shaped curve.

Worked example: one passenger, by hand

The notebook trains LogisticRegression(random_state=0) on six features: Age, Sex_female, Sex_male, Pclass_1, Pclass_2 and Pclass_3 (Part 2 shows how these columns are made). We printed the learned weights and rounded them to three decimals:

Reading the signs: being female and being in first class push the score up, being male and being in third class push it down, and every extra year of age lowers the score by 0.039.

The notebook asks: will a 30-year-old female traveling in first class survive?

Step 1: the feature vector

The notebook writes this passenger as female = [[30, 1, 0, 1, 0, 0]], in the column order Age, Sex_female, Sex_male, Pclass_1, Pclass_2, Pclass_3. Age is 30, the female column is 1, the first-class column is 1, and every other column is 0.

Step 2: the score

Only the non-zero features contribute:

z = 1.212 + (-0.039)(30) + 1.149(1) + 1.200(1)
  = 1.212 - 1.170 + 1.149 + 1.200
  = 2.391

Step 3: the probability

p = 1 / (1 + e^-2.391) = 1 / (1 + 0.0915) = 0.916

The notebook prints Probability of survival: 91.6% for this passenger, the same value.

Step 4: the class

p = 0.916 ≥ 0.5, so the prediction is 1, survived. The notebook's model.predict(female)[0] returns 1.

Worked example: a 60-year-old male in third class

The same steps for male = [[60, 0, 1, 0, 0, 1]]:

z = 1.212 + (-0.039)(60) + (-1.150) + (-1.207) = -3.485
p = 1 / (1 + e^3.485) = 1 / (1 + 32.622) = 0.030

The prediction is 0, perished. The notebook prints Probability of survival: 2.9%. The small difference comes from rounding the age weight: its full value is -0.03915, and 60 years multiply the rounding error. With the unrounded weights, z = -3.494 and p = 0.029.

Part 2: Notebook 1, a compact Titanic model

Open the notebook

Open Titanic (Logistic Regression) in Colab

The first cell reads Data/titanic.csv, a folder that does not exist when the notebook opens in Colab. Read the file from the repository instead:

import pandas as pd
 
url = ("https://raw.githubusercontent.com/"
       "jeffprosise/Machine-Learning/master/Data/titanic.csv")
df = pd.read_csv(url)
df.head()

Look for missing values

df.info() reports 891 rows and 12 columns. Three columns have missing values:

Choose the features, encode, and drop missing rows

df = df[['Survived', 'Age', 'Sex', 'Pclass']]
df = pd.get_dummies(df, columns=['Sex', 'Pclass'])
df.dropna(inplace=True)
df.head()

• The first line keeps the target Survived and three features: Age, Sex and Pclass.
• get_dummies one-hot encodes a column: it replaces it with one new column per value, holding 1 where the row has that value and 0 elsewhere. Sex becomes Sex_female and Sex_male; Pclass becomes Pclass_1, Pclass_2 and Pclass_3. A model needs numbers, and the text male is not a number.
• dropna removes the 177 rows with no age, which leaves 714 rows: 424 perished and 290 survived.

Why one-hot encode Pclass, which is already a number? Class 2 is not "twice" class 1. With three separate columns, the model learns a separate weight for each class.

On current pandas the new columns hold True and False instead of the 1 and 0 in the notebook's saved output. scikit-learn reads them as 1 and 0, so nothing else changes.

Worked example: encode two passengers

Each passenger gets exactly one 1 among the sex columns and exactly one 1 among the class columns.

Split with stratification

from sklearn.model_selection import train_test_split
 
x = df.drop('Survived', axis=1)
y = df['Survived']
 
x_train, x_test, y_train, y_test = train_test_split(
    x, y, test_size=0.2, stratify=y, random_state=0)

• x holds the six feature columns, y the target.
• test_size=0.2 keeps 143 of the 714 rows for testing and trains on 571.
• stratify=y keeps the share of survivors the same in both parts. In all 714 rows, 290 survived, a share of 0.406. The training set has 232 survivors out of 571 (0.406) and the test set 58 out of 143 (0.406).

Train and score

from sklearn.linear_model import LogisticRegression
 
model = LogisticRegression(random_state=0)
model.fit(x_train, y_train)
model.score(x_test, y_test)

model.score returns the accuracy on the test set: 0.8322, which is 119 of 143 passengers classified correctly.

Cross-validation

One split can be lucky or unlucky. The notebook also runs five-fold cross-validation: it cuts the 714 rows into five parts, trains five models, each time testing on a different part, and averages the five accuracies.

from sklearn.model_selection import cross_val_score
 
cross_val_score(model, x, y, cv=5).mean()

The five accuracies are 0.748, 0.832, 0.783, 0.755 and 0.810, and their mean is 0.7857. It is lower than 0.8322: our single test split happened to be one of the easier ones.

Predict new passengers

female = [[30, 1, 0, 1, 0, 0]]
model.predict(female)[0]                  # 1
 
probability = model.predict_proba(female)[0][1]
print(f'Probability of survival: {probability:.1%}')   # 91.6%

predict_proba returns two numbers per row: the probability of class 0 and of class 1. [0][1] takes the first row, second number: the probability of survival. The two numbers of a row always add up to 1.

A plain list has no column names, so current scikit-learn prints a warning, X does not have valid feature names. The result is still correct. To silence it, pass a DataFrame with the same columns as x.

Part 3: Evaluating a classifier

The confusion matrix

from sklearn.metrics import confusion_matrix
 
y_predicted = model.predict(x_test)
confusion_matrix(y_test, y_predicted)

Rows are the true class and columns the prediction, so the diagonal holds the correct predictions:

• 78 perished and were predicted to perish.
• 41 survived and were predicted to survive.
• 7 perished but were predicted to survive.
• 17 survived but were predicted to perish.

The notebook draws this picture with plot_confusion_matrix. That function was removed from scikit-learn; with version 1.9.1 the import fails with ImportError. Use the class that replaced it:

from sklearn.metrics import ConfusionMatrixDisplay
 
ConfusionMatrixDisplay.from_estimator(
    model, x_test, y_test,
    display_labels=['Perished', 'Survived'], cmap='Blues')

Worked example: four measures from one matrix

Take class 1, survived, as the positive class.

Accuracy, the share of all passengers classified correctly:

accuracy = (78 + 41) / 143 = 119 / 143 = 0.832

Precision of class 1: of the passengers the model called survivors, the share who really survived.

precision = 41 / (41 + 7) = 41 / 48 = 0.854

Recall of class 1: of the passengers who really survived, the share the model found.

recall = 41 / (41 + 17) = 41 / 58 = 0.707

F1-score, one number that is high only when precision and recall are both high:

F1 = 2 · precision · recall / (precision + recall)
   = 2 · 0.854 · 0.707 / (0.854 + 0.707) = 0.774

The model is careful when it says "survived" (precision 0.854) but it misses 17 of the 58 survivors (recall 0.707).

The classification report

from sklearn.metrics import classification_report
 
print(classification_report(y_test, y_predicted))

              precision    recall  f1-score   support
 
           0       0.82      0.92      0.87        85
           1       0.85      0.71      0.77        58
 
    accuracy                           0.83       143
   macro avg       0.84      0.81      0.82       143
weighted avg       0.83      0.83      0.83       143

Row 1 holds the values we just computed, rounded to two decimals. Row 0 does the same with perished as the positive class: precision 78 / (78 + 17) = 0.821 and recall 78 / (78 + 7) = 0.918. support is the number of test passengers in each class. The weighted avg row averages the two classes weighted by their support; notebook 2 uses its F1 value, 0.83 here.

The ROC curve

The notebook ends its evaluation with a ROC curve. plot_roc_curve was removed too; the current form is:

from sklearn.metrics import RocCurveDisplay
 
RocCurveDisplay.from_estimator(model, x_test, y_test)

The curve shows, for every possible threshold, the share of survivors found (true positive rate, which is recall) against the share of non-survivors wrongly called survivors (false positive rate). The dashed diagonal is a model that guesses. The area under the curve, AUC = 0.88, summarises the whole curve: 1.0 would be perfect and 0.5 is guessing.

Moving the threshold

predict always uses 0.5, but you can choose any threshold from the probabilities:

p = model.predict_proba(x_test)[:, 1]
y_pred = (p >= 0.3).astype(int)
confusion_matrix(y_test, y_pred)

We ran this for three thresholds on the same 143 test passengers:

TN, FP, FN and TP are the four cells of the matrix in reading order: true negatives, false positives, false negatives, true positives.

• A lower threshold calls more passengers survivors. It finds more of the real survivors (recall up) but makes more false alarms (precision down).
• A higher threshold is stricter. When it says "survived" it is nearly always right (precision up), but it misses many survivors (recall down).
• Which threshold is right depends on which mistake costs more in your problem.

Open the threshold explorer full screen. Each row is one sex and class group, each dot a test passenger, and the shaded part of a row is where the model predicts survival. Move the threshold and watch the boundary in each group, the confusion matrix, precision and recall change together.

Part 4: Notebook 2, exploring and cleaning the same data

Open the notebook

Open Logistic_Regression_Classification in Colab

The notebook reads titanic_train.csv from its own folder, but in the repository the file lives in the Datasets folder. In Colab read it from there:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
 
url = ("https://raw.githubusercontent.com/tirthajyoti/"
       "Machine-Learning-with-Python/master/Datasets/titanic_train.csv")
train = pd.read_csv(url)

It is the same 891-row file as in notebook 1.

Explore before you model

The notebook draws a count plot of Survived split by Sex:

Of 314 women, 233 survived (0.742). Of 577 men, 109 survived (0.189). This is why the Sex columns received large weights in the model.

It then plots the fraction of passengers who survived in each class, with train.groupby('Pclass')['Survived'].mean():

The fraction falls from 0.630 in first class to 0.473 in second and 0.242 in third.

Fill in missing ages instead of dropping them

Notebook 1 dropped the 177 passengers with no age. Notebook 2 keeps them and imputes each missing age with the average age of that passenger's class, because the boxplot of age by class shows that the classes have different ages.

f_class_Age = pd.DataFrame(train.groupby('Pclass')['Age'].mean())
a = list(f_class_Age['Age'])
 
def impute_age(cols):
    Age = cols['Age']
    Pclass = cols['Pclass']
    if pd.isnull(Age):
        if Pclass == 1:
            return a[0]
        elif Pclass == 2:
            return a[1]
        else:
            return a[2]
    else:
        return Age
 
train['Age'] = train[['Age', 'Pclass']].apply(impute_age, axis=1)

The notebook writes Age = cols[0] and Pclass = cols[1]. On pandas 3.0 this stops with KeyError: 0, because cols is a row whose labels are Age and Pclass. Reading by name, as above, works on every version.

Worked example: one filled age

Row 5 of the file is a third-class passenger with no age. impute_age receives Age = NaN and Pclass = 3, so it returns a[2], 25.14. After the fill, Age has 891 values, and its mean moves from 29.699 to 29.293.

Drop, then encode

train.drop('Cabin', axis=1, inplace=True)
train.dropna(inplace=True)
train.drop(['PassengerId', 'Name', 'Ticket'], axis=1, inplace=True)
 
sex = pd.get_dummies(train['Sex'], drop_first=True)
embark = pd.get_dummies(train['Embarked'], drop_first=True)
train.drop(['Sex', 'Embarked'], axis=1, inplace=True)
train = pd.concat([train, sex, embark], axis=1)

• Cabin is missing in 687 rows, so the whole column goes.
• dropna then removes the 2 rows with no Embarked, leaving 889 rows.
• PassengerId, Name and Ticket identify a passenger but say nothing general about survival, so they are not features.
• drop_first=True drops the first dummy column. Sex keeps only male; female is simply male = 0. Embarked (ports C, Q, S) keeps Q and S; port C is Q = 0 and S = 0.

The final table has the target Survived and eight features: Pclass, Age, SibSp, Parch, Fare, male, Q and S.

Train and read the F1-score

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
 
X_train, X_test, y_train, y_test = train_test_split(
    train.drop('Survived', axis=1), train['Survived'],
    test_size=0.30, random_state=111)
 
logmodel = LogisticRegression(C=1, max_iter=1000)
logmodel.fit(X_train, y_train)

The split keeps 267 test passengers. This model scores an accuracy of 0.835, with the confusion matrix [[145, 18], [26, 78]].

How the F1-score depends on the settings

The rest of the notebook runs three loops. Each trains many models and records the weighted avg F1-score of the classification report.

The penalty parameter. C controls regularization: the smaller C, the harder the model is pushed towards small weights. The notebook tries C from 0.001 to 0.2 and plots against the penalty 1/C.

A very strong penalty (1/C = 1000) keeps the weights so small that the F1-score drops to 0.63. As the penalty falls towards 5 (C = 0.2), the F1-score climbs to between 0.83 and 0.85.

The test size. Changing the test fraction from 0.10 to about 0.79 moves the F1-score between 0.80 and 0.87.

The random seed of the split. Changing only random_state from 101 to 198, with everything else fixed, moves the F1-score between 0.76 and 0.86.

The last plot carries the key lesson: a difference of a few hundredths between two models can come from the split alone. That is why notebook 1 also reports a cross-validated score.

Running notebook 2 on current libraries

We ran the notebook with pandas 3.0.6 and scikit-learn 1.9.1. Four things need attention:

The last row matters most. With text values, matplotlib treats every F1 value as a category and spaces them evenly in the order it first meets them. On the seed plot the vertical axis then reads 0.83, 0.77, 0.81, 0.80 and so on from the bottom up, which makes the plot meaningless. Converting to float fixes it; the figures above were drawn that way. Raising max_iter changes the F1-scores of the C loop by at most 0.02 (at C = 0.076, from 0.83 to 0.81), so the trend in the plot stays the same.

Common mistakes

• Reading the notebook's data file from a folder that does not exist in Colab.
• Feeding a text column such as Sex to the model without encoding it.
• Keeping identifier columns such as PassengerId or Name as features.
• Forgetting that predict_proba returns two columns: [:, 1] is the probability of class 1.
• Judging a classifier by accuracy alone; look at precision and recall of the class you care about.
• Crowning a model because of a difference that a different random split could erase.

Project milestone: features and target

This week's milestone is Project Data Features Extraction and Target Variable Identification. Using what both notebooks did to the Titanic data:

1. Name your target variable: the one column your model will predict. Say whether it is a class, as Survived is, or a number, as a house price is.
2. List candidate features and drop identifier columns, the way the notebooks dropped PassengerId, Name and Ticket.
3. Count missing values per column with df.info(), then decide per column: drop the column, drop the rows, or impute.
4. One-hot encode every text column with pd.get_dummies.
5. Write down the final feature list and the shape of your table.

Keep this table: next week you will train your first model on it.

Key takeaways

1. Logistic regression computes a score z = b + w1·x1 + ... + wn·xn, turns it into a probability with the sigmoid, and cuts the probability at a threshold.
2. σ(0) = 0.5, so with threshold 0.5 the prediction is 1 exactly when z ≥ 0.
3. Models need numeric features: one-hot encode text columns, and drop or impute missing values.
4. Evaluate a classifier with the confusion matrix, accuracy, precision, recall and F1, on test data.
5. Moving the threshold trades precision against recall; the right trade depends on the problem.
6. One split can mislead; cross-validation and repeated splits show how much a score can move.

Practice

About 30 minutes. Try each task before you read its answer at the end of the page.

Practice 1: two passengers by hand (about 10 minutes)

Use the rounded weights from the worked example: b = 1.212, Age -0.039, Sex_female 1.149, Sex_male -1.150, Pclass_1 1.200, Pclass_2 0.007, Pclass_3 -1.207. For each passenger, write the feature vector, then compute z, p and the prediction at threshold 0.5.

1. An 8-year-old boy traveling in second class.
2. A 30-year-old man traveling in first class.

Practice 2: read a confusion matrix (about 5 minutes)

With the threshold lowered to 0.3, the notebook 1 model gives this matrix on the 143 test passengers:

Compute the accuracy, and the precision, recall and F1-score of class 1. Compare them with the values at threshold 0.5.

Practice 3: impute and encode by hand (about 5 minutes)

Eight passengers, two of them with no age:

1. Fill each missing age with the mean age of the passenger's class, as impute_age does.
2. Encode Embarked with pd.get_dummies(..., drop_first=True). Which columns appear, and what are their values for A, E and H?

Practice 4: in Colab (about 10 minutes)

1. Run notebook 1 up to model.score(x_test, y_test), reading the data by URL.
2. Print the probability of survival for the boy of Practice 1, [[8, 0, 1, 0, 1, 0]].
3. Predict the test set with threshold 0.4 instead of 0.5, and print the confusion matrix and classification_report.
4. Did precision or recall of class 1 go up? Which one went down?

Answers

Answer 1

The boy. The feature vector is [8, 0, 1, 0, 1, 0].

z = 1.212 + (-0.039)(8) + (-1.150) + 0.007
  = 1.212 - 0.312 - 1.150 + 0.007 = -0.243
p = 1 / (1 + e^0.243) = 1 / (1 + 1.2751) = 0.440

p is below 0.5, so the prediction is 0, perished. The trained model, with unrounded weights, gives 0.4394.

The man. The feature vector is [30, 0, 1, 1, 0, 0].

z = 1.212 + (-0.039)(30) + (-1.150) + 1.200
  = 1.212 - 1.170 - 1.150 + 1.200 = 0.092
p = 1 / (1 + e^-0.092) = 1 / (1 + 0.9121) = 0.523

p ≥ 0.5, so the prediction is 1, survived. The trained model gives 0.5218. Both passengers sit close to the threshold, so a small change to the model or to the threshold could flip them.

Answer 2

accuracy  = (68 + 51) / 143 = 0.832
precision = 51 / (51 + 17) = 0.750
recall    = 51 / (51 + 7)  = 0.879
F1        = 2 · 0.7500 · 0.8793 / (0.7500 + 0.8793) = 0.8095

The F1-score rounds to 0.810, higher than 0.774 at threshold 0.5. Compared with threshold 0.5 (precision 0.854, recall 0.707), the lower threshold finds 10 more survivors, so recall rises, and makes 10 more false alarms, so precision falls. The accuracy is the same, 0.832, because the number of mistakes is 24 in both cases: here 17 plus 7, at 0.5 it was 7 plus 17.

Answer 3

The class means use only the known ages:

• Class 1: (40 + 50) / 2 = 45, so B gets 45.
• Class 2: only D is known, so E gets 30.
• Class 3: (20 + 28) / 2 = 24, so G gets 24.

Embarked has the values C, Q and S. With drop_first=True the first one, C, is dropped, so the columns are Q and S:

The other five passengers embarked at S, so they have Q = 0 and S = 1.

Answer 4

• The probability of survival for the boy is 43.9%, so predict returns 0.
• With y_pred = (model.predict_proba(x_test)[:, 1] >= 0.4).astype(int):

[[75 10]
 [10 48]]
 
              precision    recall  f1-score   support
 
           0       0.88      0.88      0.88        85
           1       0.83      0.83      0.83        58
 
    accuracy                           0.86       143
   macro avg       0.85      0.85      0.85       143
weighted avg       0.86      0.86      0.86       143

• Recall of class 1 went up, from 0.71 to 0.83: the model now finds 48 of the 58 survivors instead of 41. Precision went down, from 0.85 to 0.83. On this test set the accuracy also rose, from 0.832 to 0.860, because the number of mistakes fell from 24 to 20.