Logo
Machine Learning (2026-2027) - Decision Trees and Random Forest

In the print dialog, choose "Save as PDF" as the destination.

Machine Learning, Week 6

Decision Trees and Random Forest

Classify a borrower by asking questions, then let many trees vote: the Lending Club loans.

Objectives

  • Explain how a decision tree classifies: a chain of threshold questions that ends in a leaf
  • Compute the Gini impurity of a node and the weighted Gini of a split by hand
  • Explain why a tree grown until every leaf is pure overfits
  • Explain a random forest: bootstrap samples, random features at each split, a vote
  • Train DecisionTreeClassifier and RandomForestClassifier on the Lending Club data
  • Spot when a high accuracy only reflects an unbalanced target, and apply it to your project

Week 6 of the plan

Where This Sits in the Course

  • One notebook: DecisionTrees_RandomForest_Classification, a tree and a forest on public lending data
  • Real-world scenario: Lending Club connects borrowers with investors; predict whether a borrower paid back the loan in full
  • Project milestone: Machine Learning Model Implementation, continued on your own data

Plan for the Two Hours

PartWhat we doTime
1The idea of a tree, and a tree grown by hand30 min
2The Lending Club notebook: data and exploration15 min
3One decision tree on the loans10 min
4Random forest: by hand, then on the loans25 min
5Mistakes, project, practice with answers, takeaways40 min

Part 1

The Idea of a Decision Tree

Ask questions until the answer is clear

What Is a Decision Tree?

Decision tree
A classifier that asks a chain of yes or no questions, each comparing one feature with a threshold, such as fico ≤ 677.
  • Yes sends the sample down the left branch, no down the right one
  • The next question waits at the end of each branch
  • A node with no question is a leaf: its class is the prediction

The Vocabulary, on the Loans

TermIn the Lending Club data
Features Xcredit data, such as fico (credit score) and int.rate (interest rate)
Label ynot.fully.paid: 1 did not pay back in full, 0 paid in full
Root nodethe first question, asked of every borrower
Leafa node with no question: it predicts its majority class
Depththe number of questions on the longest path to a leaf

How Mixed Is a Node? Gini Impurity

The notebook trains with criterion='gini'

G = 1 − (p02 + p12)
  • p_0 is the fraction of the node in class 0 (paid), p_1 in class 1 (not paid)
p0 = 1: G = 0 p0 = p1 = 0.5: G = 0.5

Choosing a Question: the Weighted Gini

Gw = nLn GL + nRn GR
  • A question splits n rows into n_L on the left and n_R on the right
  • G_L, G_R are the Gini of each side; each side counts by its size
  • Candidate thresholds are the midpoints between neighbouring values

By hand

Worked Example: Ten Real Borrowers

Rows of loan_data.csv, sorted by fico. y = 1: not paid in full. Four of ten did not pay.

Borrowerficoint.ratey
B16470.14821
B26620.15070
B36670.12991
B46720.13471
B56820.11030
Borrowerficoint.ratey
B66870.13870
B76920.12840
B87070.10911
B97270.10590
B107570.11890

Step 1: The Gini of the Root

All ten borrowers: 6 paid, 4 not paid

G = 1 − (0.62 + 0.42) = 1 − (0.36 + 0.16) = 0.48

Step 2: Try the Question fico ≤ 677

677 is the midpoint between B4 (672) and B5 (682)

SideBorrowersNot paidPaid
yes, ≤ 677B1 to B431
no, > 677B5 to B101 (B8)5

Step 3: The Gini of Each Side

GL = 1 − (34)2 − (14)2 = 1 − 0.5625 − 0.0625 = 0.375
GR = 1 − (16)2 − (56)2 = 1036 ≈ 0.2778

Step 4: The Weighted Gini of the Split

Gw = 410 × 0.375 + 610 × 0.2778 = 0.15 + 0.1667 = 0.3167
ΔG = 0.48 − 0.3167 = 0.1633

Step 5: The Other Feature

The best question on int.rate is int.rate ≤ 0.12915

SideBorrowersNot paidPaid
yes, lower ratesB5, B7, B8, B9, B1014
no, higher ratesB1, B2, B3, B4, B632
Gw = 510 × 0.32 + 510 × 0.48 = 0.40 > 0.3167

Step 6: The Search Tries Every Threshold

Nine fico midpoints and nine int.rate midpoints: eighteen candidates

fico thresholdLeft: not paid, paidRight: not paid, paidWeighted Gini
654.51, 03, 60.4000
669.52, 12, 50.4190
6773, 11, 50.3167
684.53, 21, 40.4000
7174, 40, 20.4000

Step 7: Grow the Left Child

B1 to B4: 3 not paid, 1 paid, G = 0.375

Gw = 34 × 0 + 14 × 0 = 0

Step 8: Grow the Right Child

B5 to B10: 1 not paid (B8), 5 paid, G = 0.2778

Gw = 26 × 0.5 + 46 × 0 ≈ 0.1667

The Tree We Built

text
fico <= 677 ?
|-- yes: int.rate <= 0.14945 ?
|        |-- yes: leaf [3 not paid, 0 paid] -> not paid
|        |-- no:  leaf [0 not paid, 1 paid] -> paid
|-- no:  int.rate <= 0.1097 ?
         |-- yes: leaf [1 not paid, 1 paid] -> paid
         |-- no:  leaf [0 not paid, 4 paid] -> paid

Predict Two New Borrowers

Follow the questions from the root

  • A: 670 ≤ 677 yes, then 0.14 ≤ 0.14945 yes, so the leaf says not paid
  • B: 700 ≤ 677 no, then 0.12 ≤ 0.1097 no, so the leaf says paid

Try It: Grow the Tree Yourself

Part 2

The Lending Club Notebook

9578 real loans, 2007 to 2010

Open the Notebook in Colab

text
https://colab.research.google.com/github/
tirthajyoti/Machine-Learning-with-Python/blob/
master/Classification/
DecisionTrees_RandomForest_Classification.ipynb
Join the lines into one address, or click "Open in Colab" on the lesson page.

Load and Inspect the Data

python
import pandas as pd

url = ("https://raw.githubusercontent.com/"
       "tirthajyoti/Machine-Learning-with-Python/"
       "master/Datasets/loan_data.csv")
df = pd.read_csv(url)
df.info()
  • 9578 rows, 14 columns, no missing values
  • fico: credit score; int.rate: interest rate as a proportion
  • purpose: the loan purpose, as text
  • Label not.fully.paid: 1 if not paid back in full

The Count That Matters

python
print(df['not.fully.paid'].value_counts())
not.fully.paidBorrowersShare
0, paid804584%
1, not paid153316%

Explore Before You Model

Histogram of FICO scores for borrowers who meet the credit policy (blue) and who do not (red): below 660 almost everyone fails it
Count of loans by purpose, paid in full (red) and not fully paid (blue); debt consolidation is the most common purpose
  • Below fico 660: 487 borrowers fail the credit policy, only 2 pass. One threshold separates a group
  • Not paid in full: 27.8% of small_business loans, 11.6% of credit_card loans

Turn the Text Column into Numbers

python
df_final = pd.get_dummies(df, ['purpose'],
                          drop_first=True)
  • Seven purposes become six 0 or 1 columns, such as purpose_credit_card
  • drop_first=True drops all_other: all six at 0 means that purpose
  • df_final has 19 columns

Split into Training and Test Sets

python
X = df_final.drop('not.fully.paid', axis=1)
y = df_final['not.fully.paid']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.30, random_state=101)

Part 3

One Decision Tree on the Loans

Let the tree grow until every leaf is pure

Fit the Tree and Predict

python
from sklearn.tree import DecisionTreeClassifier

dtree = DecisionTreeClassifier(criterion='gini', max_depth=None,
                               random_state=101)
dtree.fit(X_train, y_train)
predictions = dtree.predict(X_test)
  • criterion='gini': the impurity from Part 1
  • max_depth=None: keep splitting until every leaf is pure
  • random_state=101: ties between equally good splits break the same way every run

The Tree's Confusion Matrix

A: accuracy. R_1, P_1: recall and precision of class 1

Predicted 0 (paid)Predicted 1 (not paid)
Actual 01975456
Actual 1336107
A = 1975 + 1072874 ≈ 0.724

The Tree Memorised Its Training Set

python
print(dtree.get_depth(), dtree.get_n_leaves())
print(dtree.score(X_train, y_train))
MeasureValue
Depth27
Leaves1005
Training accuracy1.0

Part 4

Random Forest

Many different trees, one vote

The Idea: Many Different Trees

  1. Bootstrap: each tree gets n rows drawn at random with replacement from the n training rows
  2. Random features: at every split a tree searches only m of the p features, m = ⌊√p⌋
  3. Vote: every tree classifies the new sample; the majority class wins

By hand

Worked Example: One Bootstrap Sample

RandomForestClassifier(n_estimators=5, random_state=0) on the ten borrowers. With p = 2, every split searches m = 1 random feature

  • Tree 1 drew: B4, B3, B1, B1, B3, B7, B10, B8, B4, B10
  • Twice: B1, B3, B4, B10. Once: B7, B8
  • Never drawn: B2, B5, B6, B9, left out of this tree

Five Trees Vote on a New Borrower

New borrower: fico 710, int.rate 0.11

TreeLeft out of its sampleVote
1B2, B5, B6, B9not paid
2B3, B6, B7paid
3B1, B3, B7, B10paid
4B2, B3, B7, B8, B10paid
5B1, B3, B5, B9not paid

Try It: Watch the Forest Vote

Fit the Forest

python
from sklearn.ensemble import RandomForestClassifier

rfc = RandomForestClassifier(n_estimators=600, random_state=101)
rfc.fit(X_train, y_train)
rfc_pred = rfc.predict(X_test)
print(confusion_matrix(y_test, rfc_pred))
print(classification_report(y_test, rfc_pred))
With p = 18, each split searches m = 4 features: rfc.estimators_[0].max_features_ prints 4.

The Forest's Confusion Matrix

A: accuracy on the 2874 test borrowers

Predicted 0 (paid)Predicted 1 (not paid)
Actual 0242011
Actual 143112
A = 2420 + 122874 ≈ 0.846

Do You Get Anything Strange?

A_0: accuracy of "paid" for everyone. R_1: recall of class 1

A0 = 24312874 ≈ 0.846 R1 = 12443 ≈ 0.03
  • The forest flags only 23 borrowers and beats "paid for everyone" by one
  • No test borrower gets more than 375 of the 600 trees voting "not paid"

How Many Trees?

Test accuracy of forests of 5 to 100 trees: 0.824 with 5 trees, then between 0.8455 and 0.8473 from 30 trees on, next to the 0.8459 of answering paid for everyone
  • The notebook loops over 5 to 100 trees, min_samples_split=10; we add random_state=101 to each forest
  • 0.8239 with 5 trees, then 0.8455 to 0.8473 from 30 trees on
  • More trees steady the vote, but the curve settles at the level of "paid for everyone"

Before you practise

Common Mistakes

  • Reading loan_data.csv in Colab from a folder that does not have it
  • Forgetting random_state, then comparing numbers from two different splits
  • Trusting the perfect training score of a tree grown with max_depth=None
  • Judging a model by accuracy alone when one class is rare
  • Printing the report of predictions when you meant rfc_pred
  • Passing a text column such as purpose without dummy variables

Your team project

Project Milestone: Model Implementation

  1. Take last week's features and target; turn text columns into dummies
  2. Split with a fixed random_state
  3. Fit DecisionTreeClassifier: compare its training and test scores
  4. Fit RandomForestClassifier: report the confusion matrix and classification_report
  5. Compare tree, forest and KNN on the same test set, using the recall of every class

Part 5

Practice: Your Turn

About 30 minutes, answers follow each task

About 10 minutes

Practice 1: Choose the First Question

Eight other real borrowers, y = 1: not paid

Borrowerficoint.ratey
P16470.14821
P26670.13430
P36770.17341
P46820.11970
Borrowerficoint.ratey
P56920.12870
P66970.13161
P77420.11140
P87470.07510
  • Compute the Gini of the root
  • Compute G_w for fico ≤ 679.5 and for int.rate ≤ 0.13015
  • Which question wins, and by how much does the Gini fall?

Answers

Practice 1: Answer

G = 1 − (0.3752 + 0.6252) ≈ 0.4688
QuestionWeighted Gini
fico ≤ 679.5G_w = 0.375 × 0.4444 + 0.625 × 0.32 ≈ 0.3667
int.rate ≤ 0.13015G_w = 0.5 × 0 + 0.5 × 0.375 = 0.1875

About 5 minutes

Practice 2: Read the Forest's Matrix

The 600-tree forest: [[2420, 11], [431, 12]]

P1 = 1223 ≈ 0.52 R1 = 12443 ≈ 0.03

About 5 minutes

Practice 3: Entropy of the First Split

The notebook also trains with criterion='entropy'

H = −(p0 log2 p0 + p1 log2 p1)
HL ≈ 0.8113 HR ≈ 0.6500 Hw ≈ 0.7145

About 10 minutes

Practice 4: In Colab, on the Loans

  1. After the split with random_state=101, fit DecisionTreeClassifier(max_depth=5, random_state=101)
  2. Print its confusion matrix, its test and training accuracy; compare with max_depth=None
  3. Rerun the loop over the number of trees with max_depth=5 and random_state=101 in each forest
TreeConfusion matrixTestTraining
max_depth=5[[2396, 35], [431, 12]]0.83790.845
max_depth=None[[1975, 456], [336, 107]]0.7241.0

Key Takeaways

  1. A tree classifies by a chain of threshold questions; a leaf predicts its majority class
  2. Each question is the one with the lowest weighted Gini over every feature and threshold
  3. A tree grown until every leaf is pure memorises its training set
  4. A random forest votes over trees grown on bootstrap samples with random features
  5. When one class is rare, read its recall, not only the accuracy

Open this lesson

Mahmoud AbasDecision Trees and Random Forest