Mahmoud Abas, Full-Stack Software Engineer & AI

Machine Learning (2026-2027) - K-Nearest Neighbors Classification

In the print dialog, choose "Save as PDF" as the destination.

Machine Learning, Week 5

K-Nearest Neighbors Classification

Classify a new sample by asking its K closest neighbours, then build, score and tune the model with scikit-learn.

Objectives

Explain how KNN predicts a class: distance, the K nearest rows, a majority vote
Compute Euclidean distances and a KNN prediction by hand for K = 1, 3 and 5
Scale features with StandardScaler and say why distance needs it
Fit KNeighborsClassifier, then read a confusion matrix, accuracy, precision and recall
Choose K with the elbow method
Build and use a KNN model on the iris dataset, and apply it to your project

Week 5 of the plan

Where This Sits in the Course

Two notebooks: KNN_Classification (a 1000-row dataset with 10 anonymous features) and Nearest Neighbors (iris)
Real-world scenario: a simple supervised-learning model on the famous iris dataset, 150 samples of three species
Project milestone this week: Machine Learning Model Implementation

Plan for the Two Hours

Part	What we do	Time
1	The idea, and a worked example by hand	30 min
2	Notebook 1: scaling, fitting, evaluating	25 min
3	Choosing K with the elbow method	10 min
4	Notebook 2 on iris, mistakes, your project	20 min
5	Practice with answers, then takeaways	35 min

Part 1

The Idea of KNN

Similar inputs have similar labels

What Is K-Nearest Neighbors?

K-nearest neighbors (KNN): A simple supervised-learning algorithm that predicts an outcome by identifying the data points closest in proximity to the input data.

The Vocabulary, on Iris

Term	In the iris data
Features `X`	sepal length, sepal width, petal length, petal width (cm)
Label `y`	species: `0` setosa, `1` versicolor, `2` virginica
Training set	the flowers the model stores
Query	a new flower whose species we want
K	how many neighbours vote, `n_neighbors` in scikit-learn

The Algorithm in Four Steps

Store the training rows and their labels (this is all fit needs to do)
Compute the distance from the query to every training row
Sort the distances and keep the K smallest
Count the labels of those K rows: the majority is the prediction

Measuring Closeness: Euclidean Distance

d = √((x₁ − q₁)² + (x₂ − q₂)² + ... + (x_n − q_n)²)

x is a training row, q is the query, n is the number of features
Subtract feature by feature, square, add, take the square root
Sorting by the squared distance gives the same order, so by hand you can skip the root until the end

By hand

Worked Example: One New Flower

Nine real iris flowers, two features: petal length and petal width in cm. The new flower is (4.7, 1.7).

Flower	Species	(length, width)
F1	setosa	(1.4, 0.2)
F2	versicolor	(4.7, 1.6)
F3	versicolor	(5.0, 1.7)
F4	versicolor	(5.1, 1.6)
F5	versicolor	(4.9, 1.5)

Flower	Species	(length, width)
F6	virginica	(4.9, 1.8)
F7	virginica	(4.8, 1.8)
F8	virginica	(5.0, 1.5)
F9	virginica	(5.1, 1.8)

Step 1: Squared Distances, F1 to F5

Difference = flower minus query, for length and for width

Flower	Differences	Squared distance
F1	-3.30, -1.50	10.89 + 2.25 = 13.14
F2	0, -0.10	0 + 0.01 = 0.01
F3	+0.30, 0	0.09 + 0 = 0.09
F4	+0.40, -0.10	0.16 + 0.01 = 0.17
F5	+0.20, -0.20	0.04 + 0.04 = 0.08

Step 1: Squared Distances, F6 to F9

Flower	Differences	Squared distance
F6	+0.20, +0.10	0.04 + 0.01 = 0.05
F7	+0.10, +0.10	0.01 + 0.01 = 0.02
F8	+0.30, -0.20	0.09 + 0.04 = 0.13
F9	+0.40, +0.10	0.16 + 0.01 = 0.17

Step 2: Sort and Take the Roots

Rank	Flower	Species	Distance d
1	F2	versicolor	√0.01 = 0.100
2	F7	virginica	√0.02 = 0.141
3	F6	virginica	√0.05 = 0.224
4	F5	versicolor	√0.08 = 0.283
5	F3	versicolor	√0.09 = 0.300

Step 3: Vote with K = 1

K = 1: 1 versicolor ⇒ versicolor

Step 3: Vote with K = 3

K = 3: 2 virginica, 1 versicolor ⇒ virginica

Step 3: Vote with K = 5

K = 5: 3 versicolor, 2 virginica ⇒ versicolor

Try It: The Worked Example, Live

K Is a Choice, and Ties Happen

K is not learned from the data: you choose it, then check it on test data (Part 3)
A small K follows the single closest row, even if that row is unusual
A larger K listens to more rows, so one odd row matters less
With an even K a vote can tie, for example 1 to 1 with K = 2

Part 2

Notebook 1: The Classified Data

Scale, split, fit, evaluate

Open the Notebook in Colab

text

https://colab.research.google.com/github/
tirthajyoti/Machine-Learning-with-Python/blob/
master/Classification/KNN_Classification.ipynb

Join the three lines into one address, or click "Open in Colab" on the lesson page.

Load and Inspect the Data

python

import pandas as pd
import numpy as np

url = ("https://raw.githubusercontent.com/"
       "tirthajyoti/Machine-Learning-with-Python/"
       "master/Datasets/Classified%20Data")
df = pd.read_csv(url, index_col=0)
df.info()

1000 rows, 10 feature columns with made-up names (WTT, PTI, ...)
One label column, TARGET CLASS, with values 0 and 1
Balanced: 500 rows of each class
The notebook draws a boxplot per feature, split by class, to see which features separate the classes

Why KNN Needs Scaled Features

The distance adds the squared differences of all features
A feature measured on a bigger scale adds bigger numbers, so it controls who is "near"
StandardScaler rescales every feature to mean 0 and standard deviation 1, so each one gets a fair say

z = x − μσ

Worked Example: Scale One Value

Row 0 of WTT: x = 0.913917, mean 0.949682, standard deviation 0.289490

z = 0.913917 − 0.9496820.289490 ≈ -0.12354

Scale the Features

python

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(df.drop('TARGET CLASS', axis=1))
scaled_features = scaler.transform(df.drop('TARGET CLASS', axis=1))

df_feat = pd.DataFrame(scaled_features, columns=df.columns[:-1])
df_feat.head()

Split, Fit, Predict

python

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    scaled_features, df['TARGET CLASS'], test_size=0.50, random_state=101)

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=1)
knn.fit(X_train, y_train)
pred = knn.predict(X_test)

The Confusion Matrix at K = 1

print(confusion_matrix(y_test, pred))

	Predicted 0	Predicted 1
Actual 0	233	17
Actual 1	24	226

Accuracy and Error Rate

accuracy = 233 + 226500 = 0.918

error rate = 17 + 24500 = 0.082

Precision and Recall, from the Same Matrix

Precision of a class: Of the rows predicted as that class, the share that really are.

Recall of a class: Of the rows that really are that class, the share we found.

P₀ = 233233 + 24 = 0.907

R₀ = 233233 + 17 = 0.932

Part 3

Choosing K

The elbow method

The Elbow Loop

python

error_rate = []

for i in range(1, 60):
    knn = KNeighborsClassifier(n_neighbors=i)
    knn.fit(X_train, y_train)
    pred_i = knn.predict(X_test)
    error_rate.append(np.mean(pred_i != y_test))

Train one model per K, from 1 to 59
Record each test error rate
Plot error against K and look for the bend where the curve flattens

What the Loop Finds

Error rate on the 500 test rows

K	Error rate	Wrong rows
1	0.082	41
2	0.094	47
5	0.054	27
11	0.048	24
24	0.046	23

Try It: Run the Elbow Loop

Part 4

Notebook 2: The Iris Flowers

150 samples, three species

Load Iris into a DataFrame

python

from sklearn.datasets import load_iris

iris = load_iris()
df = pd.DataFrame(iris.data, columns=iris.feature_names)
df['class'] = iris.target
df['class name'] = iris.target_names[iris['target']]
df.head()

sns.pairplot(df)

150 flowers, 50 of each species
4 features, all in cm
Ships with scikit-learn: nothing to download
The pair plot shows the petal features separating the species best

Split, Train, Score

python

x_train, x_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=0)

model = KNeighborsClassifier()
model.fit(x_train, y_train)
model.score(x_test, y_test)

Predict a Flower the Model Has Never Seen

python

predicted_class = model.predict([[5.6, 4.4, 1.2, 0.4]])
print(predicted_class)
print(iris.target_names[predicted_class[0]])

Why Setosa? The Five Neighbours

model.kneighbors lists them: all five are setosa, a 5 to 0 vote

Neighbour	Species	Distance
(5.7, 4.4, 1.5, 0.4)	setosa	0.3162
(5.8, 4, 1.2, 0.2)	setosa	0.4899
(5.2, 4.1, 1.5, 0.1)	setosa	0.6557
(5.4, 3.9, 1.7, 0.4)	setosa	0.7348
(5.7, 3.8, 1.7, 0.3)	setosa	0.7937

d₁ = √((5.6 − 5.7)² + 0 + (1.2 − 1.5)² + 0) = √0.1 = 0.3162

Where the Iris Model Goes Wrong

confusion_matrix(y_test, model.predict(x_test))

	Pred. setosa	Pred. versicolor	Pred. virginica
setosa	11	0	0
versicolor	0	12	1
virginica	0	0	6

Explore: All 150 Iris Flowers

Before you practise

Common Mistakes

Forgetting to scale before a distance-based model
Passing one flat list to predict: it needs [[...]]
Reading "Classified Data" in Colab from a folder that does not have it
Judging the model on the training rows instead of the test rows
Crowning a K from a difference of one or two test rows
Using an even K for two classes and getting ties

Your team project

Project Milestone: Model Implementation

Take the features and target you chose in week 4
Scale the features, then split into train and test sets
Fit KNeighborsClassifier and predict the test set
Report the confusion matrix and classification_report
Run the elbow loop, pick K, and write down why

Part 5

Practice: Your Turn

About 30 minutes, answers follow each task

About 10 minutes

Practice 1: A Second New Flower

Use the same nine flowers F1 to F9 (petal length, petal width)
The new flower is (5.1, 1.75)
Compute the squared distance to every flower and rank them
Predict the species with K = 1, K = 3 and K = 5

Answers

Practice 1: Answer, the Ranking

Rank	Flower	Species	Squared distance
1	F9	virginica	0.0025
2	F3	versicolor	0.0125
3	F4	versicolor	0.0225
4	F6	virginica	0.0425
5	F8	virginica	0.0725

Answers

Practice 1: Answer, the Votes

K = 1: 1 virginica ⇒ virginica

K = 3: 2 versicolor, 1 virginica ⇒ versicolor

K = 5: 3 virginica, 2 versicolor ⇒ virginica

About 5 minutes

Practice 2: Read a Confusion Matrix

	Predicted 0	Predicted 1
Actual 0	238	12
Actual 1	12	238

accuracy = 238 + 238500 = 0.952, error = 0.048

P₁ = 238238 + 12 = 0.952, R₁ = 238238 + 12 = 0.952

About 5 minutes

Practice 3: Standardize by Hand

One feature has the five values 4, 6, 8, 10, 12

μ = 8, σ = √16 + 4 + 0 + 4 + 165 = √8 = 2.8284

z = -1.4142, -0.7071, 0, 0.7071, 1.4142

About 10 minutes

Practice 4: In Colab, on Iris

Run notebook 2 up to model.score(x_test, y_test)
Predict the flower [[6.0, 2.9, 4.5, 1.5]] and print its species name
Call model.kneighbors on it: how did its five neighbours vote?
Loop K from 1 to 15 on the same split and print each score

Answers

Practice 4: Answer

Question	Answer
Species of `[6.0, 2.9, 4.5, 1.5]`	versicolor
The five neighbours	4 versicolor, 1 virginica
Nearest neighbour	(6.1, 3, 4.6, 1.4), distance 0.2000
Score 1.0 at K =	1, 4 and 6 to 15
Score 0.9667 at K =	2, 3, 5

Key Takeaways

KNN predicts by a majority vote of the K training rows closest to the query
Closeness is Euclidean distance, so features must be on the same scale
The same query can change class as K changes: K is a setting you choose
Judge a model on test data with the confusion matrix, accuracy, precision and recall
Pick K with the elbow method, and do not chase differences of one or two rows

Open this lesson

Mahmoud AbasK-Nearest Neighbors Classification

Part

What we do

Time

The idea, and a worked example by hand

30 min

Notebook 1: scaling, fitting, evaluating

25 min

Choosing K with the elbow method

10 min

Notebook 2 on iris, mistakes, your project

20 min

Practice with answers, then takeaways

35 min

Term

In the iris data

Features X

sepal length, sepal width, petal length, petal width (cm)

Label y

species: 0 setosa, 1 versicolor, 2 virginica

Training set

the flowers the model stores

Query

a new flower whose species we want

how many neighbours vote, n_neighbors in scikit-learn

Flower

Species

(length, width)

setosa

(1.4, 0.2)

versicolor

(4.7, 1.6)

versicolor

(5.0, 1.7)

versicolor

(5.1, 1.6)

versicolor

(4.9, 1.5)

Flower

Species

(length, width)

virginica

(4.9, 1.8)

virginica

(4.8, 1.8)

virginica

(5.0, 1.5)

virginica

(5.1, 1.8)

Flower

Differences

Squared distance

-3.30, -1.50

10.89 + 2.25 = 13.14

0, -0.10

0 + 0.01 = 0.01

+0.30, 0

0.09 + 0 = 0.09

+0.40, -0.10

0.16 + 0.01 = 0.17

+0.20, -0.20

0.04 + 0.04 = 0.08

Flower

Differences

Squared distance

+0.20, +0.10

0.04 + 0.01 = 0.05

+0.10, +0.10

0.01 + 0.01 = 0.02

+0.30, -0.20

0.09 + 0.04 = 0.13

+0.40, +0.10

0.16 + 0.01 = 0.17

Rank

Flower

Species

Distance d

versicolor

√0.01 = 0.100

virginica

√0.02 = 0.141

virginica

√0.05 = 0.224

versicolor

√0.08 = 0.283

versicolor

√0.09 = 0.300

import pandas as pd import numpy as np url = ("https://raw.githubusercontent.com/" "tirthajyoti/Machine-Learning-with-Python/" "master/Datasets/Classified%20Data") df = pd.read_csv(url, index_col=0) df.info()

from sklearn.preprocessing import StandardScaler scaler = StandardScaler() scaler.fit(df.drop('TARGET CLASS', axis=1)) scaled_features = scaler.transform(df.drop('TARGET CLASS', axis=1)) df_feat = pd.DataFrame(scaled_features, columns=df.columns[:-1]) df_feat.head()

from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split( scaled_features, df['TARGET CLASS'], test_size=0.50, random_state=101) from sklearn.neighbors import KNeighborsClassifier knn = KNeighborsClassifier(n_neighbors=1) knn.fit(X_train, y_train) pred = knn.predict(X_test)

Predicted 0

Predicted 1

Actual 0

233

Actual 1

226

Error rate

Wrong rows

0.082

0.094

0.054

0.048

0.046

from sklearn.datasets import load_iris iris = load_iris() df = pd.DataFrame(iris.data, columns=iris.feature_names) df['class'] = iris.target df['class name'] = iris.target_names[iris['target']] df.head() sns.pairplot(df)

Neighbour

Species

Distance

(5.7, 4.4, 1.5, 0.4)

setosa

0.3162

(5.8, 4, 1.2, 0.2)

setosa

0.4899

(5.2, 4.1, 1.5, 0.1)

setosa

0.6557

(5.4, 3.9, 1.7, 0.4)

setosa

0.7348

(5.7, 3.8, 1.7, 0.3)

setosa

0.7937

Pred. setosa

Pred. versicolor

Pred. virginica

setosa

versicolor

virginica

Rank

Flower

Species

Squared distance

virginica

0.0025

versicolor

0.0125

versicolor

0.0225

virginica

0.0425

virginica

0.0725

Predicted 0

Predicted 1

Actual 0

238

Actual 1

238

Question

Answer

Species of [6.0, 2.9, 4.5, 1.5]

versicolor

The five neighbours

4 versicolor, 1 virginica

Nearest neighbour

(6.1, 3, 4.6, 1.4), distance 0.2000

Score 1.0 at K =

1, 4 and 6 to 15

Score 0.9667 at K =

2, 3, 5