Logo
Machine Learning (2026-2027) - K-Nearest Neighbors Classification

In the print dialog, choose "Save as PDF" as the destination.

Machine Learning, Week 5

K-Nearest Neighbors Classification

Classify a new sample by asking its K closest neighbours, then build, score and tune the model with scikit-learn.

Objectives

  • Explain how KNN predicts a class: distance, the K nearest rows, a majority vote
  • Compute Euclidean distances and a KNN prediction by hand for K = 1, 3 and 5
  • Scale features with StandardScaler and say why distance needs it
  • Fit KNeighborsClassifier, then read a confusion matrix, accuracy, precision and recall
  • Choose K with the elbow method
  • Build and use a KNN model on the iris dataset, and apply it to your project

Week 5 of the plan

Where This Sits in the Course

  • Two notebooks: KNN_Classification (a 1000-row dataset with 10 anonymous features) and Nearest Neighbors (iris)
  • Real-world scenario: a simple supervised-learning model on the famous iris dataset, 150 samples of three species
  • Project milestone this week: Machine Learning Model Implementation

Plan for the Two Hours

PartWhat we doTime
1The idea, and a worked example by hand30 min
2Notebook 1: scaling, fitting, evaluating25 min
3Choosing K with the elbow method10 min
4Notebook 2 on iris, mistakes, your project20 min
5Practice with answers, then takeaways35 min

Part 1

The Idea of KNN

Similar inputs have similar labels

What Is K-Nearest Neighbors?

K-nearest neighbors (KNN)
A simple supervised-learning algorithm that predicts an outcome by identifying the data points closest in proximity to the input data.

The Vocabulary, on Iris

TermIn the iris data
Features Xsepal length, sepal width, petal length, petal width (cm)
Label yspecies: 0 setosa, 1 versicolor, 2 virginica
Training setthe flowers the model stores
Querya new flower whose species we want
Khow many neighbours vote, n_neighbors in scikit-learn

The Algorithm in Four Steps

  1. Store the training rows and their labels (this is all fit needs to do)
  2. Compute the distance from the query to every training row
  3. Sort the distances and keep the K smallest
  4. Count the labels of those K rows: the majority is the prediction

Measuring Closeness: Euclidean Distance

d = √((x1 − q1)2 + (x2 − q2)2 + ... + (xn − qn)2)
  • x is a training row, q is the query, n is the number of features
  • Subtract feature by feature, square, add, take the square root
  • Sorting by the squared distance gives the same order, so by hand you can skip the root until the end

By hand

Worked Example: One New Flower

Nine real iris flowers, two features: petal length and petal width in cm. The new flower is (4.7, 1.7).

FlowerSpecies(length, width)
F1setosa(1.4, 0.2)
F2versicolor(4.7, 1.6)
F3versicolor(5.0, 1.7)
F4versicolor(5.1, 1.6)
F5versicolor(4.9, 1.5)
FlowerSpecies(length, width)
F6virginica(4.9, 1.8)
F7virginica(4.8, 1.8)
F8virginica(5.0, 1.5)
F9virginica(5.1, 1.8)

Step 1: Squared Distances, F1 to F5

Difference = flower minus query, for length and for width

FlowerDifferencesSquared distance
F1-3.30, -1.5010.89 + 2.25 = 13.14
F20, -0.100 + 0.01 = 0.01
F3+0.30, 00.09 + 0 = 0.09
F4+0.40, -0.100.16 + 0.01 = 0.17
F5+0.20, -0.200.04 + 0.04 = 0.08

Step 1: Squared Distances, F6 to F9

FlowerDifferencesSquared distance
F6+0.20, +0.100.04 + 0.01 = 0.05
F7+0.10, +0.100.01 + 0.01 = 0.02
F8+0.30, -0.200.09 + 0.04 = 0.13
F9+0.40, +0.100.16 + 0.01 = 0.17

Step 2: Sort and Take the Roots

RankFlowerSpeciesDistance d
1F2versicolor√0.01 = 0.100
2F7virginica√0.02 = 0.141
3F6virginica√0.05 = 0.224
4F5versicolor√0.08 = 0.283
5F3versicolor√0.09 = 0.300

Step 3: Vote with K = 1

K = 1: 1 versicolor ⇒ versicolor

Step 3: Vote with K = 3

K = 3: 2 virginica, 1 versicolor ⇒ virginica

Step 3: Vote with K = 5

K = 5: 3 versicolor, 2 virginica ⇒ versicolor

Try It: The Worked Example, Live

K Is a Choice, and Ties Happen

  • K is not learned from the data: you choose it, then check it on test data (Part 3)
  • A small K follows the single closest row, even if that row is unusual
  • A larger K listens to more rows, so one odd row matters less
  • With an even K a vote can tie, for example 1 to 1 with K = 2

Part 2

Notebook 1: The Classified Data

Scale, split, fit, evaluate

Open the Notebook in Colab

text
https://colab.research.google.com/github/
tirthajyoti/Machine-Learning-with-Python/blob/
master/Classification/KNN_Classification.ipynb
Join the three lines into one address, or click "Open in Colab" on the lesson page.

Load and Inspect the Data

python
import pandas as pd
import numpy as np

url = ("https://raw.githubusercontent.com/"
       "tirthajyoti/Machine-Learning-with-Python/"
       "master/Datasets/Classified%20Data")
df = pd.read_csv(url, index_col=0)
df.info()
  • 1000 rows, 10 feature columns with made-up names (WTT, PTI, ...)
  • One label column, TARGET CLASS, with values 0 and 1
  • Balanced: 500 rows of each class
  • The notebook draws a boxplot per feature, split by class, to see which features separate the classes

Why KNN Needs Scaled Features

  • The distance adds the squared differences of all features
  • A feature measured on a bigger scale adds bigger numbers, so it controls who is "near"
  • StandardScaler rescales every feature to mean 0 and standard deviation 1, so each one gets a fair say
z = x − μσ

Worked Example: Scale One Value

Row 0 of WTT: x = 0.913917, mean 0.949682, standard deviation 0.289490

z = 0.913917 − 0.9496820.289490 ≈ -0.12354

Scale the Features

python
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(df.drop('TARGET CLASS', axis=1))
scaled_features = scaler.transform(df.drop('TARGET CLASS', axis=1))

df_feat = pd.DataFrame(scaled_features, columns=df.columns[:-1])
df_feat.head()

Split, Fit, Predict

python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    scaled_features, df['TARGET CLASS'], test_size=0.50, random_state=101)

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=1)
knn.fit(X_train, y_train)
pred = knn.predict(X_test)

The Confusion Matrix at K = 1

print(confusion_matrix(y_test, pred))

Predicted 0Predicted 1
Actual 023317
Actual 124226

Accuracy and Error Rate

accuracy = 233 + 226500 = 0.918
error rate = 17 + 24500 = 0.082

Precision and Recall, from the Same Matrix

Precision of a class
Of the rows predicted as that class, the share that really are.
Recall of a class
Of the rows that really are that class, the share we found.
P0 = 233233 + 24 = 0.907
R0 = 233233 + 17 = 0.932

Part 3

Choosing K

The elbow method

The Elbow Loop

python
error_rate = []

for i in range(1, 60):
    knn = KNeighborsClassifier(n_neighbors=i)
    knn.fit(X_train, y_train)
    pred_i = knn.predict(X_test)
    error_rate.append(np.mean(pred_i != y_test))
  • Train one model per K, from 1 to 59
  • Record each test error rate
  • Plot error against K and look for the bend where the curve flattens

What the Loop Finds

Error rate on the 500 test rows

KError rateWrong rows
10.08241
20.09447
50.05427
110.04824
240.04623

Try It: Run the Elbow Loop

Part 4

Notebook 2: The Iris Flowers

150 samples, three species

Load Iris into a DataFrame

python
from sklearn.datasets import load_iris

iris = load_iris()
df = pd.DataFrame(iris.data, columns=iris.feature_names)
df['class'] = iris.target
df['class name'] = iris.target_names[iris['target']]
df.head()

sns.pairplot(df)
  • 150 flowers, 50 of each species
  • 4 features, all in cm
  • Ships with scikit-learn: nothing to download
  • The pair plot shows the petal features separating the species best

Split, Train, Score

python
x_train, x_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=0)

model = KNeighborsClassifier()
model.fit(x_train, y_train)
model.score(x_test, y_test)

Predict a Flower the Model Has Never Seen

python
predicted_class = model.predict([[5.6, 4.4, 1.2, 0.4]])
print(predicted_class)
print(iris.target_names[predicted_class[0]])

Why Setosa? The Five Neighbours

model.kneighbors lists them: all five are setosa, a 5 to 0 vote

NeighbourSpeciesDistance
(5.7, 4.4, 1.5, 0.4)setosa0.3162
(5.8, 4, 1.2, 0.2)setosa0.4899
(5.2, 4.1, 1.5, 0.1)setosa0.6557
(5.4, 3.9, 1.7, 0.4)setosa0.7348
(5.7, 3.8, 1.7, 0.3)setosa0.7937
d1 = √((5.6 − 5.7)2 + 0 + (1.2 − 1.5)2 + 0) = √0.1 = 0.3162

Where the Iris Model Goes Wrong

confusion_matrix(y_test, model.predict(x_test))

Pred. setosaPred. versicolorPred. virginica
setosa1100
versicolor0121
virginica006

Explore: All 150 Iris Flowers

Before you practise

Common Mistakes

  • Forgetting to scale before a distance-based model
  • Passing one flat list to predict: it needs [[...]]
  • Reading "Classified Data" in Colab from a folder that does not have it
  • Judging the model on the training rows instead of the test rows
  • Crowning a K from a difference of one or two test rows
  • Using an even K for two classes and getting ties

Your team project

Project Milestone: Model Implementation

  1. Take the features and target you chose in week 4
  2. Scale the features, then split into train and test sets
  3. Fit KNeighborsClassifier and predict the test set
  4. Report the confusion matrix and classification_report
  5. Run the elbow loop, pick K, and write down why

Part 5

Practice: Your Turn

About 30 minutes, answers follow each task

About 10 minutes

Practice 1: A Second New Flower

  • Use the same nine flowers F1 to F9 (petal length, petal width)
  • The new flower is (5.1, 1.75)
  • Compute the squared distance to every flower and rank them
  • Predict the species with K = 1, K = 3 and K = 5

Answers

Practice 1: Answer, the Ranking

RankFlowerSpeciesSquared distance
1F9virginica0.0025
2F3versicolor0.0125
3F4versicolor0.0225
4F6virginica0.0425
5F8virginica0.0725

Answers

Practice 1: Answer, the Votes

K = 1: 1 virginica ⇒ virginica
K = 3: 2 versicolor, 1 virginica ⇒ versicolor
K = 5: 3 virginica, 2 versicolor ⇒ virginica

About 5 minutes

Practice 2: Read a Confusion Matrix

Predicted 0Predicted 1
Actual 023812
Actual 112238
accuracy = 238 + 238500 = 0.952, error = 0.048
P1 = 238238 + 12 = 0.952, R1 = 238238 + 12 = 0.952

About 5 minutes

Practice 3: Standardize by Hand

One feature has the five values 4, 6, 8, 10, 12

μ = 8, σ = √16 + 4 + 0 + 4 + 165 = √8 = 2.8284
z = -1.4142, -0.7071, 0, 0.7071, 1.4142

About 10 minutes

Practice 4: In Colab, on Iris

  1. Run notebook 2 up to model.score(x_test, y_test)
  2. Predict the flower [[6.0, 2.9, 4.5, 1.5]] and print its species name
  3. Call model.kneighbors on it: how did its five neighbours vote?
  4. Loop K from 1 to 15 on the same split and print each score

Answers

Practice 4: Answer

QuestionAnswer
Species of [6.0, 2.9, 4.5, 1.5]versicolor
The five neighbours4 versicolor, 1 virginica
Nearest neighbour(6.1, 3, 4.6, 1.4), distance 0.2000
Score 1.0 at K =1, 4 and 6 to 15
Score 0.9667 at K =2, 3, 5

Key Takeaways

  1. KNN predicts by a majority vote of the K training rows closest to the query
  2. Closeness is Euclidean distance, so features must be on the same scale
  3. The same query can change class as K changes: K is a setting you choose
  4. Judge a model on test data with the confusion matrix, accuracy, precision and recall
  5. Pick K with the elbow method, and do not chase differences of one or two rows

Open this lesson

Mahmoud AbasK-Nearest Neighbors Classification

Machine Learning (2026-2027) - K-Nearest Neighbors Classification