Mahmoud Abas, Full-Stack Software Engineer & AI

Software Construction - ML Week01 Lab (KNN)

In the print dialog, choose "Save as PDF" as the destination.

ML Week 1 Lab: K-Nearest Neighbours

From the k-NN algorithm and model evaluation to regression, SVMs, and decision trees

Term Plan Recap

Fundamentals of ML
K-Nearest Neighbours
this lab
Model Evaluation & Selection
Support Vector Machines
Decision Trees
Naive Bayes
Neural Networks

The k-Nearest Neighbor (`k-NN`) Algorithm

k-NN: An instance/memory-based supervised learning method: given a training set X_train/y_train and a new instance x_test, it finds similar neighbors and votes on a label.

Find the most similar instances to x_test in X_train (call them X_NN)
Get their labels y_NN
Predict x_test's label by combining y_NN, for example a simple majority vote

Needs four things: a distance metric, how many neighbors (k) to look at, an optional weighting function on neighbors, and a method to aggregate neighbor classes

Bias-Variance Tradeoff and Generalization

Bias: difference between the model's predictions and the correct value; high bias means large error on both training and test data, and the model underfits

Variance: how much predictions vary for a given point; high variance means great training performance but poor test performance, and the model overfits

Small k (e.g. k=1) leads to low bias, high variance: overfitting
Large k leads to higher bias, lower variance: underfitting
Generalization is the algorithm's ability to predict accurately on new, unseen data, the real goal

Cross-Validation

Uses multiple train-test splits instead of just one; each split trains and evaluates a separate model
A single train-test split's accuracy score can vary depending on which samples land in training vs test
Multiple splits give a more stable, reliable estimate of average classifier performance: results are averaged across different training sets rather than trusting one split

`KNN` Classification and Regression

python

from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier, KNeighborsRegressor

X_train, X_test, y_train, y_test = train_test_split(X_C2, y_C2, random_state=0)

# k=1 overfits (very jagged boundary); larger k generalizes better
clf = KNeighborsClassifier(n_neighbors=11).fit(X_train, y_train)

knnreg = KNeighborsRegressor(n_neighbors=5).fit(X_train, y_train)
print('R-squared test score: {:.3f}'.format(knnreg.score(X_test, y_test)))

k=1 overfits with a very jagged boundary; a larger k generalizes better.

model.score(X, y) for regression returns R^2 (coefficient of determination): 0 predicts the mean, 1 is a perfect prediction
k-NN works for both classification and regression

Linear Regression: Least-Squares

python

from sklearn.linear_model import LinearRegression

linreg = LinearRegression().fit(X_train, y_train)
print('coeff (w): {}'.format(linreg.coef_))
print('intercept (b): {:.3f}'.format(linreg.intercept_))
print('R-squared (train): {:.3f}'.format(linreg.score(X_train, y_train)))
print('R-squared (test): {:.3f}'.format(linreg.score(X_test, y_test)))

y = wx + b

Fits the line that minimizes squared error between predictions and true values

Ridge, Lasso, and Polynomial Regression

python

from sklearn.linear_model import Ridge, Lasso
from sklearn.preprocessing import MinMaxScaler, PolynomialFeatures

scaler = MinMaxScaler()
X_train_scaled = scaler.fit_transform(X_train)

linridge = Ridge(alpha=20.0).fit(X_train_scaled, y_train)   # L2 penalty
linlasso = Lasso(alpha=2.0, max_iter=10000).fit(X_train_scaled, y_train)  # L1 penalty, drives coefficients to 0

poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X_F1)   # adds quadratic feature interactions

Ridge (L2): shrinks all coefficients toward zero

Lasso (L1): can zero out features entirely, a form of feature selection

Logistic Regression

python

from sklearn.linear_model import LogisticRegression

clf = LogisticRegression(C=100).fit(X_train, y_train)
print('Accuracy on training set: {:.2f}'.format(clf.score(X_train, y_train)))
print('Accuracy on test set: {:.2f}'.format(clf.score(X_test, y_test)))

Used for classification (for example apple vs not-apple from fruit height/width), despite the name 'regression'
Parameter C controls regularization strength, try multiple values of C to find the best fit

Support Vector Machines (Linear)

python

from sklearn.svm import SVC, LinearSVC

clf = SVC(kernel='linear', C=1.0).fit(X_train, y_train)
clf2 = LinearSVC(C=100).fit(X_train, y_train)

SVMs: find the decision boundary that best separates classes

C controls the tolerance for misclassified points: a small C gives a wider margin (more tolerant), a large C fits training data more tightly

Kernelized `SVM`

python

from sklearn.svm import SVC

clf = SVC(kernel='rbf', gamma=1.0, C=15).fit(X_train, y_train)   # default kernel is RBF
clf_poly = SVC(kernel='poly', degree=3).fit(X_train, y_train)

RBF (radial basis function): the default kernel, good for non-linearly separable classes

gamma controls how complex or tight the decision boundary is; combine with C to tune the fit
Scaling input features (MinMaxScaler) before fitting significantly improves test accuracy on unnormalized real datasets

Decision Trees and Feature Importance

python

from sklearn.tree import DecisionTreeClassifier

clf = DecisionTreeClassifier().fit(X_train, y_train)          # tends to overfit (train acc = 1.00)
clf2 = DecisionTreeClassifier(max_depth=3).fit(X_train, y_train)  # limit depth to reduce overfitting

print('Feature importances: {}'.format(clf.feature_importances_))

An unrestricted tree memorizes training data (100% train accuracy) but generalizes worse
Limiting max_depth trades a little training accuracy for better generalization
feature_importances_ shows which input features drove the tree's decisions most

Key Takeaways

k-NN classifies and predicts by combining the labels of the k most similar training instances; small k overfits, large k underfits
Bias-variance tradeoff governs generalization: high bias underfits, high variance overfits, and cross-validation gives a stable performance estimate across multiple train-test splits
Linear regression minimizes squared error; Ridge (L2) and Lasso (L1) add regularization, with Lasso capable of zeroing out features entirely
Logistic regression, linear SVM, and kernelized SVM (RBF/polynomial) are the standard classifiers, all tuned via C (and gamma for RBF)
Feature scaling (e.g. MinMaxScaler) meaningfully improves regularized regression and SVM performance
Decision trees overfit without a depth limit; feature_importances_ reveals which features matter most to predictions

Open this lesson

Mahmoud AbasML Week01 Lab (KNN)

from sklearn.model_selection import train_test_split from sklearn.neighbors import KNeighborsClassifier, KNeighborsRegressor X_train, X_test, y_train, y_test = train_test_split(X_C2, y_C2, random_state=0) # k=1 overfits (very jagged boundary); larger k generalizes better clf = KNeighborsClassifier(n_neighbors=11).fit(X_train, y_train) knnreg = KNeighborsRegressor(n_neighbors=5).fit(X_train, y_train) print('R-squared test score: {:.3f}'.format(knnreg.score(X_test, y_test)))

from sklearn.linear_model import LinearRegression linreg = LinearRegression().fit(X_train, y_train) print('coeff (w): {}'.format(linreg.coef_)) print('intercept (b): {:.3f}'.format(linreg.intercept_)) print('R-squared (train): {:.3f}'.format(linreg.score(X_train, y_train))) print('R-squared (test): {:.3f}'.format(linreg.score(X_test, y_test)))

from sklearn.linear_model import Ridge, Lasso from sklearn.preprocessing import MinMaxScaler, PolynomialFeatures scaler = MinMaxScaler() X_train_scaled = scaler.fit_transform(X_train) linridge = Ridge(alpha=20.0).fit(X_train_scaled, y_train) # L2 penalty linlasso = Lasso(alpha=2.0, max_iter=10000).fit(X_train_scaled, y_train) # L1 penalty, drives coefficients to 0 poly = PolynomialFeatures(degree=2) X_poly = poly.fit_transform(X_F1) # adds quadratic feature interactions

from sklearn.linear_model import LogisticRegression clf = LogisticRegression(C=100).fit(X_train, y_train) print('Accuracy on training set: {:.2f}'.format(clf.score(X_train, y_train))) print('Accuracy on test set: {:.2f}'.format(clf.score(X_test, y_test)))

from sklearn.tree import DecisionTreeClassifier clf = DecisionTreeClassifier().fit(X_train, y_train) # tends to overfit (train acc = 1.00) clf2 = DecisionTreeClassifier(max_depth=3).fit(X_train, y_train) # limit depth to reduce overfitting print('Feature importances: {}'.format(clf.feature_importances_))

ML Week 1 Lab: K-Nearest Neighbours

Term Plan Recap

The k-Nearest Neighbor (k-NN) Algorithm

Bias-Variance Tradeoff and Generalization

Cross-Validation

KNN Classification and Regression

Linear Regression: Least-Squares

Ridge, Lasso, and Polynomial Regression

Logistic Regression

Support Vector Machines (Linear)

Kernelized SVM

Decision Trees and Feature Importance

Key Takeaways

Open this lesson

ML Week 1 Lab: K-Nearest Neighbours

Term Plan Recap

The k-Nearest Neighbor (k-NN) Algorithm

Bias-Variance Tradeoff and Generalization

Cross-Validation

KNN Classification and Regression

Linear Regression: Least-Squares

Ridge, Lasso, and Polynomial Regression

Logistic Regression

Support Vector Machines (Linear)

Kernelized SVM

Decision Trees and Feature Importance

Key Takeaways

Open this lesson

The k-Nearest Neighbor (`k-NN`) Algorithm

`KNN` Classification and Regression

Kernelized `SVM`

The k-Nearest Neighbor (`k-NN`) Algorithm

`KNN` Classification and Regression

Kernelized `SVM`