Mahmoud Abas, Full-Stack Software Engineer & AI

AI308 Machine Learning - Lesson 6

Linear Regression Lab

From closed-form equations to scikit-learn

Introduction to Linear Regression

Models the relationship between a dependent variable (the target) and one or more independent variables (the features)
Assumes a linear relationship and fits the best straight line through the data
Applications: sales forecasting, stock prices, house prices, trend analysis
Simple, interpretable, and the foundation for more complex algorithms

The Algorithm

Simple linear regression uses one feature; multiple linear regression extends to several

y = β₀ + β₁ x

y = β₀ + β₁ x₁ + β₂ x₂ + ... + β_n x_n

β₀ is the intercept (bias), β₁ is the slope (weight)
Goal: find the line that best fits all the data points

Mathematical Formulation: The Cost Function

Mean Squared Error (MSE) measures the average squared prediction error

J(β₀, β₁) = 1n∑ni=1(y_i - ŷ_i)²

Training means finding β₀, β₁ that minimize this cost function

The Normal Equation

Normal equation: A closed-form solution for simple linear regression: no iteration needed, solved directly from the data

β₁ = ∑(x_i - x̄)(y_i - ȳ)∑(x_i - x̄)²

β₀ = ȳ - β₁ x̄

Algorithm Steps

Collect and clean the data
Compute the means of x and y
Compute each point's deviation from the mean
Compute the products and squares of those deviations
Calculate the slope, then the intercept, then form ŷ = β₀ + β₁x
Evaluate with R², MSE, and RMSE

Worked Example

Student Exam Scores: Setup

5 students: hours studied (x) versus exam score (y)
Task: find the regression equation and predict the score for a student who studies 6 hours

python

x = np.array([1, 2, 3, 4, 5])
y = np.array([2, 4, 5, 4, 5])
x_mean, y_mean = np.mean(x), np.mean(y)

Mean of x = 3, mean of y = 4
Sum of cross-deviations = 6, sum of squared x-deviations = 10

Worked Example

Student Exam Scores: Slope and Intercept

Using the sums from the previous slide, find β₀ and β₁

β₁ = 610 = 0.6

β₀ = 4 - (0.6 × 3) = 2.2

Each extra study hour adds 0.6 points to the predicted score
Final equation: ŷ = 2.2 + 0.6x

Worked Example

Student Exam Scores: Prediction and Evaluation

Predict the score for a student who studies 6 hours

ŷ = 2.2 + 0.6(6) = 5.8

Metric	Value
MSE	0.48
RMSE	≈ 0.69
R²	0.6

R² = 0.6 means the model explains 60% of the variance in exam scores

Worked Example

House Price Prediction

House size (sq ft) predicts price for 5 sample houses
Task: fit the line and predict the price of a 2000 sq ft house

Price = -27.76 + 0.2052 × Size

Each extra square foot adds roughly $205 to the predicted price
R² = 0.9966, RMSE = 5.81, a much tighter fit than Example 1
Prediction for a 2000 sq ft house: `$382.59k`

Scikit-Learn Implementation

LinearRegression reproduces the manual math exactly

python

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X.reshape(-1, 1), y)
slope, intercept = model.coef_[0], model.intercept_
predicted = model.predict([[6]])

Model Evaluation Metrics

MSE: average squared error; sensitive to outliers

RMSE: square root of MSE; same units as the target, easier to interpret

R-squared (R²): proportion of variance explained, from 0 to 1; higher is better

Also useful: MAE (mean absolute error) and MAPE (mean absolute percentage error)

Residual Analysis

Residual: the actual value minus the predicted value, plotted against the predicted values

A consistent spread indicates homoscedasticity (constant variance)

When to Use Linear Regression

Good fit

Approximately linear relationship
Need interpretability
Few outliers
Independent features

Not suitable

Highly non-linear relationships
Many outliers
Multicollinearity
Complex feature interactions

Next steps: multiple regression, polynomial regression, regularization (Ridge, Lasso)

Key Takeaways

Linear regression finds the line that minimizes squared prediction error (MSE)
The normal equation gives a direct, closed-form solution: no iteration needed
Always validate with R², MSE, and RMSE together, never just one metric
Residual plots reveal whether the model's assumptions actually hold
Manual calculation and scikit-learn should always agree: a great trust-building exercise

Open this lesson

Mahmoud AbasLinear Regression Lab