Logo
Machine Learning (2026-2027) - Linear Regression

In the print dialog, choose "Save as PDF" as the destination.

Machine Learning, Week 3

Linear Regression

Predict a house price with a straight line: fit it by hand with least squares, then model 5000 real houses in scikit-learn.

Objectives

  • Explain what linear regression predicts, and simple versus multiple regression
  • Define a residual and the least-squares line, and compute the slope and intercept by hand
  • Compute MAE, MSE, RMSE and R² for a fitted line
  • Explore and prepare a dataset with info, describe, a histogram and a correlation heatmap
  • Fit LinearRegression, read its coefficients, and predict a house by hand
  • Evaluate the model on test data and apply the steps to your project

Week 3 of the plan

Where This Sits in the Course

  • Notebook: Linear_Regression_Practice, the prices of 5000 houses in the USA
  • Real-world scenario: Predict House Pricing
  • Project milestone this week: Project Data Exploration and Data Cleaning

Plan for the Two Hours

PartWhat we doTime
1The idea, least squares, a worked example by hand35 min
2Notebook: explore and prepare the housing data20 min
3Notebook: fit the model, read the coefficients20 min
4Evaluate on the test set, your project15 min
5Practice with answers, then takeaways30 min

Part 1

The Idea of Linear Regression

A straight line through the data

What Is Linear Regression?

Linear regression
A linear approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables X.
  • One explanatory variable: simple linear regression
  • More than one: multiple linear regression
  • Usually fitted with the least squares approach

The Model Is a Straight Line

ŷ = b0 + b1 x
  • x: the input, for example the area income
  • ŷ ("y hat"): the predicted value, for example the price
  • b₁: the slope, the change in ŷ when x grows by 1
  • b₀: the intercept, the value of ŷ at x = 0

Residuals: The Error of Each Point

e = y − ŷ
  • y is the true value, ŷ the prediction
  • A point above the line has a positive residual, below it a negative one
  • On a plot, the residual is the vertical distance from the point to the line

Least Squares

SSE = ∑ (yi − ŷi)2
  • SSE: the sum of squared errors over all n points
  • The least-squares line is the line with the smallest SSE
  • Squaring makes every error positive and punishes big errors more
b1 = ∑ (xi − x̄)(yi − ȳ)∑ (xi − x̄)2
b0 = ȳ − b1 x̄

By hand

Worked Example: Five Real Houses

Rows of USA_Housing.csv. x: income in $10,000, y: price in $100,000, rounded

HouseIncomePrice(x, y)
H1$50,016$497,368(5, 5)
H2$59,769$897,291(6, 9)
H3$69,529$995,137(7, 10)
H4$79,628$1,196,064(8, 12)
H5$89,548$1,403,803(9, 14)

Step 1: The Means

x̄ = 5 + 6 + 7 + 8 + 95 = 7
ȳ = 5 + 9 + 10 + 12 + 145 = 10

Step 2: Deviations from the Means

House(x − x̄, y − ȳ)Product(x − x̄)²
H1(-2, -5)104
H2(-1, -1)11
H3(0, 0)00
H4(1, 2)21
H5(2, 4)84

Step 3: Slope and Intercept

b1 = 2110 = 2.1
b0 = 10 − 2.1 × 7 = −4.7

Step 4: Predictions and Residuals

Houseŷe = y − ŷe²
H15.8-0.80.64
H27.91.11.21
H310.00.00.00
H412.1-0.10.01
H514.2-0.20.04

Step 5: Is It Really the Best Line?

LineSSE
ŷ = −3 + 2x7.0
ŷ = −4 + 2x2.0
ŷ = −4.7 + 2.1x1.9
The five houses, the least-squares line and the residuals in red

Step 6: Predict a New Area

ŷ = −4.7 + 2.1 × 7.5 = 11.05

Try It: Least Squares, Live

Measuring the Error of a Line

MAE = 1n ∑ |ei|
MSE = 1n ∑ ei2
RMSE = √MSE
  • MAE: the average size of an error, in the units of y
  • MSE: the average squared error, in squared units
  • RMSE: back in the units of y, and big errors weigh more than in MAE

Worked Example: MAE, MSE, RMSE

MAE = 0.8 + 1.1 + 0 + 0.1 + 0.25 = 0.44
MSE = 1.905 = 0.38, RMSE = √0.38 = 0.6164

R²: Better Than the Mean?

R2 = 1 − SSESST, SST = ∑ (yi − ȳ)2
  • SST: the SSE of the flat line ŷ = ȳ, the best guess without a feature
  • R² = the share of that error the model removes: 1 is perfect, 0 is no better than the mean
SST = 25 + 1 + 0 + 4 + 16 = 46, R2 = 1 − 1.9046 = 0.9587

Try It: What R² Measures

Part 2

Explore the Housing Data

The notebook, and the milestone of this week

Open the Notebook in Colab

text
https://colab.research.google.com/github/
tirthajyoti/Machine-Learning-with-Python/blob/
master/Regression/Linear_Regression_Practice.ipynb
Join the three lines into one address, or click "Open in Colab" on the lesson page.

Load and Inspect the Data

python
url = ("https://raw.githubusercontent.com/"
       "tirthajyoti/Machine-Learning-with-Python/"
       "master/Datasets/USA_Housing.csv")
df = pd.read_csv(url)
df.info(verbose=True)
  • 5000 rows, 7 columns, all 5000 non-null: no missing values
  • Five numeric features and the target Price
  • Address is text: a linear model only multiplies numbers, so it stays out of X

The Distribution of the Price

python
df['Price'].plot.hist(bins=25, figsize=(8,4))
  • One hill around the mean of about $1.23 million
  • Only 97 houses below $0.5 million and 74 above $2 million
  • 1e6 under the axis: the ticks are in millions
Histogram of the 5000 house prices with 25 bins

Correlations with the Price

python
df.corr(numeric_only=True)

sns.heatmap(df.corr(numeric_only=True),
            annot=True, linewidths=2)
Correlation heatmap of the six numeric columns

Reading the Price Row

FeatureCorrelation with Price
Avg. Area Income0.640
Avg. Area House Age0.453
Area Population0.409
Avg. Area Number of Rooms0.336
Avg. Area Number of Bedrooms0.171

Part 3

Fit the Model

LinearRegression in scikit-learn

Features, Target, and the Split

python
X = df[l_column[0:len_feature-2]]   # 5 features
y = df[l_column[len_feature-2]]     # Price

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=123)
The notebook imports from sklearn.cross_validation, which no longer exists: use sklearn.model_selection.

Fit and Read the Coefficients

python
lm = LinearRegression()
lm.fit(X_train, y_train)
print(lm.intercept_)

cdf = pd.DataFrame(data=lm.coef_,
    index=X_train.columns,
    columns=['Coefficients'])
FeatureCoefficient
Income21.597602
House age165201.104954
Rooms119061.463868
Bedrooms3212.585606
Population15.228121

Reading a Coefficient

  • A coefficient is the change in ŷ when its feature grows by 1 and the others stay the same

Worked Example: One Test House by Hand

Row 2648, the first test house. Real price: $800,146.23

Featurexb × x
Income63824.3945391,378,453.87
House age4.991750824,642.62
Rooms5.003836595,764.04
Bedrooms4.0012,850.34
Population40086.458749610,441.44

Worked Example: Add the Intercept

ŷ = −2,631,028.90 + 3,422,152.31 = 791,123.41
e = 800,146.23 − 791,123.41 = 9,022.82

Which Features Matter Most?

The notebook divides each coefficient by its standard error: the t-statistic

Featuret-statistic
Avg. Area Income134.68
Avg. Area House Age95.91
Area Population89.64
Avg. Area Number of Rooms70.18
Avg. Area Number of Bedrooms2.33

Part 4

Evaluate the Model

Predict the 1500 test houses

Actual Against Predicted

python
predictions = lm.predict(X_test)

plt.scatter(x=y_test, y=predictions)
  • predictions: 1500 prices, one per test house
  • A perfect model puts every point on the 45 degree line
  • The points form a narrow band along it, for cheap and expensive houses alike
Scatter of actual against predicted prices for the 1500 test houses

Checking the Residuals

Histogram of the test residuals with a density curve
Residuals plotted against the predicted prices

The Error Measures

text
Mean absolute error (MAE): 81739.77482718184
Mean square error (MSE): 10489638335.804983
Root mean square error (RMSE): 102418.93543581179
R-squared value of predictions: 0.919
RMSE = √10,489,638,335.80 = 102,418.94

Reading the Numbers

MeasureValueMeaning
MAE$81,740average size of an error
RMSE$102,419typical error, about 8% of the mean price
Test R²0.919about 92% of the price spread explained
Training R²0.917as good on new houses as on seen ones

Before you practise

Common Mistakes

  • Running the old calls as they are: cross_validation, df.corr() with a text column, distplot
  • Reading ./Datasets/USA_Housing.csv in Colab
  • Leaving the text column Address in X
  • Ranking features by coefficient size when their units differ
  • Reporting only the training R²
  • Trusting a prediction far outside the training data, like the intercept at income 0

Your team project

Project Milestone: Explore and Clean Your Data

  1. Check shape, the columns and the types with info()
  2. Count missing values with isnull().sum(), then fillna or dropna (week 1)
  3. Run describe() and look for impossible values
  4. Plot the target as a histogram, and each feature against it
  5. Draw the correlation heatmap with numeric_only=True
  6. Drop columns a model cannot use, such as free text or IDs

Part 5

Practice: Your Turn

About 30 minutes, answers follow each task

About 10 minutes

Practice 1: Fit a Line by Hand

Areax (income, $10,000)y (price, $100,000)
A146
A257
A3610
A4711
A5811
  • Compute x̄, ȳ, the slope b₁ and the intercept b₀
  • Compute the residuals, SSE, MSE, RMSE, MAE and R²
  • Predict the price for an income of $65,000

Answers

Practice 1: Answer, the Line

b1 = 6 + 2 + 0 + 2 + 44 + 1 + 0 + 1 + 4 = 1410 = 1.4
b0 = ȳ − b1 x̄ = 9 − 1.4 × 6 = 0.6

Answers

Practice 1: Answer, the Error

Areaŷee²
A16.2-0.20.04
A27.6-0.60.36
A39.01.01.00
A410.40.60.36
A511.8-0.80.64
SSE = 2.40, MSE = 0.48, RMSE = 0.6928
MAE = 0.64, SST = 22, R2 = 1 − 2.4022 = 0.8909

About 5 minutes

Practice 2: A Second Test House

Row 2456, real price $707,345.06, intercept -2,631,028.90. Predict, then find the residual.

FeaturexCoefficient b
Income67041.96766121.597602
House age6.021458165201.104954
Rooms5.346830119061.463868
Bedrooms3.393212.585606
Population15633.09904815.228121
ŷ = −2,631,028.90 + 3,328,252.05 = 697,223.15

About 5 minutes

Practice 3: Read the Coefficients

  • Two areas differ only by 2 years of house age. How far apart are the predictions?
  • They differ only by $5,000 of income. How far apart now?
  • Why does the bedrooms coefficient, 3212.59, not mean bedrooms matter more than income?
2 × 165,201.10 ≈ 330,402, 5,000 × 21.597602 ≈ 107,988

About 10 minutes

Practice 4: In Colab

  1. Run the notebook with the three fixes of today, up to the error measures
  2. Fit lm1 on X_train[['Avg. Area Income']] only: print intercept, coefficient, test R², MAE, RMSE
  3. Fit a model on all features except Avg. Area Number of Bedrooms (drop(..., axis=1)) and print its test R²
  4. Compare the three R² values: what do they say about the features?

Answers

Practice 4: Answer

ModelTest R²Test RMSE
All five features0.919102,418.94
Income only0.417275,407.48
All except bedrooms0.920102,282.54

Key Takeaways

  1. Linear regression predicts a number as b0 plus a weighted sum of the features
  2. A residual is y − ŷ; least squares makes the sum of their squares smallest
  3. Explore and clean first: types, missing values, text columns, correlations
  4. A coefficient is the change per unit of its feature: rank features by t-statistic
  5. Judge the model on the test set with MAE, RMSE, R² and the residual plots

Open this lesson

Mahmoud AbasLinear Regression