Mahmoud Abas, Full-Stack Software Engineer & AI

Machine Learning & AI - Lab 3: Data Preprocessing and Clustering with Real-World Data using Pandas DataFrames

In the print dialog, choose "Save as PDF" as the destination.

Machine Learning & AI - Lab 3

Data Preprocessing and Clustering with Real-World Data using Pandas

From raw DataFrames to a K-Means vs. SOM comparison on the Iris dataset

Lab Objectives

Load and explore real datasets with pandas
Master DataFrame indexing, filtering, and selection
Preprocess data: handle missing values, encode categories, and normalize features
Apply K-Means and SOM to real-world data with proper preparation
Compare clustering algorithms and interpret their results

What is a DataFrame?

DataFrame: A 2D labeled data structure, like a spreadsheet or SQL table.

Rows and columns carry labels; columns can hold different types
Size-mutable: add or remove rows and columns freely

python

sample_data = {'Name': ['Ahmed','Fatima'], 'Age': [22,21], 'Grade': [85,92]}
df = pd.DataFrame(sample_data)
df.shape          # (rows, columns)
df.columns.tolist()
df.dtypes

Loading the Iris Dataset

Classic ML dataset: 3 species (setosa, versicolor, virginica), 4 features
Load via sklearn.datasets.load_iris() or pd.read_csv()

python

iris = load_iris()
df_iris = pd.DataFrame(data=iris.data, columns=iris.feature_names)
df_iris['species'] = iris.target
df_iris['species_name'] = df_iris['species'].map(
    {0: 'setosa', 1: 'versicolor', 2: 'virginica'})

Result: 150 rows x 6 columns

Exploratory Data Analysis

df.describe(): count, mean, std, min/max, quartiles per column
df.dtypes and df.isnull().sum(): types and missing-value check
df['species_name'].value_counts(): 50 samples per species, perfectly balanced
No missing values detected in this dataset

Selection and Filtering

Column selection: df[['sepal length (cm)', 'sepal width (cm)']]
Row filtering: df[df['species_name'] == 'setosa']
Multiple conditions combine with &

python

filtered = df_iris[(df_iris['species_name'] == 'setosa') &
                    (df_iris['sepal length (cm)'] > 5.0)]

Feature Normalization

Features have different scales (sepal length 4 to 8 cm vs petal width 0.1 to 2.5 cm)

z = x - μσ

StandardScaler centers on mean 0 and scales to std 1 after transform

python

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

Finding the Optimal K

Sweep k

2 to 10

Track WCSS + Silhouette

at each k

Read the elbow plot

diminishing returns

Choose k = 3

matches true species count

python

for k in range(2, 11):
    km = KMeans(n_clusters=k, init='k-means++', n_init=10, random_state=42)
    km.fit(X_scaled)
    wcss.append(km.inertia_)
    silhouette_scores.append(silhouette_score(X_scaled, km.labels_))

Training K-Means

Fit with k=3 on the scaled features, then attach labels back to the DataFrame

python

kmeans_final = KMeans(n_clusters=3, init='k-means++', n_init=10, random_state=42)
df_iris['kmeans_cluster'] = kmeans_final.fit_predict(X_scaled)

Result: cluster sizes [53, 50, 47], Silhouette Score = 0.459

Training a Self-Organizing Map

sklearn_som.SOM: a scikit-learn-compatible SOM implementation
Grid 3x1 for 3 target clusters, 4 input dimensions

python

som_model = SOM(m=3, n=1, dim=4, lr=0.5, max_iter=1000, random_state=42)
som_model.fit(X_scaled)
df_iris['som_cluster'] = som_model.predict(X_scaled)

Result: cluster sizes [70, 30, 50], Silhouette Score = 0.412

K-Means vs. SOM: Cross-Tabulation

pd.crosstab(species_name, cluster) shows how well clusters match true species

	K-Means	SOM
Setosa isolation	Perfect (50/50)	Perfect (50/50)
Versicolor / virginica	Some overlap	More overlap
Silhouette Score	0.459	0.412

`K-Means` wins on Silhouette Score here

Grouping, Aggregation and Derived Features

groupby('species_name')[cols].mean(): per-group statistics
.agg(['mean', 'std', 'min', 'max']): multiple stats at once
New derived columns: sepal_area = length * width, sepal_to_petal_ratio

python

df_iris['petal_area'] = df_iris['petal length (cm)'] * df_iris['petal width (cm)']

Lab Summary

Item	Value
Dataset	150 flowers, 4 features, 3 true species classes
Preprocessing	Feature selection + `StandardScaler`, zero missing values
K-Means (k=3)	Silhouette `0.459`, best performer
SOM (3x1 grid)	Silhouette `0.412`

Both algorithms find natural groupings, but K-Means edges out SOM here

Key Takeaways

DataFrames give powerful, readable tools for real-world data manipulation
Feature scaling is essential before any distance-based clustering
Both K-Means and SOM can uncover natural structure in real data
Cross-tabulation against known labels is a great way to sanity-check clusters
Visualization plus metrics together validate what "good clustering" means

Open this lesson

Mahmoud AbasLab 3: Data Preprocessing and Clustering with Real-World Data using Pandas DataFrames

iris = load_iris() df_iris = pd.DataFrame(data=iris.data, columns=iris.feature_names) df_iris['species'] = iris.target df_iris['species_name'] = df_iris['species'].map( {0: 'setosa', 1: 'versicolor', 2: 'virginica'})

for k in range(2, 11): km = KMeans(n_clusters=k, init='k-means++', n_init=10, random_state=42) km.fit(X_scaled) wcss.append(km.inertia_) silhouette_scores.append(silhouette_score(X_scaled, km.labels_))

K-Means

SOM

Setosa isolation

Perfect (50/50)

Versicolor / virginica

Some overlap

More overlap

Silhouette Score

0.459

0.412

Item

Value

Dataset

150 flowers, 4 features, 3 true species classes

Preprocessing

Feature selection + StandardScaler, zero missing values

K-Means (k=3)

Silhouette 0.459, best performer

SOM (3x1 grid)

Silhouette 0.412