Mahmoud Abas, Full-Stack Software Engineer & AI

Arabic NLP - Text Normalization

In the print dialog, choose "Save as PDF" as the destination.

Advanced NLP - Lesson 2

Text Normalization

Reducing text variation with stemming, lemmatization, and Arabic-specific normalization

The Problem: Text Variation

The same word can appear as many surface variations: lovely, luv, loveee
To a model, these should often be treated as the same underlying concept
Preprocessing must reduce variation before feature extraction or modeling
This lesson covers stemming, lemmatization, and Arabic normalization

Key Concept

Lemmatization vs Stemming

Stemming: Cuts word endings and beginnings using fixed rules, which can destroy the word

Lemmatization: Uses morphological analysis and keeps the real dictionary word, the lemma

Lemmatization is more informative: a stem may not be a real word, a lemma always is
Trade-off: stemming is faster and simpler; lemmatization is more accurate

Stemming

Porter Stemmer

Reduces text using predefined rules, for example removing ing from verbs

python

from nltk.stem.porter import PorterStemmer
stemmer = PorterStemmer()
singles = [stemmer.stem(w) for w in plurals]
# 'caresses' -> 'caress', 'flies' -> 'fli', 'meeting' -> 'meet'

Results like flies to fli show stemming can produce non-words

Lemmatization

WordNetLemmatizer

Looks up the word in WordNet to find its dictionary root

python

from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
lemmatizer.lemmatize("rocks")            # "rock"
lemmatizer.lemmatize("better", pos="a")  # "good"

The pos argument matters: default noun assumption vs. adjective form

Arabic Character Normalization

Standardizes different Arabic letter forms into one canonical form

python

def normalize(sentence):
    sentence = re.sub("[إأآا]", "ا", sentence)
    sentence = re.sub("ى", "ي", sentence)
    sentence = re.sub("ؤ", "ء", sentence)
    sentence = re.sub("ئ", "ء", sentence)
    sentence = re.sub("ة", "ه", sentence)
    return sentence

Removes glyph variation (hamza forms, alef forms, ta marbuta) before further processing

Function

Removing Arabic Stopwords

Tokenize, then drop any word found in NLTK's Arabic stopword list

python

def removing_ar_stopwords(text):
    words = word_tokenize(text)
    return " ".join(w for w in words if w not in stopwords)

Applied per-document, before stemming, in the full pipeline

Arabic Stemming

ISRIStemmer

Built into NLTK; strips prefixes and suffixes using Arabic-specific rules

python

from nltk import ISRIStemmer
st = ISRIStemmer()
stemmed = " ".join(st.stem(w) for w in word_tokenize(text))

Arabic Stemming

Tashaphyne (Arabic Light Stemmer)

Third-party library; light stemming (removes prefixes and suffixes) with possible segmentations

python

from tashaphyne.stemming import ArabicLightStemmer
ArListem = ArabicLightStemmer()
ArListem.light_stem(word)
stem = ArListem.get_stem()

Decision

Stemming vs Lemmatization: Which to Use

Approach	Speed	Word Validity	Arabic Tools
Stemming	Fast, rule-based	May destroy words	`ISRIStemmer`, `Tashaphyne`
Lemmatization	Slower, morphology-aware	Keeps real words	`Farasa` (external)

Experimentally, Farasa lemmatization gave the best results for Arabic text
Choice depends on the downstream task and available tooling

End-to-End Pipeline on Real Data

Applied to a Hadith corpus (Malik's Muwatta), loaded via pandas

Raw hadith text

Remove stopwords

removing_ar_stopwords

Normalize characters

normalize

Stem

stemming_1, ISRIStemmer

Cleaned column

python

cleared = [removing_ar_stopwords(h) for h in hadiths]
cleared = [normalize(h) for h in cleared]
cleared = [stemming_1(h) for h in cleared]  # ISRIStemmer

Pipeline order: remove stopwords, then normalize characters, then stem
Result stored back into a DataFrame column for downstream modeling

Key Takeaways

Stemming destroys word structure with fixed rules; lemmatization preserves real dictionary words
PorterStemmer (English) and ISRIStemmer / Tashaphyne (Arabic) are common stemmers
Arabic character normalization (alef, hamza, ta marbuta forms) must happen before stemming
Stopword removal, normalization, and stemming form a standard Arabic preprocessing pipeline
Farasa lemmatization outperformed the tested stemmers on Arabic text in this lesson

Open this lesson

Mahmoud AbasText Normalization

def normalize(sentence): sentence = re.sub("[إأآا]", "ا", sentence) sentence = re.sub("ى", "ي", sentence) sentence = re.sub("ؤ", "ء", sentence) sentence = re.sub("ئ", "ء", sentence) sentence = re.sub("ة", "ه", sentence) return sentence

Approach

Speed

Word Validity

Arabic Tools

Stemming

Fast, rule-based

May destroy words

ISRIStemmer, Tashaphyne

Lemmatization

Slower, morphology-aware

Keeps real words

Farasa (external)