Mahmoud Abas, Full-Stack Software Engineer & AI

Arabic NLP - Arabic Corpora: A Key Resource for Arabic NLP

In the print dialog, choose "Save as PDF" as the destination.

Arabic NLP - Lesson 1

Arabic Corpora: A Key Resource for Arabic NLP

Understanding corpora, tokenization, segmentation, and stopword removal for Arabic text.

What is an Arabic Corpus?

Corpus: Plural corpora: a large collection of Arabic text used for training and evaluating NLP models.

Powers tasks like tokenization, POS tagging, NER, sentiment analysis, machine translation
Corpus quality and domain shape downstream model performance
Arabic corpora vary by domain, purpose, and structure

Corpora Types: General-Purpose & Speech

Category	Examples	Supports
General-purpose	Arabic Gigaword (news), Arabic Web Corpus (crawled text), Tashkeela Corpus (diacritics/Tashkeel)	Tokenization, parsing, embeddings
Speech & conversational	MGB-2 (TV broadcasts), CALLHOME Arabic (phone calls), ArzEn (Egyptian Arabic-English bilingual speech)	Recognition, chatbots, dialogue systems

Corpora Types: Sentiment, Religious & Historical

Category	Examples
Sentiment/opinion	ASTD (annotated tweets), LABR (book reviews), OCA (movie reviews)
Quranic/religious	Quranic Arabic Corpus (annotated Quran), Al-Hadith Corpus
Historical/literary	Al-Khalil Corpus (classical texts), Doha Historical Dictionary of Arabic

Each domain corpus enables specialized NLP research and tools

Text Normalization Tasks (Overview)

Tokenization

Splitting text into words or subwords

Segmentation

Dividing text into sentences or phrases

Stemming

Reducing words to root form by removing affixes

Lemmatization

Converting words to base dictionary form

These are the essential preprocessing steps before any Arabic NLP task

Tokenization: Three Methods

Tokenization breaks a phrase, sentence, or document into tokens (words, subwords, punctuation)
Method 1: Python's built-in split() - basic, whitespace-based
Method 2: Regular expressions - re.findall() / re.split()
Method 3: NLP library - nltk.word_tokenize (or Farasa for Arabic)
Preprocessing first: strip punctuation with regex before tokenizing

python

import re
pre_processed = re.sub(r'[^\w\s]', '', text)  # keep only whitespace & words

Method 1

Python `split()`

Splits text purely on whitespace
Fast, but does not handle punctuation or special cases

python

tokens_1 = pre_processed.split()
# ['إن', 'التاريخ', 'العسكري', 'سوف', 'يتوقف', ...]

Method 2

Regular Expressions

re.findall(PATTERN, TEXT) extracts tokens matching a pattern
Pattern \w+ matches word characters, giving a result similar to split() here

python

PATTERN = r"\w+"
tokens_2 = re.findall(PATTERN, pre_processed)

Method 3

NLTK `word_tokenize`

Context-aware, language-specific tokenization
Requires the punkt_tab pretrained model (replaces the deprecated punkt)

python

import nltk
nltk.download('punkt_tab')
from nltk.tokenize import word_tokenize
tokens_3 = word_tokenize(pre_processed)

Sentence Segmentation with NLTK

Segmentation divides a paragraph into individual sentences
Uses NLTK's Punkt Sentence Tokenizer

python

def segment(text):
    segmenter = nltk.data.load("tokenizers/punkt/english.pickle")
    sents = segmenter.tokenize(text)
    for sent in sents:
        print(sent)

Example paragraph was correctly segmented into 2 sentences at the period boundary

Removing Arabic Stopwords

Stopwords: Common words (the, and, is) that add little meaning to text analysis.

NLTK's Arabic stopword list has 754 entries
Tokenize first, then filter out stopwords

python

from nltk.corpus import stopwords
stopwords = stopwords.words('arabic')
X_set = {w for w in word_tokenize(X) if w not in stopwords}

Example: أحب المشي على الشاطئ becomes the set of words المشي, الشاطئ, أحب

Key Takeaways

Arabic corpora come in many flavors - general, speech, sentiment, religious, historical
Text normalization = tokenization + segmentation + stemming + lemmatization
Three tokenization approaches: split(), regex, and NLTK - each trades simplicity for accuracy
NLTK's Punkt tokenizer segments paragraphs into sentences
Removing stopwords (754 Arabic stopwords in NLTK) reduces noise before downstream tasks

Open this lesson

Mahmoud AbasArabic Corpora: A Key Resource for Arabic NLP

Arabic Corpora: A Key Resource for Arabic NLP

What is an Arabic Corpus?

Corpora Types: General-Purpose & Speech

Corpora Types: Sentiment, Religious & Historical

Text Normalization Tasks (Overview)

Tokenization: Three Methods

Python split()

Regular Expressions

NLTK word_tokenize

Sentence Segmentation with NLTK

Removing Arabic Stopwords

Key Takeaways

Open this lesson

Arabic Corpora: A Key Resource for Arabic NLP

What is an Arabic Corpus?

Corpora Types: General-Purpose & Speech

Corpora Types: Sentiment, Religious & Historical

Text Normalization Tasks (Overview)

Tokenization: Three Methods

Python split()

Regular Expressions

NLTK word_tokenize

Sentence Segmentation with NLTK

Removing Arabic Stopwords

Key Takeaways

Open this lesson

Python `split()`

NLTK `word_tokenize`

Python `split()`

NLTK `word_tokenize`