Mahmoud Abas, Full-Stack Software Engineer & AI

Arabic NLP - CAMeL Tools Documentation

In the print dialog, choose "Save as PDF" as the destination.

CAMeL Tools Documentation

A tour of CAMeL Tools: normalization, morphology, disambiguation, and pretrained Arabic NLP models

What is CAMeL Tools?

CAMeL Tools: An open-source Python toolkit for Arabic natural language processing.

Docs at camel-tools.readthedocs.io, source at github.com/CAMeL-Lab/camel_tools
Covers normalization, tokenization, morphology, disambiguation, POS tagging, sentiment analysis, and NER

bash

pip install camel-tools
camel_data -i all   # install all pretrained databases/models

camel_data -i all installs every pretrained database and model.

Unicode Normalization

Expands special Unicode ligature characters into their full Arabic text

python

from camel_tools.utils.normalize import normalize_unicode

sent_norm = normalize_unicode('ﷺ')
print(sent_norm)
# 'صلى الله عليه وسلم'

Orthographic Normalization

Standardizes alef, alef-maksura, and teh-marbuta variants into one canonical form

python

from camel_tools.utils.normalize import (
    normalize_alef_ar, normalize_alef_maksura_ar, normalize_teh_marbuta_ar)

sent_norm = normalize_alef_ar("هل ذهبت إلى المكتبة؟")
sent_norm = normalize_alef_maksura_ar(sent_norm)
sent_norm = normalize_teh_marbuta_ar(sent_norm)
# 'هل ذهبت الي المكتبه؟'

Dediacritization & Tokenization

Dediacritization: Strips diacritics (tashkeel) from text.

python

from camel_tools.utils.dediac import dediac_ar
dediac_ar("هَلْ ذَهَبْتَ إِلَى المَكْتَبَةِ؟")
# 'هل ذهبت إلى المكتبة؟'

Tokenization: simple_word_tokenize splits on whitespace and punctuation.

python

from camel_tools.tokenizers.word import simple_word_tokenize
simple_word_tokenize("هَلْ ذَهَبْتَ")
# ['هَلْ', 'ذَهَبْتَ']

Transliteration

Converts Arabic script to the Buckwalter transliteration scheme (Latin characters) via a CharMapper

python

from camel_tools.utils.charmap import CharMapper

ar2bw = CharMapper.builtin_mapper('ar2bw')
ar2bw("ذهبت إلى المكتبة.")
# '*hbt <lY Almktbp.'

Morphological Analysis

Given a word, returns every possible morphological analysis: diacritized form, lemma, POS, root, gloss, and features

python

from camel_tools.morphology.database import MorphologyDB
from camel_tools.morphology.analyzer import Analyzer

db = MorphologyDB.builtin_db()
analyzer = Analyzer(db)
analyses = analyzer.analyze('وسيكتبونها')

A single word can yield many candidate analyses, disambiguation picks the right one in context

Morphological Generation & Reinflection

Generation: Produce surface forms from a lemma plus target features.

python

generator = Generator(MorphologyDB.builtin_db(flags='g'))
generator.generate('مُوَظَّف', {'pos': 'noun', 'gen': 'm', 'num': 'p'})
# 'مُوَظَّفُونَ', 'مُوَظَّفِينَ', ...

Reinflection: Change an existing word's features, for example singular to dual.

python

reinflector = Reinflector(MorphologyDB.builtin_db(flags='r'))
reinflector.reinflect('شوارع', {'num': 'd', 'prc1': 'bi_prep'})
# 'بِشارِعَيْنِ', 'بِشارِعَيْ'

Morphological Disambiguation & POS Tagging

MLEDisambiguator picks the most likely analysis for each word in context (MLE = maximum likelihood estimate)
Used to derive diacritization, POS tags, and lemmas for a full sentence

python

mle = MLEDisambiguator.pretrained()
disambig = mle.disambiguate('سوف نقرأ الكتب'.split())
# ('سَوْفَ', 'part_fut', 'سَوْفَ')
# ('نَقْرَأ', 'verb', 'قَرَأ')
# ('الكُتُبِ', 'noun', 'كِتاب')

Morphological Tokenization

Splits words into morphemes (prefixes, stems, suffixes) using different tokenization schemes
Separate MSA and Egyptian dialect disambiguators feed the tokenizer

Scheme	What it produces
`d3tok`	Diacritized morpheme segments
`atbtok`	ATB-style segments, undiacritized
`bwtok`	Buckwalter-style segments

python

mle_msa = MLEDisambiguator.pretrained('calima-msa-r13')
tokenizer = MorphologicalTokenizer(disambiguator=mle_msa, scheme='atbtok')
tokenizer.tokenize(['فتنفست', 'الصعداء'])
# ['ف+_تنفست', 'الصعداء']

Pretrained Sentiment Analysis

Ready-to-use sentiment classifier, no training required

python

from camel_tools.sentiment import SentimentAnalyzer

sa = SentimentAnalyzer.pretrained()
sa.predict_sentence('أنا بخير')          # 'positive'
sa.predict(['أنا بخير', 'أنا لست بخير'])  # ['positive', 'negative']

Pretrained Named Entity Recognition

BERT-based NER tagger; input must be pre-tokenized by whitespace and punctuation
Uses IOB-style tags (B-LOC, I-LOC, O)

python

from camel_tools.ner import NERecognizer

ner = NERecognizer.pretrained()
sentence = 'إمارة أبوظبي هي إحدى إمارات دولة الإمارات العربية المتحدة السبع .'.split()
labels = ner.predict_sentence(sentence)
# ('أبوظبي', 'B-LOC'), ('الإمارات', 'B-LOC'), ('العربية', 'I-LOC'), ('المتحدة', 'I-LOC')

Key Takeaways

Normalization

Tokenization

Morphology

Disambiguation

Tagging

Normalization functions (Unicode, orthographic, dediacritization) standardize noisy Arabic text before modeling
Morphological analysis, generation, and reinflection work with lemmas, roots, and grammatical features, not just surface words
MLEDisambiguator resolves ambiguous word analyses using sentence context
Pretrained models ship out of the box for sentiment analysis and NER, no training required

Open this lesson

Mahmoud AbasCAMeL Tools Documentation

from camel_tools.utils.normalize import ( normalize_alef_ar, normalize_alef_maksura_ar, normalize_teh_marbuta_ar) sent_norm = normalize_alef_ar("هل ذهبت إلى المكتبة؟") sent_norm = normalize_alef_maksura_ar(sent_norm) sent_norm = normalize_teh_marbuta_ar(sent_norm) # 'هل ذهبت الي المكتبه؟'

from camel_tools.morphology.database import MorphologyDB from camel_tools.morphology.analyzer import Analyzer db = MorphologyDB.builtin_db() analyzer = Analyzer(db) analyses = analyzer.analyze('وسيكتبونها')

mle = MLEDisambiguator.pretrained() disambig = mle.disambiguate('سوف نقرأ الكتب'.split()) # ('سَوْفَ', 'part_fut', 'سَوْفَ') # ('نَقْرَأ', 'verb', 'قَرَأ') # ('الكُتُبِ', 'noun', 'كِتاب')

Scheme

What it produces

d3tok

Diacritized morpheme segments

atbtok

ATB-style segments, undiacritized

bwtok

Buckwalter-style segments

mle_msa = MLEDisambiguator.pretrained('calima-msa-r13') tokenizer = MorphologicalTokenizer(disambiguator=mle_msa, scheme='atbtok') tokenizer.tokenize(['فتنفست', 'الصعداء']) # ['ف+_تنفست', 'الصعداء']

from camel_tools.sentiment import SentimentAnalyzer sa = SentimentAnalyzer.pretrained() sa.predict_sentence('أنا بخير') # 'positive' sa.predict(['أنا بخير', 'أنا لست بخير']) # ['positive', 'negative']

from camel_tools.ner import NERecognizer ner = NERecognizer.pretrained() sentence = 'إمارة أبوظبي هي إحدى إمارات دولة الإمارات العربية المتحدة السبع .'.split() labels = ner.predict_sentence(sentence) # ('أبوظبي', 'B-LOC'), ('الإمارات', 'B-LOC'), ('العربية', 'I-LOC'), ('المتحدة', 'I-LOC')