Mahmoud Abas, Full-Stack Software Engineer & AI

Arabic NLP - Intro To NN

In the print dialog, choose "Save as PDF" as the destination.

Arabic NLP - Lesson 4

Intro to NN

Building an Arabic sentiment classifier with Keras: trainable vs. pretrained embeddings and LSTM layers

Environment Setup

Core stack: tensorflow, keras, numpy, pandas, scikit-learn, gensim, nltk
Arabic stemming via NLTK's ISRIStemmer

python

!pip install tensorflow keras numpy pandas scikit-learn gensim requests tqdm nltk
import nltk
nltk.download('stopwords')
from nltk.stem.isri import ISRIStemmer

The Dataset

Arabic Sentiment Analysis Dataset (SS2030) - Arabic tweets with binary sentiment labels
Loaded from Google Drive as CSV, missing values dropped

python

df = pd.read_csv(csv_path, encoding="utf-8")
df = df.dropna()
# columns: text, Sentiment (1 = positive)

Arabic Preprocessing Pipeline

Raw tweet

Strip punctuation

، . ؟ ! and newlines

Tokenize

split on whitespace

Remove stopwords

Arabic stopword list

Stem

ISRIStemmer

Processed text

python

stemmer = ISRIStemmer()
stop_words = set(stopwords.words("arabic"))

def preprocess_arabic_text(text):
    text = text.replace("،", "").replace(".", "").replace("؟", "").replace("!", "")
    words = text.split()
    words = [stemmer.stem(w) for w in words if w not in stop_words]
    return " ".join(words)

Tokenization & Padding

Keras Tokenizer builds a word index capped at the top 10K words
Sequences are padded/truncated to a fixed length of 100 tokens
Data split 80/20 with train_test_split

python

tokenizer = Tokenizer(num_words=10000)
tokenizer.fit_on_texts(df["processed_text"])
X_sequences = tokenizer.texts_to_sequences(df["processed_text"])
X_padded = pad_sequences(X_sequences, maxlen=100)

Model 1: Trainable Embedding + LSTM

Embedding layer weights are learned from scratch during training

Embedding

10000 x 300, trainable=True

LSTM(64)

return_sequences=True

LSTM(32)

Dense(1)

sigmoid

python

trainable_model = Sequential([
    Embedding(input_dim=10000, output_dim=300, trainable=True),
    LSTM(64, return_sequences=True),
    LSTM(32),
    Dense(1, activation='sigmoid')
])
trainable_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

Model 1: Training Results

Trained 5 epochs, batch size 32
Training accuracy quickly reaches ~99%, a sign of overfitting on a small dataset
Final test accuracy: `0.85`

text

Epoch 5/5 - accuracy: 0.9962 - val_accuracy: 0.8496
Trainable Embedding Model Accuracy: 0.85

Pretrained Embeddings: AraVec

AraVec: A pretrained Word2Vec model for Arabic, trained on Twitter data (CBOW, 300 dimensions)

Downloaded as a zip archive and loaded with gensim

python

word2vec_model = gensim.models.Word2Vec.load(
    './aravec/full_grams_cbow_300_twitter.mdl')

Building the Embedding Matrix

Start with a zero matrix sized (vocab_size + 1) x 300
For every word in the tokenizer's word index, look it up in AraVec
Found in AraVec: copy its vector into that word's row
Not found: the row stays all zeros

python

embedding_matrix = np.zeros((len(tokenizer.word_index) + 1, embedding_dim))
for word, i in tokenizer.word_index.items():
    if word in word2vec_model.wv:
        embedding_matrix[i] = word2vec_model.wv[word]
# Embedding matrix shape: (11710, 300)

Model 2: Frozen Pretrained Embedding + LSTM

Embedding layer is initialized with AraVec vectors and frozen (trainable=False)
Deeper stack: 2 LSTM layers with Dropout, plus an extra Dense layer

Embedding

AraVec weights, trainable=False

LSTM(128)

return_sequences=True

Dropout(0.3)

LSTM(64)

Dense(32)

relu

Dense(1)

sigmoid

python

pretrained_model = Sequential([
    Embedding(input_dim=len(tokenizer.word_index) + 1, output_dim=300,
              weights=[embedding_matrix], trainable=False),
    LSTM(128, return_sequences=True),
    Dropout(0.3),
    LSTM(64),
    Dense(32, activation='relu'),
    Dense(1, activation='sigmoid')
])

Model 2: Training Results

Same 5 epochs, batch size 32
Training accuracy rises more gradually, less overfitting than Model 1
Final test accuracy: `0.86`

text

Epoch 5/5 - accuracy: 0.9760 - val_accuracy: 0.8555
Pretrained Word2Vec Model Accuracy: 0.86

Comparing the Two Approaches

Model	Test Accuracy
Trainable embedding (from scratch)	`0.85`
Pretrained `AraVec` (frozen)	`0.86`

Pretrained embeddings slightly outperform trainable-from-scratch embeddings
Pretrained + frozen embeddings also show less overfitting during training
Dropout in Model 2 further regularizes the deeper LSTM stack

Key Takeaways

Preprocess

Tokenize / pad

Embed

LSTM

Classify

Preprocessing combines punctuation stripping, stopword removal, and ISRI stemming
Trainable embeddings learn from your data but can overfit small datasets
Pretrained AraVec embeddings (frozen) transfer external knowledge and slightly boost accuracy
Stacking LSTM layers with Dropout helps build deeper, more regularized sequence models

Open this lesson

Mahmoud AbasIntro To NN

stemmer = ISRIStemmer() stop_words = set(stopwords.words("arabic")) def preprocess_arabic_text(text): text = text.replace("،", "").replace(".", "").replace("؟", "").replace("!", "") words = text.split() words = [stemmer.stem(w) for w in words if w not in stop_words] return " ".join(words)

trainable_model = Sequential([ Embedding(input_dim=10000, output_dim=300, trainable=True), LSTM(64, return_sequences=True), LSTM(32), Dense(1, activation='sigmoid') ]) trainable_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

embedding_matrix = np.zeros((len(tokenizer.word_index) + 1, embedding_dim)) for word, i in tokenizer.word_index.items(): if word in word2vec_model.wv: embedding_matrix[i] = word2vec_model.wv[word] # Embedding matrix shape: (11710, 300)

pretrained_model = Sequential([ Embedding(input_dim=len(tokenizer.word_index) + 1, output_dim=300, weights=[embedding_matrix], trainable=False), LSTM(128, return_sequences=True), Dropout(0.3), LSTM(64), Dense(32, activation='relu'), Dense(1, activation='sigmoid') ])

Model

Test Accuracy

Trainable embedding (from scratch)

0.85

Pretrained AraVec (frozen)

0.86