Mahmoud Abas, Full-Stack Software Engineer & AI

Arabic NLP - Types of sequence problems

In the print dialog, choose "Save as PDF" as the destination.

Arabic NLP - Lesson 6

Types of Sequence Problems

Why order matters, sequence problem types, and RNN, GRU, LSTM architectures

Why Recurrent Networks?

A feed-forward network treats each word independently - its activation is a fixed weighted sum of the words, regardless of order
Shuffling the words in a sentence would not change a feed-forward network's output at all
Recurrent Neural Networks (RNNs) feed input sequentially through time
Each step's hidden state is computed from the current input and the previous hidden state - so word order matters

Types of Sequence Problems

Type	Examples
One to Many	song generator, text generator
Many to One	sentiment analysis, voice verification
Many to Many (aligned)	named entity recognition, part-of-speech tagging
Many to Many (encoder-decoder)	machine translation, question answering

One-to-one problems do not need a sequence model at all

Inside the RNN Block

Each RNN cell computes a hidden state h_t from the current input and the previous hidden state
The hidden state can optionally feed a sigmoid/softmax output layer, producing an additional output a_t
This recurrence is what lets the network remember earlier parts of the sequence

Input

xₜ

RNN cell

combines input with previous hidden state

Hidden state

hₜ

sigmoid / softmax

Output

aₜ, optional

Preparing RNN Inputs: Padding

Padding: All inputs must be padded to the same length before batching.

Example: the sentence the product was very good with few problems needs 8 steps
A shorter input like a good product (3 steps) gets padded with PAD tokens to match the longest sequence length

Input Representation

Input representation: RNN inputs are represented as embeddings, the same embedding techniques used at the character, word, or sentence level.

Embeddings can be learned from scratch or loaded from a pretrained model, e.g. Word2Vec, GloVe, AraVec

Multi-Layer & Bidirectional RNNs

Layer 1

builds a representation from the raw input

Layer 2

builds a richer representation on top of layer 1's output

Multi-layer (stacked) RNN: Cells are stacked so the second layer builds a richer representation on top of the first layer's output.

Bidirectional RNN (BiRNN): Captures both past and future context by processing the sequence in both directions.

Building an RNN with TensorFlow

Example task: IMDB sentiment classification (binary), from tf.keras.datasets.imdb
Sequences are padded/truncated to a fixed length of 100 tokens before training

python

model = tf.keras.models.Sequential([
    tf.keras.layers.Embedding(input_dim=VOCAB_SIZE+1, output_dim=100),
    tf.keras.layers.SimpleRNN(64),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

Result: about 60% test accuracy after 5 epochs - a plain SimpleRNN struggles on this task

Bidirectional SimpleRNN

Wrapping the recurrent layer in Bidirectional lets the model see both past and future context

python

model_bi = tf.keras.models.Sequential([
    tf.keras.layers.Embedding(VOCAB_SIZE, 64),
    tf.keras.layers.Bidirectional(tf.keras.layers.SimpleRNN(64)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

RNN's Weakness: Vanishing Gradients

Two gated architectures were proposed to fix this: GRU and LSTM

Gated Recurrent Unit (GRU)

GRU: Adds an update gate that lets the network choose what to keep from the previous state.

C̃ = tanh(W_c · [C^t-1, X^t] + b_c)

Γ_u = σ(W_u · [C^t-1, X^t] + b_u)

C^t = Γ_u · C̃ + (1 - Γ_u) · C^t-1

python

gru_bi = tf.keras.models.Sequential([
    tf.keras.layers.Embedding(VOCAB_SIZE, 64),
    tf.keras.layers.Bidirectional(tf.keras.layers.GRU(64, return_sequences=True)),
    tf.keras.layers.Bidirectional(tf.keras.layers.GRU(64)),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

GRU swaps in for SimpleRNN inside the same Bidirectional wrapper.

Long Short-Term Memory (LSTM)

LSTM: Adds a long-term memory component alongside the short-term one, plus three gates: update, forget, and output.

C̃ = tanh(W_c · [a^t-1, x^t] + b_c)

C^t = Γ_u · C̃ + Γ_f · C^t-1

a^t = Γ_o · tanh(C^t)

Keep the forget gate's share of the past, add the update gate's share of the new candidate
The three gates let LSTM retain information from far earlier in the sequence, solving the vanishing-gradient problem

Bidirectional LSTM

Same bidirectional idea as RNN/GRU - just swap in an LSTM cell
No fixed max length needed anymore; sequences can be padded to their true max length

python

lstm_bi = tf.keras.models.Sequential([
    tf.keras.layers.Embedding(VOCAB_SIZE, 64),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

Predicting on New Text

Save and reload a trained model, then run inference on raw text

python

def predict(text, clf, word2idx, padding_size):
    padded_text = np.zeros((padding_size))
    padded_text[:len(text.split())] = [word2idx.get(w, 0) for w in text.split()]
    return clf.predict(tf.expand_dims(padded_text, 0))

predict("the movie was super awesome!", lstm_bi, word2idx, lstm_padding)

Key Takeaways

RNNs process sequences step by step, so, unlike feed-forward networks, word order changes the output
Sequence problems fall into four shapes: one-to-many, many-to-one, aligned many-to-many, and encoder-decoder many-to-many
Inputs must be padded to a common length and represented as embeddings before entering the RNN
Plain SimpleRNN suffers from vanishing gradients and forgets early context on long sequences
GRU (update gate) and LSTM (update/forget/output gates) fix this, with LSTM adding an explicit long-term memory
Bidirectional wrappers let any of these cells use both past and future context, at the cost of needing the full sequence first

Open this lesson

Mahmoud AbasTypes of sequence problems

Type

Examples

One to Many

song generator, text generator

Many to One

sentiment analysis, voice verification

Many to Many (aligned)

named entity recognition, part-of-speech tagging

Many to Many (encoder-decoder)

machine translation, question answering

model = tf.keras.models.Sequential([ tf.keras.layers.Embedding(input_dim=VOCAB_SIZE+1, output_dim=100), tf.keras.layers.SimpleRNN(64), tf.keras.layers.Dense(64, activation='relu'), tf.keras.layers.Dense(1, activation='sigmoid') ])

model_bi = tf.keras.models.Sequential([ tf.keras.layers.Embedding(VOCAB_SIZE, 64), tf.keras.layers.Bidirectional(tf.keras.layers.SimpleRNN(64)), tf.keras.layers.Dense(64, activation='relu'), tf.keras.layers.Dense(1, activation='sigmoid') ])

gru_bi = tf.keras.models.Sequential([ tf.keras.layers.Embedding(VOCAB_SIZE, 64), tf.keras.layers.Bidirectional(tf.keras.layers.GRU(64, return_sequences=True)), tf.keras.layers.Bidirectional(tf.keras.layers.GRU(64)), tf.keras.layers.Dense(1, activation='sigmoid') ])

lstm_bi = tf.keras.models.Sequential([ tf.keras.layers.Embedding(VOCAB_SIZE, 64), tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)), tf.keras.layers.Dense(64, activation='relu'), tf.keras.layers.Dense(1, activation='sigmoid') ])

def predict(text, clf, word2idx, padding_size): padded_text = np.zeros((padding_size)) padded_text[:len(text.split())] = [word2idx.get(w, 0) for w in text.split()] return clf.predict(tf.expand_dims(padded_text, 0)) predict("the movie was super awesome!", lstm_bi, word2idx, lstm_padding)