Mahmoud Abas, Full-Stack Software Engineer & AI

Arabic NLP - Text Representation Techniques in NLP (Including Arabic Data)

In the print dialog, choose "Save as PDF" as the destination.

Advanced NLP - Lesson 3

Text Representation Techniques in NLP (Including Arabic Data)

Turning text into numbers: one-hot encoding, bag-of-words, TF-IDF, and word embeddings

Why Text Representation?

ML models need numeric input, not raw text
Text representation techniques map words and documents into numeric feature spaces
A corpus of 5 English sentences is used as the running example throughout this lesson

Text Representation

maps words and documents to numbers

Sparse / count-based

One-Hot, BoW, TF-IDF

Dense / learned

Word2Vec, GloVe

Building a Vocabulary

python

vocab = sorted(set(word for sentence in texts for word in sentence.split()))

Vocabulary = every unique word across the corpus
Every representation technique below is built relative to this vocabulary
Test text used for examples: Machine learning is awesome

One-Hot Encoding

One-Hot Encoding: Each unique word becomes a vector of size equal to the vocabulary. Only one position is set to 1, and the rest are 0.

python

def one_hot_encode(word):
    vec = np.zeros(len(vocab))
    if word in word_to_index:
        vec[word_to_index[word]] = 1
    return vec

One-Hot Encoding - scikit-learn

python

from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder(sparse_output=False, handle_unknown='ignore')
encoder.fit(vocab_np)

handle_unknown='ignore' avoids errors on out-of-vocabulary words
Production code typically uses the library implementation over hand-rolled loops

Binary Encoding (Simplest Bag-of-Words)

Binary Encoding: One vector per document; each position corresponds to a vocabulary word. 1 if the word appears in the document, 0 otherwise: no counts.

python

def binary_transform(text):
    output = np.zeros(len(vocab))
    words = set(text.split())
    for i, v in enumerate(vocab):
        output[i] = v in words
    return output

Binary Encoding with CountVectorizer

python

from sklearn.feature_extraction.text import CountVectorizer
vec = CountVectorizer(binary=True)
vec.fit(texts)

binary=True gives presence/absence instead of counts
Produces a document-term matrix (5 documents by 43-word vocabulary in the example)

Bag of Words (Word Counts)

python

from sklearn.feature_extraction.text import CountVectorizer
vec = CountVectorizer()
vec.fit(texts)

Same idea as Binary Encoding, but records how many times each word appears
Example: the appears twice in sentence 1, giving a count of 2 rather than just 1

TF-IDF - Weighted Bag of Words

TF-IDF: Term Frequency times Inverse Document Frequency: reduces the weight of common words, and boosts rare-but-meaningful words.

Useful for text classification and search engines

python

from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer()
vec.fit(texts)

Analogy

From Points to Vectors: Personality Embeddings

Represent a person's traits (for example Big Five personality scores) as a vector
Vectors, not raw numbers, let us measure similarity mathematically
The same idea scales from personality traits to word meaning: words become vectors

1 trait

a point on a line

add a trait

2 traits

a 2D vector

add more traits

5 traits

a 5D vector (Big Five)

same idea, applied to words

Word meaning

an N-D vector

Cosine Similarity

Cosine Similarity: The standard method to measure similarity between two vectors. Works for any number of dimensions, from 2D traits to 300D word embeddings.

A value close to 1 means the vectors point in a similar direction: more similar
Core mechanism behind comparing word and document embeddings

Word2Vec: CBOW and Skip-Gram

Learns dense vectors from context-based prediction

CBOW: Continuous Bag of Words: predicts a word from its surrounding context.

Skip-Gram: Predicts context words from a single center word.

python

import gensim
word2vec_model = gensim.models.KeyedVectors.load_word2vec_format(
    "GoogleNews-vectors-negative300.bin", binary=True)
word2vec_model["king"][:10]

GloVe: Co-occurrence-Based Embeddings

GloVe: Unlike Word2Vec, GloVe factorizes a word-word co-occurrence matrix, capturing global word relationships across the whole corpus.

python

glove_embeddings = {}
with open("glove.6B.100d.txt", encoding="utf-8") as f:
    for line in f:
        values = line.split()
        glove_embeddings[values[0]] = np.asarray(values[1:], dtype="float32")

Key Takeaways

Sparse methods (one-hot, binary, BoW, TF-IDF) count words but ignore meaning and order
TF-IDF weights rare-but-meaningful words higher than common words
Representing entities as vectors enables similarity measurement via cosine similarity
Word2Vec (CBOW/Skip-Gram) and GloVe learn dense embeddings that capture semantic relationships
Representations progress from sparse counting to dense, semantically-aware vectors

Open this lesson

Mahmoud AbasText Representation Techniques in NLP (Including Arabic Data)

glove_embeddings = {} with open("glove.6B.100d.txt", encoding="utf-8") as f: for line in f: values = line.split() glove_embeddings[values[0]] = np.asarray(values[1:], dtype="float32")