Text Representation Techniques in NLP (Including Arabic Data)
Introduction
Machine learning models cannot process raw text data directly; they require numeric inputs. This is where text representation techniques come into play. Below, we explore various methods to transform textual data into numerical feature spaces.
Preparation
texts = [
"The quick brown fox jumps over the lazy dog",
"Natural language processing is a subfield of artificial intelligence",
"Machine learning enables computers to learn from data without explicit programming",
"Deep learning is a branch of machine learning focused on neural networks",
"Word embeddings capture semantic relationships between words in a text corpus"
]
test_txt = "Machine learning is awesome"
# Build Vocabulary
vocab = sorted(set(word for sentence in texts for word in sentence.split()))One-Hot-Encoding
One-Hot Encoding is technique to represent text numerically.
What is One-Hot Encoding?
- Each unique word is represented as a vector of size equal to the vocabulary.
- Only one position in the vector is set to
1, and the rest are0.
import numpy as np
word_to_index = {word: idx for idx, word in enumerate(vocab)}
def one_hot_encode(word):
"""Convert a single English word into a one-hot encoded vector."""
one_hot_vector = np.zeros(len(vocab))
if word in word_to_index:
one_hot_vector[word_to_index[word]] = 1 # Set index position to 1
return one_hot_vector
# Example: Encode individual words
example_words = test_txt.split()
one_hot_vectors = {word: one_hot_encode(word) for word in example_words}
# Print One-Hot Encodings
for word, vector in one_hot_vectors.items():
print(f"{word}: {vector}")Machine: [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
learning: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
is: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
awesome: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]import pandas as pd
import numpy as np
from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder(sparse_output=False,handle_unknown='ignore')
vocab_np = np.array(vocab).reshape(-1, 1)
encoder.fit(vocab_np)
test_txt_words = np.array(test_txt.lower().split()).reshape(-1, 1)
encoded_words = encoder.transform(test_txt_words)
for word, vector in zip(test_txt_words.flatten(), encoded_words):
print(f"{word}: {vector}")machine: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
learning: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
is: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
awesome: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]Binary Encoding - Simplest version of BoW
What is Binary Encoding?
Binary Encoding is a simple way of representing text numerically. The key idea is:
- Construct a vocabulary from all unique words in the dataset.
- Create a vector for each document, where each position corresponds to a word from the vocabulary.
- If a word is present in a document, mark
1; otherwise, mark0.
def binary_transform(text):
"""Convert text into a binary vector."""
output = np.zeros(len(vocab))
words = set(text.split()) # Tokenize the input
for i, v in enumerate(vocab):
output[i] = v in words # Set 1 if the word exists
return output
print(binary_transform(test_txt))[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.
0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd
vec = CountVectorizer(binary=True)
vec.fit(texts)
df = pd.DataFrame(vec.transform(texts).toarray(), columns=sorted(vec.vocabulary_.keys()))
print(df) artificial between branch brown capture computers corpus data deep \
0 0 0 0 1 0 0 0 0 0
1 1 0 0 0 0 0 0 0 0
2 0 0 0 0 0 1 0 1 0
3 0 0 1 0 0 0 0 0 1
4 0 1 0 0 1 0 1 0 0
dog ... quick relationships semantic subfield text the to without \
0 1 ... 1 0 0 0 0 1 0 0
1 0 ... 0 0 0 1 0 0 0 0
2 0 ... 0 0 0 0 0 0 1 1
3 0 ... 0 0 0 0 0 0 0 0
4 0 ... 0 1 1 0 1 0 0 0
word words
0 0 0
1 0 0
2 0 0
3 0 0
4 1 1
[5 rows x 43 columns]print(sorted(vec.vocabulary_.keys()))
print(vec.transform([test_txt]).toarray())['artificial', 'between', 'branch', 'brown', 'capture', 'computers', 'corpus', 'data', 'deep', 'dog', 'embeddings', 'enables', 'explicit', 'focused', 'fox', 'from', 'in', 'intelligence', 'is', 'jumps', 'language', 'lazy', 'learn', 'learning', 'machine', 'natural', 'networks', 'neural', 'of', 'on', 'over', 'processing', 'programming', 'quick', 'relationships', 'semantic', 'subfield', 'text', 'the', 'to', 'without', 'word', 'words']
[[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0]]df = pd.DataFrame(vec.transform([test_txt]).toarray(), columns=sorted(vec.vocabulary_.keys()))
print(df) artificial between branch brown capture computers corpus data deep \
0 0 0 0 0 0 0 0 0 0
dog ... quick relationships semantic subfield text the to without \
0 0 ... 0 0 0 0 0 0 0 0
word words
0 0 0
[1 rows x 43 columns]Bag of words
What is Bag of Words?
Bag of Words (BoW) is similar to Binary Encoding but records how many times a word appears in a document instead of just presence (1 or 0).
from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd
vec = CountVectorizer()
vec.fit(texts)
df = pd.DataFrame(vec.transform(texts).toarray(), columns=sorted(vec.vocabulary_.keys()))
print(df) artificial between branch brown capture computers corpus data deep \
0 0 0 0 1 0 0 0 0 0
1 1 0 0 0 0 0 0 0 0
2 0 0 0 0 0 1 0 1 0
3 0 0 1 0 0 0 0 0 1
4 0 1 0 0 1 0 1 0 0
dog ... quick relationships semantic subfield text the to without \
0 1 ... 1 0 0 0 0 2 0 0
1 0 ... 0 0 0 1 0 0 0 0
2 0 ... 0 0 0 0 0 0 1 1
3 0 ... 0 0 0 0 0 0 0 0
4 0 ... 0 1 1 0 1 0 0 0
word words
0 0 0
1 0 0
2 0 0
3 0 0
4 1 1
[5 rows x 43 columns]TF-IDF(Term Frequency - Inverse Document Frequency) - Weighted BoW
Why Use TF-IDF?
- Reduces the weight of common words while increasing importance of rare but meaningful words.
- Helps in tasks like text classification and search engines.
from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer()
vec.fit(texts)
df = pd.DataFrame(vec.transform(texts).toarray(), columns=sorted(vec.vocabulary_.keys()))
print(df) artificial between branch brown capture computers corpus \
0 0.00000 0.000000 0.000000 0.301511 0.000000 0.000000 0.000000
1 0.37007 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
2 0.00000 0.000000 0.000000 0.000000 0.000000 0.311561 0.000000
3 0.00000 0.000000 0.307781 0.000000 0.000000 0.000000 0.000000
4 0.00000 0.316228 0.000000 0.000000 0.316228 0.000000 0.316228
data deep dog ... quick relationships semantic \
0 0.000000 0.000000 0.301511 ... 0.301511 0.000000 0.000000
1 0.000000 0.000000 0.000000 ... 0.000000 0.000000 0.000000
2 0.311561 0.000000 0.000000 ... 0.000000 0.000000 0.000000
3 0.000000 0.307781 0.000000 ... 0.000000 0.000000 0.000000
4 0.000000 0.000000 0.000000 ... 0.000000 0.316228 0.316228
subfield text the to without word words
0 0.00000 0.000000 0.603023 0.000000 0.000000 0.000000 0.000000
1 0.37007 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
2 0.00000 0.000000 0.000000 0.311561 0.311561 0.000000 0.000000
3 0.00000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
4 0.00000 0.316228 0.000000 0.000000 0.000000 0.316228 0.316228
[5 rows x 43 columns]df = pd.DataFrame(vec.transform([test_txt]).toarray(), columns=sorted(vec.vocabulary_.keys()))
print(df) artificial between branch brown capture computers corpus data deep \
0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
dog ... quick relationships semantic subfield text the to \
0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0
without word words
0 0.0 0.0 0.0
[1 rows x 43 columns]Embeddings
Personality Embeddings: What Are You Like?
Read : https://jalammar.github.io/illustrated-word2vec/
“I give you the desert chameleon, whose ability to blend itself into the background tells you all you need to know about the roots of ecology and the foundations of a personal identity.”
~ Children of Dune
Understanding Personality Traits
On a scale of 0 to 100, how introverted or extraverted are you? (Where 0 is the most introverted and 100 is the most extraverted). Have you ever taken a personality test like MBTI or, even better, the Big Five Personality Traits test?
If not, these tests ask a series of questions and then score you on a number of axes, introversion/extraversion being one of them.
Example Personality Test Result
The Big Five Personality Traits test can provide deep insights into personality and has been shown to predict academic, personal, and professional success. Here’s an example of how it might look:
Openness: 72/100
Conscientiousness: 85/100
Extraversion: 38/100
Agreeableness: 65/100
Neuroticism: 47/100Visualizing Personality Scores
Imagine I’ve scored 38/100 on the introversion/extraversion scale. We can visualize this as:
To make it more useful for mathematical operations, let’s normalize the range from -1 to 1:
Adding More Dimensions
How well do you feel you know a person based on just one piece of information? Not much! People are complex. Let’s add another trait score to represent personality in two dimensions:
Instead of just points, we can represent personality as vectors. Why? Because vectors allow us to measure similarity mathematically!
Comparing Two People’s Personalities
Imagine I need to be replaced by someone with a similar personality. Below, we compare two other individuals:
Which person is more similar to me?
A common method to measure similarity in vector space is cosine similarity:
If cosine similarity is close to 1, the vectors point in the same direction, meaning they are more similar.
- Person #1 is more similar because their vector points in the same general direction as mine.
- Person #2 is less similar because their vector is in a different quadrant.
Scaling to More Dimensions
Two dimensions aren’t enough to fully capture human personality. Decades of psychology research suggest that personality can be best described with five major traits (The Big Five), along with many sub-traits.
The problem? We can’t draw neat little arrows for five dimensions! But in machine learning, we often deal with high-dimensional spaces. Luckily, cosine similarity still works for any number of dimensions!
Key Takeaways
At the end of this section, we should remember two key ideas:
- We can represent people (and things) as vectors of numbers – which is great for machines!
- We can easily calculate how similar these vectors are to each other using cosine similarity.
Word Embeddings
Word2Vec (Trained on Context-Based Prediction)
- CBOW (Continuous Bag of Words): Predicts a word from its context.
- Skip-Gram: Predicts context words from a single word.
import gensim
word2vec_model = gensim.models.KeyedVectors.load_word2vec_format(
"GoogleNews-vectors-negative300.bin", binary=True
)
print("Word2Vec (King):", word2vec_model["king"][:10]) # First 10 valuesWord2Vec (King): [ 0.12597656 0.02978516 0.00860596 0.13964844 -0.02563477 -0.03613281
0.11181641 -0.19824219 0.05126953 0.36328125]print("Word2Vec (Queen):", word2vec_model["Queen"][:10]) # First 10 valuesWord2Vec (Queen): [-0.22070312 -0.17480469 -0.10498047 0.2578125 0.16210938 -0.13085938
-0.16699219 0.07373047 -0.07226562 0.02404785]GloVe (Trained on Word Co-Occurrence Statistics)
- Unlike Word2Vec, GloVe is based on matrix factorization of a word-word co-occurrence matrix.
- Captures global word relationships.
import numpy as np
# Load GloVe embeddings
glove_embeddings = {}
with open("glove.6B.100d.txt", "r", encoding="utf-8") as f:
for line in f:
values = line.split()
word = values[0]
vector = np.asarray(values[1:], dtype="float32")
glove_embeddings[word] = vector
print("GloVe 'king':", glove_embeddings["king"][:10]) # Show first 10 valuesGloVe 'king': [-0.32307 -0.87616 0.21977 0.25268 0.22976 0.7388 -0.37954 -0.35307
-0.84369 -1.1113 ]