Mahmoud Abas, Full-Stack Software Engineer & AI

Neural Networks - LSTM on 20 Newsgroups Dataset using Keras

In the print dialog, choose "Save as PDF" as the destination.

Neural Networks - Lesson 10

LSTM on 20 Newsgroups Dataset using Keras

Building and comparing a standard LSTM vs. a Bidirectional LSTM for text classification

The Task: 20 Newsgroups Classification

Classic NLP classification problem: predict which newsgroup a post came from
20 possible groups exist (comp.graphics, sci.space, rec.autos, talk.politics.misc, ...); this lab uses 10 selected categories

python

from sklearn.datasets import fetch_20newsgroups
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.layers import Embedding, LSTM, Dense, Bidirectional

Loading the Data

Fetch the training split for 10 chosen categories

python

categories = ['sci.space', 'comp.graphics', 'rec.sport.baseball', 'talk.politics.misc',
              'sci.electronics', 'soc.religion.christian', 'rec.autos',
              'talk.politics.mideast', 'comp.sys.mac.hardware', 'misc.forsale']

newsgroups = fetch_20newsgroups(subset='train', categories=categories)
texts = newsgroups.data
labels = newsgroups.target
# len(texts) -> 5750

Each example is a raw newsgroup post; each label is an integer 0-9 mapping to newsgroups.target_names

Tokenizer: Text to Integer Sequences

Raw text

one post

fit_on_texts

Tokenizer

top vocab_size words

Integer sequence

texts_to_sequences

pad_sequences

Padded sequence

fixed length

Tokenizer(num_words=vocab_size) restricts vocabulary to the top N most frequent words; rarer words are dropped
fit_on_texts builds the word-to-integer dictionary; texts_to_sequences converts text into integer lists
pad_sequences pads/truncates every sequence to the same fixed length

python

vocab_size, maxlen = 10000, 200
tokenizer = Tokenizer(num_words=vocab_size)
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(texts)
padded_sequences = pad_sequences(sequences, maxlen=maxlen, padding='post')

encoder = LabelBinarizer()
one_hot_labels = encoder.fit_transform(labels)

Padded output is a fixed-length integer vector per document, zero-padded at the end

The Embedding Layer

Embedding: Converts each integer word index into a dense, learnable vector.

input_dim: vocabulary size; output_dim: embedding vector dimensionality; input_length: fixed sequence length

python

Embedding(input_dim=vocab_size, output_dim=128, input_length=maxlen)

Embeddings let the model learn semantic relationships between words during training, instead of using raw one-hot vectors

The LSTM Layer

units: number of hidden units / memory cells
return_sequences=True: returns the full output sequence, needed when stacking LSTM layers
dropout: dropout on input connections; recurrent_dropout: dropout on the recurrent (memory-state) connections between time steps

python

LSTM(units=64, return_sequences=True)
LSTM(units=32, dropout=0.2, recurrent_dropout=0.2)

Building and Training a Standard LSTM

Stack two LSTM layers, then classify into 10 categories with softmax

Embedding

128-dim

LSTM

64 units, return_sequences

LSTM

32 units, dropout 0.2

Dense

10 units, softmax

python

model_lstm = Sequential()
model_lstm.add(Embedding(input_dim=vocab_size, output_dim=128, input_length=maxlen))
model_lstm.add(LSTM(units=64, return_sequences=True))
model_lstm.add(LSTM(units=32, dropout=0.2, recurrent_dropout=0.2))
model_lstm.add(Dense(10, activation='softmax'))
model_lstm.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model_lstm.fit(padded_sequences, one_hot_labels, epochs=10, batch_size=32, validation_split=0.2)

text

Epoch 1/10  accuracy: 0.1436  val_accuracy: 0.2157
Epoch 10/10 accuracy: 0.5796  val_accuracy: 0.5191

Learns slowly, only reaches ~52% validation accuracy after 10 epochs

The Bidirectional Layer

Bidirectional: Wraps an LSTM to process the sequence in both forward and backward directions.

Often improves NLP performance because context from both sides of a word helps disambiguate meaning

python

Bidirectional(LSTM(units=64, dropout=0.2, recurrent_dropout=0.2))

Building and Training a Bidirectional LSTM

Same overall architecture, but the LSTM is wrapped in Bidirectional

python

model_bi = Sequential()
model_bi.add(Embedding(input_dim=vocab_size, output_dim=128, input_length=maxlen))
model_bi.add(Bidirectional(LSTM(units=64, dropout=0.2, recurrent_dropout=0.2)))
model_bi.add(Dense(10, activation='softmax'))
model_bi.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model_bi.fit(padded_sequences, one_hot_labels, epochs=10, batch_size=32, validation_split=0.2)

text

Epoch 1/10  accuracy: 0.1717  val_accuracy: 0.3843
Epoch 5/10  accuracy: 0.9387  val_accuracy: 0.7783
Epoch 10/10 accuracy: 0.9971  val_accuracy: 0.8070

Making a Prediction

Tokenize and pad new text exactly like training data, then predict with the trained model

python

sample_text = ["NASA is planning a mission to Jupiter"]
seq = tokenizer.texts_to_sequences(sample_text)
pad = pad_sequences(seq, maxlen=maxlen, padding='post')

prediction = model_bi.predict(pad)
predicted_class_index = np.argmax(prediction)
print("Actual Category:", newsgroups.target_names[predicted_class_index])

Summary Comparison & Exercises

Standard LSTM: processes sequences only from past to future
Bidirectional LSTM: processes sequences in both directions, learns richer context, generally better for NLP

Suggested Extensions

Stack another LSTM layer and add Dropout after it
Replace LSTM with GRU and compare
Use the full 20-newsgroups train/test split; try different sequence lengths and hidden-state sizes

Key Takeaways

Text must be tokenized, integer-encoded, and padded to a fixed length before feeding a neural network
The Embedding layer learns dense word representations instead of using raw one-hot vectors
return_sequences=True is required when stacking multiple LSTM layers
dropout and recurrent_dropout regularize LSTMs against overfitting on input and recurrent connections respectively
Bidirectional LSTMs read context from both directions and substantially outperformed the standard LSTM here (80.7% vs 51.9% val accuracy)
A large gap between train and validation accuracy (Bidirectional hit 99.7% train vs 80.7% val) is a clear overfitting signal worth addressing

Open this lesson

Mahmoud AbasLSTM on 20 Newsgroups Dataset using Keras

from sklearn.datasets import fetch_20newsgroups from tensorflow.keras.preprocessing.text import Tokenizer from tensorflow.keras.preprocessing.sequence import pad_sequences from tensorflow.keras.layers import Embedding, LSTM, Dense, Bidirectional

categories = ['sci.space', 'comp.graphics', 'rec.sport.baseball', 'talk.politics.misc', 'sci.electronics', 'soc.religion.christian', 'rec.autos', 'talk.politics.mideast', 'comp.sys.mac.hardware', 'misc.forsale'] newsgroups = fetch_20newsgroups(subset='train', categories=categories) texts = newsgroups.data labels = newsgroups.target # len(texts) -> 5750

vocab_size, maxlen = 10000, 200 tokenizer = Tokenizer(num_words=vocab_size) tokenizer.fit_on_texts(texts) sequences = tokenizer.texts_to_sequences(texts) padded_sequences = pad_sequences(sequences, maxlen=maxlen, padding='post') encoder = LabelBinarizer() one_hot_labels = encoder.fit_transform(labels)

model_lstm = Sequential() model_lstm.add(Embedding(input_dim=vocab_size, output_dim=128, input_length=maxlen)) model_lstm.add(LSTM(units=64, return_sequences=True)) model_lstm.add(LSTM(units=32, dropout=0.2, recurrent_dropout=0.2)) model_lstm.add(Dense(10, activation='softmax')) model_lstm.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy']) model_lstm.fit(padded_sequences, one_hot_labels, epochs=10, batch_size=32, validation_split=0.2)

model_bi = Sequential() model_bi.add(Embedding(input_dim=vocab_size, output_dim=128, input_length=maxlen)) model_bi.add(Bidirectional(LSTM(units=64, dropout=0.2, recurrent_dropout=0.2))) model_bi.add(Dense(10, activation='softmax')) model_bi.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy']) model_bi.fit(padded_sequences, one_hot_labels, epochs=10, batch_size=32, validation_split=0.2)

sample_text = ["NASA is planning a mission to Jupiter"] seq = tokenizer.texts_to_sequences(sample_text) pad = pad_sequences(seq, maxlen=maxlen, padding='post') prediction = model_bi.predict(pad) predicted_class_index = np.argmax(prediction) print("Actual Category:", newsgroups.target_names[predicted_class_index])