Logo
Courses/Neural Networks

LSTM on 20 Newsgroups Dataset using Keras

8 min read
Lesson slides
1 / 13

LSTM on 20 Newsgroups Dataset using Keras

Building and comparing a standard LSTM vs. a Bidirectional LSTM for text classification

This notebook demonstrates how to build and compare a standard LSTM and a Bidirectional LSTM model using the 20 Newsgroups dataset. We use 10 categories and perform multi-class classification.

The 20 Newsgroups data set is a well known classification problem. The goal is to classify which newsgroup a particular post came from. The 20 possible groups are:

comp.graphics comp.os.ms-windows.misc comp.sys.ibm.pc.hardware comp.sys.mac.hardware comp.windows.x rec.autos rec.motorcycles rec.sport.baseball rec.sport.hockey sci.crypt sci.electronics sci.med sci.space misc.forsale talk.politics.misc talk.politics.guns talk.politics.mideast talk.religion.misc alt.atheism soc.religion.christian

Here we select 10 classes to work on.

from sklearn.datasets import fetch_20newsgroups
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, Dense, Bidirectional
import numpy as np
from sklearn.preprocessing import LabelBinarizer
# Load dataset
categories = [
    'sci.space', 'comp.graphics', 'rec.sport.baseball', 'talk.politics.misc',
    'sci.electronics', 'soc.religion.christian', 'rec.autos',
    'talk.politics.mideast', 'comp.sys.mac.hardware', 'misc.forsale'
]
newsgroups = fetch_20newsgroups(subset='train', categories=categories)
texts = newsgroups.data
labels = newsgroups.target
 
len(texts)
5750
# โœ… Display a sample
print("Sample text:")
print(texts[1][:1000])
print("\nLabel:", labels[1])
print("Actual Category:", newsgroups.target_names[labels[1]])
Sample text:
From: [email protected] (Wolfgang Pest)
Subject: Speedstar 24 - how to program the TrueColor mode ?
Distribution: world
Organization: Kontron Elektronik GmbH Eching, Germany
Lines: 17
 
Hello,
I purchased my new 486 with a NoName graphics card installed which is obviously 
Speedstar 24 compatible. Its name is "VGA 4000 TrueColor".
It is accompanied with some drivers and the utilities VMODE, XMODE and
at least one more MODE, as well as some drivers for Lotus, Windows, etc.
Only one of the drivers is told to provide the TrueColor mode, namely
the Windows 3.1 driver.
Nowhere else, except in the ad, is any pointer to the TrueColor mode.
Some articles in this group about the Speedstar 24 and some other facts
made me believe that my card is compatible to that one.
 
Does anybody out there know how this mode can be adjusted? How can I write
a driver which allows me to have 16.7 millions of colors with a resolution
of 640 x 480 with 45 Hz interlaced ?
 
Greetings,
    Wolfgang
 
 
Label: 0
Actual Category: comp.graphics
for i, name in enumerate(newsgroups.target_names):
    print(f"Label {i}: {name}")
Label 0: comp.graphics
Label 1: comp.sys.mac.hardware
Label 2: misc.forsale
Label 3: rec.autos
Label 4: rec.sport.baseball
Label 5: sci.electronics
Label 6: sci.space
Label 7: soc.religion.christian
Label 8: talk.politics.mideast
Label 9: talk.politics.misc

๐Ÿ”น Tokenizer Explanation

The Tokenizer is used to vectorize a text corpus by:

  1. Creating a word index (mapping words to integers)
  2. Converting each document into a list of word indices

Tokenizer: Converts words to integers (e.g., "space" โ†’ 345). It restricts the tokenizer to only use the top vocab_size words (e.g., 10,000). Any word beyond the top 10,000 most frequent is ignored during text-to-sequence conversion.

fit_on_texts: Builds the word dictionary.

texts_to_sequences: Converts each text to a list of numbers.

pad_sequences: Pads all sequences to the same length (200 words).

# Tokenization
vocab_size = 10000
maxlen = 200
tokenizer = Tokenizer(num_words=vocab_size)
 
# Learns the word -> number mapping
tokenizer.fit_on_texts(texts)
# Converts each article into a sequence of word indices
sequences = tokenizer.texts_to_sequences(texts)
 
padded_sequences = pad_sequences(sequences, maxlen=maxlen, padding='post')
 
# Encode labels to one-hot
encoder = LabelBinarizer()
one_hot_labels = encoder.fit_transform(labels)
# โœ… Display a sample
print("Sample text:")
print(padded_sequences[500])
print("\nLabel:", labels[500])
Sample text:
[  13 4108 1631   14 1822 8471   28  209   10  411 1317   31  548  487
    3  372 2498   30  343   85   74   81 1631   14  388 1317  411  146
   56   36    6   20    4 2581  209   24  119   13   38  249 2974 3020
 1317   27   12   47  310   66    2  504   11  169    3   12    1  231
  232    8   22    4  467  503  146   73    3   55   94 9770 1108   24
   80   31    9   40 1620 1317  184    4 4010 4108 1631   14    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0]
 
Label: 5

๐Ÿ”น Layer Explanations

Embedding Layer

  • Converts each word index into a dense vector.
  • input_dim: vocabulary size.
  • output_dim: dimensionality of embedding vectors.
  • input_length: fixed length of input sequences.

LSTM Layer

  • units: number of hidden units or memory cells.
  • return_sequences: True returns full sequence; use when stacking LSTM layers.
  • dropout: dropout rate for input connections.
  • recurrent_dropout: dropout rate for memory-state connections between time steps
 
model_lstm = Sequential()
model_lstm.add(Embedding(input_dim=vocab_size, output_dim=128, input_length=maxlen))
model_lstm.add(LSTM(units=64, return_sequences=True))
model_lstm.add(LSTM(units=32, dropout=0.2, recurrent_dropout=0.2))
model_lstm.add(Dense(10, activation='softmax'))
model_lstm.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model_lstm.fit(padded_sequences, one_hot_labels, epochs=10, batch_size=32, validation_split=0.2)
Epoch 1/10
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/embedding.py:90: UserWarning: Argument `input_length` is deprecated. Just remove it.
  warnings.warn(
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 64s 355ms/step - accuracy: 0.1436 - loss: 2.2707 - val_accuracy: 0.2157 - val_loss: 2.1740
Epoch 2/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 52s 361ms/step - accuracy: 0.2254 - loss: 2.1460 - val_accuracy: 0.2365 - val_loss: 2.1035
Epoch 3/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 80s 343ms/step - accuracy: 0.2947 - loss: 1.9751 - val_accuracy: 0.2365 - val_loss: 2.0300
Epoch 4/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 51s 353ms/step - accuracy: 0.3294 - loss: 1.8010 - val_accuracy: 0.3339 - val_loss: 1.7684
Epoch 5/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 82s 357ms/step - accuracy: 0.3438 - loss: 1.7629 - val_accuracy: 0.2504 - val_loss: 2.0128
Epoch 6/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 81s 352ms/step - accuracy: 0.3253 - loss: 1.7569 - val_accuracy: 0.3365 - val_loss: 1.8139
Epoch 7/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 83s 363ms/step - accuracy: 0.4165 - loss: 1.5375 - val_accuracy: 0.3530 - val_loss: 1.7347
Epoch 8/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 81s 355ms/step - accuracy: 0.4726 - loss: 1.3878 - val_accuracy: 0.4209 - val_loss: 1.6340
Epoch 9/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 82s 352ms/step - accuracy: 0.5262 - loss: 1.2354 - val_accuracy: 0.4609 - val_loss: 1.5254
Epoch 10/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 51s 350ms/step - accuracy: 0.5796 - loss: 1.0998 - val_accuracy: 0.5191 - val_loss: 1.4216
<keras.src.callbacks.history.History at 0x7fa0e2224190>

Bidirectional Layer

  • Processes the input sequence in both forward and backward directions.
  • Often improves performance for NLP tasks where context from both sides is useful.
 
model_bi = Sequential()
model_bi.add(Embedding(input_dim=vocab_size, output_dim=128, input_length=maxlen))
model_bi.add(Bidirectional(LSTM(units=64, dropout=0.2, recurrent_dropout=0.2)))
model_bi.add(Dense(10, activation='softmax'))
model_bi.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model_bi.fit(padded_sequences, one_hot_labels, epochs=10, batch_size=32, validation_split=0.2)
Epoch 1/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 92s 581ms/step - accuracy: 0.1717 - loss: 2.2183 - val_accuracy: 0.3843 - val_loss: 1.8057
Epoch 2/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 81s 566ms/step - accuracy: 0.4948 - loss: 1.5136 - val_accuracy: 0.6157 - val_loss: 1.1395
Epoch 3/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 82s 568ms/step - accuracy: 0.7449 - loss: 0.8354 - val_accuracy: 0.7070 - val_loss: 0.8469
Epoch 4/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 82s 565ms/step - accuracy: 0.8816 - loss: 0.4428 - val_accuracy: 0.7530 - val_loss: 0.7618
Epoch 5/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 82s 565ms/step - accuracy: 0.9387 - loss: 0.2518 - val_accuracy: 0.7783 - val_loss: 0.7177
Epoch 6/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 82s 564ms/step - accuracy: 0.9585 - loss: 0.1622 - val_accuracy: 0.8026 - val_loss: 0.6888
Epoch 7/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 82s 563ms/step - accuracy: 0.9787 - loss: 0.0978 - val_accuracy: 0.8096 - val_loss: 0.6960
Epoch 8/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 82s 569ms/step - accuracy: 0.9895 - loss: 0.0547 - val_accuracy: 0.8113 - val_loss: 0.7218
Epoch 9/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 80s 553ms/step - accuracy: 0.9871 - loss: 0.0556 - val_accuracy: 0.8061 - val_loss: 0.7460
Epoch 10/10
144/144 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 86s 579ms/step - accuracy: 0.9971 - loss: 0.0288 - val_accuracy: 0.8070 - val_loss: 0.7524
<keras.src.callbacks.history.History at 0x7fa0f4273cd0>
sample_text = ["NASA is planning a mission to Jupiter"]
seq = tokenizer.texts_to_sequences(sample_text)
pad = pad_sequences(seq, maxlen=maxlen, padding='post')
 
prediction = model_bi.predict(pad)
predicted_class_index = np.argmax(prediction)
print("Predicted class index:", predicted_class_index)
print("Actual Category:", newsgroups.target_names[predicted_class_index])
1/1 โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 0s 74ms/step
Predicted class index: 6
Actual Category: sci.space

๐Ÿ”š Summary Comparison

  • Standard LSTM: Processes sequences from past to future.
  • Bidirectional LSTM: Processes sequences from both directions.
  • Bidirectional models can learn richer context and often yield better results in NLP tasks.

In the previous model, try to stack another LSTM Layer and Add a Dropout Layer After the LSTM

##Exercise: Train a GRU Model

  • Replace your LSTM with a GRU.
  • Download the full 20 newsgroups data; there is already a designated "train" and "test" set and use the the test set to evaulate your model
  • Try different sequence lengths, and dimensions for the hidden state of the LSTM. Can you improve the model?