Content-Based Recommendations

This section builds a small recommendation system. Given a movie that a viewer liked, it returns the movies whose content is most similar: the same genres, the same keywords, the same cast and director. You will first compute cosine similarity by hand on five real movies, then run the week 9 notebook on a database of 4,803 movies and read the recommendations it produces, including the cases where it gets them wrong.

Objectives

By the end of the section you should be able to:

• Explain how a content-based recommender picks items: the items most similar to one you liked are recommended first.
• Turn text into count vectors with CountVectorizer, and explain what stop_words and min_df do.
• Compute a cosine similarity by hand: the dot product, the two lengths, and the division.
• Build the similarity matrix with cosine_similarity and read one row of it.
• Trace get_recommendations line by line and explain where it goes wrong.
• Prepare your team's project presentation for this week's discussion.

Where this sits in the course

Week 9 of the plan has three parts:

• The notebook. Movie Recommendations loads a database of 4,803 movies that includes the director, the cast, the genres and keywords describing each movie, and builds a model that takes a movie title as input and returns a list of similar movies.
• A real-world scenario. Movie recommendations.
• A project milestone. Project Teams Presentations and Discussions.

You met CountVectorizer in week 7, where the count vectors of e-mails were fed to a Naive Bayes classifier. This week there is no classifier and no target column: the count vectors of the movies are compared with each other.

Part 1: The idea

What a content-based recommender is

Machine learning can be used to build recommendation systems that recommend products such as books and movies based on other products that you like. A content-based system looks at the products themselves: it describes every product by its content and recommends the products whose content is closest to a product the user liked. Content-based recommendation systems frequently use a technique called cosine similarity to quantify the similarity between two products.

In one sentence: describe every movie by its words, find the movies whose words are closest to a movie you liked, and recommend those.

The recipe in four steps

1. Describe each movie as one text: its title, genres, keywords, cast and director.
2. Vectorize: count how many times each vocabulary word appears in that text.
3. Compare: compute the cosine similarity of every pair of movies.
4. Recommend: for a liked movie, return the movies with the highest similarity.

Nothing is predicted against a label. The model is the similarity matrix itself.

From text to counts: CountVectorizer

CountVectorizer turns a list of texts into a table of counts:

from sklearn.feature_extraction.text import CountVectorizer
 
texts = ['Action Adventure Fantasy Science Fiction',
         'Adventure Fantasy Action']
cv = CountVectorizer(stop_words='english')
m = cv.fit_transform(texts)
print(cv.get_feature_names_out())
print(m.toarray())

['action' 'adventure' 'fantasy' 'fiction' 'science']
[[1 1 1 1 1]
 [1 1 1 0 0]]

• fit builds the vocabulary: every distinct word, lowercased and sorted.
• transform gives each text one row of counts, with one column per vocabulary word.
• stop_words='english' removes very common English words such as the and of.
• The order of the words is lost. Only the counts remain.

Cosine similarity

For two count vectors A and B over a vocabulary of n words:

cos(A, B) = (A · B) / (‖A‖ × ‖B‖)
 
A · B = a1 b1 + a2 b2 + ... + an bn
‖A‖   = √(a1² + a2² + ... + an²)

• a1 ... an and b1 ... bn are how many times each vocabulary word appears in the two movies.
• A · B is the dot product: multiply word by word and add.
• ‖A‖ is the length of A: the square root of its sum of squares.

How to read a cosine:

• 1 means the two vectors point the same way: the same mix of words.
• 0 means the two movies share no word at all.
• Counts are never negative, so here the cosine always lies between 0 and 1.
• Only the direction matters: doubling every count of a movie leaves all its cosines unchanged (Practice 2 checks this).

Why not simply count the shared words? A movie with a long list of keywords would share words with almost everything. Dividing by the two lengths turns the overlap into a share of each movie's words.

Worked example: five movies by hand

We take the first five rows of movies.csv and use only the genres column. A viewer liked Avatar. Which movie do we recommend?

Step 1: the vocabulary and the vectors

The tokenizer splits on spaces and lowercases, so Science Fiction becomes two words. The five strings use 8 distinct words:

['action' 'adventure' 'crime' 'drama' 'fantasy' 'fiction' 'science' 'thriller']

With the columns in that order, each movie becomes a vector of 0s and 1s:

Step 2: the lengths

With only 0s and 1s, the sum of squares is the number of words.

Step 3: the dot products with Avatar

Multiply Avatar's vector by each other vector word by word and add. With 0/1 counts, the dot product is simply the number of shared words.

Step 4: the cosines

For Pirates of the Caribbean:

cos(A, B) = 3 / (√5 × √3) = 3 / √15 = 3 / 3.8730 = 0.7746

All three of its words are in Avatar, but Avatar has two more that Pirates lacks, so the cosine is below 1. The other three:

Step 5: rank and recommend

The top two recommendations for a viewer who liked Avatar are John Carter and Pirates of the Caribbean. John Carter shares 4 of Avatar's 5 genre words and has nothing else: the closest mix of words wins.

Check with scikit-learn

from sklearn.metrics.pairwise import cosine_similarity
 
cv = CountVectorizer(stop_words='english')
m = cv.fit_transform(df['genres'].head(5))
sim = cosine_similarity(m)
sim[0]

array([1.        , 0.77459667, 0.51639778, 0.2236068 , 0.89442719])

Row 0 is Avatar, and it matches the hand results. The whole 5 by 5 matrix:

The matrix is symmetric: the cosine of Avatar and Spectre is the same as the cosine of Spectre and Avatar.

To replay the example one comparison at a time, open the cosine step-by-step widget full screen. Pick the liked movie, press Step, and click any count to see how the cosines change.

Part 2: The notebook on 4,803 movies

Open the notebook

Open Movie Recommendations in Colab

The first code cell reads Data/movies.csv, a folder that does not exist next to the notebook in Colab. Read the file from the repository instead (it is about 23 MB, so the cell takes a few seconds):

import pandas as pd
 
url = ('https://raw.githubusercontent.com/'
       'jeffprosise/Machine-Learning/'
       'master/Data/movies.csv')
df = pd.read_csv(url)
df.shape

The output is (4803, 24): 4,803 movies and 24 columns. Among them are title, genres, keywords, cast and director, together with columns such as the budget, revenue, runtime and votes that this model does not use.

Keep five columns and fill the gaps

df = df[['title', 'genres', 'keywords', 'cast', 'director']]
df = df.fillna('') # Fill missing values with empty strings

Some cells are empty: keywords is missing in 412 rows, cast in 43, director in 30 and genres in 28, and 426 rows miss at least one of them. The next cell glues the columns together with +, and adding a string to a missing value gives a missing value. Without fillna('') those 426 movies would have no text at all, and CountVectorizer stops with:

ValueError: np.nan is an invalid document, expected byte or unicode string.

One text per movie: the features column

df['features'] = df['title'] + ' ' + df['genres'] + ' ' + df['keywords'] + ' ' + df['cast'] + ' ' + df['director']

The five columns become one string per movie, with a space between them. Names become ordinary words: a director's first and last name are two words of the text. The title The Dark Knight Rises gives the words dark, knight and rises; the is a stop word and is removed.

Vectorize all the movies

from sklearn.feature_extraction.text import CountVectorizer
 
vectorizer = CountVectorizer(stop_words='english', min_df=20)
word_matrix = vectorizer.fit_transform(df['features'])
word_matrix.shape

The output is (4803, 918): one row per movie and 918 vocabulary words.

min_df=20 keeps a word only if it appears in at least 20 movies. Rarer words are dropped. On the same features column, with stop_words='english':

A consequence you will see in Part 3: die appears in 17 movies and hard in 15, so the title of Die Hard adds no word to its vector. Neither do skyfall or rises, which appear in one movie each.

Each movie keeps a median of 12 words, so only 56,451 of the 4,803 × 918 cells, about 1.3%, are not zero. fit_transform returns a sparse matrix that stores only those cells.

The similarity matrix

from sklearn.metrics.pairwise import cosine_similarity
 
sim = cosine_similarity(word_matrix)
sim.shape

The output is (4803, 4803):

• sim[i, j] is the cosine similarity of movie i and movie j.
• The matrix is symmetric: sim[i, j] equals sim[j, i].
• The diagonal is 1.0 for 4,794 movies. The other 9 movies keep no word after min_df=20, their vector is all zeros, and cosine_similarity returns 0 for every pair that involves them, the diagonal included.

Worked example: Skyfall and Spectre

We look at two real rows of word_matrix. Every count in both vectors is 1. Grouped by the column each word comes from:

With 0/1 counts, A · B is the number of shared words, and each length is the square root of the number of words:

cos(A, B) = 11 / (√15 × √15) = 11 / 15 = 0.7333

The notebook's sim holds exactly 0.7333 for this pair. Six of the eleven shared words are names: the two movies have the same lead actors and the same director.

Part 3: Generating recommendations

The function, line by line

The notebook defines a function that takes a title and returns a list of similar movies:

def get_recommendations(title, df, sim, count=10):
    # Get the row index of the specified title in the DataFrame
    index = df.index[df['title'].str.lower() == title.lower()]
 
    # Return an empty list if there is no entry for the specified title
    if (len(index) == 0):
        return []
 
    # Get the corresponding row in the similarity matrix
    similarities = list(enumerate(sim[index[0]]))
 
    # Sort the similarity scores in that row in descending order
    recommendations = sorted(similarities, key=lambda x: x[1], reverse=True)
 
    # Get the top n recommendations, ignoring the first entry in the list since
    # it corresponds to the title itself (and thus has a similarity of 1.0)
    top_recs = recommendations[1:count + 1]
 
    # Generate a list of titles from the indexes in top_recs
    titles = []
 
    for i in range(len(top_recs)):
        title = df.iloc[top_recs[i][0]]['title']
        titles.append(title)
 
    return titles

• The title match lowercases both sides, so 'skyfall' gives the same list as 'Skyfall'.
• An unknown title returns an empty list instead of raising an error.
• enumerate pairs every score with its movie number. For Skyfall the row begins (0, 0.1732), (1, 0.1333), (2, 0.7333), ... (rounded here).
• sorted(..., reverse=True) puts the highest scores first. For equal scores Python's sorted keeps the original order, that is, the order of the file.
• The slice starts at 1 because entry 0 is normally the movie itself, with a similarity of 1.0.

Skyfall

get_recommendations('Skyfall', df, sim) returns:

['Spectre',
 'Quantum of Solace',
 'Johnny English Reborn',
 'Clash of the Titans',
 'Die Another Day',
 'Diamonds Are Forever',
 'Wrath of the Titans',
 'I Spy',
 'Sanctum',
 'Blackthorn']

The scores behind that list:

Spectre shares the cast and the director. Quantum of Solace and Die Another Day share the keywords secret and agent and the genres. Sanctum and Blackthorn tie at 0.3651, and sorted keeps them in file order.

Worked example: why does Mulan get Shrek?

get_recommendations('Mulan', df, sim) returns:

['Shrek',
 'Frozen',
 '1911',
 'Kung Fu Panda',
 'Shrek the Third',
 'The Polar Express',
 'Tangled',
 'Shrek Forever After',
 'Shrek 2',
 'Jungle Shuffle']

Mulan has 10 words in its vector and Shrek has 15, all with a count of 1. They share the three genre words adventure, animation and family, and two words from the name of a voice actor who appears in both.

cos(A, B) = (3 + 2) / (√10 × √15) = 5 / 12.2474 = 0.4082

That is the top score in Mulan's row. A shared voice actor counts exactly as much as a shared genre: the model only sees words.

Die Hard: sequels found through the cast

get_recommendations('Die Hard', df, sim) returns:

['Die Hard 2',
 'The Prince',
 'Sphinx',
 'Die Hard: With a Vengeance',
 '13 Hours: The Secret Soldiers of Benghazi',
 'Act of Valor',
 'Live Free or Die Hard',
 'A Good Day to Die Hard',
 'Broken Arrow',
 'Surrogates']

Four sequels are in the list, at ranks 1, 4, 7 and 8, but not because of the title: die and hard fell under min_df=20. The sequels are found through cast names and the genres action and thriller. Die Hard 2 is first, with 0.7303, because it also shares four keywords: based, novel, helicopter and journalist.

To explore any of these lists, open the similarity explorer full screen. Pick a movie, press Play to rank its row, and click any bar to see the shared words and the division behind its score.

Where it goes wrong

• Empty vectors. 9 movies keep no word after min_df=20, so every score in their row is 0.
• Duplicate titles. Two different movies are called Batman (rows 1359 and 4267); The Host and Out of the Blue are also shared by two movies each. The function always uses the first match.
• Ties with itself. In 14 rows the movie is not entry 0 of its own sorted row: the 9 empty vectors, and 5 movies whose one or two words (for example only drama, or only documentary) are exactly the words of a movie earlier in the file, which then also scores 1.0. In those rows [1:count + 1] skips the wrong movie.
• Content only. The model never sees ratings or what other viewers watched, so it cannot tell a good movie from a bad one with similar words.

Before trusting a list, print the scores next to the titles.

Worked example: Sharkskin

Sharkskin is one of the 9 empty vectors: its genres and keywords are empty, and its other words appear in fewer than 20 movies. Every score in its row is 0, so sorted keeps the whole row in file order.

get_recommendations('Sharkskin', df, sim, count=3) returns:

["Pirates of the Caribbean: At World's End", 'Spectre', 'The Dark Knight Rises']

These are simply rows 1, 2 and 3 of the file. Entry 0 of the sorted row is Avatar, row 0 of the file, not Sharkskin, so the slice even skipped the wrong movie.

Part 4: Project Teams Presentations and Discussions

This week's milestone is a presentation of your team project followed by a discussion. Structure it along the milestones of the plan so far:

1. The idea, or the research paper, and why it matters.
2. The data: its source, the exploration and the cleaning.
3. The features and the target variable you chose.
4. The model you implemented and how you evaluated it.
5. What you will improve next.

During the discussion:

• Every member should be able to explain every step, not only their own part.
• Show the notebook running and the numbers it printed, not only screenshots.
• Name one thing that did not work and what you learned from it.
• Write down the questions you are asked: they are your to-do list for next week.

If your project contains text, such as titles, reviews or descriptions, it can be vectorized and compared exactly as in this lesson.

Common mistakes

• Reading Data/movies.csv in Colab, where that folder does not exist.
• Forgetting fillna(''), so a missing value breaks CountVectorizer.
• Returning entry 0 of the sorted row: the liked movie recommends itself.
• Adding up shared words and forgetting to divide by the two lengths.
• Trusting a list without its scores: empty vectors, ties and duplicate titles all give misleading lists.
• Expecting a rare title word to count when min_df has dropped it.

Key takeaways

1. A content-based recommender suggests the items whose content is closest to an item you liked.
2. CountVectorizer turns text into count vectors; stop_words and min_df decide which words count.
3. Cosine similarity is the dot product divided by the two lengths, between 0 and 1 for counts, and it ignores how long a text is.
4. cosine_similarity gives the full matrix, and one sorted row of it is a recommendation list.
5. Always look at the scores: empty vectors, ties and duplicate titles give misleading lists.

Practice

About 30 minutes. Try each task before you read its answer at the end of the page.

Practice 1: a sixth movie (about 10 minutes)

Row 11 of the file, Quantum of Solace, has the genres Adventure Action Thriller Crime.

1. Write its count vector over the same 8 words as the worked example, and compute its length.
2. Compute its cosine with each of the five movies of the worked example.
3. Which two movies would you recommend to a viewer who liked it?

Hint: all four of its genres are already in the vocabulary, so the columns stay the same.

Practice 2: counts and length (about 5 minutes)

Three short texts: X = space war space alien, Y = space alien robot, and Z = X written twice (space war space alien space war space alien).

1. With the vocabulary alien, robot, space, war, write X, Y and Z as count vectors.
2. Compute cos(X, Y), cos(Z, Y) and cos(X, Z).
3. What does repeating a text do to its cosine?

Practice 3: scores in Colab (about 10 minutes)

1. Run the notebook up to sim = cosine_similarity(word_matrix).
2. Copy get_recommendations and change it so that it returns (title, score) pairs, with each score rounded to 4 decimal places.
3. Print the top 5 for 'The Dark Knight Rises'.
4. Check the first score by hand from word_matrix: compute the dot product of the two rows and their two sums of squares.

Practice 4: change min_df (about 5 minutes)

Rebuild the vectors with min_df=5 instead of 20, print the new shape of the word matrix, and rerun the recommendations for 'Skyfall' with scores. What happens to the score of Spectre, and why?

Answers

Answer 1

The vector of Quantum of Solace, in the order action, adventure, crime, drama, fantasy, fiction, science, thriller, is [1 1 1 0 0 0 0 1]. It has 4 words, so its length is √4 = 2.

Recommend Spectre and The Dark Knight Rises. Spectre shares action, adventure and crime; The Dark Knight Rises shares action, crime and thriller, but it has 4 words instead of 3, so its cosine is lower. cosine_similarity on the six vectors gives the same row: [0.4472, 0.5774, 0.866, 0.75, 0.5, 1.0].

Answer 2

• X = [1, 0, 2, 1], Y = [1, 1, 1, 0], Z = [2, 0, 4, 2].
• cos(X, Y): X · Y = 1 + 0 + 2 + 0 = 3, ‖X‖ = √6 = 2.4495, ‖Y‖ = √3 = 1.7321, so 3 / 4.2426 = 0.7071.
• cos(Z, Y): Z · Y = 2 + 0 + 4 + 0 = 6, ‖Z‖ = √24 = 4.8990, so 6 / 8.4853 = 0.7071.
• cos(X, Z): X · Z = 2 + 0 + 8 + 2 = 12, and ‖X‖ × ‖Z‖ = 2.4495 × 4.8990 = 12, so 1.

Repeating a text doubles every count, which doubles the dot product and the length together. The direction does not change, so the cosine does not change: Z is exactly as similar to Y as X is.

Answer 3

def get_recommendations_with_scores(title, df, sim, count=10):
    index = df.index[df['title'].str.lower() == title.lower()]
    if (len(index) == 0):
        return []
    similarities = list(enumerate(sim[index[0]]))
    recommendations = sorted(similarities, key=lambda x: x[1], reverse=True)
    top_recs = recommendations[1:count + 1]
    return [(df.iloc[i]['title'], round(float(s), 4)) for i, s in top_recs]
 
for pair in get_recommendations_with_scores('The Dark Knight Rises', df, sim, count=5):
    print(pair)

('The Dark Knight', 0.7924)
('Batman Begins', 0.7348)
('Harsh Times', 0.4667)
('The Killer Inside Me', 0.4648)
("Amidst the Devil's Wings", 0.4619)

The check by hand:

a = df.index[df['title'] == 'The Dark Knight Rises'][0]
b = df.index[df['title'] == 'The Dark Knight'][0]
A = word_matrix[a].toarray()[0]
B = word_matrix[b].toarray()[0]
print(A @ B, A @ A, B @ B)

19 25 23

cos(A, B) = 19 / (√25 × √23) = 19 / (5 × 4.7958) = 19 / 23.9792 = 0.7924

This time the counts are not all 1: crime appears twice in both movies (once in the genres and once in the keywords), so the sums of squares, 25 and 23, are larger than the numbers of words, 22 and 20. The two movies share the title words dark and knight, four genres, several keywords and five name words.

Answer 4

vectorizer5 = CountVectorizer(stop_words='english', min_df=5)
word_matrix5 = vectorizer5.fit_transform(df['features'])
print(word_matrix5.shape)
sim5 = cosine_similarity(word_matrix5)
print(get_recommendations_with_scores('Skyfall', df, sim5, count=10))

(4803, 3852)
[('Spectre', 0.55), ('Quantum of Solace', 0.4564), ('Casino Royale', 0.35), ('Diamonds Are Forever', 0.313), ('I Spy', 0.2739), ('Johnny English Reborn', 0.2739), ('Coriolanus', 0.2712), ('Goldfinger', 0.2635), ('Sanctum', 0.2582), ('Lara Croft: Tomb Raider', 0.25)]

• The vocabulary grows from 918 to 3,852 words.
• Spectre is still first, but its score drops from 0.7333 to 0.55. With min_df=5, Skyfall and Spectre each keep 20 words instead of 15, while they still share the same 11:

cos(A, B) = 11 / (√20 × √20) = 11 / 20 = 0.55

Longer vectors with the same overlap give a lower cosine. The list changes too: Casino Royale, Coriolanus, Goldfinger and Lara Croft: Tomb Raider enter, while Clash of the Titans, Die Another Day, Wrath of the Titans and Blackthorn leave.