-
Notifications
You must be signed in to change notification settings - Fork 0
/
eda.html
220 lines (199 loc) · 10.2 KB
/
eda.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta name="description" content="">
<meta name="author" content="">
<title>CS109 Spotify Project</title>
<!-- Bootstrap core CSS -->
<link href="vendor/bootstrap/css/bootstrap.min.css" rel="stylesheet">
<!-- Custom styles for this template -->
<link href="css/logo-nav.css" rel="stylesheet">
</head>
<body>
<!-- Navigation -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark fixed-top">
<div class="container">
<a class="navbar-brand" href="#">
<img src="spotify4.png" width="100" height="30" alt="">
</a>
<button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarResponsive" aria-controls="navbarResponsive" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
<div class="collapse navbar-collapse" id="navbarResponsive">
<ul class="navbar-nav ml-auto">
<li class="nav-item active">
<li class="nav-item">
<a class="nav-link" href="index.html">Home</a>
</li>
<li class="nav-item">
<a class="nav-link" href="literaturereview.html">Literature Review</a>
</li>
<li class="nav-item">
<a class="nav-link" href="eda.html">EDA</a>
</li>
<li class="nav-item">
<a class="nav-link" href="model.html">Model</a>
</li>
<li class="nav-item">
<a class="nav-link" href="results.html">Results</a>
</li>
<li class="nav-item">
<a class="nav-link" href="conclusion.html">Conclusion</a>
</li>
<li class="nav-item">
<a class="nav-link" href="citations.html">References</a>
</li>
</ul>
</div>
</div>
</nav>
<!-- Page Content -->
<div class="container">
<h1 class="mt-5">Exploratory Data Analysis</h1>
<p>
The only data set we are using to solve the playlist prediction problem is the Spotify Million Playlists data set. This data set contains individual statistics for each of the one million spotify playlists, such as playlist name, number of followers, last modified, a list of each individual song in that playlist, and several other columns.
</p>
<p>
The first potential approach we explored was based on the assumption of latent classes, then using maximum likelihood methods to parameterize the latent classes and classify new songs. We theorized these latent classes might end up capturing some genre information, given the large amount of playlists with similar, genre defining names like “metal” or “disney”. Each spotify playlist in the data set would be categorized into an arbitrary latent class. Then, given an inputted playlist, we could predict new songs based on the playlist’s likely latent class. To explore the possibility of playlists with similar names containing similar songs, we first looked at the top 20 most common playlist names. Below is a graph of the similarity index for the 20 names, which we define as the average percentage of songs present in both of 2 randomly selected playlists of the same name. We ran this exercise on 10 percent of the data - 100,000 playlists. Names with a high similarity index would theoretically be more prone to suggesting the most popular songs of those playlists, while names with a lower similarity index might be more prone to suggesting a wider variety of songs rather than the few most popular songs of playlists with those names.
</p>
<img style="padding: 0 15px float: left," src="similarity_scores.jpg" width="599" height="442" align=”right” alt="" class='center'>
<p>
Based on these findings, we believe that there are a decent number of playlists that are similar. That is to say: many of the millions of songs are rarely touched. An alternative approach that evaluates on a metric such as novelness of suggestion might hold that effective playlists would introduce listeners to some of those songs that are currently untouched. However, considering our current metric of evaluation is based on successfully completing playlists based off a slice of the initial part of the playlist, this is less of a concern. This is all a somewhat roundabout way of saying that the fact that there are many playlists with fairly high similarity indexes plus the fact that playlists are using many of the same songs over and over again signals that collaborative filtering can be an effective route to tackling this problem, as collaborative filtering is good at predicting similar songs but is less effective at generating novel suggestions.
</p>
<p>
The main benefit of collaborative filtering in relation to this problem, however, lies with the nature of the data we are working with. Collaborative filtering completely ignores all features relating to specifics of the actual songs themselves. Content-based models might use features such as lyrics of a song, or waveforms of their audio data. Collaborative filtering, on the other hand, only utilizes information about which playlists each song is included in. This makes it perfect for use on the Million Playlists Dataset, given that this dataset has no song-specific feature information. The dataset is in essence simply a list of songs in playlists, which is exactly the type of data collaborative filtering uses as inputs. Collaborative filtering does not leverage or make inferences about any meta-information about playlists such as intended genre or mood in the way that a hybrid of content-based approach might. However, our EDA showed that in many cases, the problem is a fairly straightforward one where people have fairly straightforward aims and create fairly similar playlists. This signals that other approaches may overfit the data and miss the elegance of a simple model.
</p>
<p>
We also investigated how we might weight playlists. We believed that playlists with more followers might deserve more influence on the final model than models without many followers. We investigated the distribution of followers for playlists and found the following breakdown.
</p>
<table border= '1px solid black'>
<tr>
<th>Percentile</th>
<th>Number of followers</th>
</tr>
<tr>
<td>0.50 </td>
<td> 1</td>
</tr>
<tr>
<td> 0.75</td>
<td> 1</td>
</tr>
<tr>
<td> 0.90</td>
<td> </td>
</tr>
<tr>
<td> 0.95</td>
<td> 3</td>
</tr>
<tr>
<td> 0.99</td>
<td> 8</td>
</tr>
<tr>
<td> 0.9999 (top 100 playlists)</td>
<td> 1995</td>
</tr>
</table>
<p>
75 percent of playlists have only one follower. In fact, the 99th percentile only has 8 followers. Within the most popular 1% of playlists, there is small subset of playlists that have 1000 followers.
</p>
<p>
One might be tempted to consider these popular playlists as outliers, perhaps to be discarded. However, we theorized that popular models might have important insights into listener preference and therefore should actually be given more weight in potential models. Songs that appear disproportionately in popular playlists but not in the rest of the population could serve as good recommendation choices. Thus, we calculated the most popular choices in the subset of popular playlists and the most popular choices in the data set as a whole:
</p>
<table border= '1px solid black'>
<tr>
<th colspan="2">Whole dataset</th>
<th colspan="2">100 most popular playlists</th>
</tr>
<tr>
<td>Song name</td>
<td>Percent of playlists</td>
<td>Song name</td>
<td>Percent of playlists</td>
</tr>
<tr>
<td>Humble</td>
<td>4.35%</td>
<td>Sign of the Times</td>
<td>11%</td>
</tr>
<tr>
<td>One Dance</td>
<td>4.34%</td>
<td>Chasing Cars</td>
<td>7%</td>
</tr>
<tr>
<td>Broccoli</td>
<td>4.13%</td>
<td>Iris</td>
<td>6%</td>
</tr>
<tr>
<td>Closer</td>
<td>4.11%</td>
<td>Let it Go</td>
<td>5%</td>
</tr>
<tr>
<td>Congratulations</td>
<td>4.00%</td>
<td>Feel it Still</td>
<td>5%</td>
</tr>
<tr>
<td>Caroline</td>
<td>3.52%</td>
<td>Scar Tissue</td>
<td>5%</td>
</tr>
<tr>
<td>iSpy</td>
<td>3.51%</td>
<td>Drops of Jupiter</td>
<td>5%</td>
</tr>
<tr>
<td>Bad and Boujee</td>
<td>3.50%</td>
<td>Fix You</td>
<td>5%</td>
</tr>
<tr>
<td>Location</td>
<td>3.50%</td>
<td>The Scientist</td>
<td>5%</td>
</tr>
<tr>
<td>XO TOUR Llif3</td>
<td>3.49%</td>
<td>She Will be Loved</td>
<td>5%</td>
</tr>
</table>
<p>
The sample size of the popular playlist is somewhat limited - it’s only 100 playlists. The songs in the top 10 of the popular playlists covers a wider range of songs. While not explicitly included in the data, all the songs in the overall playlist count were released in the last three years. All but one (Closer) could be classified as Hip-Hop and R&B. Most songs in the most popular playlists were not exclusively from Hip-Hop and R&B and spanned a more expansive range of release dates: “Feel it Still” was released in 2017 but “Iris” was released in 1998.
</p>
<p>
The similarity score reinforces this finding that the most popular playlists used a broad range of songs. The similarity score between the top 100 playlists was quite small, at 0.002. This means the most popular playlists might be useful for broad recommendation for many different types of playlists, due to the large range of songs they include.
</p>
</div>
<!-- /.container -->
<!-- Bootstrap core JavaScript -->
<script src="vendor/jquery/jquery.min.js"></script>
<script src="vendor/bootstrap/js/bootstrap.bundle.min.js"></script>
<footer class="py-5 bg-dark">
<div class="container">
<p class="m-0 text-center text-white">Template from Startbootstrap.com Logonav design </p>
<p class="m-0 text-center text-white">By David Gibson and Theo Lebryk</p>
<p class="m-0 text-center text-white">Group #53</p>
</div>
<!-- /.container -->
</footer>
</body>
</html>