Team3_Modeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Team 3 Final Project - Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using a variety of Machine Learning Models in an unsupervised manner to produce a Collaborative Filtering based Recommender System.\n",
    "\n",
    "K Nearest Neigbors will produce the actual recommendations based on cosine similarity between anime titles. Recommendations will be made based on user's last watched anime, this will be the query for KNN. Titles already watched (completed) by the user will be excluded from the dataset for KNN (don't want to recommend already watched titles). The K (default 10) nearest neighbors to the last watched title will be the recommendations.\n",
    "\n",
    "BERT Model (Bi-Directional Encoder Representations from Transformers) - Default Pre-Trained BERT model will be used on the synopsis table to generate a vector embedding or semantic representation that can be used by KNN to produce more distinct recommendations. **Note: This will significantly increase the complexity of the project and will only be used if there is enough time to implement**\n",
    "\n",
    "K-Means Clustering - There are currently 36 unique genres that may be too similar and titles can be attached to many genres. We want to simplify the genres by generating clusters that will serve to reduce the dimensionality from 36 genres to k - clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/secretstorage/dhcrypto.py:16: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead\n",
      "  from cryptography.utils import int_from_bytes\n",
      "/opt/conda/lib/python3.7/site-packages/secretstorage/util.py:25: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead\n",
      "  from cryptography.utils import int_from_bytes\n",
      "Collecting pyathena\n",
      "  Using cached PyAthena-2.5.1-py3-none-any.whl (40 kB)\n",
      "Requirement already satisfied: boto3>=1.4.4 in /opt/conda/lib/python3.7/site-packages (from pyathena) (1.20.23)\n",
      "Requirement already satisfied: botocore>=1.5.52 in /opt/conda/lib/python3.7/site-packages (from pyathena) (1.23.23)\n",
      "Requirement already satisfied: tenacity>=4.1.0 in /opt/conda/lib/python3.7/site-packages (from pyathena) (8.0.1)\n",
      "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3>=1.4.4->pyathena) (0.10.0)\n",
      "Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /opt/conda/lib/python3.7/site-packages (from boto3>=1.4.4->pyathena) (0.5.0)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /opt/conda/lib/python3.7/site-packages (from botocore>=1.5.52->pyathena) (2.8.1)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /opt/conda/lib/python3.7/site-packages (from botocore>=1.5.52->pyathena) (1.26.7)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore>=1.5.52->pyathena) (1.14.0)\n",
      "Installing collected packages: pyathena\n",
      "Successfully installed pyathena-2.5.1\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\n",
      "\u001b[33mWARNING: You are using pip version 21.3.1; however, version 22.0.4 is available.\n",
      "You should consider upgrading via the '/opt/conda/bin/python -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install pyathena"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is for MAR@K metrics\n",
    "#!pip install recmetrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install ml_metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import boto3\n",
    "from botocore.exceptions import ClientError\n",
    "import sagemaker\n",
    "import pyathena as pa\n",
    "from pyathena.pandas.cursor import PandasCursor\n",
    "import os\n",
    "\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.preprocessing import MinMaxScaler, StandardScaler\n",
    "\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.neighbors import NearestNeighbors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess = sagemaker.Session()\n",
    "bucket = sess.default_bucket()\n",
    "role = sagemaker.get_execution_role()\n",
    "region = boto3.Session().region_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Athena modeling bucket\n",
    "s3_modeling_dir = 's3://ads508team3/modeling/'\n",
    "# Athena staging bucket\n",
    "s3_staging_dir = 's3://ads508team3/athena-staging/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create Connection for pyathena\n",
    "conn = pa.connect(region_name=region, s3_staging_dir=s3_staging_dir)\n",
    "cursor = pa.connect(s3_staging_dir=s3_staging_dir, region_name='us-east-1',cursor_class=PandasCursor).cursor()\n",
    "dbname = 'myanimelist'\n",
    "table_list = ['animelist', 'anime', 'synopsis', 'ratings', 'watching_status']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download Files from S3\n",
    "\n",
    "s3 = boto3.resource('s3')\n",
    "\n",
    "try:\n",
    "    s3.Bucket('ads508team3').download_file('modeling/training.csv', 'Data/training.csv')\n",
    "    s3.Bucket('ads508team3').download_file('modeling/semantics.csv', 'Data/semantics.csv')\n",
    "    s3.Bucket('ads508team3').download_file('modeling/anime_subset.csv', 'Data/anime_subset.csv')\n",
    "except botocore.exceptions.ClientError as e:\n",
    "    if e.response['Error']['Code'] == \"404\":\n",
    "        print(\"The object does not exist.\")\n",
    "    else:\n",
    "        raise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "training = pd.read_csv('Data/training.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>anime_id</th>\n",
       "      <th>user_rating</th>\n",
       "      <th>watching_status</th>\n",
       "      <th>watched_episodes</th>\n",
       "      <th>mal_id</th>\n",
       "      <th>aggregated_score</th>\n",
       "      <th>episodes</th>\n",
       "      <th>rating</th>\n",
       "      <th>ranked</th>\n",
       "      <th>...</th>\n",
       "      <th>Yaoi</th>\n",
       "      <th>Yuri</th>\n",
       "      <th>Movie</th>\n",
       "      <th>Music.2</th>\n",
       "      <th>Music.3</th>\n",
       "      <th>ONA</th>\n",
       "      <th>OVA</th>\n",
       "      <th>Other</th>\n",
       "      <th>Special</th>\n",
       "      <th>TV</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>336145</td>\n",
       "      <td>2251</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>13</td>\n",
       "      <td>2251</td>\n",
       "      <td>8.420000</td>\n",
       "      <td>13.0</td>\n",
       "      <td>5</td>\n",
       "      <td>143.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>228773</td>\n",
       "      <td>2251</td>\n",
       "      <td>9</td>\n",
       "      <td>2</td>\n",
       "      <td>13</td>\n",
       "      <td>2251</td>\n",
       "      <td>8.420000</td>\n",
       "      <td>13.0</td>\n",
       "      <td>5</td>\n",
       "      <td>143.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>61090</td>\n",
       "      <td>2251</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>13</td>\n",
       "      <td>2251</td>\n",
       "      <td>8.420000</td>\n",
       "      <td>13.0</td>\n",
       "      <td>5</td>\n",
       "      <td>143.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>26848</td>\n",
       "      <td>2251</td>\n",
       "      <td>9</td>\n",
       "      <td>2</td>\n",
       "      <td>13</td>\n",
       "      <td>2251</td>\n",
       "      <td>8.420000</td>\n",
       "      <td>13.0</td>\n",
       "      <td>5</td>\n",
       "      <td>143.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>195185</td>\n",
       "      <td>2251</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>13</td>\n",
       "      <td>2251</td>\n",
       "      <td>8.420000</td>\n",
       "      <td>13.0</td>\n",
       "      <td>5</td>\n",
       "      <td>143.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999991</th>\n",
       "      <td>300428</td>\n",
       "      <td>13503</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>13503</td>\n",
       "      <td>6.509999</td>\n",
       "      <td>11.0</td>\n",
       "      <td>3</td>\n",
       "      <td>13708.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999992</th>\n",
       "      <td>30459</td>\n",
       "      <td>34369</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>34369</td>\n",
       "      <td>6.509999</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3</td>\n",
       "      <td>14403.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999993</th>\n",
       "      <td>275530</td>\n",
       "      <td>7779</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>7779</td>\n",
       "      <td>6.210000</td>\n",
       "      <td>52.0</td>\n",
       "      <td>2</td>\n",
       "      <td>7295.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999994</th>\n",
       "      <td>272555</td>\n",
       "      <td>40406</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>40406</td>\n",
       "      <td>6.509999</td>\n",
       "      <td>16.0</td>\n",
       "      <td>1</td>\n",
       "      <td>15754.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999995</th>\n",
       "      <td>300428</td>\n",
       "      <td>35037</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>35037</td>\n",
       "      <td>6.509999</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>11589.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>999996 rows × 76 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        user_id  anime_id  user_rating  watching_status  watched_episodes  \\\n",
       "0        336145      2251           10                2                13   \n",
       "1        228773      2251            9                2                13   \n",
       "2         61090      2251           10                2                13   \n",
       "3         26848      2251            9                2                13   \n",
       "4        195185      2251            8                2                13   \n",
       "...         ...       ...          ...              ...               ...   \n",
       "999991   300428     13503            0                4                 0   \n",
       "999992    30459     34369            0                4                 0   \n",
       "999993   275530      7779            0                4                 0   \n",
       "999994   272555     40406            0                4                 0   \n",
       "999995   300428     35037            0                4                 0   \n",
       "\n",
       "        mal_id  aggregated_score  episodes  rating   ranked  ...  Yaoi  Yuri  \\\n",
       "0         2251          8.420000      13.0       5    143.0  ...     0     0   \n",
       "1         2251          8.420000      13.0       5    143.0  ...     0     0   \n",
       "2         2251          8.420000      13.0       5    143.0  ...     0     0   \n",
       "3         2251          8.420000      13.0       5    143.0  ...     0     0   \n",
       "4         2251          8.420000      13.0       5    143.0  ...     0     0   \n",
       "...        ...               ...       ...     ...      ...  ...   ...   ...   \n",
       "999991   13503          6.509999      11.0       3  13708.0  ...     0     0   \n",
       "999992   34369          6.509999       5.0       3  14403.0  ...     0     0   \n",
       "999993    7779          6.210000      52.0       2   7295.0  ...     0     0   \n",
       "999994   40406          6.509999      16.0       1  15754.0  ...     0     0   \n",
       "999995   35037          6.509999       1.0       1  11589.0  ...     0     0   \n",
       "\n",
       "        Movie  Music.2  Music.3  ONA  OVA  Other  Special   TV  \n",
       "0         0.0        0      0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "1         0.0        0      0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "2         0.0        0      0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "3         0.0        0      0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "4         0.0        0      0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "...       ...      ...      ...  ...  ...    ...      ...  ...  \n",
       "999991    0.0        0      0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "999992    0.0        0      0.0  0.0  0.0    0.0      1.0  0.0  \n",
       "999993    0.0        0      0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "999994    0.0        0      0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "999995    0.0        0      0.0  0.0  1.0    0.0      0.0  0.0  \n",
       "\n",
       "[999996 rows x 76 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['user_id', 'anime_id', 'user_rating', 'watching_status',\n",
       "       'watched_episodes', 'mal_id', 'aggregated_score', 'episodes', 'rating',\n",
       "       'ranked', 'popularity', 'members', 'score_10', 'score_9', 'score_8',\n",
       "       'score_7', 'score_6', 'score_5', 'score_4', 'score_3', 'score_2',\n",
       "       'score_1', 'days_since_aired', 'Action', 'Adventure', 'Cars', 'Comedy',\n",
       "       'Dementia', 'Demons', 'Drama', 'Ecchi', 'Fantasy', 'Game', 'Harem',\n",
       "       'Hentai', 'Historical', 'Horror', 'Josei', 'Kids', 'Magic',\n",
       "       'Martial Arts', 'Mecha', 'Military', 'Music', 'Music.1', 'Mystery',\n",
       "       'Parody', 'Police', 'Psychological', 'Romance', 'Samurai', 'School',\n",
       "       'Sci-Fi', 'Seinen', 'Shoujo', 'Shoujo Ai', 'Shounen', 'Shounen Ai',\n",
       "       'Slice of Life', 'Space', 'Sports', 'Super Power', 'Supernatural',\n",
       "       'Thriller', 'Unknown', 'Vampire', 'Yaoi', 'Yuri', 'Movie', 'Music.2',\n",
       "       'Music.3', 'ONA', 'OVA', 'Other', 'Special', 'TV'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import anime subset\n",
    "\n",
    "Used for KNN of titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "anime_subset = pd.read_csv('Data/anime_subset.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mal_id</th>\n",
       "      <th>score</th>\n",
       "      <th>episodes</th>\n",
       "      <th>rating</th>\n",
       "      <th>ranked</th>\n",
       "      <th>popularity</th>\n",
       "      <th>members</th>\n",
       "      <th>score_10</th>\n",
       "      <th>score_9</th>\n",
       "      <th>score_8</th>\n",
       "      <th>...</th>\n",
       "      <th>Yaoi</th>\n",
       "      <th>Yuri</th>\n",
       "      <th>Movie</th>\n",
       "      <th>Music.2</th>\n",
       "      <th>Music.3</th>\n",
       "      <th>ONA</th>\n",
       "      <th>OVA</th>\n",
       "      <th>Other</th>\n",
       "      <th>Special</th>\n",
       "      <th>TV</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>8.780000</td>\n",
       "      <td>26.0</td>\n",
       "      <td>5</td>\n",
       "      <td>28.0</td>\n",
       "      <td>39</td>\n",
       "      <td>1251960</td>\n",
       "      <td>229170.0</td>\n",
       "      <td>182126.0</td>\n",
       "      <td>131625.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>8.390000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5</td>\n",
       "      <td>159.0</td>\n",
       "      <td>518</td>\n",
       "      <td>273145</td>\n",
       "      <td>30043.0</td>\n",
       "      <td>49201.0</td>\n",
       "      <td>49505.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>8.240000</td>\n",
       "      <td>26.0</td>\n",
       "      <td>3</td>\n",
       "      <td>266.0</td>\n",
       "      <td>201</td>\n",
       "      <td>558913</td>\n",
       "      <td>50229.0</td>\n",
       "      <td>75651.0</td>\n",
       "      <td>86142.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>7.270000</td>\n",
       "      <td>26.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2481.0</td>\n",
       "      <td>1467</td>\n",
       "      <td>94683</td>\n",
       "      <td>2182.0</td>\n",
       "      <td>4806.0</td>\n",
       "      <td>10128.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>6.980000</td>\n",
       "      <td>52.0</td>\n",
       "      <td>2</td>\n",
       "      <td>3710.0</td>\n",
       "      <td>4369</td>\n",
       "      <td>13224</td>\n",
       "      <td>312.0</td>\n",
       "      <td>529.0</td>\n",
       "      <td>1242.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17557</th>\n",
       "      <td>48481</td>\n",
       "      <td>6.509999</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>7896.0</td>\n",
       "      <td>13116</td>\n",
       "      <td>354</td>\n",
       "      <td>39.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>92.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17558</th>\n",
       "      <td>48483</td>\n",
       "      <td>6.509999</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>7896.0</td>\n",
       "      <td>17562</td>\n",
       "      <td>7010</td>\n",
       "      <td>39.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>92.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17559</th>\n",
       "      <td>48488</td>\n",
       "      <td>6.509999</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5</td>\n",
       "      <td>7896.0</td>\n",
       "      <td>17558</td>\n",
       "      <td>11309</td>\n",
       "      <td>1.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>92.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17560</th>\n",
       "      <td>48491</td>\n",
       "      <td>6.509999</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3</td>\n",
       "      <td>7896.0</td>\n",
       "      <td>17565</td>\n",
       "      <td>1386</td>\n",
       "      <td>39.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>92.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17561</th>\n",
       "      <td>48492</td>\n",
       "      <td>6.509999</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>7896.0</td>\n",
       "      <td>17563</td>\n",
       "      <td>310</td>\n",
       "      <td>39.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>92.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>17562 rows × 71 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       mal_id     score  episodes  rating  ranked  popularity  members  \\\n",
       "0           1  8.780000      26.0       5    28.0          39  1251960   \n",
       "1           5  8.390000       1.0       5   159.0         518   273145   \n",
       "2           6  8.240000      26.0       3   266.0         201   558913   \n",
       "3           7  7.270000      26.0       3  2481.0        1467    94683   \n",
       "4           8  6.980000      52.0       2  3710.0        4369    13224   \n",
       "...       ...       ...       ...     ...     ...         ...      ...   \n",
       "17557   48481  6.509999       2.0       6  7896.0       13116      354   \n",
       "17558   48483  6.509999       2.0       6  7896.0       17562     7010   \n",
       "17559   48488  6.509999       2.0       5  7896.0       17558    11309   \n",
       "17560   48491  6.509999       2.0       3  7896.0       17565     1386   \n",
       "17561   48492  6.509999       2.0       6  7896.0       17563      310   \n",
       "\n",
       "       score_10   score_9   score_8  ...  Yaoi  Yuri  Movie  Music.2  Music.3  \\\n",
       "0      229170.0  182126.0  131625.0  ...     0     0    0.0        0      0.0   \n",
       "1       30043.0   49201.0   49505.0  ...     0     0    1.0        0      0.0   \n",
       "2       50229.0   75651.0   86142.0  ...     0     0    0.0        0      0.0   \n",
       "3        2182.0    4806.0   10128.0  ...     0     0    0.0        0      0.0   \n",
       "4         312.0     529.0    1242.0  ...     0     0    0.0        0      0.0   \n",
       "...         ...       ...       ...  ...   ...   ...    ...      ...      ...   \n",
       "17557      39.0      71.0      92.0  ...     0     0    0.0        0      0.0   \n",
       "17558      39.0      71.0      92.0  ...     0     0    0.0        0      0.0   \n",
       "17559       1.0      71.0      92.0  ...     0     0    0.0        0      0.0   \n",
       "17560      39.0      71.0      92.0  ...     0     0    0.0        0      0.0   \n",
       "17561      39.0      71.0      92.0  ...     0     0    0.0        0      0.0   \n",
       "\n",
       "       ONA  OVA  Other  Special   TV  \n",
       "0      0.0  0.0    0.0      0.0  1.0  \n",
       "1      0.0  0.0    0.0      0.0  0.0  \n",
       "2      0.0  0.0    0.0      0.0  1.0  \n",
       "3      0.0  0.0    0.0      0.0  1.0  \n",
       "4      0.0  0.0    0.0      0.0  1.0  \n",
       "...    ...  ...    ...      ...  ...  \n",
       "17557  1.0  0.0    0.0      0.0  0.0  \n",
       "17558  0.0  0.0    0.0      0.0  1.0  \n",
       "17559  0.0  0.0    0.0      0.0  1.0  \n",
       "17560  0.0  0.0    0.0      0.0  1.0  \n",
       "17561  0.0  0.0    0.0      0.0  1.0  \n",
       "\n",
       "[17562 rows x 71 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anime_subset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "anime_subset.set_index(['mal_id'], inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Standardization and Min-max scaling variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['score', 'episodes', 'rating', 'ranked', 'popularity', 'members',\n",
       "       'score_10', 'score_9', 'score_8', 'score_7', 'score_6', 'score_5',\n",
       "       'score_4', 'score_3', 'score_2', 'score_1', 'days_since_aired',\n",
       "       'Action', 'Adventure', 'Cars', 'Comedy', 'Dementia', 'Demons', 'Drama',\n",
       "       'Ecchi', 'Fantasy', 'Game', 'Harem', 'Hentai', 'Historical', 'Horror',\n",
       "       'Josei', 'Kids', 'Magic', 'Martial Arts', 'Mecha', 'Military', 'Music',\n",
       "       'Music.1', 'Mystery', 'Parody', 'Police', 'Psychological', 'Romance',\n",
       "       'Samurai', 'School', 'Sci-Fi', 'Seinen', 'Shoujo', 'Shoujo Ai',\n",
       "       'Shounen', 'Shounen Ai', 'Slice of Life', 'Space', 'Sports',\n",
       "       'Super Power', 'Supernatural', 'Thriller', 'Unknown', 'Vampire', 'Yaoi',\n",
       "       'Yuri', 'Movie', 'Music.2', 'Music.3', 'ONA', 'OVA', 'Other', 'Special',\n",
       "       'TV'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anime_subset.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop Music.2 and Music.3\n",
    "anime_subset = anime_subset.drop(columns=['Music.2', 'Music.3'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standard Scale the aggregated user score\n",
    "anime_subset['score'] = StandardScaler().fit_transform(anime_subset[['score']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Min-max scaling\n",
    "mm_scaling_cols = ['episodes', 'rating', 'ranked', 'popularity', 'members',\n",
    "       'score_10', 'score_9', 'score_8', 'score_7', 'score_6', 'score_5',\n",
    "       'score_4', 'score_3', 'score_2', 'score_1', 'days_since_aired']\n",
    "anime_subset[mm_scaling_cols] = MinMaxScaler().fit_transform(anime_subset[mm_scaling_cols])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "      <th>episodes</th>\n",
       "      <th>rating</th>\n",
       "      <th>ranked</th>\n",
       "      <th>popularity</th>\n",
       "      <th>members</th>\n",
       "      <th>score_10</th>\n",
       "      <th>score_9</th>\n",
       "      <th>score_8</th>\n",
       "      <th>score_7</th>\n",
       "      <th>...</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Vampire</th>\n",
       "      <th>Yaoi</th>\n",
       "      <th>Yuri</th>\n",
       "      <th>Movie</th>\n",
       "      <th>ONA</th>\n",
       "      <th>OVA</th>\n",
       "      <th>Other</th>\n",
       "      <th>Special</th>\n",
       "      <th>TV</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mal_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.044157e+00</td>\n",
       "      <td>0.008181</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.001774</td>\n",
       "      <td>0.002220</td>\n",
       "      <td>0.483466</td>\n",
       "      <td>0.320601</td>\n",
       "      <td>0.340261</td>\n",
       "      <td>0.286693</td>\n",
       "      <td>0.205156</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2.521152e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.010076</td>\n",
       "      <td>0.029490</td>\n",
       "      <td>0.105479</td>\n",
       "      <td>0.042028</td>\n",
       "      <td>0.091919</td>\n",
       "      <td>0.107826</td>\n",
       "      <td>0.074490</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2.319997e+00</td>\n",
       "      <td>0.008181</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.016857</td>\n",
       "      <td>0.011443</td>\n",
       "      <td>0.215834</td>\n",
       "      <td>0.070268</td>\n",
       "      <td>0.141336</td>\n",
       "      <td>0.187625</td>\n",
       "      <td>0.162703</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.019190e+00</td>\n",
       "      <td>0.008181</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.157224</td>\n",
       "      <td>0.083518</td>\n",
       "      <td>0.036563</td>\n",
       "      <td>0.003051</td>\n",
       "      <td>0.008977</td>\n",
       "      <td>0.022058</td>\n",
       "      <td>0.038237</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>6.302889e-01</td>\n",
       "      <td>0.016688</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.235108</td>\n",
       "      <td>0.248733</td>\n",
       "      <td>0.005106</td>\n",
       "      <td>0.000435</td>\n",
       "      <td>0.000986</td>\n",
       "      <td>0.002703</td>\n",
       "      <td>0.005635</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48481</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.746712</td>\n",
       "      <td>0.000136</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48483</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.999829</td>\n",
       "      <td>0.002707</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48488</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.999601</td>\n",
       "      <td>0.004367</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48491</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000535</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48492</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.999886</td>\n",
       "      <td>0.000119</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>17562 rows × 68 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               score  episodes  rating    ranked  popularity   members  \\\n",
       "mal_id                                                                   \n",
       "1       3.044157e+00  0.008181     0.8  0.001774    0.002220  0.483466   \n",
       "5       2.521152e+00  0.000000     0.8  0.010076    0.029490  0.105479   \n",
       "6       2.319997e+00  0.008181     0.4  0.016857    0.011443  0.215834   \n",
       "7       1.019190e+00  0.008181     0.4  0.157224    0.083518  0.036563   \n",
       "8       6.302889e-01  0.016688     0.2  0.235108    0.248733  0.005106   \n",
       "...              ...       ...     ...       ...         ...       ...   \n",
       "48481   7.146486e-15  0.000327     1.0  0.500380    0.746712  0.000136   \n",
       "48483   7.146486e-15  0.000327     1.0  0.500380    0.999829  0.002707   \n",
       "48488   7.146486e-15  0.000327     0.8  0.500380    0.999601  0.004367   \n",
       "48491   7.146486e-15  0.000327     0.4  0.500380    1.000000  0.000535   \n",
       "48492   7.146486e-15  0.000327     1.0  0.500380    0.999886  0.000119   \n",
       "\n",
       "        score_10   score_9   score_8   score_7  ...  Unknown  Vampire  Yaoi  \\\n",
       "mal_id                                          ...                           \n",
       "1       0.320601  0.340261  0.286693  0.205156  ...        0        0     0   \n",
       "5       0.042028  0.091919  0.107826  0.074490  ...        0        0     0   \n",
       "6       0.070268  0.141336  0.187625  0.162703  ...        0        0     0   \n",
       "7       0.003051  0.008977  0.022058  0.038237  ...        0        0     0   \n",
       "8       0.000435  0.000986  0.002703  0.005635  ...        0        0     0   \n",
       "...          ...       ...       ...       ...  ...      ...      ...   ...   \n",
       "48481   0.000053  0.000131  0.000198  0.000000  ...        0        0     0   \n",
       "48483   0.000053  0.000131  0.000198  0.000458  ...        0        0     0   \n",
       "48488   0.000000  0.000131  0.000198  0.000458  ...        0        0     0   \n",
       "48491   0.000053  0.000131  0.000198  0.000458  ...        0        0     0   \n",
       "48492   0.000053  0.000131  0.000198  0.000458  ...        0        0     0   \n",
       "\n",
       "        Yuri  Movie  ONA  OVA  Other  Special   TV  \n",
       "mal_id                                              \n",
       "1          0    0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "5          0    1.0  0.0  0.0    0.0      0.0  0.0  \n",
       "6          0    0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "7          0    0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "8          0    0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "...      ...    ...  ...  ...    ...      ...  ...  \n",
       "48481      0    0.0  1.0  0.0    0.0      0.0  0.0  \n",
       "48483      0    0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "48488      0    0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "48491      0    0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "48492      0    0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "\n",
       "[17562 rows x 68 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Post scaling\n",
    "anime_subset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clustering using Kmeans\n",
    "\n",
    "We want to cluster the titles into distinct groups to prevent same similarity measures when using KNN. This will also reduce dimensionality of the dataset by replacing the genre columns with these clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "genres_list = ['Dementia', 'Shounen Ai', 'Adventure', 'Music', 'Martial Arts', 'Kids',\n",
    "               'Sports', 'Action', 'Cars', 'Hentai', 'Slice of Life', 'Vampire', 'Seinen',\n",
    "               'Thriller', 'Romance', 'Samurai', 'Historical', 'Drama', 'Unknown', 'Psychological', 'Comedy', \n",
    "               'Mystery', 'Josei', 'Military', 'Ecchi', 'Magic', 'Supernatural', 'Yaoi', 'Shounen', 'Sci-Fi', \n",
    "               'Super Power', 'Demons', 'Parody', 'Horror', 'Police', 'Fantasy', 'Game', 'Shoujo', 'Harem',\n",
    "               'School', 'Space', 'Mecha', 'Shoujo Ai', 'Yuri']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "      <th>episodes</th>\n",
       "      <th>rating</th>\n",
       "      <th>ranked</th>\n",
       "      <th>popularity</th>\n",
       "      <th>members</th>\n",
       "      <th>score_10</th>\n",
       "      <th>score_9</th>\n",
       "      <th>score_8</th>\n",
       "      <th>score_7</th>\n",
       "      <th>...</th>\n",
       "      <th>Unknown</th>\n",
       "      <th>Vampire</th>\n",
       "      <th>Yaoi</th>\n",
       "      <th>Yuri</th>\n",
       "      <th>Movie</th>\n",
       "      <th>ONA</th>\n",
       "      <th>OVA</th>\n",
       "      <th>Other</th>\n",
       "      <th>Special</th>\n",
       "      <th>TV</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mal_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.044157e+00</td>\n",
       "      <td>0.008181</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.001774</td>\n",
       "      <td>0.002220</td>\n",
       "      <td>0.483466</td>\n",
       "      <td>0.320601</td>\n",
       "      <td>0.340261</td>\n",
       "      <td>0.286693</td>\n",
       "      <td>0.205156</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2.521152e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.010076</td>\n",
       "      <td>0.029490</td>\n",
       "      <td>0.105479</td>\n",
       "      <td>0.042028</td>\n",
       "      <td>0.091919</td>\n",
       "      <td>0.107826</td>\n",
       "      <td>0.074490</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2.319997e+00</td>\n",
       "      <td>0.008181</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.016857</td>\n",
       "      <td>0.011443</td>\n",
       "      <td>0.215834</td>\n",
       "      <td>0.070268</td>\n",
       "      <td>0.141336</td>\n",
       "      <td>0.187625</td>\n",
       "      <td>0.162703</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.019190e+00</td>\n",
       "      <td>0.008181</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.157224</td>\n",
       "      <td>0.083518</td>\n",
       "      <td>0.036563</td>\n",
       "      <td>0.003051</td>\n",
       "      <td>0.008977</td>\n",
       "      <td>0.022058</td>\n",
       "      <td>0.038237</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>6.302889e-01</td>\n",
       "      <td>0.016688</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.235108</td>\n",
       "      <td>0.248733</td>\n",
       "      <td>0.005106</td>\n",
       "      <td>0.000435</td>\n",
       "      <td>0.000986</td>\n",
       "      <td>0.002703</td>\n",
       "      <td>0.005635</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48481</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.746712</td>\n",
       "      <td>0.000136</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48483</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.999829</td>\n",
       "      <td>0.002707</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48488</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.999601</td>\n",
       "      <td>0.004367</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48491</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000535</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48492</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.999886</td>\n",
       "      <td>0.000119</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>17562 rows × 68 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               score  episodes  rating    ranked  popularity   members  \\\n",
       "mal_id                                                                   \n",
       "1       3.044157e+00  0.008181     0.8  0.001774    0.002220  0.483466   \n",
       "5       2.521152e+00  0.000000     0.8  0.010076    0.029490  0.105479   \n",
       "6       2.319997e+00  0.008181     0.4  0.016857    0.011443  0.215834   \n",
       "7       1.019190e+00  0.008181     0.4  0.157224    0.083518  0.036563   \n",
       "8       6.302889e-01  0.016688     0.2  0.235108    0.248733  0.005106   \n",
       "...              ...       ...     ...       ...         ...       ...   \n",
       "48481   7.146486e-15  0.000327     1.0  0.500380    0.746712  0.000136   \n",
       "48483   7.146486e-15  0.000327     1.0  0.500380    0.999829  0.002707   \n",
       "48488   7.146486e-15  0.000327     0.8  0.500380    0.999601  0.004367   \n",
       "48491   7.146486e-15  0.000327     0.4  0.500380    1.000000  0.000535   \n",
       "48492   7.146486e-15  0.000327     1.0  0.500380    0.999886  0.000119   \n",
       "\n",
       "        score_10   score_9   score_8   score_7  ...  Unknown  Vampire  Yaoi  \\\n",
       "mal_id                                          ...                           \n",
       "1       0.320601  0.340261  0.286693  0.205156  ...        0        0     0   \n",
       "5       0.042028  0.091919  0.107826  0.074490  ...        0        0     0   \n",
       "6       0.070268  0.141336  0.187625  0.162703  ...        0        0     0   \n",
       "7       0.003051  0.008977  0.022058  0.038237  ...        0        0     0   \n",
       "8       0.000435  0.000986  0.002703  0.005635  ...        0        0     0   \n",
       "...          ...       ...       ...       ...  ...      ...      ...   ...   \n",
       "48481   0.000053  0.000131  0.000198  0.000000  ...        0        0     0   \n",
       "48483   0.000053  0.000131  0.000198  0.000458  ...        0        0     0   \n",
       "48488   0.000000  0.000131  0.000198  0.000458  ...        0        0     0   \n",
       "48491   0.000053  0.000131  0.000198  0.000458  ...        0        0     0   \n",
       "48492   0.000053  0.000131  0.000198  0.000458  ...        0        0     0   \n",
       "\n",
       "        Yuri  Movie  ONA  OVA  Other  Special   TV  \n",
       "mal_id                                              \n",
       "1          0    0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "5          0    1.0  0.0  0.0    0.0      0.0  0.0  \n",
       "6          0    0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "7          0    0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "8          0    0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "...      ...    ...  ...  ...    ...      ...  ...  \n",
       "48481      0    0.0  1.0  0.0    0.0      0.0  0.0  \n",
       "48483      0    0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "48488      0    0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "48491      0    0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "48492      0    0.0  0.0  0.0    0.0      0.0  1.0  \n",
       "\n",
       "[17562 rows x 68 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anime_subset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Elbow curve')"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmIAAAE/CAYAAADlrq9SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXhV1b3/8fc38zyPJGGeQWYZRXFGq6JWvdahagc72Gpvr221vfenrVo73LZXa6vVOtZWtApKFUWqRBxA5pkgEIYECIGEQMKcZP3+OBsMGCAHQvZJ8nk9z3myzzprn3zPeo7k4x7WMuccIiIiItLywvwuQERERKS9UhATERER8YmCmIiIiIhPFMREREREfKIgJiIiIuITBTERERERnyiIiUjIMrNbzeyjBs+dmXX3syYRkeakICYivjKz9Wa218xqGjwe87suEZGWEOF3ASIiwOXOuX/7XcTpYmYRzrlav+sQkdCjI2Ii0tpcambFZrbdzH5rZmEAZhZmZv9tZhvMrNzMXjCzZO+1583sv7ztPO8U53e9593NrNLMrLFfZmbfNLOVZlZtZivMbIjXfsRpUjN7zswe9LbHmVmpmf3EzMqAZ733uKxB/wjvMxx6v5Fm9omZVZnZYjMbdzoGT0RCi4KYiLQ2VwHDgCHABOBrXvut3uNcoCuQABw6xfkBMM7bPgco9n4CnA186BpZ783MrgXuB74KJAFXABVNrDMHSAM6AbcDLwFfafD6xcB259wCM8sD3gIe9Pa5G3jNzDKb+LtEpJVSEBORUPC6dyTo0OObx+n7a+dcpXNuI/B/fB5ubgR+75wrds7VAPcC15tZBIEgNtY7enY28BtgjLffOd7rjfkG8Bvn3FwXsMY5t6GJn6keuM85t985txf4B3CFmcV5r9/gtQHcBEx1zk11ztU756YD84BLm/i7RKSVUhATkVBwpXMupcHjqeP0LWmwvQHo4G138J43fC0CyHbOrQVqgEHAWOBNYLOZ9eL4QawAWBv0pwnY5pzbd+iJc24NsBK43AtjV/B5EOsEXNswjAJnAbkn+btFpJXQxfoi0toUAMu97Y7AZm97M4FAQ4PXaoGt3vMPgGuAKOfcJjP7gMApx1Rg0TF+VwnQ7Riv7QHiGjzPAUobPP/CqU4+Pz0ZBqzwwtmh3/M359zxjgSKSBukI2Ii0tr8yMxSzawAuAt42Wt/CfhPM+tiZgnAL4GXG9yt+AHwPWCm97wQ+D7wkXOu7hi/66/A3WY21AK6m9mhsLcIuMHMws1sPJ9fc3Y8E4GLgO/w+dEwgBcJHCm72Hu/GO+C//wmvKeItGIKYiISCv511Dxik4/T9w1gPoEg9BbwtNf+DPA3AkFrHbCPQNA65AMgkc+D2EcEjmjN5Bicc/8EHiIQmqqB1wlcTA+BEHg5UEXg+rTXT/QhnXNbgFnAaD4PkDjnSgjcePBTYBuBI2Q/Qv9Gi7R51siNQiIiIiLSAvR/WyIiIiI+URATERER8YmCmIiIiIhPFMREREREfKIgJiIiIuKTVjuha0ZGhuvcubPfZYSk3bt3Ex8f73cZrYrGLHgas+BpzIKnMQuOxit4LTVm8+fP3+6c+8L6sa02iHXu3Jl58+b5XUZIKiwsZNy4cX6X0apozIKnMQuexix4GrPgaLyC11JjZmaNrlOrU5MiIiIiPlEQExEREfGJgpiIiIiITxTERERERHyiICYiIiLiEwUxEREREZ8oiImIiIj4REFMRERExCcKYiIiIiI+URA7hnnrK3lryRa/yxAREZE2rNUucXS6/WVmMYtKqri4XzYR4cqrIiIi0vyUMI7h6sF5bKvez8drK/wuRURERNooBbFjOK9PFkkxEUxeUOp3KSIiItJGKYgdQ3REOJcN7MC05VvZvb/W73JERESkDVIQO46rB+ex92Ad7ywr87sUERERaYMUxI5jaKdUOqbFMXnhJr9LERERkTZIQew4zIwrB+fx8drtlO3c53c5IiIi0sYoiJ3AVYPzcA7eWKSjYiIiItK8FMROoEtGPIM7pjBpwSacc36XIyIiIm2IglgTXD04j1Vbq1mxZZffpYiIiEgboiDWBJcN6EBkuDF5gU5PioiISPNREGuC1PgoxvXK4o3Fm6mtq/e7HBEREWkjFMSaSEseiYiISHNTEGsiLXkkIiIizU1BrImiI8L50gAteSQiIiLNR0EsCFcP0ZJHIiIi0nwUxIIwrFMqBWmxWvJIREREmoWCWBDMjKsG52vJIxEREWkWCmJB0pJHIiIi0lwUxIJ0aMkjnZ4UERGRU6UgdhKuHpxHUVk1KzZrySMRERE5eQpiJ+HwkkcLNaeYiIiInDwFsZNwaMmj1xdpySMRERE5eQpiJ0lLHomIiMipalIQM7MUM3vVzIrMbKWZjTKz+81sk5kt8h6XNuh/r5mtMbNVZnZxg/bxXtsaM7unQXsXM/vUzFab2ctmFtW8H7P5ackjEREROVVNPSL2CPCOc643MBBY6bX/wTk3yHtMBTCzvsD1QD9gPPBnMws3s3DgT8AlQF/gK15fgF9779UD2AF8vRk+22mlJY9ERETkVJ0wiJlZEnA28DSAc+6Ac67qOLtMACY65/Y759YBa4Dh3mONc67YOXcAmAhMMDMDzgNe9fZ/HrjyZD9QS9KSRyIiInIqmnJErCuwDXjWzBaa2V/NLN577XtmtsTMnjGzVK8tDyhpsH+p13as9nSgyjlXe1R7yNOSRyIiInIqIprYZwjwfefcp2b2CHAP8BjwAOC8n78DvgZYI+/haDz0ueP0/wIzux24HSA7O5vCwsImlH96DU6t5V9rtjP5nfdJjQmNex9qampCYmxaE41Z8DRmwdOYBU9jFhyNV/D8HrOmBLFSoNQ596n3/FXgHufc1kMdzOwp4M0G/Qsa7J8PbPa2G2vfDqSYWYR3VKxh/yM4554EngQYNmyYGzduXBPKP7069d/NlP8tpDy2E1ed083vcgAoLCwkFMamNdGYBU9jFjyNWfA0ZsHReAXP7zE74SEc51wZUGJmvbym84EVZpbboNtVwDJvewpwvZlFm1kXoAcwB5gL9PDukIwicEH/FOecA2YA13j73wK8cYqfq8V0yYhnUIGWPBIREZHgNfVc2veBv5vZEmAQ8EvgN2a21Gs7F/hPAOfccuAVYAXwDnCHc67OO9r1PWAagbsuX/H6AvwE+KGZrSFwzdjTzfLpWsiXh2jJIxEREQleU05N4pxbBAw7qvnm4/R/CHiokfapwNRG2osJ3FXZKl02oAO/eHMFkxeW0rdD3xPvICIiIoJm1m8WWvJIREREToaCWDPRkkciIiISLAWxZqIlj0RERCRYCmLNREseiYiISLAUxJqRljwSERGRYCiINSMteSQiIiLBUBBrRmbGVYPy+Hjtdsp27vO7HBEREQlxCmLN7Koh+TgHbyzSUTERERE5PgWxZqYlj0RERKSpFMROg0NLHn2ydrvfpYiIiEgIUxA7Db48NJ+CtFh+NnkZ+w7W+V2OiIiIhCgFsdMgLiqCh68awLrtu3n0vdV+lyMiIiIhSkHsNDmrRwbXDM3nLzOLWb55p9/liIiISAhSEDuN/vtLfUiNi+Qnry3RYuAiIiLyBQpip1FKXBT3X9GPZZt28czH6/wuR0REREKMgthp9qUzcrmgTza/n/4ZGyp2+12OiIiIhBAFsdPMzHjwyv5EhoVx76SlOOf8LklERERChIJYC8hJjuEnl/Tmk7UV/HNeqd/liIiISIhQEGshNwzvyPAuaTz41grKd2kdShEREVEQazFhYcbDV5/Bvtp67v/Xcr/LERERkRCgINaCumUmcNf5PZi6tIxpy8v8LkdERER8piDWwm4/uyt9cpP4n9eXsXPvQb/LERERER8piLWwyPAwfv3lM9hes59fvV3kdzkiIiLiIwUxHwzIT+HrZ3XhpTkbmV1c4Xc5IiIi4hMFMZ/88MJedEyL495JS9l3sM7vckRERMQHCmI+iY0K55dXncG67bt55L3VfpcjIiIiPlAQ89FZPTK4dmg+T84sZtmmnX6XIyIiIi1MQcxnP/tSH1Ljorhn0hJq6+r9LkdERERakIKYz1Liovj5Ff1YtmkXT3+0zu9yREREpAUpiIWAS8/I4cK+2fx++mes377b73JERESkhSiIhQAz44EJ/YkKD+PeSUtxzvldkoiIiLQABbEQkZMcwz2X9mZWcQWvzCvxuxwRERFpAU0KYmaWYmavmlmRma00s1FmlmZm081stfcz1etrZvaoma0xsyVmNqTB+9zi9V9tZrc0aB9qZku9fR41M2v+jxr6vnJmR4Z3SePBt1ZSvmuf3+WIiIjIadbUI2KPAO8453oDA4GVwD3Ae865HsB73nOAS4Ae3uN24HEAM0sD7gNGAMOB+w6FN6/P7Q32G39qH6t1CgszfnX1GeyvrdcpShERkXbghEHMzJKAs4GnAZxzB5xzVcAE4Hmv2/PAld72BOAFFzAbSDGzXOBiYLpzrtI5twOYDoz3Xktyzs1ygeTxQoP3ane6ZiZwz/jevFdUzt9mb/C7HBERETmNmnJErCuwDXjWzBaa2V/NLB7Ids5tAfB+Znn984CGFzmVem3Hay9tpL3dum1MZ8b1yuSht1by2dZqv8sRERGR0ySiiX2GAN93zn1qZo/w+WnIxjR2fZc7ifYvvrHZ7QROYZKdnU1hYeFxymjdrurgWLCunq899SH/b1QsUeFNv2yupqamTY/N6aAxC57GLHgas+BpzIKj8Qqe32PWlCBWCpQ65z71nr9KIIhtNbNc59wW7/RieYP+BQ32zwc2e+3jjmov9NrzG+n/Bc65J4EnAYYNG+bGjRvXWLc2I6lTObc9N5dZe7K47/J+Td6vsLCQtj42zU1jFjyNWfA0ZsHTmAVH4xU8v8fshKcmnXNlQImZ9fKazgdWAFOAQ3c+3gK84W1PAb7q3T05EtjpnbqcBlxkZqneRfoXAdO816rNbKR3t+RXG7xXu3Zu7yxuHd2ZZz9ez4xV5SfeQURERFqVphwRA/g+8HcziwKKgdsIhLhXzOzrwEbgWq/vVOBSYA2wx+uLc67SzB4A5nr9fuGcq/S2vwM8B8QCb3sPAe65pDez1lbwo38u5u27ziYzMdrvkkRERKSZNCmIOecWAcMaeen8Rvo64I5jvM8zwDONtM8D+jellvYmJjKcR78ymMsf+4gfv7qYZ249k3Y6zZqIiEibo5n1W4FeOYn87NI+zFi1jec/We93OSIiItJMFMRaia+O6sR5vbP45dtFFJXt8rscERERaQYKYq2EmfGbawaQFBPJnS8tZN/BOr9LEhERkVOkINaKZCRE87vrBvLZ1hoenrrS73JERETkFCmItTLn9Mzka2O68PysDbxftNXvckREROQUKIi1Qj+5pBd9cpO4+59LKK/e53c5IiIicpIUxFqh6IhwHr1+ELv313L3P5dQX9/oilAiIiIS4hTEWqke2Yn892V9mfnZNp7VlBYiIiKtkoJYK3bTiI5c0CebX79dxIrNmtJCRESktVEQa8UOTWmREhfJnRMXsveAprQQERFpTRTEWrm0+Ch+d91A1pTX8NDUFX6XIyIiIkFQEGsDxvbI5Jtju/Di7I1MX6EpLURERFoLBbE24u6Le9GvQxI/fnUxO/bV+12OiIiINIGCWBsRHRHOI9cPZu/BOv66dD91mtJCREQk5CmItSHdsxK4//J+LK+o59H3VvtdjoiIiJyAglgb8x9nFjCmQwSPvr+awlXlfpcjIiIix6Eg1saYGV/tF0Wv7ER+8PIiNlXt9bskEREROQYFsTYoOtx4/Kah1NU5vvv3Beyv1fxiIiIioUhBrI3qkhHPb68dwOKSKh56a6Xf5YiIiEgjFMTasPH9c/nm2C68MGsDbyza5Hc5IiIichQFsTbux+N7c2bnVO55bSmrt1b7XY6IiIg0oCDWxkWGh/HYDUOIj47g2y/Op2Z/rd8liYiIiEdBrB3ITorhj18ZzLrtu7nntSU4p8leRUREQoGCWDsxqls6d1/cizeXbOH5T9b7XY6IiIigINaufPvsblzQJ4uHpq5kwcYdfpcjIiLS7imItSNhYcbvrh1ETnIMd/x9ARU1+/0uSUREpF1TEGtnkuMiefzGoVTsPsAPXl6kxcFFRER8pCDWDvXPS+YXV/Tjw9XbeUSLg4uIiPhGQayd+o8zC7hmaD5/1OLgIiIivlEQa6fMjAcm9D+8OHjpjj1+lyQiItLuKIi1Y7FR4TzhLQ5+hxYHFxERaXEKYu1c54x4fnvtQBaX7uTBN7U4uIiISEtqUhAzs/VmttTMFpnZPK/tfjPb5LUtMrNLG/S/18zWmNkqM7u4Qft4r22Nmd3ToL2LmX1qZqvN7GUzi2rODynHN75/Dref3ZW/zdbi4CIiIi0pmCNi5zrnBjnnhjVo+4PXNsg5NxXAzPoC1wP9gPHAn80s3MzCgT8BlwB9ga94fQF+7b1XD2AH8PVT+1gSrB9f3IvhndO457WlzFpb4Xc5IiIi7cLpODU5AZjonNvvnFsHrAGGe481zrli59wBYCIwwcwMOA941dv/eeDK01CXHEdEeBiP3TCY3OQYbvzrbB4vXEu95hgTERE5rawpC0Cb2ToCR6oc8Bfn3JNmdj9wK7ALmAf8l3Nuh5k9Bsx2zr3o7fs08Lb3VuOdc9/w2m8GRgD3e/27e+0FwNvOuf6N1HE7cDtAdnb20IkTJ57kx27bampqSEhIOKl999Y6nlm2n7lldQzKDOebA6KJj7RmrjD0nMqYtVcas+BpzIKnMQuOxit4LTVm55577vyjzioCENHE/cc45zabWRYw3cyKgMeBBwiEsweA3wFfAxr7q+1o/OibO07/LzY69yTwJMCwYcPcuHHjmlh++1JYWMipjM348x3PfbKeX05dycMLHI/fOIQz8pObr8AQdKpj1h5pzIKnMQuexiw4Gq/g+T1mTTo16Zzb7P0sByYDw51zW51zdc65euApAqceAUqBgga75wObj9O+HUgxs4ij2sUnZsZtY7rw8rdGUV/v+PLjn/D3TzfQlKOnIiIi0nQnDGJmFm9miYe2gYuAZWaW26DbVcAyb3sKcL2ZRZtZF6AHMAeYC/Tw7pCMInBB/xQX+Os+A7jG2/8W4I1T/2hyqoZ0TOXNO8cysls6P5u8jB++spg9B2r9LktERKTNaMqpyWxgcuCaeiKAfzjn3jGzv5nZIAKnEdcD3wJwzi03s1eAFUAtcIdzrg7AzL4HTAPCgWecc8u93/ETYKKZPQgsBJ5ups8npygtPornbj2Tx2as4Q///ozlm3fy5xuH0j1L1yCIiIicqhMGMedcMTCwkfabj7PPQ8BDjbRPBaYe43cMP7pdQkNYmHHn+T0Y3DGFuyYuYsJjH/HwlwdwxcAOfpcmIiLSqmlmfWmysT0yeevOs+idm8SdLy3kvjeWaVkkERGRU6AgJkHJTY5l4u0j+cZZXXh+1gau+8tsLRguIiJykhTEJGiR4WH892V9efzGIRSX13DZHz9ixqpyv8sSERFpdRTE5KRdckYuU75/FjlJMdz27Fx+9+4q6jQbv4iISJMpiMkp6ZIRz+t3jOG6Yfn88f013PrsHHbuPeh3WSIiIq2CgpicspjIcH5zzUB+dfUZzFpbwdV//pj123f7XZaIiEjIUxCTZnP98I68+I0RVOw+wJV//phZayv8LklERCSkKYhJsxrZNZ037hhDenwUNz/9KRPnbPS7JBERkZClICbNrlN6PJPvGMPo7hncM2kpD7y5Qhfxi4iINEJBTE6LpJhInrllGLeO7szTH63jG8/PpXqfLuIXERFpSEFMTpuI8DDuv6IfD17Zn5mrt/Plxz+hpFKTv4qIiByiICan3U0jO/HC14ZTtnMfE/70MXPXV/pdkoiISEhQEJMWMaZ7Bq/fMYbk2EhufOpTXp1f6ndJIiIivlMQkxbTNTOByd8dzbDOqdz9z8X86u0i6nURv4iItGMKYtKiUuKieP5rw7lhREee+GAt33pxPrv31/pdloiIiC8UxKTFRYaH8dCV/bn/8r68t3Ir1zwxi01Ve/0uS0REpMUpiIkvzIxbx3ThmVvPpLRyDxMe+5gFG3f4XZaIiEiLUhATX43rlcWk744mLiqc65+czdtLt/hdkoiISItREBPf9chO5PU7xtC/QxLf/ccCnvt4nd8liYiItAgFMQkJafFR/P0bI7mgTzb3/2sFD09dqTsqRUSkzVMQk5ARGxXOEzcN5aaRHfnLzGL+85VFHKit97ssERGR0ybC7wJEGgoPMx6Y0J/c5Fh+O20V26r388TNQ0mKifS7NBERkWanI2IScsyMO87tzu+vG8icdZVc98Qsynbu87ssERGRZqcgJiHr6iH5PHvbmZRU7uHqP3/MZ1ur/S5JRESkWSmISUgb2yOTV749ioP1jmse/4RPiyv8LklERKTZKIhJyOvXIZlJ3xlNZmI0Nz89h7eWaK4xERFpGxTEpFUoSIvjte+MZkB+Mt97aQFPf6S5xkREpPVTEJNWIyUuihe/MYKL++bwwJsrePDNFZprTEREWjUFMWlVYiLD+dONQ7h1dGf++tE67py4kP21dX6XJSIiclI0j5i0OuFhxn2X9yU3OYaH3y5ie81+/nLzMJJjNdeYiIi0LjoiJq2SmfGtc7rxyPWDmL9hB9c9MYuSyj1+lyUiIhKUJgUxM1tvZkvNbJGZzfPa0sxsupmt9n6meu1mZo+a2RozW2JmQxq8zy1e/9VmdkuD9qHe+6/x9rXm/qDSNk0YlMfztw1nc9VevvToh7y9VHdUiohI6xHMEbFznXODnHPDvOf3AO8553oA73nPAS4BeniP24HHIRDcgPuAEcBw4L5D4c3rc3uD/caf9CeSdmd09wzeunMsXTLi+c7fF/DTyUvZd1DXjYmISOg7lVOTE4Dnve3ngSsbtL/gAmYDKWaWC1wMTHfOVTrndgDTgfHea0nOuVnOOQe80OC9RJqkY3oc//z2aL51dlf+8elGJjymmfhFRCT0NTWIOeBdM5tvZrd7bdnOuS0A3s8srz0PKGmwb6nXdrz20kbaRYISFRHGvZf24fmvDWd7zX6ueOwjXpqzkUC+FxERCT1NvWtyjHNus5llAdPNrOg4fRu7vsudRPsX3zgQAm8HyM7OprCw8LhFt1c1NTXtfmz+58xwnlpay72TljL5kxXc2i+auMhjX3qoMQuexix4GrPgacyCo/EKnt9j1qQg5pzb7P0sN7PJBK7x2mpmuc65Ld7pxXKveylQ0GD3fGCz1z7uqPZCrz2/kf6N1fEk8CTAsGHD3Lhx4xrr1u4VFhaisYErLnI8MXMtv3v3Mzbvdzz6lUEM6ZjaaF+NWfA0ZsHTmAVPYxYcjVfw/B6zE56aNLN4M0s8tA1cBCwDpgCH7ny8BXjD254CfNW7e3IksNM7dTkNuMjMUr2L9C8CpnmvVZvZSO9uya82eC+RkxYWZnx3XHde+dYonIPrnpjF44VrNRu/iIiEjKZcI5YNfGRmi4E5wFvOuXeAXwEXmtlq4ELvOcBUoBhYAzwFfBfAOVcJPADM9R6/8NoAvgP81dtnLfD2qX80kYChnVKZetdYLuqXza/fKeKWZ+dQXr3P77JEREROfGrSOVcMDGykvQI4v5F2B9xxjPd6BnimkfZ5QP8m1CtyUpJjI/nTDUN4aU4JP//Xci595EN+f90gzu6Z6XdpIiLSjmlmfWk3zIwbRnRkyvfOIi0+iq8+M4eHp67kQG2936WJiEg7pSAm7U6vnETeuOMsvjK8I3+ZWcy1f5lF+R6FMRERaXkKYtIuxUaF8/DVZ/CnG4ZQvK2Gn360l19OXUnVngN+lyYiIu2Igpi0a18akMu0H5zNiJwInvqwmLN/M4MnPlirJZJERKRFKIhJu9chJZZvDohm6p1jGdoplV+9XcS5/1vIK/NKqNNUFyIichopiIl4+uQm8extw3npmyPJSozmx68u4ZJHZvLeyq1aJklERE4LBTGRo4zqls7rd4zhTzcM4UBtPV9/fh7/8eRsFmzc4XdpIiLSxiiIiTTCzPjSgFym//AcHpjQj+JtNVz950/4zovzWbutxu/yRESkjWjqot8i7VJkeBg3j+rM1UPyeerDYp6aWcy7K7Zy/ZkF3HV+D7KSYvwuUUREWjEdERNpgvjoCH5wQU8Kf3QuN47oyMtzSzjnt4X8/t1VVO876Hd5IiLSSimIiQQhMzGaX0zoz79/eA7n9cni0ffXcPZvZnD/lOUsLd2pi/pFRCQoOjUpchI6Z8TzpxuGcPvYKp74YC3/+HQjz32ynu5ZCVw9JI8rB+XRISXW7zJFRCTEKYiJnIKBBSk8ftNQdu45yFtLtzBpQSm/eWcVv522ilFd07lqcB6XnJFLQrT+UxMRkS/SXweRZpAcF8kNIzpyw4iObKjYzeSFm5i8cBM/enUJ//PGMi7ul8PVQ/IZ0y2diHBdESAiIgEKYiLNrFN6PD+4oCd3nd+DBRt3MGnBJv61eDNvLNpMZmI0Vw7qwFWD8+nbIcnvUkVExGcKYiKniZkxtFMaQzul8f8u78uMonJeW7CJ5z5Zz1MfrqN3TmLgerLBeWQlahoMEZH2SEFMpAVER4Qzvn8u4/vnsmP3Ad5cspnXFmzil1OL+PU7qzi/dxbXDy/g7B6ZOnUpItKOKIiJtLDU+ChuHtWZm0d1Zu22Gl6ZV8Jr80t5d8VWcpJiuHZYPtcNK6AgLc7vUkVE5DRTEBPxUbfMBO69pA93X9SL91aW8/Lcjfxpxhr++P4azuqewXVnFnBR32xiIsP9LlVERE4DBTGREBAZHsb4/jmM75/D5qq9vDq/lJfnlnDnSwtJiYvkqsF5XH9mR3rlJPpdqoiINCMFMZEQ0yElljvP78H3zu3Ox2u3M3FuCS/O3sCzH69nUEEK159ZwGUDO2huMhGRNkD/kouEqLAwY2yPTMb2yKSiZj+TF27i5bkl3DNpKb94cwWXD+jANcPyGdoxlbAw87tcERE5CQpiIq1AekI03xjbla+f1YUFG6t4ee5GpizezMvzSshMjObCvtlc3C+HUV3TiYrQXSpv4qoAABlBSURBVJciIq2FgphIKxKYmyyVoZ1S+Z/L+vJ+UTnTlpfx+sJN/OPTjSTGRHB+7ywu7pfDOb0yiYvSf+IiIqFM/0qLtFKJMZFMGJTHhEF57DtYx4ertzNteRn/XrmV1xdtJjoijLE9Mrm4XzYX9MkmNT7K75JFROQoCmIibUBMZDgX9s3mwr7Z1NbVM2d9Je8u33o4mIWHGSO6pHFxvxwu6pdNbnKs3yWLiAgKYiJtTkR4GKO7ZTC6Wwb3Xd6XpZt2Mm15GdOWb+W+Kcu5b8pyBuYnc1G/HMb2yKBfh2TCdbG/iIgvFMRE2jAzY0B+CgPyU/jRxb1ZU17DtOVlvLu8jN9OW8Vvp60iMSaCEV3SGdUtndHd0umVnai7MEVEWoiCmEg70j0rge5Z3bnj3O6UV+9j1toKZhdX8MnaCv69cisAqXGRjOwaCGWjuqXTLTMBMwUzEZHTQUFMpJ3KSow5fLE/wKaqvcxaW+E9tvP2sjIAMhOjGdU1EMpGdU2nU3qcgpmISDNREBMRAPJSYrlmaD7XDM3HOcfGyj3MWhs4WjaruIIpizcD0CE5hpHd0imglrH1TteXiYicAgUxEfkCM6NTejyd0uO5fnhHnHOs3babWWu3M6u4ghlF5ezYc5BJ62dwy6jOXDusgOTYSL/LFhFpdZo8BbeZhZvZQjN703v+nJmtM7NF3mOQ125m9qiZrTGzJWY2pMF73GJmq73HLQ3ah5rZUm+fR03nPURCipnRPSuBm0d15s83DmXuzy7gjkHR5CbF8uBbKxn18Hv8z+vLWFNe43epIiKtSjBHxO4CVgJJDdp+5Jx79ah+lwA9vMcI4HFghJmlAfcBwwAHzDezKc65HV6f24HZwFRgPPB28B9HRFpCRHgYZ+ZE8KPrR7Fs006e+2Q9L88t4W+zN3B2z0xuG92Zc3pm6u5LEZETaNIRMTPLB74E/LUJ3ScAL7iA2UCKmeUCFwPTnXOVXviaDoz3Xktyzs1yzjngBeDKk/kwItLy+ucl87/XDuSTe8/jvy7sSdGWXdz23FzO//0HPPfxOqr3HfS7RBGRkNXUU5P/B/wYqD+q/SHv9OMfzCzaa8sDShr0KfXajtde2ki7iLQiGQnRfP/8Hnz0k/N45PpBpMRFcv+/VjDq4fe5f8py1m/f7XeJIiIh54SnJs3sMqDcOTffzMY1eOleoAyIAp4EfgL8AmjsXIQ7ifbGarmdwClMsrOzKSwsPFH57VJNTY3GJkgas+Adb8ySgbv6QnGHGKZvOMjfZq3n+U/WMyAznAs7RdIvPaxdToGh71nwNGbB0XgFz+8xa8o1YmOAK8zsUiAGSDKzF51zN3mv7zezZ4G7veelQEGD/fOBzV77uKPaC732/Eb6f4Fz7kkCoY9hw4a5cePGNdat3SssLERjExyNWfCaMmbjgK8B5bv28eKnG/nHpxv433n7yE+NZUB+Ej2zE+mdk0ivnCQ6psW1+akw9D0LnsYsOBqv4Pk9ZicMYs65ewkc/cI7Ina3c+4mM8t1zm3x7nC8Eljm7TIF+J6ZTSRwsf5Or9804Jdmlur1uwi41zlXaWbVZjYS+BT4KvDHZvyMIuKzrKQYfnhhT+44txtvLt7CtOVlLN+8i7eXleG8498xkWH0zE6kV3YivXI+f2QmRLfLo2ci0j6cyjxifzezTAKnFhcB3/bapwKXAmuAPcBtAF7gegCY6/X7hXOu0tv+DvAcEEvgbkndMSnSBkVHhPPlofl8eWjgIPieA7Ws3lrDqrJqisqq+WxrNTNWbeOf8z+/bDQtPuqIcNY7J5F+HZKJimjy7DsiIiErqCDmnCskcDoR59x5x+jjgDuO8dozwDONtM8D+gdTi4i0fnFREQwsSGFgQcoR7RU1+1lVVs2qrdWHQ9or80rYc6DO2y+cUV3TGdsjg7N7ZtIlI15HzUSkVdLM+iISctITohndPZrR3TMOt9XXOzZV7WX55p18vKaCD1dv472iciCwPNPZPTM4u0cmo7tlkBynWf5FpHVQEBORViEszChIi6MgLY7x/XMB2Fixh5mrt/Hh6m28uXgLL80pIcxgYEEKY3tkck7PDAbmpxARrtOYIhKaFMREpNXqmB7HTemduGlkJ2rr6llUUsXM1duZ+dk2Hnt/NY++t5rE6AhGd0/3glkmBWlxfpctInKYgpiItAkR4WEM65zGsM5p/PDCnlTtOcAnayuY+dk2Zn62jWnLtwKQmxxDn9wkeuck0js3ib65iXROj9dRMxHxhYKYiLRJKXFRXHpGLpeekYtzjuLtu5n52TYWl1Sxcks1Mz/bRm19YO6MqIgwemYn0Ccnid65SfTJSaRPbhKp8VE+fwoRaesUxESkzTMzumUm0C0z4XDb/to61pbvZuWWXRSV7aKorJoZq8qPmDojOyma3jlJ9MlNok9uYNqMbpm6Q1NEmo+CmIi0S9ER4fTtkETfDklHtG+r3k9R2a5AQNtSzcqyaj5ZW8zBusDRs945iVw1OI8rB+eRnRTjR+ki0oYoiImINJCZGE1mYiZje2QebjtQW0/x9ho+La5k8sJNPPx2Eb9+p4gx3TP48pB8LuqXTVyU/jkVkeDpXw4RkROIigijd04SvXOSuGV0Z4q31TB54SYmLdjED15eRHxUOOP753L1kDxGdk1v82tmikjzURATEQlS18wE/uuiXvznBT2Zu76SSQs28dbSLby2oJTc5BiuHJzH1YPz6JGd6HepIhLiFMRERE5SWJgxoms6I7qm8/MJ/Zi+YiuTFpTy5MxiHi9cyxl5yVw9JI/LB3YgIyHa73JFJAQpiImINIOYyHAuH9iBywd2YFv1fqYs3sykBaX8/F8reOitlZzTM5PkugOUJ5SQkxRDTnIM2UkxJMVE6C5MkXZMQUxEpJllJkbz9bO68PWzurCqrJpJC0t5c/EWNlUdZNLqJUf0jY0MJzspmuwG4Sw7KcYLa9FkJQaeR0VowlmRtkhBTETkNOqVk8i9l/Th3kv68O57M+g9aARlu/ax1XuU7dx3+PmCjTvYums/B2rrv/A++amxjOiSzsiuaYzsmq6lmkTaCAUxEZEWEhVudEyPo2P6sUOUc46qPQcp2+UFNC+orSqr5v2irby2IDDhbH5qLCO7pnuPNPJTFcxEWiMFMRGREGJmpMZHkRofRZ/cIyebra93rC6vYXZxBbOLK3hv5VZenX9kMBvVNZ2R3dLJS4n1o3wRCZKCmIhIKxEWZvTKSaRXTiK3jO5Mfb3js/JqZq+tYHZx5RHBrCAtlpFdvCNm3dLpkByjmwJEQpCCmIhIKxUWZocnmr11TJfDwWzW2sARs+krtx5eOzMzMZqB+ckMyE9hYEEKA/KStai5SAhQEBMRaSMaBrPbvGC2ams1c9ZVsrikisWlVbxXVI4LLJtJx7Q4BuQnMzA/hQH5yfTPSyY+Wn8WRFqS/osTEWmjwsKMPrlJR1xrVr3vIEs37WRJ6U4Wl1SxcGMVby7ZEuhv0D0rIXDULD+ZgQUp9M5J0tQZIqeRgpiISDuSGBPJ6G4ZjO6Wcbhte81+lpRWsbhkJ0tKq5hRVH74WrOo8DAGd0zhvN5ZnN8ni26ZCbrWTKQZKYiJiLRzGQnRnNc7m/N6ZwOBKTQ2Ve1lcclOFpdWMfOzbTz8dhEPv11EQVos5/fO5tzeWYzokkZMZLjP1Yu0bgpiIiJyBDMjPzWO/NQ4vjQgl59e2odNVXuZUVTOjKJyXpqzkec+WU9cVDhjumdwfu8szu2dRXZSjN+li7Q6CmIiInJCeSmx3DSyEzeN7MS+g3XMWlvBe0VbmVG0jekrtgLQPy+J83plcV6fbAbkJRMWplOYIieiICYiIkGJiQznXO8omHOBOzPfLyrn/ZXlPDZjDY++v4aMhCjG9cpibI8MumUm0DUznrgo/ckROZr+qxARkZNm9vmUGd8d150duw/wwWfbeL+onHeXlx2+6B8gNzmGrpnxdM1IoEtGPF0z4+mWmUCHlFjCdfRM2ikFMRERaTap8VFcOTiPKwfnUVtXz+ryGoq37aZ4Ww3rtu9m7fbdvL5oE9X7ag/vExURRuf0OLpmBI6cdc0MBLVumfE+fhKRlqEgJiIip0VEeNgX5jGDwF2Z22sOHA5nxdsDQe2zrdX8e+VWauvd4b6ZscaYsoUM6ZTKkI6p9M5JJCJc85pJ26EgJiIiLcrMyEyMJjMxmhFd04947WBdPSWVeyjetps122qYvmA1H6+t4PVFmwGIiwpnQH4yQzqmMrRTKoM7ppKmpZqkFVMQExGRkBEZHkbXzAS6ZiZwAdn0diWcc845lO7Yy4KNO1i4sYr5G3bwl5nF1HlHzrpkxDO4YwpDvaNmPbMTdc2ZtBoKYiIiEtLMjIK0OArS4pgwKA+AvQfqWFJaxQIvmH2wahuTFmwCICE6goEFyQzrlMaILmkM7phKbJQmnpXQpCAmIiKtTmxUOCO6ph8+temcY2PlHhZs3MGCDYFw9sf3V/OIg4gwY0B+MsO7pDOiSxpDO6eSFBPp8ycQCWhyEDOzcGAesMk5d5mZdQEmAmnAAuBm59wBM4sGXgCGAhXAfzjn1nvvcS/wdaAOuNM5N81rHw88AoQDf3XO/aqZPp+IiLQDZkan9Hg6pcdz1eB8AHbtO8j8DTuYs66SOesqefqjYp74YC1hBn1ykxjeJXDE7MzOaaQnRPv8CaS9CuaI2F3ASuDQ7S+/Bv7gnJtoZk8QCFiPez93OOe6m9n1Xr//MLO+wPVAP6AD8G8z6+m915+AC4FSYK6ZTXHOrTjFzyYiIu1YUkwk5/bK4txeWUDgdObCks+D2UtzNvLsx+sB6J6VcDiYDe+SRm5yrI+VS3vSpCBmZvnAl4CHgB+amQHnATd4XZ4H7icQxCZ42wCvAo95/ScAE51z+4F1ZrYGGO71W+OcK/Z+10Svr4KYiIg0m9iocEZ3y2B0twwADtTWs3TTTi+YVfCvRZv5x6cbAchJiiElLpKE6AjioyNIiIkg8dD2oUfMkdvxUREkem0pcZEE/vSJHF9Tj4j9H/BjINF7ng5UOecOzchXCuR523lACYBzrtbMdnr984DZDd6z4T4lR7WPCOIziIiIBC0qIoyhnQLTYHxnXDfq6h0rt+xizrpKlm/eRfW+g+w+UEvVngOU7NjD7v211OyrZfeBuhO+d1JMBL1zk+ibm0Sf3ER65yTRKyeRmEjdNCBHOmEQM7PLgHLn3HwzG3eouZGu7gSvHau9sZn5XCNtmNntwO0A2dnZFBYWHrvwdqympkZjEySNWfA0ZsHTmAXPjzHrCnTNauyVCCCCeufYXwd7ax17a2HfET8de2phy+56SndU8dLGSvZ7uc2AnHijIDHsiEdajDXb0TN9x4Ln95g15YjYGOAKM7sUiCFwjdj/ASlmFuEdFcsHNnv9S4ECoNTMIoBkoLJB+yEN9zlW+xGcc08CTwIMGzbMjRs3rgnltz+FhYVobIKjMQuexix4GrPgtfYxq68P3M1ZVLaLFVuqWbllF0Vlu5izeu/hPsmxkfTOSaSPdwSte3YCBalxZCREBR3QWvt4+cHvMTthEHPO3QvcC+AdEbvbOXejmf0TuIbAnZO3AG94u0zxns/yXn/fOefMbArwDzP7PYGL9XsAcwj8T0IP7y7MTQQu6D907ZmIiEirFRZmdM6Ip3NGPOP75x5u37XvIKvKqina8nlAe3luCXsPfn7aMzoijLzUWPJT48hPjfUen29nJkTrOrQ24FTmEfsJMNHMHgQWAk977U8Df/Muxq8kEKxwzi03s1cIXIRfC9zhnKsDMLPvAdMITF/xjHNu+SnUJSIiEtKSYiI5s3Ng6oxD6usdGyr3ULythtIdeyndscf7uZelpVXs2HPwiPdoLKjt2lJLSkkVndLidMNAKxFUEHPOFQKF3nYxn9/12LDPPuDaY+z/EIE7L49unwpMDaYWERGRtiQszOiSEU+XjPhGX6/ZX8umIwJa40HticUfA5AYHUFBWhyd0uPomBZHR+9np7R4clNiiNTi6SFBM+uLiIi0AgnREfTKSaRXTmKjr+/eX8ukd2eS060fGyp2U1K5hw2Ve1i1tZr3VpZzoK7+cN/wMKNDSgyd0uKPCGs9sxPpmhFPmNbqbDEKYiIiIm1AfHQEBYlhjOub/YXX6usdZbv2sbFyDxsr9gR+ekFt2vIyKncfONw3MTqC/nnJDChIZmB+CgPyk8lLidVpztNEQUxERKSNCwszOqTE0iEllpHe+pwN7dp3kI0Ve1ixZRdLSqtYUrqTZz5ax8G6wGxS6fFRDMhPZkB+CgMLkjkjL4XMRC0L1RwUxERERNq5pJhI+ucl0z8vmeuGBWaU2l9bR9GWapaUVrG4dCdLSqso/Gwbzpvps0NyDAPyUw4fOevfIZnkOC2mHiwFMREREfmC6IhwBhakMLAghZu9tt37a1m2aSdLSney2Dty9s7yssP7pMZF0ik9ns7pcd4i7HGHn6fFBz8vWnugICYiIiJNEh8dwYiu6YxocHpzx+4DLNm0k6Itu9hQuYcNFbuZu34HbyzefPjoGQSuPeuU8Xkw65QWCGqdM+LJSmy/c6IpiImIiMhJS42P4pyemZzTM/OI9v21dZRU7mVDxW42VAQC2vqKPSzftJN3lpVRV/95SouJDCMnKYasxBgyk6LJSowmKzEm8DMpsJ2dFE1ybNubG01BTERERJpddEQ43bMS6J6V8IXXDtbVs7lqL+u9gLahYg9bd+2jvHo/KzbvonDXvkYXV4+KCCMz4VA4+zysdUiJpXNGHB3T4k9qaSg/KYiJiIhIi4oMD/OuIYsHMhvtU7O/lnIvnJVX76d81z62Hdqu3kfxtt3MLq5k594jVxyIjwqnY3o8nbz50T6/Vi2O3ORYwkNsjjQFMREREQk5CdERJGQm0DXzi0fUGtp3sI5NVXvZWLGH9Q1Og35WXs37RUdOZBsVHkZ+auwRAW1neS0jDtQRGxV+uj9SoxTEREREpNWKiQynW2YC3RoJbHX1ji07D4W0PWyo3H14e866ysOnP2+6tFZBTERERKQ5hYeZtyh6HKO7H/mac47tNQd4/d8fkR4f5U+BgFb8FBERkXbHzMhMjKZHarivF/criImIiIj4REFMRERExCcKYiIiIiI+URATERER8YmCmIiIiIhPFMREREREfKIgJiIiIuITBTERERERnyiIiYiIiPhEQUxERETEJ+ac87uGk2Jm24ANftcRojKA7X4X0cpozIKnMQuexix4GrPgaLyC11Jj1sk5l3l0Y6sNYnJsZjbPOTfM7zpaE41Z8DRmwdOYBU9jFhyNV/D8HjOdmhQRERHxiYKYiIiIiE8UxNqmJ/0uoBXSmAVPYxY8jVnwNGbB0XgFz9cx0zViIiIiIj7RETERERERnyiItTFmtt7MlprZIjOb53c9ocjMnjGzcjNb1qAtzcymm9lq72eqnzWGmmOM2f1mtsn7ri0ys0v9rDGUmFmBmc0ws5VmttzM7vLa9T07huOMmb5nx2BmMWY2x8wWe2P2c6+9i5l96n3PXjazKL9rDRXHGbPnzGxdg+/ZoBarSacm2xYzWw8Mc85pHpljMLOzgRrgBedcf6/tN0Clc+5XZnYPkOqc+4mfdYaSY4zZ/UCNc+5//awtFJlZLpDrnFtgZonAfOBK4Fb0PWvUccbsOvQ9a5SZGRDvnKsxs0jgI+Au4IfAJOfcRDN7AljsnHvcz1pDxXHG7NvAm865V1u6Jh0Rk3bHOTcTqDyqeQLwvLf9PIE/AOI5xpjJMTjntjjnFnjb1cBKIA99z47pOGMmx+ACarynkd7DAecBhwKFvmcNHGfMfKMg1vY44F0zm29mt/tdTCuS7ZzbAoE/CECWz/W0Ft8zsyXeqUudZmuEmXUGBgOfou9Zkxw1ZqDv2TGZWbiZLQLKgenAWqDKOVfrdSlFgfYIR4+Zc+7Q9+wh73v2BzOLbql6FMTanjHOuSHAJcAd3iklkdPhcaAbMAjYAvzO33JCj5klAK8BP3DO7fK7ntagkTHT9+w4nHN1zrlBQD4wHOjTWLeWrSq0HT1mZtYfuBfoDZwJpAEtdsmAglgb45zb7P0sByYT+A9TTmyrd43KoWtVyn2uJ+Q557Z6/6DVA0+h79oRvOtPXgP+7pyb5DXre3YcjY2ZvmdN45yrAgqBkUCKmUV4L+UDm/2qK5Q1GLPx3qlx55zbDzxLC37PFMTaEDOL9y5yxczigYuAZcffSzxTgFu87VuAN3yspVU4FCg8V6Hv2mHeBcFPAyudc79v8JK+Z8dwrDHT9+zYzCzTzFK87VjgAgLX1s0ArvG66XvWwDHGrKjB/yAZgWvqWux7prsm2xAz60rgKBhABPAP59xDPpYUkszsJWAckAFsBe4DXgdeAToCG4FrnXO6ON1zjDEbR+B0kQPWA986dP1Te2dmZwEfAkuBeq/5pwSuedL3rBHHGbOvoO9Zo8xsAIGL8cMJHFh5xTn3C+9vwUQCp9gWAjd5R3raveOM2ftAJmDAIuDbDS7qP701KYiJiIiI+EOnJkVERER8oiAmIiIi4hMFMRERERGfKIiJiIiI+ERBTERERMQnCmIiIiIiPlEQExEREfGJgpiIiIiIT/4/VbR6cjE8VcEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "distorsions = []\n",
    "for k in range(2, 36):\n",
    "    kmeans = KMeans(n_clusters=k)\n",
    "    kmeans.fit(anime_subset)\n",
    "    distorsions.append(kmeans.inertia_)\n",
    "\n",
    "fig = plt.figure(figsize=(10, 5))\n",
    "plt.plot(range(2, 36), distorsions)\n",
    "plt.grid(True)\n",
    "plt.title('Elbow curve')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### K-Means Results\n",
    "\n",
    "Based on inertia, 10 clusters will best represent the anime titles and the cluster assignments will replace the genres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "km_10 = KMeans(10)\n",
    "g_clusters = km_10.fit(anime_subset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "anime_subset['cluster'] = g_clusters.labels_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "      <th>episodes</th>\n",
       "      <th>rating</th>\n",
       "      <th>ranked</th>\n",
       "      <th>popularity</th>\n",
       "      <th>members</th>\n",
       "      <th>score_10</th>\n",
       "      <th>score_9</th>\n",
       "      <th>score_8</th>\n",
       "      <th>score_7</th>\n",
       "      <th>...</th>\n",
       "      <th>Vampire</th>\n",
       "      <th>Yaoi</th>\n",
       "      <th>Yuri</th>\n",
       "      <th>Movie</th>\n",
       "      <th>ONA</th>\n",
       "      <th>OVA</th>\n",
       "      <th>Other</th>\n",
       "      <th>Special</th>\n",
       "      <th>TV</th>\n",
       "      <th>cluster</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mal_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.044157e+00</td>\n",
       "      <td>0.008181</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.001774</td>\n",
       "      <td>0.002220</td>\n",
       "      <td>0.483466</td>\n",
       "      <td>0.320601</td>\n",
       "      <td>0.340261</td>\n",
       "      <td>0.286693</td>\n",
       "      <td>0.205156</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2.521152e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.010076</td>\n",
       "      <td>0.029490</td>\n",
       "      <td>0.105479</td>\n",
       "      <td>0.042028</td>\n",
       "      <td>0.091919</td>\n",
       "      <td>0.107826</td>\n",
       "      <td>0.074490</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2.319997e+00</td>\n",
       "      <td>0.008181</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.016857</td>\n",
       "      <td>0.011443</td>\n",
       "      <td>0.215834</td>\n",
       "      <td>0.070268</td>\n",
       "      <td>0.141336</td>\n",
       "      <td>0.187625</td>\n",
       "      <td>0.162703</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.019190e+00</td>\n",
       "      <td>0.008181</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.157224</td>\n",
       "      <td>0.083518</td>\n",
       "      <td>0.036563</td>\n",
       "      <td>0.003051</td>\n",
       "      <td>0.008977</td>\n",
       "      <td>0.022058</td>\n",
       "      <td>0.038237</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>6.302889e-01</td>\n",
       "      <td>0.016688</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.235108</td>\n",
       "      <td>0.248733</td>\n",
       "      <td>0.005106</td>\n",
       "      <td>0.000435</td>\n",
       "      <td>0.000986</td>\n",
       "      <td>0.002703</td>\n",
       "      <td>0.005635</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48481</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.746712</td>\n",
       "      <td>0.000136</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48483</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.999829</td>\n",
       "      <td>0.002707</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48488</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.999601</td>\n",
       "      <td>0.004367</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48491</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000535</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48492</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.999886</td>\n",
       "      <td>0.000119</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>17562 rows × 69 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               score  episodes  rating    ranked  popularity   members  \\\n",
       "mal_id                                                                   \n",
       "1       3.044157e+00  0.008181     0.8  0.001774    0.002220  0.483466   \n",
       "5       2.521152e+00  0.000000     0.8  0.010076    0.029490  0.105479   \n",
       "6       2.319997e+00  0.008181     0.4  0.016857    0.011443  0.215834   \n",
       "7       1.019190e+00  0.008181     0.4  0.157224    0.083518  0.036563   \n",
       "8       6.302889e-01  0.016688     0.2  0.235108    0.248733  0.005106   \n",
       "...              ...       ...     ...       ...         ...       ...   \n",
       "48481   7.146486e-15  0.000327     1.0  0.500380    0.746712  0.000136   \n",
       "48483   7.146486e-15  0.000327     1.0  0.500380    0.999829  0.002707   \n",
       "48488   7.146486e-15  0.000327     0.8  0.500380    0.999601  0.004367   \n",
       "48491   7.146486e-15  0.000327     0.4  0.500380    1.000000  0.000535   \n",
       "48492   7.146486e-15  0.000327     1.0  0.500380    0.999886  0.000119   \n",
       "\n",
       "        score_10   score_9   score_8   score_7  ...  Vampire  Yaoi  Yuri  \\\n",
       "mal_id                                          ...                        \n",
       "1       0.320601  0.340261  0.286693  0.205156  ...        0     0     0   \n",
       "5       0.042028  0.091919  0.107826  0.074490  ...        0     0     0   \n",
       "6       0.070268  0.141336  0.187625  0.162703  ...        0     0     0   \n",
       "7       0.003051  0.008977  0.022058  0.038237  ...        0     0     0   \n",
       "8       0.000435  0.000986  0.002703  0.005635  ...        0     0     0   \n",
       "...          ...       ...       ...       ...  ...      ...   ...   ...   \n",
       "48481   0.000053  0.000131  0.000198  0.000000  ...        0     0     0   \n",
       "48483   0.000053  0.000131  0.000198  0.000458  ...        0     0     0   \n",
       "48488   0.000000  0.000131  0.000198  0.000458  ...        0     0     0   \n",
       "48491   0.000053  0.000131  0.000198  0.000458  ...        0     0     0   \n",
       "48492   0.000053  0.000131  0.000198  0.000458  ...        0     0     0   \n",
       "\n",
       "        Movie  ONA  OVA  Other  Special   TV  cluster  \n",
       "mal_id                                                 \n",
       "1         0.0  0.0  0.0    0.0      0.0  1.0        5  \n",
       "5         1.0  0.0  0.0    0.0      0.0  0.0        5  \n",
       "6         0.0  0.0  0.0    0.0      0.0  1.0        5  \n",
       "7         0.0  0.0  0.0    0.0      0.0  1.0        5  \n",
       "8         0.0  0.0  0.0    0.0      0.0  1.0        5  \n",
       "...       ...  ...  ...    ...      ...  ...      ...  \n",
       "48481     0.0  1.0  0.0    0.0      0.0  0.0        8  \n",
       "48483     0.0  0.0  0.0    0.0      0.0  1.0        9  \n",
       "48488     0.0  0.0  0.0    0.0      0.0  1.0        9  \n",
       "48491     0.0  0.0  0.0    0.0      0.0  1.0        9  \n",
       "48492     0.0  0.0  0.0    0.0      0.0  1.0        9  \n",
       "\n",
       "[17562 rows x 69 columns]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anime_subset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "enc = OneHotEncoder(handle_unknown='ignore', sparse=False)\n",
    "enc.fit(anime_subset[['cluster']])\n",
    "cluster_10 = pd.DataFrame(enc.transform(anime_subset[['cluster']]), columns=['g_cluster_' + str(i) for i in range(1, 11)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster_10 = cluster_10.astype('int')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# need the index (mal_id) from the genres dataframe\n",
    "cluster_10.index = anime_subset.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>g_cluster_1</th>\n",
       "      <th>g_cluster_2</th>\n",
       "      <th>g_cluster_3</th>\n",
       "      <th>g_cluster_4</th>\n",
       "      <th>g_cluster_5</th>\n",
       "      <th>g_cluster_6</th>\n",
       "      <th>g_cluster_7</th>\n",
       "      <th>g_cluster_8</th>\n",
       "      <th>g_cluster_9</th>\n",
       "      <th>g_cluster_10</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mal_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48481</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48483</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48488</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48491</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48492</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>17562 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        g_cluster_1  g_cluster_2  g_cluster_3  g_cluster_4  g_cluster_5  \\\n",
       "mal_id                                                                    \n",
       "1                 0            0            0            0            0   \n",
       "5                 0            0            0            0            0   \n",
       "6                 0            0            0            0            0   \n",
       "7                 0            0            0            0            0   \n",
       "8                 0            0            0            0            0   \n",
       "...             ...          ...          ...          ...          ...   \n",
       "48481             0            0            0            0            0   \n",
       "48483             0            0            0            0            0   \n",
       "48488             0            0            0            0            0   \n",
       "48491             0            0            0            0            0   \n",
       "48492             0            0            0            0            0   \n",
       "\n",
       "        g_cluster_6  g_cluster_7  g_cluster_8  g_cluster_9  g_cluster_10  \n",
       "mal_id                                                                    \n",
       "1                 1            0            0            0             0  \n",
       "5                 1            0            0            0             0  \n",
       "6                 1            0            0            0             0  \n",
       "7                 1            0            0            0             0  \n",
       "8                 1            0            0            0             0  \n",
       "...             ...          ...          ...          ...           ...  \n",
       "48481             0            0            0            1             0  \n",
       "48483             0            0            0            0             1  \n",
       "48488             0            0            0            0             1  \n",
       "48491             0            0            0            0             1  \n",
       "48492             0            0            0            0             1  \n",
       "\n",
       "[17562 rows x 10 columns]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster_10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Replace the genres columns with clusters\n",
    "\n",
    "Dimensionality was reduced from 74 to 39 columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop the original genres columns\n",
    "training_w_clusters = training.drop(columns=genres_list)\n",
    "training_w_clusters = pd.merge(training_w_clusters, cluster_10, left_on='mal_id', right_index=True)\n",
    "training_w_clusters.drop(columns=['mal_id'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>user_rating</th>\n",
       "      <th>watching_status</th>\n",
       "      <th>watched_episodes</th>\n",
       "      <th>aggregated_score</th>\n",
       "      <th>episodes</th>\n",
       "      <th>rating</th>\n",
       "      <th>ranked</th>\n",
       "      <th>popularity</th>\n",
       "      <th>members</th>\n",
       "      <th>score_10</th>\n",
       "      <th>...</th>\n",
       "      <th>g_cluster_1</th>\n",
       "      <th>g_cluster_2</th>\n",
       "      <th>g_cluster_3</th>\n",
       "      <th>g_cluster_4</th>\n",
       "      <th>g_cluster_5</th>\n",
       "      <th>g_cluster_6</th>\n",
       "      <th>g_cluster_7</th>\n",
       "      <th>g_cluster_8</th>\n",
       "      <th>g_cluster_9</th>\n",
       "      <th>g_cluster_10</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th>anime_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <th>269</th>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>64</td>\n",
       "      <td>7.80</td>\n",
       "      <td>366.0</td>\n",
       "      <td>3</td>\n",
       "      <td>818.0</td>\n",
       "      <td>33</td>\n",
       "      <td>1286382</td>\n",
       "      <td>116063.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>33486</th>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>25</td>\n",
       "      <td>8.33</td>\n",
       "      <td>25.0</td>\n",
       "      <td>3</td>\n",
       "      <td>203.0</td>\n",
       "      <td>13</td>\n",
       "      <td>1611771</td>\n",
       "      <td>188165.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">3</th>\n",
       "      <th>10379</th>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>13</td>\n",
       "      <td>8.60</td>\n",
       "      <td>13.0</td>\n",
       "      <td>3</td>\n",
       "      <td>67.0</td>\n",
       "      <td>738</td>\n",
       "      <td>200904</td>\n",
       "      <td>23877.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9863</th>\n",
       "      <td>9</td>\n",
       "      <td>2</td>\n",
       "      <td>77</td>\n",
       "      <td>8.25</td>\n",
       "      <td>77.0</td>\n",
       "      <td>3</td>\n",
       "      <td>262.0</td>\n",
       "      <td>804</td>\n",
       "      <td>185346</td>\n",
       "      <td>11534.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>24</td>\n",
       "      <td>7.65</td>\n",
       "      <td>24.0</td>\n",
       "      <td>5</td>\n",
       "      <td>1146.0</td>\n",
       "      <td>322</td>\n",
       "      <td>411230</td>\n",
       "      <td>17675.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">353398</th>\n",
       "      <th>17247</th>\n",
       "      <td>8</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>7.15</td>\n",
       "      <td>12.0</td>\n",
       "      <td>4</td>\n",
       "      <td>3003.0</td>\n",
       "      <td>553</td>\n",
       "      <td>264089</td>\n",
       "      <td>7926.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1571</th>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>25</td>\n",
       "      <td>7.81</td>\n",
       "      <td>25.0</td>\n",
       "      <td>3</td>\n",
       "      <td>811.0</td>\n",
       "      <td>766</td>\n",
       "      <td>194354</td>\n",
       "      <td>11328.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36144</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7.17</td>\n",
       "      <td>24.0</td>\n",
       "      <td>5</td>\n",
       "      <td>2907.0</td>\n",
       "      <td>2034</td>\n",
       "      <td>59132</td>\n",
       "      <td>1023.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32979</th>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>13</td>\n",
       "      <td>7.68</td>\n",
       "      <td>13.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1084.0</td>\n",
       "      <td>949</td>\n",
       "      <td>156048</td>\n",
       "      <td>5540.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>353401</th>\n",
       "      <th>34662</th>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>7.16</td>\n",
       "      <td>25.0</td>\n",
       "      <td>5</td>\n",
       "      <td>2937.0</td>\n",
       "      <td>402</td>\n",
       "      <td>339526</td>\n",
       "      <td>12121.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>999996 rows × 39 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                  user_rating  watching_status  watched_episodes  \\\n",
       "user_id anime_id                                                   \n",
       "0       269                 9                3                64   \n",
       "1       33486               8                2                25   \n",
       "3       10379               8                2                13   \n",
       "        9863                9                2                77   \n",
       "        71                  7                2                24   \n",
       "...                       ...              ...               ...   \n",
       "353398  17247               8                3                 3   \n",
       "        1571                7                2                25   \n",
       "        36144               0                1                 0   \n",
       "        32979               7                2                13   \n",
       "353401  34662               0                6                 0   \n",
       "\n",
       "                  aggregated_score  episodes  rating  ranked  popularity  \\\n",
       "user_id anime_id                                                           \n",
       "0       269                   7.80     366.0       3   818.0          33   \n",
       "1       33486                 8.33      25.0       3   203.0          13   \n",
       "3       10379                 8.60      13.0       3    67.0         738   \n",
       "        9863                  8.25      77.0       3   262.0         804   \n",
       "        71                    7.65      24.0       5  1146.0         322   \n",
       "...                            ...       ...     ...     ...         ...   \n",
       "353398  17247                 7.15      12.0       4  3003.0         553   \n",
       "        1571                  7.81      25.0       3   811.0         766   \n",
       "        36144                 7.17      24.0       5  2907.0        2034   \n",
       "        32979                 7.68      13.0       3  1084.0         949   \n",
       "353401  34662                 7.16      25.0       5  2937.0         402   \n",
       "\n",
       "                  members  score_10  ...  g_cluster_1  g_cluster_2  \\\n",
       "user_id anime_id                     ...                             \n",
       "0       269       1286382  116063.0  ...            0            0   \n",
       "1       33486     1611771  188165.0  ...            0            0   \n",
       "3       10379      200904   23877.0  ...            0            0   \n",
       "        9863       185346   11534.0  ...            1            0   \n",
       "        71         411230   17675.0  ...            0            0   \n",
       "...                   ...       ...  ...          ...          ...   \n",
       "353398  17247      264089    7926.0  ...            0            0   \n",
       "        1571       194354   11328.0  ...            1            0   \n",
       "        36144       59132    1023.0  ...            0            0   \n",
       "        32979      156048    5540.0  ...            0            0   \n",
       "353401  34662      339526   12121.0  ...            0            0   \n",
       "\n",
       "                  g_cluster_3  g_cluster_4  g_cluster_5  g_cluster_6  \\\n",
       "user_id anime_id                                                       \n",
       "0       269                 0            0            0            1   \n",
       "1       33486               0            0            0            1   \n",
       "3       10379               0            0            0            1   \n",
       "        9863                0            0            0            0   \n",
       "        71                  0            0            0            1   \n",
       "...                       ...          ...          ...          ...   \n",
       "353398  17247               0            0            0            1   \n",
       "        1571                0            0            0            0   \n",
       "        36144               0            0            0            1   \n",
       "        32979               0            0            0            1   \n",
       "353401  34662               0            0            0            1   \n",
       "\n",
       "                  g_cluster_7  g_cluster_8  g_cluster_9  g_cluster_10  \n",
       "user_id anime_id                                                       \n",
       "0       269                 0            0            0             0  \n",
       "1       33486               0            0            0             0  \n",
       "3       10379               0            0            0             0  \n",
       "        9863                0            0            0             0  \n",
       "        71                  0            0            0             0  \n",
       "...                       ...          ...          ...           ...  \n",
       "353398  17247               0            0            0             0  \n",
       "        1571                0            0            0             0  \n",
       "        36144               0            0            0             0  \n",
       "        32979               0            0            0             0  \n",
       "353401  34662               0            0            0             0  \n",
       "\n",
       "[999996 rows x 39 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_w_clusters.sort_values(['user_id']).set_index(['user_id', 'anime_id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do the same for anime_subset\n",
    "anime_clusters = anime_subset.drop(columns=genres_list)\n",
    "anime_clusters = pd.merge(anime_clusters, cluster_10, left_on='mal_id', right_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "      <th>episodes</th>\n",
       "      <th>rating</th>\n",
       "      <th>ranked</th>\n",
       "      <th>popularity</th>\n",
       "      <th>members</th>\n",
       "      <th>score_10</th>\n",
       "      <th>score_9</th>\n",
       "      <th>score_8</th>\n",
       "      <th>score_7</th>\n",
       "      <th>...</th>\n",
       "      <th>g_cluster_1</th>\n",
       "      <th>g_cluster_2</th>\n",
       "      <th>g_cluster_3</th>\n",
       "      <th>g_cluster_4</th>\n",
       "      <th>g_cluster_5</th>\n",
       "      <th>g_cluster_6</th>\n",
       "      <th>g_cluster_7</th>\n",
       "      <th>g_cluster_8</th>\n",
       "      <th>g_cluster_9</th>\n",
       "      <th>g_cluster_10</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mal_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.044157e+00</td>\n",
       "      <td>0.008181</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.001774</td>\n",
       "      <td>0.002220</td>\n",
       "      <td>0.483466</td>\n",
       "      <td>0.320601</td>\n",
       "      <td>0.340261</td>\n",
       "      <td>0.286693</td>\n",
       "      <td>0.205156</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2.521152e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.010076</td>\n",
       "      <td>0.029490</td>\n",
       "      <td>0.105479</td>\n",
       "      <td>0.042028</td>\n",
       "      <td>0.091919</td>\n",
       "      <td>0.107826</td>\n",
       "      <td>0.074490</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2.319997e+00</td>\n",
       "      <td>0.008181</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.016857</td>\n",
       "      <td>0.011443</td>\n",
       "      <td>0.215834</td>\n",
       "      <td>0.070268</td>\n",
       "      <td>0.141336</td>\n",
       "      <td>0.187625</td>\n",
       "      <td>0.162703</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.019190e+00</td>\n",
       "      <td>0.008181</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.157224</td>\n",
       "      <td>0.083518</td>\n",
       "      <td>0.036563</td>\n",
       "      <td>0.003051</td>\n",
       "      <td>0.008977</td>\n",
       "      <td>0.022058</td>\n",
       "      <td>0.038237</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>6.302889e-01</td>\n",
       "      <td>0.016688</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.235108</td>\n",
       "      <td>0.248733</td>\n",
       "      <td>0.005106</td>\n",
       "      <td>0.000435</td>\n",
       "      <td>0.000986</td>\n",
       "      <td>0.002703</td>\n",
       "      <td>0.005635</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48481</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.746712</td>\n",
       "      <td>0.000136</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48483</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.999829</td>\n",
       "      <td>0.002707</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48488</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.999601</td>\n",
       "      <td>0.004367</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48491</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000535</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48492</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.999886</td>\n",
       "      <td>0.000119</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>17562 rows × 35 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               score  episodes  rating    ranked  popularity   members  \\\n",
       "mal_id                                                                   \n",
       "1       3.044157e+00  0.008181     0.8  0.001774    0.002220  0.483466   \n",
       "5       2.521152e+00  0.000000     0.8  0.010076    0.029490  0.105479   \n",
       "6       2.319997e+00  0.008181     0.4  0.016857    0.011443  0.215834   \n",
       "7       1.019190e+00  0.008181     0.4  0.157224    0.083518  0.036563   \n",
       "8       6.302889e-01  0.016688     0.2  0.235108    0.248733  0.005106   \n",
       "...              ...       ...     ...       ...         ...       ...   \n",
       "48481   7.146486e-15  0.000327     1.0  0.500380    0.746712  0.000136   \n",
       "48483   7.146486e-15  0.000327     1.0  0.500380    0.999829  0.002707   \n",
       "48488   7.146486e-15  0.000327     0.8  0.500380    0.999601  0.004367   \n",
       "48491   7.146486e-15  0.000327     0.4  0.500380    1.000000  0.000535   \n",
       "48492   7.146486e-15  0.000327     1.0  0.500380    0.999886  0.000119   \n",
       "\n",
       "        score_10   score_9   score_8   score_7  ...  g_cluster_1  g_cluster_2  \\\n",
       "mal_id                                          ...                             \n",
       "1       0.320601  0.340261  0.286693  0.205156  ...            0            0   \n",
       "5       0.042028  0.091919  0.107826  0.074490  ...            0            0   \n",
       "6       0.070268  0.141336  0.187625  0.162703  ...            0            0   \n",
       "7       0.003051  0.008977  0.022058  0.038237  ...            0            0   \n",
       "8       0.000435  0.000986  0.002703  0.005635  ...            0            0   \n",
       "...          ...       ...       ...       ...  ...          ...          ...   \n",
       "48481   0.000053  0.000131  0.000198  0.000000  ...            0            0   \n",
       "48483   0.000053  0.000131  0.000198  0.000458  ...            0            0   \n",
       "48488   0.000000  0.000131  0.000198  0.000458  ...            0            0   \n",
       "48491   0.000053  0.000131  0.000198  0.000458  ...            0            0   \n",
       "48492   0.000053  0.000131  0.000198  0.000458  ...            0            0   \n",
       "\n",
       "        g_cluster_3  g_cluster_4  g_cluster_5  g_cluster_6  g_cluster_7  \\\n",
       "mal_id                                                                    \n",
       "1                 0            0            0            1            0   \n",
       "5                 0            0            0            1            0   \n",
       "6                 0            0            0            1            0   \n",
       "7                 0            0            0            1            0   \n",
       "8                 0            0            0            1            0   \n",
       "...             ...          ...          ...          ...          ...   \n",
       "48481             0            0            0            0            0   \n",
       "48483             0            0            0            0            0   \n",
       "48488             0            0            0            0            0   \n",
       "48491             0            0            0            0            0   \n",
       "48492             0            0            0            0            0   \n",
       "\n",
       "        g_cluster_8  g_cluster_9  g_cluster_10  \n",
       "mal_id                                          \n",
       "1                 0            0             0  \n",
       "5                 0            0             0  \n",
       "6                 0            0             0  \n",
       "7                 0            0             0  \n",
       "8                 0            0             0  \n",
       "...             ...          ...           ...  \n",
       "48481             0            1             0  \n",
       "48483             0            0             1  \n",
       "48488             0            0             1  \n",
       "48491             0            0             1  \n",
       "48492             0            0             1  \n",
       "\n",
       "[17562 rows x 35 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anime_clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# K Nearest Neighbors Model\n",
    "\n",
    "Since we're using KNN in an unsupervised manner (no labels, no classifications) we will use NearestNeigbors vs KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notes**\n",
    "KNN will use the anime_subset with cluster assignments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NearestNeighbors(algorithm='auto', leaf_size=30, metric='cosine',\n",
       "                 metric_params=None, n_jobs=None, n_neighbors=5, p=2,\n",
       "                 radius=1.0)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn = NearestNeighbors(metric=\"cosine\")\n",
    "knn.fit(anime_clusters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test Query\n",
    "\n",
    "The test query will include itself as a recommendation, this will be excluded for final inferences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the original anime table for title information from athena\n",
    "anime = cursor.execute('SELECT * FROM %s.%s' % (dbname, 'anime')).as_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Query Title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mal_id</th>\n",
       "      <th>name</th>\n",
       "      <th>score</th>\n",
       "      <th>genres</th>\n",
       "      <th>english_name</th>\n",
       "      <th>japanese_name</th>\n",
       "      <th>type</th>\n",
       "      <th>episodes</th>\n",
       "      <th>aired</th>\n",
       "      <th>premiered</th>\n",
       "      <th>...</th>\n",
       "      <th>score_10</th>\n",
       "      <th>score_9</th>\n",
       "      <th>score_8</th>\n",
       "      <th>score_7</th>\n",
       "      <th>score_6</th>\n",
       "      <th>score_5</th>\n",
       "      <th>score_4</th>\n",
       "      <th>score_3</th>\n",
       "      <th>score_2</th>\n",
       "      <th>score_1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1117</th>\n",
       "      <td>5</td>\n",
       "      <td>Cowboy Bebop: Tengoku no Tobira</td>\n",
       "      <td>8.39</td>\n",
       "      <td>Action, Drama, Mystery, Sci-Fi, Space</td>\n",
       "      <td>Cowboy Bebop:The Movie</td>\n",
       "      <td>カウボーイビバップ 天国の扉</td>\n",
       "      <td>Movie</td>\n",
       "      <td>1</td>\n",
       "      <td>Sep 1, 2001</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>...</td>\n",
       "      <td>30043.0</td>\n",
       "      <td>49201.0</td>\n",
       "      <td>49505.0</td>\n",
       "      <td>22632.0</td>\n",
       "      <td>5805.0</td>\n",
       "      <td>1877.0</td>\n",
       "      <td>577.0</td>\n",
       "      <td>221.0</td>\n",
       "      <td>109.0</td>\n",
       "      <td>379.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 35 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      mal_id                             name score  \\\n",
       "1117       5  Cowboy Bebop: Tengoku no Tobira  8.39   \n",
       "\n",
       "                                     genres            english_name  \\\n",
       "1117  Action, Drama, Mystery, Sci-Fi, Space  Cowboy Bebop:The Movie   \n",
       "\n",
       "       japanese_name   type episodes        aired premiered  ... score_10  \\\n",
       "1117  カウボーイビバップ 天国の扉  Movie        1  Sep 1, 2001   Unknown  ...  30043.0   \n",
       "\n",
       "      score_9  score_8  score_7 score_6 score_5 score_4  score_3  score_2  \\\n",
       "1117  49201.0  49505.0  22632.0  5805.0  1877.0   577.0    221.0    109.0   \n",
       "\n",
       "      score_1  \n",
       "1117    379.0  \n",
       "\n",
       "[1 rows x 35 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anime[anime['mal_id'] == anime_clusters.iloc[1, :].name]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "distances, titles = knn.kneighbors(anime_clusters.iloc[1, :].values.reshape(1, -1), n_neighbors=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.        , 0.00019806, 0.00026596, 0.00031216, 0.00069917,\n",
       "        0.00073396, 0.00076065, 0.00080426, 0.00085417, 0.00086291]])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Sort by distances, smallest to largest\n",
    "distances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 2215, 2813, 6893, 6229, 8139, 2373, 14023, 5897, 409]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Recommendation rows ordered by distances\n",
    "list(titles[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 2418, 5205, 9260, 7472, 25537, 4282, 37055, 6675, 437]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert rows to anime_ids\n",
    "anime_clusters.iloc[list(titles[0]), :].index.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recommended Titles based on Query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mal_id</th>\n",
       "      <th>name</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1099</th>\n",
       "      <td>2418</td>\n",
       "      <td>Stranger: Mukou Hadan</td>\n",
       "      <td>Action, Adventure, Historical, Samurai</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1117</th>\n",
       "      <td>5</td>\n",
       "      <td>Cowboy Bebop: Tengoku no Tobira</td>\n",
       "      <td>Action, Drama, Mystery, Sci-Fi, Space</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1525</th>\n",
       "      <td>437</td>\n",
       "      <td>Perfect Blue</td>\n",
       "      <td>Dementia, Drama, Horror, Psychological</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3578</th>\n",
       "      <td>4282</td>\n",
       "      <td>Kara no Kyoukai 5: Mujun Rasen</td>\n",
       "      <td>Action, Mystery, Supernatural, Drama, Romance,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4018</th>\n",
       "      <td>5205</td>\n",
       "      <td>Kara no Kyoukai 7: Satsujin Kousatsu (Go)</td>\n",
       "      <td>Action, Mystery, Supernatural, Romance, Thriller</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5897</th>\n",
       "      <td>6675</td>\n",
       "      <td>Redline</td>\n",
       "      <td>Action, Sci-Fi, Cars, Sports</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6229</th>\n",
       "      <td>7472</td>\n",
       "      <td>Gintama Movie 1: Shinyaku Benizakura-hen</td>\n",
       "      <td>Action, Sci-Fi, Comedy, Historical, Parody, Sa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6893</th>\n",
       "      <td>9260</td>\n",
       "      <td>Kizumonogatari I: Tekketsu-hen</td>\n",
       "      <td>Vampire</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8139</th>\n",
       "      <td>25537</td>\n",
       "      <td>Fate/stay night Movie: Heaven's Feel - I. Pres...</td>\n",
       "      <td>Action, Fantasy, Magic, Supernatural</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14023</th>\n",
       "      <td>37055</td>\n",
       "      <td>Youjo Senki Movie</td>\n",
       "      <td>Action, Military, Magic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       mal_id                                               name  \\\n",
       "1099     2418                              Stranger: Mukou Hadan   \n",
       "1117        5                    Cowboy Bebop: Tengoku no Tobira   \n",
       "1525      437                                       Perfect Blue   \n",
       "3578     4282                     Kara no Kyoukai 5: Mujun Rasen   \n",
       "4018     5205          Kara no Kyoukai 7: Satsujin Kousatsu (Go)   \n",
       "5897     6675                                            Redline   \n",
       "6229     7472           Gintama Movie 1: Shinyaku Benizakura-hen   \n",
       "6893     9260                     Kizumonogatari I: Tekketsu-hen   \n",
       "8139    25537  Fate/stay night Movie: Heaven's Feel - I. Pres...   \n",
       "14023   37055                                  Youjo Senki Movie   \n",
       "\n",
       "                                                  genres  \n",
       "1099              Action, Adventure, Historical, Samurai  \n",
       "1117               Action, Drama, Mystery, Sci-Fi, Space  \n",
       "1525              Dementia, Drama, Horror, Psychological  \n",
       "3578   Action, Mystery, Supernatural, Drama, Romance,...  \n",
       "4018    Action, Mystery, Supernatural, Romance, Thriller  \n",
       "5897                        Action, Sci-Fi, Cars, Sports  \n",
       "6229   Action, Sci-Fi, Comedy, Historical, Parody, Sa...  \n",
       "6893                                             Vampire  \n",
       "8139                Action, Fantasy, Magic, Supernatural  \n",
       "14023                            Action, Military, Magic  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anime[anime['mal_id'].isin(anime_clusters.iloc[list(titles[0]), :].index.tolist())][['mal_id', 'name', 'genres']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding NLP Embeddings of Anime Synopsis Data using BERT\n",
    "\n",
    "BERT model inferences are on the Team3_Modeling_BERT.ipynb notebook. See the note"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pull Synopsis from Athena\n",
    "synopsis = cursor.execute('SELECT * FROM %s.%s' % (dbname, 'synopsis')).as_pandas()\n",
    "# Semantics is downloaded from S3\n",
    "embeddings = pd.read_csv('Data/semantics.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = pd.concat([synopsis[['mal_id']], embeddings], axis=1)\n",
    "embeddings.set_index(['mal_id'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>758</th>\n",
       "      <th>759</th>\n",
       "      <th>760</th>\n",
       "      <th>761</th>\n",
       "      <th>762</th>\n",
       "      <th>763</th>\n",
       "      <th>764</th>\n",
       "      <th>765</th>\n",
       "      <th>766</th>\n",
       "      <th>767</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mal_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.338000</td>\n",
       "      <td>0.280912</td>\n",
       "      <td>0.316794</td>\n",
       "      <td>-0.081068</td>\n",
       "      <td>0.229358</td>\n",
       "      <td>-0.153148</td>\n",
       "      <td>-0.447823</td>\n",
       "      <td>0.525362</td>\n",
       "      <td>0.265396</td>\n",
       "      <td>-0.106113</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.887086</td>\n",
       "      <td>0.158227</td>\n",
       "      <td>-0.095016</td>\n",
       "      <td>0.038595</td>\n",
       "      <td>0.107712</td>\n",
       "      <td>-0.243573</td>\n",
       "      <td>-0.090685</td>\n",
       "      <td>-0.082259</td>\n",
       "      <td>0.366798</td>\n",
       "      <td>-0.188894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-0.025266</td>\n",
       "      <td>-0.049519</td>\n",
       "      <td>0.394098</td>\n",
       "      <td>-0.144688</td>\n",
       "      <td>0.058020</td>\n",
       "      <td>-0.056215</td>\n",
       "      <td>-0.152939</td>\n",
       "      <td>0.453019</td>\n",
       "      <td>0.189982</td>\n",
       "      <td>0.088776</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.925842</td>\n",
       "      <td>-0.115866</td>\n",
       "      <td>-0.085770</td>\n",
       "      <td>-0.468572</td>\n",
       "      <td>-0.072172</td>\n",
       "      <td>0.015078</td>\n",
       "      <td>-0.120486</td>\n",
       "      <td>0.005637</td>\n",
       "      <td>0.211060</td>\n",
       "      <td>-0.038622</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-0.050121</td>\n",
       "      <td>0.361339</td>\n",
       "      <td>0.314151</td>\n",
       "      <td>0.030035</td>\n",
       "      <td>-0.018952</td>\n",
       "      <td>-0.169436</td>\n",
       "      <td>0.143299</td>\n",
       "      <td>0.653574</td>\n",
       "      <td>0.067216</td>\n",
       "      <td>0.066807</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.699382</td>\n",
       "      <td>0.128695</td>\n",
       "      <td>0.588282</td>\n",
       "      <td>-0.257558</td>\n",
       "      <td>0.276342</td>\n",
       "      <td>0.132950</td>\n",
       "      <td>-0.047471</td>\n",
       "      <td>-0.000429</td>\n",
       "      <td>0.647236</td>\n",
       "      <td>-0.094455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>-0.277991</td>\n",
       "      <td>-0.150378</td>\n",
       "      <td>0.051548</td>\n",
       "      <td>-0.052691</td>\n",
       "      <td>0.007375</td>\n",
       "      <td>-0.371882</td>\n",
       "      <td>0.255686</td>\n",
       "      <td>-0.042258</td>\n",
       "      <td>0.262719</td>\n",
       "      <td>0.135156</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.490768</td>\n",
       "      <td>-0.326309</td>\n",
       "      <td>0.118539</td>\n",
       "      <td>-0.256842</td>\n",
       "      <td>-0.066406</td>\n",
       "      <td>-0.026164</td>\n",
       "      <td>-0.191518</td>\n",
       "      <td>-0.156832</td>\n",
       "      <td>0.019953</td>\n",
       "      <td>0.001370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-0.460624</td>\n",
       "      <td>-0.038628</td>\n",
       "      <td>0.111783</td>\n",
       "      <td>-0.103765</td>\n",
       "      <td>-0.212746</td>\n",
       "      <td>0.000713</td>\n",
       "      <td>0.400580</td>\n",
       "      <td>0.219770</td>\n",
       "      <td>0.255605</td>\n",
       "      <td>-0.086178</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.709232</td>\n",
       "      <td>-0.420919</td>\n",
       "      <td>0.098211</td>\n",
       "      <td>-0.308744</td>\n",
       "      <td>-0.238373</td>\n",
       "      <td>0.105808</td>\n",
       "      <td>-0.230327</td>\n",
       "      <td>0.046714</td>\n",
       "      <td>0.237139</td>\n",
       "      <td>-0.073872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48481</th>\n",
       "      <td>-0.224903</td>\n",
       "      <td>-0.046536</td>\n",
       "      <td>0.105933</td>\n",
       "      <td>0.138320</td>\n",
       "      <td>0.308536</td>\n",
       "      <td>-0.486385</td>\n",
       "      <td>-0.147109</td>\n",
       "      <td>0.669781</td>\n",
       "      <td>-0.125035</td>\n",
       "      <td>-0.394241</td>\n",
       "      <td>...</td>\n",
       "      <td>0.076944</td>\n",
       "      <td>0.367996</td>\n",
       "      <td>-0.870528</td>\n",
       "      <td>-0.305757</td>\n",
       "      <td>0.010205</td>\n",
       "      <td>0.025413</td>\n",
       "      <td>-0.210676</td>\n",
       "      <td>-0.458167</td>\n",
       "      <td>0.639832</td>\n",
       "      <td>0.115506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48483</th>\n",
       "      <td>-0.334284</td>\n",
       "      <td>0.088137</td>\n",
       "      <td>-0.028772</td>\n",
       "      <td>-0.419087</td>\n",
       "      <td>0.146186</td>\n",
       "      <td>0.032084</td>\n",
       "      <td>0.498137</td>\n",
       "      <td>0.263859</td>\n",
       "      <td>0.312464</td>\n",
       "      <td>-0.129219</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.257484</td>\n",
       "      <td>-0.269723</td>\n",
       "      <td>0.116594</td>\n",
       "      <td>-0.221381</td>\n",
       "      <td>-0.304173</td>\n",
       "      <td>-0.068555</td>\n",
       "      <td>0.008158</td>\n",
       "      <td>0.153852</td>\n",
       "      <td>0.174848</td>\n",
       "      <td>0.095812</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48488</th>\n",
       "      <td>-0.818853</td>\n",
       "      <td>-0.930333</td>\n",
       "      <td>-0.327279</td>\n",
       "      <td>-0.424246</td>\n",
       "      <td>-0.954531</td>\n",
       "      <td>-0.041453</td>\n",
       "      <td>0.148937</td>\n",
       "      <td>-0.311416</td>\n",
       "      <td>0.090937</td>\n",
       "      <td>0.274851</td>\n",
       "      <td>...</td>\n",
       "      <td>0.414920</td>\n",
       "      <td>-1.015739</td>\n",
       "      <td>-0.041265</td>\n",
       "      <td>-0.171408</td>\n",
       "      <td>-0.161529</td>\n",
       "      <td>-0.068896</td>\n",
       "      <td>0.426150</td>\n",
       "      <td>-0.309882</td>\n",
       "      <td>-0.275953</td>\n",
       "      <td>0.215306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48491</th>\n",
       "      <td>-0.250934</td>\n",
       "      <td>-0.810936</td>\n",
       "      <td>-0.306815</td>\n",
       "      <td>-0.604034</td>\n",
       "      <td>-0.627260</td>\n",
       "      <td>-0.286715</td>\n",
       "      <td>-0.158840</td>\n",
       "      <td>-0.135506</td>\n",
       "      <td>-0.131089</td>\n",
       "      <td>0.065861</td>\n",
       "      <td>...</td>\n",
       "      <td>0.114063</td>\n",
       "      <td>-0.240617</td>\n",
       "      <td>0.285552</td>\n",
       "      <td>-0.297708</td>\n",
       "      <td>-0.324652</td>\n",
       "      <td>-0.156983</td>\n",
       "      <td>0.526073</td>\n",
       "      <td>-0.278961</td>\n",
       "      <td>-0.138874</td>\n",
       "      <td>-0.477214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48492</th>\n",
       "      <td>-0.370834</td>\n",
       "      <td>-0.181923</td>\n",
       "      <td>0.258452</td>\n",
       "      <td>-0.173429</td>\n",
       "      <td>0.005902</td>\n",
       "      <td>-0.063980</td>\n",
       "      <td>0.084326</td>\n",
       "      <td>0.148892</td>\n",
       "      <td>0.289342</td>\n",
       "      <td>-0.403003</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.454519</td>\n",
       "      <td>-0.253239</td>\n",
       "      <td>0.154116</td>\n",
       "      <td>-0.160369</td>\n",
       "      <td>-0.224938</td>\n",
       "      <td>0.083880</td>\n",
       "      <td>-0.174236</td>\n",
       "      <td>-0.198880</td>\n",
       "      <td>0.039738</td>\n",
       "      <td>-0.187738</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>16214 rows × 768 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               0         1         2         3         4         5         6  \\\n",
       "mal_id                                                                         \n",
       "1      -0.338000  0.280912  0.316794 -0.081068  0.229358 -0.153148 -0.447823   \n",
       "5      -0.025266 -0.049519  0.394098 -0.144688  0.058020 -0.056215 -0.152939   \n",
       "6      -0.050121  0.361339  0.314151  0.030035 -0.018952 -0.169436  0.143299   \n",
       "7      -0.277991 -0.150378  0.051548 -0.052691  0.007375 -0.371882  0.255686   \n",
       "8      -0.460624 -0.038628  0.111783 -0.103765 -0.212746  0.000713  0.400580   \n",
       "...          ...       ...       ...       ...       ...       ...       ...   \n",
       "48481  -0.224903 -0.046536  0.105933  0.138320  0.308536 -0.486385 -0.147109   \n",
       "48483  -0.334284  0.088137 -0.028772 -0.419087  0.146186  0.032084  0.498137   \n",
       "48488  -0.818853 -0.930333 -0.327279 -0.424246 -0.954531 -0.041453  0.148937   \n",
       "48491  -0.250934 -0.810936 -0.306815 -0.604034 -0.627260 -0.286715 -0.158840   \n",
       "48492  -0.370834 -0.181923  0.258452 -0.173429  0.005902 -0.063980  0.084326   \n",
       "\n",
       "               7         8         9  ...       758       759       760  \\\n",
       "mal_id                                ...                                 \n",
       "1       0.525362  0.265396 -0.106113  ... -0.887086  0.158227 -0.095016   \n",
       "5       0.453019  0.189982  0.088776  ... -0.925842 -0.115866 -0.085770   \n",
       "6       0.653574  0.067216  0.066807  ... -0.699382  0.128695  0.588282   \n",
       "7      -0.042258  0.262719  0.135156  ... -0.490768 -0.326309  0.118539   \n",
       "8       0.219770  0.255605 -0.086178  ... -0.709232 -0.420919  0.098211   \n",
       "...          ...       ...       ...  ...       ...       ...       ...   \n",
       "48481   0.669781 -0.125035 -0.394241  ...  0.076944  0.367996 -0.870528   \n",
       "48483   0.263859  0.312464 -0.129219  ... -0.257484 -0.269723  0.116594   \n",
       "48488  -0.311416  0.090937  0.274851  ...  0.414920 -1.015739 -0.041265   \n",
       "48491  -0.135506 -0.131089  0.065861  ...  0.114063 -0.240617  0.285552   \n",
       "48492   0.148892  0.289342 -0.403003  ... -0.454519 -0.253239  0.154116   \n",
       "\n",
       "             761       762       763       764       765       766       767  \n",
       "mal_id                                                                        \n",
       "1       0.038595  0.107712 -0.243573 -0.090685 -0.082259  0.366798 -0.188894  \n",
       "5      -0.468572 -0.072172  0.015078 -0.120486  0.005637  0.211060 -0.038622  \n",
       "6      -0.257558  0.276342  0.132950 -0.047471 -0.000429  0.647236 -0.094455  \n",
       "7      -0.256842 -0.066406 -0.026164 -0.191518 -0.156832  0.019953  0.001370  \n",
       "8      -0.308744 -0.238373  0.105808 -0.230327  0.046714  0.237139 -0.073872  \n",
       "...          ...       ...       ...       ...       ...       ...       ...  \n",
       "48481  -0.305757  0.010205  0.025413 -0.210676 -0.458167  0.639832  0.115506  \n",
       "48483  -0.221381 -0.304173 -0.068555  0.008158  0.153852  0.174848  0.095812  \n",
       "48488  -0.171408 -0.161529 -0.068896  0.426150 -0.309882 -0.275953  0.215306  \n",
       "48491  -0.297708 -0.324652 -0.156983  0.526073 -0.278961 -0.138874 -0.477214  \n",
       "48492  -0.160369 -0.224938  0.083880 -0.174236 -0.198880  0.039738 -0.187738  \n",
       "\n",
       "[16214 rows x 768 columns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# join embeddings on anime_clusters dataframe\n",
    "anime_embeddings = pd.merge(anime_clusters, embeddings, left_index=True, right_index=True, how='left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "      <th>episodes</th>\n",
       "      <th>rating</th>\n",
       "      <th>ranked</th>\n",
       "      <th>popularity</th>\n",
       "      <th>members</th>\n",
       "      <th>score_10</th>\n",
       "      <th>score_9</th>\n",
       "      <th>score_8</th>\n",
       "      <th>score_7</th>\n",
       "      <th>...</th>\n",
       "      <th>758</th>\n",
       "      <th>759</th>\n",
       "      <th>760</th>\n",
       "      <th>761</th>\n",
       "      <th>762</th>\n",
       "      <th>763</th>\n",
       "      <th>764</th>\n",
       "      <th>765</th>\n",
       "      <th>766</th>\n",
       "      <th>767</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mal_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.044157e+00</td>\n",
       "      <td>0.008181</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.001774</td>\n",
       "      <td>0.002220</td>\n",
       "      <td>0.483466</td>\n",
       "      <td>0.320601</td>\n",
       "      <td>0.340261</td>\n",
       "      <td>0.286693</td>\n",
       "      <td>0.205156</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.887086</td>\n",
       "      <td>0.158227</td>\n",
       "      <td>-0.095016</td>\n",
       "      <td>0.038595</td>\n",
       "      <td>0.107712</td>\n",
       "      <td>-0.243573</td>\n",
       "      <td>-0.090685</td>\n",
       "      <td>-0.082259</td>\n",
       "      <td>0.366798</td>\n",
       "      <td>-0.188894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2.521152e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.010076</td>\n",
       "      <td>0.029490</td>\n",
       "      <td>0.105479</td>\n",
       "      <td>0.042028</td>\n",
       "      <td>0.091919</td>\n",
       "      <td>0.107826</td>\n",
       "      <td>0.074490</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.925842</td>\n",
       "      <td>-0.115866</td>\n",
       "      <td>-0.085770</td>\n",
       "      <td>-0.468572</td>\n",
       "      <td>-0.072172</td>\n",
       "      <td>0.015078</td>\n",
       "      <td>-0.120486</td>\n",
       "      <td>0.005637</td>\n",
       "      <td>0.211060</td>\n",
       "      <td>-0.038622</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2.319997e+00</td>\n",
       "      <td>0.008181</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.016857</td>\n",
       "      <td>0.011443</td>\n",
       "      <td>0.215834</td>\n",
       "      <td>0.070268</td>\n",
       "      <td>0.141336</td>\n",
       "      <td>0.187625</td>\n",
       "      <td>0.162703</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.699382</td>\n",
       "      <td>0.128695</td>\n",
       "      <td>0.588282</td>\n",
       "      <td>-0.257558</td>\n",
       "      <td>0.276342</td>\n",
       "      <td>0.132950</td>\n",
       "      <td>-0.047471</td>\n",
       "      <td>-0.000429</td>\n",
       "      <td>0.647236</td>\n",
       "      <td>-0.094455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.019190e+00</td>\n",
       "      <td>0.008181</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.157224</td>\n",
       "      <td>0.083518</td>\n",
       "      <td>0.036563</td>\n",
       "      <td>0.003051</td>\n",
       "      <td>0.008977</td>\n",
       "      <td>0.022058</td>\n",
       "      <td>0.038237</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.490768</td>\n",
       "      <td>-0.326309</td>\n",
       "      <td>0.118539</td>\n",
       "      <td>-0.256842</td>\n",
       "      <td>-0.066406</td>\n",
       "      <td>-0.026164</td>\n",
       "      <td>-0.191518</td>\n",
       "      <td>-0.156832</td>\n",
       "      <td>0.019953</td>\n",
       "      <td>0.001370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>6.302889e-01</td>\n",
       "      <td>0.016688</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.235108</td>\n",
       "      <td>0.248733</td>\n",
       "      <td>0.005106</td>\n",
       "      <td>0.000435</td>\n",
       "      <td>0.000986</td>\n",
       "      <td>0.002703</td>\n",
       "      <td>0.005635</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.709232</td>\n",
       "      <td>-0.420919</td>\n",
       "      <td>0.098211</td>\n",
       "      <td>-0.308744</td>\n",
       "      <td>-0.238373</td>\n",
       "      <td>0.105808</td>\n",
       "      <td>-0.230327</td>\n",
       "      <td>0.046714</td>\n",
       "      <td>0.237139</td>\n",
       "      <td>-0.073872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48481</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.746712</td>\n",
       "      <td>0.000136</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.076944</td>\n",
       "      <td>0.367996</td>\n",
       "      <td>-0.870528</td>\n",
       "      <td>-0.305757</td>\n",
       "      <td>0.010205</td>\n",
       "      <td>0.025413</td>\n",
       "      <td>-0.210676</td>\n",
       "      <td>-0.458167</td>\n",
       "      <td>0.639832</td>\n",
       "      <td>0.115506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48483</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.999829</td>\n",
       "      <td>0.002707</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.257484</td>\n",
       "      <td>-0.269723</td>\n",
       "      <td>0.116594</td>\n",
       "      <td>-0.221381</td>\n",
       "      <td>-0.304173</td>\n",
       "      <td>-0.068555</td>\n",
       "      <td>0.008158</td>\n",
       "      <td>0.153852</td>\n",
       "      <td>0.174848</td>\n",
       "      <td>0.095812</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48488</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.999601</td>\n",
       "      <td>0.004367</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0.414920</td>\n",
       "      <td>-1.015739</td>\n",
       "      <td>-0.041265</td>\n",
       "      <td>-0.171408</td>\n",
       "      <td>-0.161529</td>\n",
       "      <td>-0.068896</td>\n",
       "      <td>0.426150</td>\n",
       "      <td>-0.309882</td>\n",
       "      <td>-0.275953</td>\n",
       "      <td>0.215306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48491</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000535</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>0.114063</td>\n",
       "      <td>-0.240617</td>\n",
       "      <td>0.285552</td>\n",
       "      <td>-0.297708</td>\n",
       "      <td>-0.324652</td>\n",
       "      <td>-0.156983</td>\n",
       "      <td>0.526073</td>\n",
       "      <td>-0.278961</td>\n",
       "      <td>-0.138874</td>\n",
       "      <td>-0.477214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48492</th>\n",
       "      <td>7.146486e-15</td>\n",
       "      <td>0.000327</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.500380</td>\n",
       "      <td>0.999886</td>\n",
       "      <td>0.000119</td>\n",
       "      <td>0.000053</td>\n",
       "      <td>0.000131</td>\n",
       "      <td>0.000198</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.454519</td>\n",
       "      <td>-0.253239</td>\n",
       "      <td>0.154116</td>\n",
       "      <td>-0.160369</td>\n",
       "      <td>-0.224938</td>\n",
       "      <td>0.083880</td>\n",
       "      <td>-0.174236</td>\n",
       "      <td>-0.198880</td>\n",
       "      <td>0.039738</td>\n",
       "      <td>-0.187738</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>17562 rows × 803 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               score  episodes  rating    ranked  popularity   members  \\\n",
       "mal_id                                                                   \n",
       "1       3.044157e+00  0.008181     0.8  0.001774    0.002220  0.483466   \n",
       "5       2.521152e+00  0.000000     0.8  0.010076    0.029490  0.105479   \n",
       "6       2.319997e+00  0.008181     0.4  0.016857    0.011443  0.215834   \n",
       "7       1.019190e+00  0.008181     0.4  0.157224    0.083518  0.036563   \n",
       "8       6.302889e-01  0.016688     0.2  0.235108    0.248733  0.005106   \n",
       "...              ...       ...     ...       ...         ...       ...   \n",
       "48481   7.146486e-15  0.000327     1.0  0.500380    0.746712  0.000136   \n",
       "48483   7.146486e-15  0.000327     1.0  0.500380    0.999829  0.002707   \n",
       "48488   7.146486e-15  0.000327     0.8  0.500380    0.999601  0.004367   \n",
       "48491   7.146486e-15  0.000327     0.4  0.500380    1.000000  0.000535   \n",
       "48492   7.146486e-15  0.000327     1.0  0.500380    0.999886  0.000119   \n",
       "\n",
       "        score_10   score_9   score_8   score_7  ...       758       759  \\\n",
       "mal_id                                          ...                       \n",
       "1       0.320601  0.340261  0.286693  0.205156  ... -0.887086  0.158227   \n",
       "5       0.042028  0.091919  0.107826  0.074490  ... -0.925842 -0.115866   \n",
       "6       0.070268  0.141336  0.187625  0.162703  ... -0.699382  0.128695   \n",
       "7       0.003051  0.008977  0.022058  0.038237  ... -0.490768 -0.326309   \n",
       "8       0.000435  0.000986  0.002703  0.005635  ... -0.709232 -0.420919   \n",
       "...          ...       ...       ...       ...  ...       ...       ...   \n",
       "48481   0.000053  0.000131  0.000198  0.000000  ...  0.076944  0.367996   \n",
       "48483   0.000053  0.000131  0.000198  0.000458  ... -0.257484 -0.269723   \n",
       "48488   0.000000  0.000131  0.000198  0.000458  ...  0.414920 -1.015739   \n",
       "48491   0.000053  0.000131  0.000198  0.000458  ...  0.114063 -0.240617   \n",
       "48492   0.000053  0.000131  0.000198  0.000458  ... -0.454519 -0.253239   \n",
       "\n",
       "             760       761       762       763       764       765       766  \\\n",
       "mal_id                                                                         \n",
       "1      -0.095016  0.038595  0.107712 -0.243573 -0.090685 -0.082259  0.366798   \n",
       "5      -0.085770 -0.468572 -0.072172  0.015078 -0.120486  0.005637  0.211060   \n",
       "6       0.588282 -0.257558  0.276342  0.132950 -0.047471 -0.000429  0.647236   \n",
       "7       0.118539 -0.256842 -0.066406 -0.026164 -0.191518 -0.156832  0.019953   \n",
       "8       0.098211 -0.308744 -0.238373  0.105808 -0.230327  0.046714  0.237139   \n",
       "...          ...       ...       ...       ...       ...       ...       ...   \n",
       "48481  -0.870528 -0.305757  0.010205  0.025413 -0.210676 -0.458167  0.639832   \n",
       "48483   0.116594 -0.221381 -0.304173 -0.068555  0.008158  0.153852  0.174848   \n",
       "48488  -0.041265 -0.171408 -0.161529 -0.068896  0.426150 -0.309882 -0.275953   \n",
       "48491   0.285552 -0.297708 -0.324652 -0.156983  0.526073 -0.278961 -0.138874   \n",
       "48492   0.154116 -0.160369 -0.224938  0.083880 -0.174236 -0.198880  0.039738   \n",
       "\n",
       "             767  \n",
       "mal_id            \n",
       "1      -0.188894  \n",
       "5      -0.038622  \n",
       "6      -0.094455  \n",
       "7       0.001370  \n",
       "8      -0.073872  \n",
       "...          ...  \n",
       "48481   0.115506  \n",
       "48483   0.095812  \n",
       "48488   0.215306  \n",
       "48491  -0.477214  \n",
       "48492  -0.187738  \n",
       "\n",
       "[17562 rows x 803 columns]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anime_embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For titles with no synopsis fill 0\n",
    "anime_embeddings = anime_embeddings.fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# KNN Recommendations using NLP Embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NearestNeighbors(algorithm='auto', leaf_size=30, metric='cosine',\n",
       "                 metric_params=None, n_jobs=None, n_neighbors=5, p=2,\n",
       "                 radius=1.0)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_e = NearestNeighbors(metric=\"cosine\")\n",
    "knn_e.fit(anime_embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mal_id</th>\n",
       "      <th>name</th>\n",
       "      <th>score</th>\n",
       "      <th>genres</th>\n",
       "      <th>english_name</th>\n",
       "      <th>japanese_name</th>\n",
       "      <th>type</th>\n",
       "      <th>episodes</th>\n",
       "      <th>aired</th>\n",
       "      <th>premiered</th>\n",
       "      <th>...</th>\n",
       "      <th>score_10</th>\n",
       "      <th>score_9</th>\n",
       "      <th>score_8</th>\n",
       "      <th>score_7</th>\n",
       "      <th>score_6</th>\n",
       "      <th>score_5</th>\n",
       "      <th>score_4</th>\n",
       "      <th>score_3</th>\n",
       "      <th>score_2</th>\n",
       "      <th>score_1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1117</th>\n",
       "      <td>5</td>\n",
       "      <td>Cowboy Bebop: Tengoku no Tobira</td>\n",
       "      <td>8.39</td>\n",
       "      <td>Action, Drama, Mystery, Sci-Fi, Space</td>\n",
       "      <td>Cowboy Bebop:The Movie</td>\n",
       "      <td>カウボーイビバップ 天国の扉</td>\n",
       "      <td>Movie</td>\n",
       "      <td>1</td>\n",
       "      <td>Sep 1, 2001</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>...</td>\n",
       "      <td>30043.0</td>\n",
       "      <td>49201.0</td>\n",
       "      <td>49505.0</td>\n",
       "      <td>22632.0</td>\n",
       "      <td>5805.0</td>\n",
       "      <td>1877.0</td>\n",
       "      <td>577.0</td>\n",
       "      <td>221.0</td>\n",
       "      <td>109.0</td>\n",
       "      <td>379.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 35 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      mal_id                             name score  \\\n",
       "1117       5  Cowboy Bebop: Tengoku no Tobira  8.39   \n",
       "\n",
       "                                     genres            english_name  \\\n",
       "1117  Action, Drama, Mystery, Sci-Fi, Space  Cowboy Bebop:The Movie   \n",
       "\n",
       "       japanese_name   type episodes        aired premiered  ... score_10  \\\n",
       "1117  カウボーイビバップ 天国の扉  Movie        1  Sep 1, 2001   Unknown  ...  30043.0   \n",
       "\n",
       "      score_9  score_8  score_7 score_6 score_5 score_4  score_3  score_2  \\\n",
       "1117  49201.0  49505.0  22632.0  5805.0  1877.0   577.0    221.0    109.0   \n",
       "\n",
       "      score_1  \n",
       "1117    379.0  \n",
       "\n",
       "[1 rows x 35 columns]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anime[anime['mal_id'] == anime_embeddings.iloc[1, :].name]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "distances, titles = knn_e.kneighbors(anime_embeddings.iloc[1, :].values.reshape(1, -1), n_neighbors=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.        , 0.07331998, 0.07454903, 0.07488714, 0.07849818,\n",
       "        0.07853637, 0.07978431, 0.08434003, 0.08455094, 0.0848173 ]])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "distances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 1234, 439, 711, 10967, 6436, 376, 12714, 3584, 2297]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(titles[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 1365, 468, 781, 31490, 8100, 400, 34944, 2593, 4106]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert rows to anime_ids\n",
    "anime_embeddings.iloc[list(titles[0]), :].index.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recommendations using KNN + NLP Embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mal_id</th>\n",
       "      <th>name</th>\n",
       "      <th>genres</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1117</th>\n",
       "      <td>5</td>\n",
       "      <td>Cowboy Bebop: Tengoku no Tobira</td>\n",
       "      <td>Action, Drama, Mystery, Sci-Fi, Space</td>\n",
       "      <td>R - 17+ (violence &amp; profanity)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>1365</td>\n",
       "      <td>Detective Conan Movie 06: The Phantom of Baker...</td>\n",
       "      <td>Adventure, Mystery, Police, Shounen</td>\n",
       "      <td>PG-13 - Teens 13 or older</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1555</th>\n",
       "      <td>468</td>\n",
       "      <td>Innocence</td>\n",
       "      <td>Military, Sci-Fi, Police, Psychological, Mecha</td>\n",
       "      <td>R - 17+ (violence &amp; profanity)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1827</th>\n",
       "      <td>781</td>\n",
       "      <td>Detective Conan Movie 03: The Last Wizard of t...</td>\n",
       "      <td>Adventure, Mystery, Comedy, Police, Shounen</td>\n",
       "      <td>PG-13 - Teens 13 or older</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10967</th>\n",
       "      <td>31490</td>\n",
       "      <td>One Piece Film: Gold</td>\n",
       "      <td>Action, Adventure, Comedy, Drama, Fantasy, Sho...</td>\n",
       "      <td>PG-13 - Teens 13 or older</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6436</th>\n",
       "      <td>8100</td>\n",
       "      <td>Mardock Scramble: The First Compression</td>\n",
       "      <td>Action, Sci-Fi, Psychological</td>\n",
       "      <td>R+ - Mild Nudity</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1492</th>\n",
       "      <td>400</td>\n",
       "      <td>Seihou Bukyou Outlaw Star</td>\n",
       "      <td>Action, Sci-Fi, Adventure, Space, Comedy</td>\n",
       "      <td>R+ - Mild Nudity</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12714</th>\n",
       "      <td>34944</td>\n",
       "      <td>Bungou Stray Dogs: Dead Apple</td>\n",
       "      <td>Action, Comedy, Mystery, Super Power, Supernat...</td>\n",
       "      <td>R - 17+ (violence &amp; profanity)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2379</th>\n",
       "      <td>2593</td>\n",
       "      <td>Kara no Kyoukai 1: Fukan Fuukei</td>\n",
       "      <td>Action, Mystery, Supernatural, Thriller</td>\n",
       "      <td>R - 17+ (violence &amp; profanity)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3502</th>\n",
       "      <td>4106</td>\n",
       "      <td>Trigun: Badlands Rumble</td>\n",
       "      <td>Action, Adventure, Comedy, Drama, Sci-Fi, Shounen</td>\n",
       "      <td>PG-13 - Teens 13 or older</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      mal_id                                               name  \\\n",
       "1117       5                    Cowboy Bebop: Tengoku no Tobira   \n",
       "118     1365  Detective Conan Movie 06: The Phantom of Baker...   \n",
       "1555     468                                          Innocence   \n",
       "1827     781  Detective Conan Movie 03: The Last Wizard of t...   \n",
       "10967  31490                               One Piece Film: Gold   \n",
       "6436    8100            Mardock Scramble: The First Compression   \n",
       "1492     400                          Seihou Bukyou Outlaw Star   \n",
       "12714  34944                      Bungou Stray Dogs: Dead Apple   \n",
       "2379    2593                    Kara no Kyoukai 1: Fukan Fuukei   \n",
       "3502    4106                            Trigun: Badlands Rumble   \n",
       "\n",
       "                                                  genres  \\\n",
       "1117               Action, Drama, Mystery, Sci-Fi, Space   \n",
       "118                  Adventure, Mystery, Police, Shounen   \n",
       "1555      Military, Sci-Fi, Police, Psychological, Mecha   \n",
       "1827         Adventure, Mystery, Comedy, Police, Shounen   \n",
       "10967  Action, Adventure, Comedy, Drama, Fantasy, Sho...   \n",
       "6436                       Action, Sci-Fi, Psychological   \n",
       "1492            Action, Sci-Fi, Adventure, Space, Comedy   \n",
       "12714  Action, Comedy, Mystery, Super Power, Supernat...   \n",
       "2379             Action, Mystery, Supernatural, Thriller   \n",
       "3502   Action, Adventure, Comedy, Drama, Sci-Fi, Shounen   \n",
       "\n",
       "                               rating  \n",
       "1117   R - 17+ (violence & profanity)  \n",
       "118         PG-13 - Teens 13 or older  \n",
       "1555   R - 17+ (violence & profanity)  \n",
       "1827        PG-13 - Teens 13 or older  \n",
       "10967       PG-13 - Teens 13 or older  \n",
       "6436                 R+ - Mild Nudity  \n",
       "1492                 R+ - Mild Nudity  \n",
       "12714  R - 17+ (violence & profanity)  \n",
       "2379   R - 17+ (violence & profanity)  \n",
       "3502        PG-13 - Teens 13 or older  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.merge(anime_embeddings.iloc[list(titles[0]), [1]], anime, left_index=True, right_on='mal_id')[['mal_id', 'name', 'genres', 'rating']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommender Systems\n",
    "\n",
    "There will be 3 for evaluation:\n",
    "\n",
    " 1. A Naive System that generates recommendations by uniformly random selection of titles\n",
    " 2. Simpler Recommendation System using only feature set and clustering\n",
    " 3. A comprehensive Recommendation system that includes feature set, clustering and Synopsis Embeddings using BERT\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Naive Recommender\n",
    "\n",
    "This is a baseline system that is unskilled and generates k uniformly random recommendations from the anime titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This version is the naive recommender\n",
    "def naive_recommendations(user_id, k=10):\n",
    "    \"\"\"\n",
    "    user_id: The user_id in the anime_list table\n",
    "    k: The top k recommendations\n",
    "\n",
    "    Returns: Dataframe of top k titles\n",
    "    \"\"\"\n",
    "\n",
    "    # Filter by user_id and remove titles from anime_embeddings if title watching status is 1 or 2\n",
    "    user = training_w_clusters[training_w_clusters['user_id'] == user_id].sort_values(['user_rating'], ascending=False)\n",
    "    # list of already watched titles\n",
    "    already_watched = user[user['watching_status'].isin([1, 2])]['anime_id'].values.tolist()\n",
    "    \n",
    "    # Filtered anime table \n",
    "    anime_filtered = anime[~anime['mal_id'].isin(already_watched)]\n",
    "    \n",
    "    return anime_filtered.sample(k, replace=False, random_state=42)[['mal_id', 'name', \n",
    "                                                                     'score', 'genres', 'rating']].reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mal_id</th>\n",
       "      <th>name</th>\n",
       "      <th>score</th>\n",
       "      <th>genres</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>41220</td>\n",
       "      <td>Huo Feng Liao Yuan</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>Action, Adventure, Historical, Martial Arts</td>\n",
       "      <td>Unknown</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>40594</td>\n",
       "      <td>Tatoeba Last Dungeon Mae no Mura no Shounen ga...</td>\n",
       "      <td>6.75</td>\n",
       "      <td>Adventure, Fantasy</td>\n",
       "      <td>Unknown</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>34659</td>\n",
       "      <td>Otokonoko ♂ Delivery</td>\n",
       "      <td>5.86</td>\n",
       "      <td>Hentai, Yaoi</td>\n",
       "      <td>Rx - Hentai</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10755</td>\n",
       "      <td>Lupin tai Holmes</td>\n",
       "      <td>6.08</td>\n",
       "      <td>Mystery</td>\n",
       "      <td>Unknown</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>41212</td>\n",
       "      <td>Niplheim no Karyuudo</td>\n",
       "      <td>5.29</td>\n",
       "      <td>Hentai</td>\n",
       "      <td>Rx - Hentai</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2030</td>\n",
       "      <td>El Cazador de la Bruja</td>\n",
       "      <td>7.24</td>\n",
       "      <td>Adventure, Drama, Mystery</td>\n",
       "      <td>PG-13 - Teens 13 or older</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>28201</td>\n",
       "      <td>Yeonghongibyeong Lazenca</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>Action, Adventure, Mecha, Sci-Fi</td>\n",
       "      <td>Unknown</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2972</td>\n",
       "      <td>Kindan no Mokushiroku: Crystal Triangle</td>\n",
       "      <td>4.62</td>\n",
       "      <td>Adventure, Military, Demons, Mystery</td>\n",
       "      <td>PG-13 - Teens 13 or older</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>36548</td>\n",
       "      <td>Kokkoku</td>\n",
       "      <td>7.04</td>\n",
       "      <td>Psychological, Supernatural, Drama, Mystery, S...</td>\n",
       "      <td>R - 17+ (violence &amp; profanity)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>8039</td>\n",
       "      <td>Oppai no Ouja 48</td>\n",
       "      <td>7.01</td>\n",
       "      <td>Hentai</td>\n",
       "      <td>Rx - Hentai</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mal_id                                               name    score  \\\n",
       "0   41220                                 Huo Feng Liao Yuan  Unknown   \n",
       "1   40594  Tatoeba Last Dungeon Mae no Mura no Shounen ga...     6.75   \n",
       "2   34659                               Otokonoko ♂ Delivery     5.86   \n",
       "3   10755                                   Lupin tai Holmes     6.08   \n",
       "4   41212                               Niplheim no Karyuudo     5.29   \n",
       "5    2030                             El Cazador de la Bruja     7.24   \n",
       "6   28201                           Yeonghongibyeong Lazenca  Unknown   \n",
       "7    2972            Kindan no Mokushiroku: Crystal Triangle     4.62   \n",
       "8   36548                                            Kokkoku     7.04   \n",
       "9    8039                                   Oppai no Ouja 48     7.01   \n",
       "\n",
       "                                              genres  \\\n",
       "0        Action, Adventure, Historical, Martial Arts   \n",
       "1                                 Adventure, Fantasy   \n",
       "2                                       Hentai, Yaoi   \n",
       "3                                            Mystery   \n",
       "4                                             Hentai   \n",
       "5                          Adventure, Drama, Mystery   \n",
       "6                   Action, Adventure, Mecha, Sci-Fi   \n",
       "7               Adventure, Military, Demons, Mystery   \n",
       "8  Psychological, Supernatural, Drama, Mystery, S...   \n",
       "9                                             Hentai   \n",
       "\n",
       "                           rating  \n",
       "0                         Unknown  \n",
       "1                         Unknown  \n",
       "2                     Rx - Hentai  \n",
       "3                         Unknown  \n",
       "4                     Rx - Hentai  \n",
       "5       PG-13 - Teens 13 or older  \n",
       "6                         Unknown  \n",
       "7       PG-13 - Teens 13 or older  \n",
       "8  R - 17+ (violence & profanity)  \n",
       "9                     Rx - Hentai  "
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test using user_id = 353398\n",
    "naive_recommendations(353398, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recommender using clustering only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This version is the simple recommender system without BERT embeddings\n",
    "def simple_recommendations(user_id, k=10):\n",
    "    \"\"\"\n",
    "    user_id: The user_id in the anime_list table\n",
    "    k: The top k recommendations\n",
    "\n",
    "    Returns: Dataframe of top k titles\n",
    "    \"\"\"\n",
    "\n",
    "    # Filter by user_id and remove titles from anime_embeddings if title watching status is 1 or 2\n",
    "    user = training_w_clusters[training_w_clusters['user_id'] == user_id].sort_values(['user_rating'], ascending=False)\n",
    "    # anime_id of highest rated\n",
    "    highest_rated = user['anime_id'].values.tolist()[0]\n",
    "    # list of already watched titles\n",
    "    already_watched = user[user['watching_status'].isin([1, 2])]['anime_id'].values.tolist()\n",
    "    \n",
    "    # Filtered anime table \n",
    "    anime_filtered = anime_clusters[~anime_clusters.index.isin(already_watched)]\n",
    "    # Query point\n",
    "    query = anime_clusters[anime_clusters.index == highest_rated]\n",
    "    \n",
    "    # KNN model on filtered anime table\n",
    "    knn = NearestNeighbors(metric=\"cosine\")\n",
    "    knn.fit(anime_filtered)\n",
    "    \n",
    "    distances, titles = knn.kneighbors(query.values.reshape(1, -1), n_neighbors=k)\n",
    "    row_list = list(titles[0])\n",
    "    # Convert row list to anime ids\n",
    "    anime_ids = anime_filtered.iloc[row_list, :].index.tolist()\n",
    "    \n",
    "    return pd.merge(anime_filtered.iloc[row_list, [1]], anime, left_index=True, right_on='mal_id')[['mal_id', 'name', \n",
    "                                                                                                    'score', 'genres', 'rating']].reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mal_id</th>\n",
       "      <th>name</th>\n",
       "      <th>score</th>\n",
       "      <th>genres</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>17247</td>\n",
       "      <td>Machine-Doll wa Kizutsukanai</td>\n",
       "      <td>7.15</td>\n",
       "      <td>Action, Ecchi, Fantasy, School</td>\n",
       "      <td>R+ - Mild Nudity</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6747</td>\n",
       "      <td>Dance in the Vampire Bund</td>\n",
       "      <td>7.06</td>\n",
       "      <td>Action, Supernatural, Vampire</td>\n",
       "      <td>R+ - Mild Nudity</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>14829</td>\n",
       "      <td>Fate/kaleid liner Prisma☆Illya</td>\n",
       "      <td>7.11</td>\n",
       "      <td>Action, Comedy, Magic, Fantasy</td>\n",
       "      <td>PG-13 - Teens 13 or older</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>34662</td>\n",
       "      <td>Fate/Apocrypha</td>\n",
       "      <td>7.16</td>\n",
       "      <td>Action, Supernatural, Drama, Magic, Fantasy</td>\n",
       "      <td>R - 17+ (violence &amp; profanity)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>34104</td>\n",
       "      <td>Knight's &amp; Magic</td>\n",
       "      <td>7.13</td>\n",
       "      <td>Action, Fantasy, Mecha, School</td>\n",
       "      <td>PG-13 - Teens 13 or older</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>34176</td>\n",
       "      <td>Zero kara Hajimeru Mahou no Sho</td>\n",
       "      <td>7.1</td>\n",
       "      <td>Action, Adventure, Magic, Fantasy</td>\n",
       "      <td>PG-13 - Teens 13 or older</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>16035</td>\n",
       "      <td>Karneval (TV)</td>\n",
       "      <td>7.14</td>\n",
       "      <td>Action, Fantasy, Josei, Mystery, Sci-Fi</td>\n",
       "      <td>PG-13 - Teens 13 or older</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>18277</td>\n",
       "      <td>Strike the Blood</td>\n",
       "      <td>7.14</td>\n",
       "      <td>Action, Harem, Supernatural, Ecchi, Vampire, F...</td>\n",
       "      <td>R+ - Mild Nudity</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>21085</td>\n",
       "      <td>Witch Craft Works</td>\n",
       "      <td>7.07</td>\n",
       "      <td>Action, Fantasy, Magic, Seinen, Supernatural</td>\n",
       "      <td>PG-13 - Teens 13 or older</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>4896</td>\n",
       "      <td>Umineko no Naku Koro ni</td>\n",
       "      <td>7.13</td>\n",
       "      <td>Mystery, Horror, Psychological, Supernatural</td>\n",
       "      <td>R - 17+ (violence &amp; profanity)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mal_id                             name score  \\\n",
       "0   17247     Machine-Doll wa Kizutsukanai  7.15   \n",
       "1    6747        Dance in the Vampire Bund  7.06   \n",
       "2   14829   Fate/kaleid liner Prisma☆Illya  7.11   \n",
       "3   34662                   Fate/Apocrypha  7.16   \n",
       "4   34104                 Knight's & Magic  7.13   \n",
       "5   34176  Zero kara Hajimeru Mahou no Sho   7.1   \n",
       "6   16035                    Karneval (TV)  7.14   \n",
       "7   18277                 Strike the Blood  7.14   \n",
       "8   21085                Witch Craft Works  7.07   \n",
       "9    4896          Umineko no Naku Koro ni  7.13   \n",
       "\n",
       "                                              genres  \\\n",
       "0                     Action, Ecchi, Fantasy, School   \n",
       "1                      Action, Supernatural, Vampire   \n",
       "2                     Action, Comedy, Magic, Fantasy   \n",
       "3        Action, Supernatural, Drama, Magic, Fantasy   \n",
       "4                     Action, Fantasy, Mecha, School   \n",
       "5                  Action, Adventure, Magic, Fantasy   \n",
       "6            Action, Fantasy, Josei, Mystery, Sci-Fi   \n",
       "7  Action, Harem, Supernatural, Ecchi, Vampire, F...   \n",
       "8       Action, Fantasy, Magic, Seinen, Supernatural   \n",
       "9       Mystery, Horror, Psychological, Supernatural   \n",
       "\n",
       "                           rating  \n",
       "0                R+ - Mild Nudity  \n",
       "1                R+ - Mild Nudity  \n",
       "2       PG-13 - Teens 13 or older  \n",
       "3  R - 17+ (violence & profanity)  \n",
       "4       PG-13 - Teens 13 or older  \n",
       "5       PG-13 - Teens 13 or older  \n",
       "6       PG-13 - Teens 13 or older  \n",
       "7                R+ - Mild Nudity  \n",
       "8       PG-13 - Teens 13 or older  \n",
       "9  R - 17+ (violence & profanity)  "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test using user_id = 353398\n",
    "simple_recommendations(353398, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recommender with clustering and BERT embeddings\n",
    "\n",
    " * Takes in a user_id from the animelist table.\n",
    " * Get the user's highest rated title. Filter out anime_embeddings table by already watched titles (watching_status 1 or 2) Don't want to recommend already watched titles\n",
    " * KNN will use the filtered anime_embeddings table\n",
    " * The query data point will be the user's highest rated title.\n",
    " * The k nearest neighbors to the query point will be the 10 top recommended titles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "def recommendations(user_id, k=10):\n",
    "    \"\"\"\n",
    "    user_id: The user_id in the anime_list table\n",
    "    k: The top k recommendations\n",
    "\n",
    "    Returns: Dataframe of top k titles\n",
    "    \"\"\"\n",
    "\n",
    "    # Filter by user_id and remove titles from anime_embeddings if title watching status is 1 or 2\n",
    "    user = training_w_clusters[training_w_clusters['user_id'] == user_id].sort_values(['user_rating'], ascending=False)\n",
    "    # anime_id of highest rated\n",
    "    highest_rated = user['anime_id'].values.tolist()[0]\n",
    "    # list of already watched titles\n",
    "    already_watched = user[user['watching_status'].isin([1, 2])]['anime_id'].values.tolist()\n",
    "    \n",
    "    # Filtered anime table \n",
    "    anime_filtered = anime_embeddings[~anime_embeddings.index.isin(already_watched)]\n",
    "    # Query point\n",
    "    query = anime_embeddings[anime_embeddings.index == highest_rated]\n",
    "    \n",
    "    # KNN model on filtered anime table\n",
    "    knn = NearestNeighbors(metric=\"cosine\")\n",
    "    knn.fit(anime_filtered)\n",
    "    \n",
    "    distances, titles = knn.kneighbors(query.values.reshape(1, -1), n_neighbors=k)\n",
    "    row_list = list(titles[0])\n",
    "    # Convert row list to anime ids\n",
    "    anime_ids = anime_filtered.iloc[row_list, :].index.tolist()\n",
    "    \n",
    "    return pd.merge(anime_filtered.iloc[row_list, [1]], anime, left_index=True, right_on='mal_id')[['mal_id', 'name', 'score', \n",
    "                                                                                                    'genres', 'rating']].reset_index(drop=True)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mal_id</th>\n",
       "      <th>name</th>\n",
       "      <th>score</th>\n",
       "      <th>genres</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>17247</td>\n",
       "      <td>Machine-Doll wa Kizutsukanai</td>\n",
       "      <td>7.15</td>\n",
       "      <td>Action, Ecchi, Fantasy, School</td>\n",
       "      <td>R+ - Mild Nudity</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>238</td>\n",
       "      <td>Rekka no Honoo</td>\n",
       "      <td>7.36</td>\n",
       "      <td>Action, Adventure, Martial Arts, Shounen, Supe...</td>\n",
       "      <td>PG-13 - Teens 13 or older</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9065</td>\n",
       "      <td>Hakuouki Hekketsuroku</td>\n",
       "      <td>7.62</td>\n",
       "      <td>Action, Drama, Historical, Josei, Samurai, Sup...</td>\n",
       "      <td>R - 17+ (violence &amp; profanity)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>14075</td>\n",
       "      <td>Zetsuen no Tempest</td>\n",
       "      <td>7.99</td>\n",
       "      <td>Action, Mystery, Psychological, Drama, Magic, ...</td>\n",
       "      <td>PG-13 - Teens 13 or older</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>34561</td>\n",
       "      <td>Re:Creators</td>\n",
       "      <td>7.57</td>\n",
       "      <td>Action, Sci-Fi, Fantasy, Mecha</td>\n",
       "      <td>R - 17+ (violence &amp; profanity)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>3375</td>\n",
       "      <td>Kirepapa.</td>\n",
       "      <td>6.7</td>\n",
       "      <td>Comedy, Drama, Romance, Yaoi</td>\n",
       "      <td>R+ - Mild Nudity</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2116</td>\n",
       "      <td>Captain Tsubasa</td>\n",
       "      <td>7.3</td>\n",
       "      <td>Action, Shounen, Sports</td>\n",
       "      <td>G - All Ages</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>9465</td>\n",
       "      <td>Break Blade 4: Sanka no Chi</td>\n",
       "      <td>7.87</td>\n",
       "      <td>Action, Fantasy, Mecha, Military, Shounen</td>\n",
       "      <td>R - 17+ (violence &amp; profanity)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>36934</td>\n",
       "      <td>Captain Tsubasa (2018)</td>\n",
       "      <td>7.42</td>\n",
       "      <td>Action, Sports, Shounen</td>\n",
       "      <td>G - All Ages</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>26159</td>\n",
       "      <td>Shinken Seminar Koukou Kouza</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>School, Shounen</td>\n",
       "      <td>PG-13 - Teens 13 or older</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  mal_id                          name    score  \\\n",
       "0  17247  Machine-Doll wa Kizutsukanai     7.15   \n",
       "1    238                Rekka no Honoo     7.36   \n",
       "2   9065         Hakuouki Hekketsuroku     7.62   \n",
       "3  14075            Zetsuen no Tempest     7.99   \n",
       "4  34561                   Re:Creators     7.57   \n",
       "5   3375                     Kirepapa.      6.7   \n",
       "6   2116               Captain Tsubasa      7.3   \n",
       "7   9465   Break Blade 4: Sanka no Chi     7.87   \n",
       "8  36934        Captain Tsubasa (2018)     7.42   \n",
       "9  26159  Shinken Seminar Koukou Kouza  Unknown   \n",
       "\n",
       "                                              genres  \\\n",
       "0                     Action, Ecchi, Fantasy, School   \n",
       "1  Action, Adventure, Martial Arts, Shounen, Supe...   \n",
       "2  Action, Drama, Historical, Josei, Samurai, Sup...   \n",
       "3  Action, Mystery, Psychological, Drama, Magic, ...   \n",
       "4                     Action, Sci-Fi, Fantasy, Mecha   \n",
       "5                       Comedy, Drama, Romance, Yaoi   \n",
       "6                            Action, Shounen, Sports   \n",
       "7          Action, Fantasy, Mecha, Military, Shounen   \n",
       "8                            Action, Sports, Shounen   \n",
       "9                                    School, Shounen   \n",
       "\n",
       "                           rating  \n",
       "0                R+ - Mild Nudity  \n",
       "1       PG-13 - Teens 13 or older  \n",
       "2  R - 17+ (violence & profanity)  \n",
       "3       PG-13 - Teens 13 or older  \n",
       "4  R - 17+ (violence & profanity)  \n",
       "5                R+ - Mild Nudity  \n",
       "6                    G - All Ages  \n",
       "7  R - 17+ (violence & profanity)  \n",
       "8                    G - All Ages  \n",
       "9       PG-13 - Teens 13 or older  "
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test using user_id = 353398\n",
    "recommendations(353398, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Evaluation\n",
    "\n",
    "Since there are over 240,000 users to create recommendations for, we will use a random sample of 100 users to generate recommendations for. There are 3 models to be evaluated. First a naive model that generates 10 recommendations (uniformly sampled from the 17,562 anime titles) for each of the 100 sampled users. Second is a simple version of the recommender system that does not use the synopsis embeddings from BERT. Third a more comprehensive model that uses the synopsis embeddings from BERT.\n",
    "\n",
    "The models will evaluated using Personalization (cosine similarity) to measure recommendation similarity or dissimilarity between users and Intra-list similarity which evaluates how similar recommendations are for each user.\n",
    "\n",
    "Ideally you would want a higher average personalization score which demonstrates more personalized experience for each user. The intra-list similarity for each user measures how similar each recommended title is a higher intra-list similarity measure means that the system is providing relevant or closely related recommendations (i.e not random)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Batch recommendations from each recommender system for evaluation\n",
    "\n",
    "Will randomly select 100 user create recommendations from each system and store in s3 for evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 100 random sample of users\n",
    "eval_users = pd.Series(training_w_clusters['user_id'].unique()).sample(100, replace=False, random_state=42).values.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "99 of 100 users\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import clear_output\n",
    "\n",
    "naive = []\n",
    "simple = []\n",
    "comprehensive = []\n",
    "# Ten recommendations for each user\n",
    "k=10\n",
    "\n",
    "for i, user_id in enumerate(eval_users):\n",
    "    clear_output(wait=True)\n",
    "    \n",
    "    n_rec = naive_recommendations(user_id, k)\n",
    "    n_rec['user_id'] = user_id\n",
    "    naive.append(n_rec)\n",
    "    \n",
    "    s_rec = simple_recommendations(user_id, k)\n",
    "    s_rec['user_id'] = user_id\n",
    "    simple.append(s_rec)\n",
    "    \n",
    "    c_rec = recommendations(user_id, k)\n",
    "    c_rec['user_id'] = user_id\n",
    "    comprehensive.append(c_rec)\n",
    "    \n",
    "    print('%d of %d users' % (i, len(eval_users)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "naive_recs = pd.concat(naive, axis=0)\n",
    "naive_recs.set_index(['user_id']).to_csv('Data/naive_recommendations.csv', index=True)\n",
    "\n",
    "simple_recs = pd.concat(simple, axis=0)\n",
    "simple_recs.set_index(['user_id']).to_csv('Data/simple_recommendations.csv', index=True)\n",
    "\n",
    "comp_recs = pd.concat(comprehensive, axis=0)\n",
    "comp_recs.set_index(['user_id']).to_csv('Data/comprehensive_recommendations.csv', index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "def upload_file(file_name, bucket, object_name=None):\n",
    "    \"\"\"Upload a file to an S3 bucket\n",
    "\n",
    "    :param file_name: File to upload\n",
    "    :param bucket: Bucket to upload to\n",
    "    :param object_name: S3 object name. If not specified then file_name is used\n",
    "    :return: True if file was uploaded, else False\n",
    "    \"\"\"\n",
    "\n",
    "    # If S3 object_name was not specified, use file_name\n",
    "    if object_name is None:\n",
    "        object_name = os.path.basename(file_name)\n",
    "\n",
    "    # Upload the file\n",
    "    s3_client = boto3.client('s3')\n",
    "    try:\n",
    "        response = s3_client.upload_file(file_name, bucket, object_name)\n",
    "    except ClientError as e:\n",
    "        logging.error(e)\n",
    "        return False\n",
    "    return True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "upload_file('Data/naive_recommendations.csv', 'ads508team3', 'modeling/naive_recommendations.csv')\n",
    "upload_file('Data/simple_recommendations.csv', 'ads508team3', 'modeling/simple_recommendations.csv')\n",
    "upload_file('Data/comprehensive_recommendations.csv', 'ads508team3', 'modeling/comprehensive_recommendations.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Metrics and Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Personalization Metric\n",
    "\n",
    "(1 - Cosine Similarity) between all user title recommendations. Metric to describe disimilarity between recommendations across users. The higher the disimilarity, demonstrates a more personalized user experience"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To compute cosine similarity matrix\n",
    "from sklearn.metrics.pairwise import cosine_similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Naive Recommender Personalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>319</th>\n",
       "      <th>320</th>\n",
       "      <th>1005</th>\n",
       "      <th>1006</th>\n",
       "      <th>1007</th>\n",
       "      <th>1008</th>\n",
       "      <th>1121</th>\n",
       "      <th>1122</th>\n",
       "      <th>1172</th>\n",
       "      <th>1174</th>\n",
       "      <th>...</th>\n",
       "      <th>40529</th>\n",
       "      <th>40530</th>\n",
       "      <th>40594</th>\n",
       "      <th>40819</th>\n",
       "      <th>40897</th>\n",
       "      <th>41212</th>\n",
       "      <th>41220</th>\n",
       "      <th>41568</th>\n",
       "      <th>41729</th>\n",
       "      <th>43467</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>864</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1661</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2119</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9362</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10099</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>329689</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>338474</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>341890</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>343786</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>344325</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 190 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         319    320    1005   1006   1007   1008   1121   1122   1172   1174   \\\n",
       "user_id                                                                         \n",
       "864          0      0      0      1      0      0      0      0      0      0   \n",
       "1661         1      0      0      0      0      0      1      0      0      0   \n",
       "2119         0      0      0      0      0      0      0      0      0      0   \n",
       "9362         0      0      0      0      1      0      0      0      0      0   \n",
       "10099        1      0      0      0      0      0      1      0      0      0   \n",
       "...        ...    ...    ...    ...    ...    ...    ...    ...    ...    ...   \n",
       "329689       1      0      0      0      0      0      1      0      0      0   \n",
       "338474       1      0      0      0      0      0      1      0      0      0   \n",
       "341890       0      1      0      0      0      0      0      1      0      0   \n",
       "343786       1      0      0      0      0      0      1      0      0      0   \n",
       "344325       0      0      0      0      0      0      0      0      0      0   \n",
       "\n",
       "         ...  40529  40530  40594  40819  40897  41212  41220  41568  41729  \\\n",
       "user_id  ...                                                                  \n",
       "864      ...      0      0      0      0      0      0      0      0      0   \n",
       "1661     ...      0      0      0      0      0      0      0      0      1   \n",
       "2119     ...      0      0      0      0      0      0      0      0      0   \n",
       "9362     ...      0      0      0      0      0      0      0      1      0   \n",
       "10099    ...      0      0      0      0      0      0      0      0      1   \n",
       "...      ...    ...    ...    ...    ...    ...    ...    ...    ...    ...   \n",
       "329689   ...      0      0      0      0      0      0      0      0      1   \n",
       "338474   ...      0      0      0      0      0      0      0      0      1   \n",
       "341890   ...      0      0      0      0      0      0      0      0      1   \n",
       "343786   ...      0      0      0      0      0      0      0      0      1   \n",
       "344325   ...      0      0      0      0      1      0      0      0      0   \n",
       "\n",
       "         43467  \n",
       "user_id         \n",
       "864          0  \n",
       "1661         1  \n",
       "2119         0  \n",
       "9362         0  \n",
       "10099        1  \n",
       "...        ...  \n",
       "329689       1  \n",
       "338474       1  \n",
       "341890       1  \n",
       "343786       1  \n",
       "344325       0  \n",
       "\n",
       "[100 rows x 190 columns]"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Pivot title ids creating binary columns of titles that have been recommended\n",
    "naive_cos_mat = naive_recs[['user_id', 'mal_id']].groupby(['user_id', 'mal_id']).size().unstack(fill_value=0).astype(bool).rename_axis(None, axis=1).astype(int)\n",
    "naive_cos_mat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Construct Pairwise Cosine Similarity Matrix\n",
    "naive_cos_mat = cosine_similarity(naive_cos_mat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8614059405940594"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute the mean of the upper triangle of the matrix (pairwise average)\n",
    "1 - np.mean(list(naive_cos_mat[np.triu_indices(100)]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Simple Recommender Personalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>18</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>30</th>\n",
       "      <th>32</th>\n",
       "      <th>...</th>\n",
       "      <th>40956</th>\n",
       "      <th>41006</th>\n",
       "      <th>41389</th>\n",
       "      <th>41491</th>\n",
       "      <th>41511</th>\n",
       "      <th>41917</th>\n",
       "      <th>42949</th>\n",
       "      <th>43299</th>\n",
       "      <th>44042</th>\n",
       "      <th>44218</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>864</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1661</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2119</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9362</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10099</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>329689</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>338474</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>341890</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>343786</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>344325</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 686 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         1      15     16     18     20     21     25     26     30     32     \\\n",
       "user_id                                                                         \n",
       "864          0      1      0      0      0      0      0      0      0      0   \n",
       "1661         0      0      0      0      0      0      0      0      0      0   \n",
       "2119         0      0      0      0      0      0      0      0      0      0   \n",
       "9362         0      0      0      0      0      0      0      0      0      0   \n",
       "10099        0      0      0      0      0      0      0      0      0      0   \n",
       "...        ...    ...    ...    ...    ...    ...    ...    ...    ...    ...   \n",
       "329689       0      0      0      0      0      0      0      0      0      0   \n",
       "338474       0      0      0      0      0      1      0      0      0      0   \n",
       "341890       0      0      0      0      0      0      0      0      0      0   \n",
       "343786       0      0      0      0      0      0      0      0      0      0   \n",
       "344325       0      0      0      0      0      0      0      0      0      0   \n",
       "\n",
       "         ...  40956  41006  41389  41491  41511  41917  42949  43299  44042  \\\n",
       "user_id  ...                                                                  \n",
       "864      ...      0      0      0      0      0      0      0      0      0   \n",
       "1661     ...      0      0      0      0      0      0      0      0      0   \n",
       "2119     ...      0      0      0      0      0      0      0      0      0   \n",
       "9362     ...      0      0      0      0      0      0      0      0      0   \n",
       "10099    ...      0      0      0      0      0      0      0      0      0   \n",
       "...      ...    ...    ...    ...    ...    ...    ...    ...    ...    ...   \n",
       "329689   ...      0      0      0      0      0      0      0      0      0   \n",
       "338474   ...      0      0      0      0      0      0      0      0      0   \n",
       "341890   ...      0      0      0      0      0      0      0      0      0   \n",
       "343786   ...      0      0      0      0      0      0      0      0      0   \n",
       "344325   ...      0      0      0      0      0      0      0      0      0   \n",
       "\n",
       "         44218  \n",
       "user_id         \n",
       "864          0  \n",
       "1661         0  \n",
       "2119         0  \n",
       "9362         0  \n",
       "10099        0  \n",
       "...        ...  \n",
       "329689       0  \n",
       "338474       0  \n",
       "341890       0  \n",
       "343786       0  \n",
       "344325       0  \n",
       "\n",
       "[100 rows x 686 columns]"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Pivot title ids creating binary columns of titles that have been recommended\n",
    "simple_cos_mat = simple_recs[['user_id', 'mal_id']].groupby(['user_id', 'mal_id']).size().unstack(fill_value=0).astype(bool).rename_axis(None, axis=1).astype(int)\n",
    "simple_cos_mat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Construct Pairwise Cosine Similarity Matrix\n",
    "simple_cos_mat = cosine_similarity(simple_cos_mat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9697623762376237"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1 - np.mean(list(simple_cos_mat[np.triu_indices(100)]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Comprehensive Recommender Personalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>7</th>\n",
       "      <th>16</th>\n",
       "      <th>24</th>\n",
       "      <th>33</th>\n",
       "      <th>45</th>\n",
       "      <th>47</th>\n",
       "      <th>50</th>\n",
       "      <th>51</th>\n",
       "      <th>64</th>\n",
       "      <th>68</th>\n",
       "      <th>...</th>\n",
       "      <th>42328</th>\n",
       "      <th>42668</th>\n",
       "      <th>42750</th>\n",
       "      <th>42803</th>\n",
       "      <th>42897</th>\n",
       "      <th>42923</th>\n",
       "      <th>42940</th>\n",
       "      <th>43299</th>\n",
       "      <th>43683</th>\n",
       "      <th>43791</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>864</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1661</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2119</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9362</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10099</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>329689</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>338474</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>341890</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>343786</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>344325</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 772 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         7      16     24     33     45     47     50     51     64     68     \\\n",
       "user_id                                                                         \n",
       "864          0      0      0      0      0      0      0      0      0      0   \n",
       "1661         0      0      0      0      0      0      0      0      0      0   \n",
       "2119         0      0      0      0      0      0      0      0      0      0   \n",
       "9362         0      0      0      0      0      0      0      0      0      0   \n",
       "10099        0      0      0      0      0      0      0      0      0      0   \n",
       "...        ...    ...    ...    ...    ...    ...    ...    ...    ...    ...   \n",
       "329689       0      0      0      0      0      0      0      0      0      0   \n",
       "338474       0      0      0      0      0      0      0      0      0      0   \n",
       "341890       0      0      0      0      0      0      0      0      0      0   \n",
       "343786       0      0      0      0      0      0      0      0      0      0   \n",
       "344325       0      0      0      0      0      0      0      0      0      0   \n",
       "\n",
       "         ...  42328  42668  42750  42803  42897  42923  42940  43299  43683  \\\n",
       "user_id  ...                                                                  \n",
       "864      ...      0      0      0      0      0      0      0      0      0   \n",
       "1661     ...      0      0      0      0      0      0      0      0      0   \n",
       "2119     ...      0      0      0      0      0      0      0      0      0   \n",
       "9362     ...      0      0      0      0      0      0      0      0      0   \n",
       "10099    ...      0      0      0      0      0      0      0      0      0   \n",
       "...      ...    ...    ...    ...    ...    ...    ...    ...    ...    ...   \n",
       "329689   ...      0      0      0      0      0      0      0      0      0   \n",
       "338474   ...      0      0      0      0      0      0      0      0      0   \n",
       "341890   ...      0      0      0      0      0      0      0      0      0   \n",
       "343786   ...      0      0      1      0      0      0      0      0      0   \n",
       "344325   ...      0      0      0      0      0      0      0      0      0   \n",
       "\n",
       "         43791  \n",
       "user_id         \n",
       "864          0  \n",
       "1661         0  \n",
       "2119         0  \n",
       "9362         0  \n",
       "10099        0  \n",
       "...        ...  \n",
       "329689       0  \n",
       "338474       0  \n",
       "341890       0  \n",
       "343786       0  \n",
       "344325       0  \n",
       "\n",
       "[100 rows x 772 columns]"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Pivot title ids creating binary columns of titles that have been recommended\n",
    "comp_cos_mat = comp_recs[['user_id', 'mal_id']].groupby(['user_id', 'mal_id']).size().unstack(fill_value=0).astype(bool).rename_axis(None, axis=1).astype(int)\n",
    "comp_cos_mat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Construct Pairwise Cosine Similarity Matrix\n",
    "comp_cos_mat = cosine_similarity(comp_cos_mat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.974"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1 - np.mean(list(comp_cos_mat[np.triu_indices(100)]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results - Personalization Metric (1 - Cosine Similarity)\n",
    "\n",
    " * Naive Recommender: **0.8614059405940594**\n",
    " * Simple Recommender: **0.9697623762376237**\n",
    " * Comprehensive Recommender: **0.974**\n",
    " \n",
    "Based on the average pairwise disimilarity (1 - cosine similarity), the Comprehensive Recommender scored highest with average of 0.974. Based on this metric the Comprehensive Recommender has the highest Personalization in comparison to the other two recommenders."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Content Coverage\n",
    "\n",
    "This is a measure of how wide the coverage of overall content is recommended (amount of overall unique titles). The wider the coverage the broader the range of content is recommended. \n",
    "\n",
    "There are a total of 17,562 available titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f6cae0fd4d0>"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAFLCAYAAAAteMwoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAZLElEQVR4nO3de5CldX3n8fdHRrwQdQAbQmaIgzre1ivpKLtaJgG1BA2DrijG6CyyTmoXo8atUrS21rhJVdBcvOymiBNJdsgaAa/MqqsiqPGyogMiXtBiQIRx0GkVESWK6Hf/OL92mpme6dMz3f0wv36/qk6d5/k9v+7+tmf89I/vec7zpKqQJPXlbkMXIElaeIa7JHXIcJekDhnuktQhw12SOmS4S1KHVowzKcmfAP8RKODLwOnAUcD5wGHAFcALq+r2JPcAzgN+C/g+8Lyqun5v3//+979/rVmzZh9/BUlani6//PLvVdXEbMfmDPckq4CXAY+oqn9NciFwGnAS8KaqOj/J3wFnAOe055ur6sFJTgPeADxvbz9jzZo1bNmyZV6/lCQtd0m+tadj47ZlVgD3SrICuDdwE3A88O52fBNwStte1/Zpx09IkvkWLUnad3OGe1V9G/gr4AZGoX4LcDnww6q6o03bBqxq26uAG9vX3tHmH77r902yIcmWJFumpqb29/eQJM0wZ7gnOZTRavwY4DeAQ4ATZ5k6fR2D2Vbpu13joKo2VtVkVU1OTMzaMpIk7aNx2jJPAb5ZVVNV9XPgvcC/A1a2Ng3AamB7294GHA3Qjt8P+MGCVi1J2qtxwv0G4Lgk92698xOArwEfB57T5qwHLmrbm9s+7fil5dXJJGlJjdNzv4zRG6NXMDoN8m7ARuDVwCuTbGXUUz+3fcm5wOFt/JXAWYtQtyRpL3JXWFRPTk6Wp0JK0vwkubyqJmc75idUJalDhrskdWisyw9I0kJYc9YHhy5hUV1/9jOGLuFXXLlLUocMd0nqkOEuSR0y3CWpQ4a7JHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUoTnDPclDk1w54/GjJK9IcliSi5Nc054PbfOT5K1Jtia5Ksmxi/9rSJJmGucG2d+oqsdW1WOB3wJuA97H6MbXl1TVWuASdt4I+0RgbXtsAM5ZjMIlSXs237bMCcC1VfUtYB2wqY1vAk5p2+uA82rkc8DKJEctSLWSpLHMN9xPA97Zto+sqpsA2vMRbXwVcOOMr9nWxu4kyYYkW5JsmZqammcZkqS9GTvckxwMnAy8a66ps4zVbgNVG6tqsqomJyYmxi1DkjSG+azcTwSuqKrvtv3vTrdb2vOONr4NOHrG160Gtu9voZKk8c0n3J/PzpYMwGZgfdteD1w0Y/xF7ayZ44Bbpts3kqSlsWKcSUnuDTwV+KMZw2cDFyY5A7gBOLWNfwg4CdjK6Mya0xesWglYc9YHhy5h0Vx/9jOGLkGdGCvcq+o24PBdxr7P6OyZXecWcOaCVCdJ2id+QlWSOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShwx3SeqQ4S5JHTLcJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUofGCvckK5O8O8nXk1yd5N8mOSzJxUmuac+HtrlJ8tYkW5NcleTYxf0VJEm7Gnfl/hbgw1X1MOAxwNXAWcAlVbUWuKTtA5wIrG2PDcA5C1qxJGlOc4Z7kvsCTwbOBaiq26vqh8A6YFObtgk4pW2vA86rkc8BK5McteCVS5L2aJyV+wOBKeAfk3wxyduTHAIcWVU3AbTnI9r8VcCNM75+Wxu7kyQbkmxJsmVqamq/fglJ0p2NE+4rgGOBc6rqccBP2NmCmU1mGavdBqo2VtVkVU1OTEyMVawkaTzjhPs2YFtVXdb2380o7L873W5pzztmzD96xtevBrYvTLmSpHHMGe5V9R3gxiQPbUMnAF8DNgPr29h64KK2vRl4UTtr5jjglun2jSRpaawYc94fA+9IcjBwHXA6oz8MFyY5A7gBOLXN/RBwErAVuK3NlSQtobHCvaquBCZnOXTCLHMLOHM/65Ik7Qc/oSpJHTLcJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nqkOEuSR0y3CWpQ4a7JHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUNjhXuS65N8OcmVSba0scOSXJzkmvZ8aBtPkrcm2ZrkqiTHLuYvIEna3XxW7r9XVY+tqul7qZ4FXFJVa4FL2j7AicDa9tgAnLNQxUqSxrM/bZl1wKa2vQk4Zcb4eTXyOWBlkqP24+dIkuZp3HAv4KNJLk+yoY0dWVU3AbTnI9r4KuDGGV+7rY3dSZINSbYk2TI1NbVv1UuSZrVizHlPrKrtSY4ALk7y9b3MzSxjtdtA1UZgI8Dk5ORuxyVJ+26slXtVbW/PO4D3AY8HvjvdbmnPO9r0bcDRM758NbB9oQqWJM1tznBPckiS+0xvA08DvgJsBta3aeuBi9r2ZuBF7ayZ44Bbpts3kqSlMU5b5kjgfUmm5/9zVX04yReAC5OcAdwAnNrmfwg4CdgK3AacvuBVS5L2as5wr6rrgMfMMv594IRZxgs4c0GqkyTtEz+hKkkdMtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShwx3SeqQ4S5JHTLcJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nqkOEuSR0aO9yTHJTki0k+0PaPSXJZkmuSXJDk4DZ+j7a/tR1fszilS5L2ZD4r95cDV8/YfwPwpqpaC9wMnNHGzwBurqoHA29q8yRJS2iscE+yGngG8Pa2H+B44N1tyibglLa9ru3Tjp/Q5kuSlsi4K/c3A68Cftn2Dwd+WFV3tP1twKq2vQq4EaAdv6XNlyQtkTnDPckzgR1VdfnM4Vmm1hjHZn7fDUm2JNkyNTU1VrGSpPGMs3J/InBykuuB8xm1Y94MrEyyos1ZDWxv29uAowHa8fsBP9j1m1bVxqqarKrJiYmJ/folJEl3Nme4V9Vrqmp1Va0BTgMuraoXAB8HntOmrQcuatub2z7t+KVVtdvKXZK0ePbnPPdXA69MspVRT/3cNn4ucHgbfyVw1v6VKEmarxVzT9mpqj4BfKJtXwc8fpY5PwVOXYDaJEn7yE+oSlKHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShwx3SeqQ4S5JHTLcJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nq0JzhnuSeST6f5EtJvprk9W38mCSXJbkmyQVJDm7j92j7W9vxNYv7K0iSdjXOyv1nwPFV9RjgscDTkxwHvAF4U1WtBW4GzmjzzwBurqoHA29q8yRJS2jOcK+RH7fdu7dHAccD727jm4BT2va6tk87fkKSLFjFkqQ5jdVzT3JQkiuBHcDFwLXAD6vqjjZlG7Cqba8CbgRox28BDp/le25IsiXJlqmpqf37LSRJdzJWuFfVL6rqscBq4PHAw2eb1p5nW6XXbgNVG6tqsqomJyYmxq1XkjSGeZ0tU1U/BD4BHAesTLKiHVoNbG/b24CjAdrx+wE/WIhiJUnjGedsmYkkK9v2vYCnAFcDHwee06atBy5q25vbPu34pVW128pdkrR4Vsw9haOATUkOYvTH4MKq+kCSrwHnJ/lz4IvAuW3+ucA/JdnKaMV+2iLULUnaiznDvaquAh43y/h1jPrvu47/FDh1QaqTJO0TP6EqSR0y3CWpQ4a7JHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShwx3SeqQ4S5JHTLcJalD49wg++gkH09ydZKvJnl5Gz8sycVJrmnPh7bxJHlrkq1Jrkpy7GL/EpKkOxtn5X4H8F+q6uHAccCZSR4BnAVcUlVrgUvaPsCJwNr22ACcs+BVS5L2as5wr6qbquqKtn0rcDWwClgHbGrTNgGntO11wHk18jlgZZKjFrxySdIezavnnmQN8DjgMuDIqroJRn8AgCPatFXAjTO+bFsb2/V7bUiyJcmWqamp+VcuSdqjscM9ya8B7wFeUVU/2tvUWcZqt4GqjVU1WVWTExMT45YhSRrDWOGe5O6Mgv0dVfXeNvzd6XZLe97RxrcBR8/48tXA9oUpV5I0jhVzTUgS4Fzg6qr6mxmHNgPrgbPb80Uzxl+a5HzgCcAt0+2bu4o1Z31w6BIW1fVnP2PoEiQNbM5wB54IvBD4cpIr29hrGYX6hUnOAG4ATm3HPgScBGwFbgNOX9CKJUlzmjPcq+rTzN5HBzhhlvkFnLmfdUmS9oOfUJWkDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShwx3SeqQ4S5JHTLcJalDhrskdchwl6QOGe6S1KE5wz3JPyTZkeQrM8YOS3Jxkmva86FtPEnemmRrkquSHLuYxUuSZjfOyv1/AU/fZews4JKqWgtc0vYBTgTWtscG4JyFKVOSNB9zhntV/Qvwg12G1wGb2vYm4JQZ4+fVyOeAlUmOWqhiJUnj2dee+5FVdRNAez6ija8Cbpwxb1sb202SDUm2JNkyNTW1j2VIkmaz0G+oZpaxmm1iVW2sqsmqmpyYmFjgMiRpedvXcP/udLulPe9o49uAo2fMWw1s3/fyJEn7Yl/DfTOwvm2vBy6aMf6idtbMccAt0+0bSdLSWTHXhCTvBH4XuH+SbcDrgLOBC5OcAdwAnNqmfwg4CdgK3Aacvgg1S5LmMGe4V9Xz93DohFnmFnDm/hYlSdo/fkJVkjpkuEtShwx3SeqQ4S5JHTLcJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nqkOEuSR0y3CWpQ4a7JHXIcJekDhnuktQhw12SOmS4S1KHFiXckzw9yTeSbE1y1mL8DEnSni14uCc5CPhb4ETgEcDzkzxioX+OJGnPFmPl/nhga1VdV1W3A+cD6xbh50iS9mDFInzPVcCNM/a3AU/YdVKSDcCGtvvjJN9YhFruKu4PfG+pfljesFQ/aVnwtTuw9f76PWBPBxYj3DPLWO02ULUR2LgIP/8uJ8mWqpocug7Nn6/dgW05v36L0ZbZBhw9Y381sH0Rfo4kaQ8WI9y/AKxNckySg4HTgM2L8HMkSXuw4G2ZqrojyUuBjwAHAf9QVV9d6J9zgFkW7adO+dod2Jbt65eq3drhkqQDnJ9QlaQOGe6S1CHDXZI6ZLhLUocM90WS5CFJLknylbb/6CT/dei6NJ4kT0pyetueSHLM0DVpfL5+hvti+nvgNcDPAarqKkbn/OsuLsnrgFczev0A7g787+Eq0nz4+o0Y7ovn3lX1+V3G7hikEs3Xs4CTgZ8AVNV24D6DVqT58PXDcF9M30vyINp1dZI8B7hp2JI0pttr9AGQ6dfukIHr0fz4+rE4Fw7TyJmMPh33sCTfBr4JvGDYkjSmC5O8DViZ5CXAixm12XRg8PXDT6gumiQHVdUv2qrhblV169A1aXxJngo8jdFVTj9SVRcPXJLmwdfPcF80SW4APgxcAFxa/g8tLYkkfwK8q6q2DV3LkOy5L56HAh9j1J75ZpL/meRJA9ekvUhya5IfzfK4NcmPhq5PY7sv8JEkn0pyZpIjhy5oCK7cl0CSQ4G3AC+oqoOGrkdaDpI8Gnge8O+BbVX1lIFLWlK+obqIkvwOo39cJzK6zv1zh61I40pyLPAkRmdcfLqqvjhwSZq/HcB3gO8DRwxcy5Jz5b5IknwTuBK4ENhcVT8ZuCSNKcl/A04F3tuGTmHUw/3z4arSuJL8J0aLqgng3cAFVfW1Yataeob7Ikly36qyT3sASnI18Liq+mnbvxdwRVU9fNjKNI4kZwPnV9WVQ9cyJNsyCyzJq6rqjcCfJ7vfK7yqXrb0VWmergfuCfy07d8DuHawajSWGQuqN7b9w2Yer6ofDFLYQAz3hXd1e7580Cq0P34GfDXJxYx67k8FPp3kreAf6Luwfwaeyej/e8XoHPdpBTxwiKKGYltG2kWS9Xs7XlWblqoWaV8Z7oskyQSjK9M9gtF/4gNQVccPVpS0DCR5InBlVf0kyR8CxwJvrqobBi5tSfkhpsXzDkYtmmOA1zPq435hyII0niTPTPLFJD/wQ0wHpHOA25I8BngV8C3gn4YtaekZ7ovn8Ko6F/h5VX2yql4MHDd0URrLm4H1jF7D+1bVfarqvkMXpbHd0S73sQ54S1W9hWV4yV/fUF08P2/PNyV5BrAdWD1gPRrfjcBXvB7QAevWJK8B/hB4cpKDGN2wY1mx575IkjwT+BRwNPA/GF3v4vVVtXnQwjSnJL8N/BnwSUZnzgBQVX8zWFEaW5JfB/4A+EJVfSrJbwK/W1XnDVzakjLcpV0k+SjwY+DLwC+nx6vq9YMVJc2TbZkF1j66vidVVX+2ZMVoXx1WVU8bugjtmyTPBt7A6HoyaY9abu+b+IbqwvvJLA+AMxidGqm7vo8lMdwPXG8ETq6q+y3nN8RtyyyiJPcBXs4o2C8E/rqqdgxbleaS5FbgEEb99p+zTFd+B6okn6mqJw5dx9BsyyyCdk2LVzK6Z+om4NiqunnYqjSuqlp2p811ZkuSC4D3c+c3xN+75y/pj+G+wJL8JfBsRjfHflRV/XjgkjSmJA+rqq+3a7nvpqquWOqatE/uC9zG6B6q04qdl3BeFmzLLLAkv2S0WriD0T+oXx3C/7S/S0uysao2JPn4jOFfvYZeOkIHEsNdapI8Hrihqr7T9tczukXb9cCfLrdLxh6okjyE0SUIjqyqR7bb7Z283G624tky0k5/B9wOkOTJwF8wes/kFkZtNh0Y/h54De1T4lV1FXDaoBUNwJ67tNNBM1bnzwM2VtV7gPckWdZ39TnA3LuqPr/LzXLuGKqYobhyl3Y6KMn0gucE4NIZx1wIHTi+l+RBtPdLkjwHuGnYkpae/2Clnd4JfDLJ94B/ZXRtIJI8mFFrRgeGMxm10R6W5NvANxldRGxZ8Q1VaYYkxwFHAR+tqp+0sYcAv+apkAeWJIcAd6uqW4euZQiGu6SuJLkHo7Oc1jCjO1FV/32omoZgW0ZSby5i1Ea7nBmfUF1uXLlL6kqSr1TVI4euY2ieLSOpN59N8qihixiaK3dJXUjyZUanP64A1gLXMWrLTF/649EDlrfkDHdJXUjygL0dr6pvLVUtdwW2ZSR1oaq+Nf1gdO/i49v2bSzDrHPlLqkrSV4HTAIPraqHJPkN4F3L7QYey+6vmaTuPQs4mXaLy6raDiy7G7AY7pJ6c3uNWhLT15Y5ZOB6BmG4S+rNhUneBqxM8hLgY4wuA7ys2HOX1J0kT2V0m70AH6mqiwcuackZ7pK6keQgRmH+lKFrGZptGUndqKpfALclud/QtQzNC4dJ6s1PgS8nuZh2xgxAVb1suJKWnuEuqTcfbI9lzZ67pO4kORh4GKPTIb9RVbcPXNKSM9wldSXJScDbgGsZnS1zDPBHVfV/By1siRnukrqS5OvAM6tqa9t/EPDBqnrYsJUtLc+WkdSbHdPB3lwH7BiqmKG4cpfUlSTnAA8ALmTUcz8V+AbwGYCqeu9w1S0dw11SV5L8414OV1W9eMmKGZDhLkkd8jx3SV1Jcgzwx8AaZmRcVZ08VE1DMNwl9eb9wLnA/wF+OXAtg7EtI6krSS6rqicMXcfQDHdJXUnyB8Ba4KPAz6bHq+qKwYoagG0ZSb15FPBC4Hh2tmWq7S8brtwldaV9QvXRy/F6MjP5CVVJvfkSsHLoIoZmW0ZSb44Evp7kC9y55+6pkJJ0AHvd0AXcFdhzl9SdJEcCv912P19Vy+7CYfbcJXUlyXOBzzO6YNhzgcuSPGfYqpaeK3dJXUnyJeCp06v1JBPAx6rqMcNWtrRcuUvqzd12acN8n2WYdb6hKqk3H07yEeCdbf95wIcGrGcQtmUkdSHJg4Ejq+ozSZ4NPInRPVRvBt5RVdcOWuASM9wldSHJB4DXVtVVu4xPAq+rqt8fprJhLLs+lKRurdk12AGqaguja7svK4a7pF7ccy/H7rVkVdxFGO6SevGFJC/ZdTDJGcDlA9QzKHvukrrQPpX6PuB2dob5JHAw8Kyq+s5QtQ3BcJfUlSS/Bzyy7X61qi4dsp6hGO6S1CF77pLUIcNdkjrk5Qe0rCQ5HLik7f468Atgqu0/frnfmk39sOeuZSvJnwI/rqq/GroWaaHZlpGAJH+R5MwZ+29I8p+TPCXJx5O8P8nXkvxtkrQ5Jyb5f0muSHJBkkOG+w2kOzPcpZG3A/8BIMlBjG70MH1VwScArwAeBTwcWJfkCOAs4ISqOha4Cnj5Etcs7ZE9dwmoqmuT3JrkUcADGN2a7ea2SP9cVV0PkOR8RlcbBHgE8Nk252Dg00teuLQHhru007mMVu9rgLfNGN/1jalidCnZD1fVC5ekMmmebMtIO70H+H3gscDHZowfl+Q3W7vmuYxW6J8FfifJAwGSHJJk7VIXLO2JK3epqaqfJvkX4DtV9csZhz4L/DXwb4BPAJurqtoFqS5IcnCb91rgmqWsWdoTT4WUmiR3A64ETqmq69rYU4CXVtUpgxYnzZNtGQlob6Rey6iPft3Q9Uj7y5W7JHXIlbskdchwl6QOGe6S1CHDXZI6ZLhLUof+P7GIRALPLcTDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cov_results = {'Type': ['Naive', 'Simple', 'Comprehensive'], \n",
    "               'Coverage': [len(naive_recs['mal_id'].unique()), len(simple_recs['mal_id'].unique()), len(comp_recs['mal_id'].unique())]}\n",
    "\n",
    "pd.DataFrame(cov_results).plot(kind='bar', x='Type', y='Coverage', legend=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Type</th>\n",
       "      <th>Coverage</th>\n",
       "      <th>Percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Naive</td>\n",
       "      <td>190</td>\n",
       "      <td>1.081881</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Simple</td>\n",
       "      <td>686</td>\n",
       "      <td>3.906161</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Comprehensive</td>\n",
       "      <td>772</td>\n",
       "      <td>4.395855</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Type  Coverage   Percent\n",
       "0          Naive       190  1.081881\n",
       "1         Simple       686  3.906161\n",
       "2  Comprehensive       772  4.395855"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cov_df = pd.DataFrame(cov_results)\n",
    "cov_df['Percent'] = (cov_df['Coverage'] / 17562) * 100\n",
    "cov_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results - Content Coverage\n",
    "\n",
    "As shown with the graph and table above, the comprehensive recommender has the highest content coverage with 772 unique titles (4.4% of all titles) being recommended across the sample of 100 users. Suprisingly, this was expected to be lower than the uniformly random naive recommender but coverage may change with a larger user sample size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intra-List Similarity Metric\n",
    "\n",
    "This measures the similarity between title recommendations for each user. This is a metric for measuring how related or relevant the recommendations are. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This calculates the mean cosine similarity across recommendations by user\n",
    "def cs_mean(user_id, recs):\n",
    "    # get the k recommendations by user id and features\n",
    "    features = anime_embeddings[anime_embeddings.index.isin(recs[recs['user_id'] == user_id]['mal_id'])]\n",
    "    # construct cosine similarity matrix. This is a k x k matrix\n",
    "    cs_mat = cosine_similarity(features)\n",
    "    # get upper triangle pairwise measure and take mean\n",
    "    triu_mean = np.mean(list(cs_mat[np.triu_indices(cs_mat.shape[0])]))\n",
    "    \n",
    "    return triu_mean\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [],
   "source": [
    "# iterate through all 100 sample of users and calculate intra-list metric\n",
    "naive_int = []\n",
    "simple_int = []\n",
    "comp_int = []\n",
    "\n",
    "for user in eval_users:\n",
    "    naive_int.append(cs_mean(user, naive_recs))\n",
    "    simple_int.append(cs_mean(user, simple_recs))\n",
    "    comp_int.append(cs_mean(user, comp_recs))\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Type</th>\n",
       "      <th>Mean Intra-Similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Naive</td>\n",
       "      <td>0.725356</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Simple</td>\n",
       "      <td>0.854268</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Comprehensive</td>\n",
       "      <td>0.923819</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Type  Mean Intra-Similarity\n",
       "0          Naive               0.725356\n",
       "1         Simple               0.854268\n",
       "2  Comprehensive               0.923819"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_res = {'Type': ['Naive', 'Simple', 'Comprehensive'], \n",
    "           'Mean Intra-Similarity': [np.mean(naive_int), np.mean(simple_int), np.mean(comp_int)]}\n",
    "\n",
    "int_df = pd.DataFrame(int_res)\n",
    "int_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results - Intra-List Similarity\n",
    "\n",
    "Based on the average intra-list similarity metric by user, the comprehensive recommender had the highest average intra-list similarity measurement of 0.92. This means the recommended titles from the comprehensive recommender system gives more relevant and related content recommendations."
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}