Movie_review_imdb_scrapping.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "#importing Beautiful soup for scrapping the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependencies\n",
    "\n",
    "`BeautifulSoup` :    pip install BeautifulSoup\n",
    "oup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd  # Importing pandas library for data manipulation\n",
    "import numpy as np  # Importing numpy library for numerical operations\n",
    "import itertools  # Importing itertools for creating efficient looping mechanisms to fetch more data at once\n",
    "import re  # Importing re for regular expression operations\n",
    "import random  # Importing random for generating random numbers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating BS4 Functions for scrapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  Movies are cateogirsed into seven and each category is processed by indivitual team members.\n",
    "- category 1: [1940 to 1980 200 movie listing with rating=10000](https://www.imdb.com/search/title/?title_type=feature&release_date=1940-01-01,1980-12-31&num_votes=10000,&count=200) \n",
    "- category 2: [2020 to 2021 200 movie listing with rating=20000](https://www.imdb.com/search/title/?title_type=feature&release_date=2020-01-01,2021-12-31&num_votes=20000,&count=200) \n",
    "- category 3: [2000 to 2021 200 movie listing with rating=60000](https://www.imdb.com/search/title/?title_type=feature&release_date=2000-01-01,2021-12-31&num_votes=60000,&count=200) \n",
    "- category 4: [1940 to 1980 200 movie listing with rating=10000](https://www.imdb.com/search/title/?title_type=feature&release_date=2005-01-01,2015-12-31&num_votes=30000,&count=200) \n",
    "- category 5: [1980 to 2019 200 movie listing with rating=500000](https://www.imdb.com/search/title/?title_type=feature&release_date=1980-01-01,2019-12-31&num_votes=500000,&count=200) \n",
    "- category 6: [1980 to 2019 200 movie listing with rating=80000](https://www.imdb.com/search/title/?title_type=tv_series&release_date=1980-01-01,2019-12-31&num_votes=80000,&count=200) \n",
    "- category 7: [2005 to 2010 200 movie listing with rating=5000](https://www.imdb.com/search/title/?title_type=feature&release_date=2005-01-01,2010-12-31&num_votes=5000,&count=200) \n",
    "\n",
    "#Sample files are put into Data_scrapped folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# URL for fetching movie data\n",
    "url = \"https://www.imdb.com/search/title/?title_type=feature&release_date=2009-01-01,2011-12-31&num_votes=2000,&count=200\"  # 2000 - 2021 6000votes filter 200 titles (IMDb does not allow filtering >200 titles at once)\n",
    "\n",
    "def getSoup(url):\n",
    "    \"\"\"\n",
    "    Utility function to fetch the HTML content of the provided URL and parse it using BeautifulSoup.\n",
    "    \"\"\"\n",
    "    response = requests.get(url)\n",
    "    soup = BeautifulSoup(response.text, 'html.parser')\n",
    "    return soup\n",
    "\n",
    "def getReviews(soup):\n",
    "    '''\n",
    "    Function to extract all reviews, including positive and negative.\n",
    "    '''\n",
    "    # Get a list of user ratings\n",
    "    user_review_ratings = [tag.previous_element for tag in \n",
    "                           soup.find_all('span', attrs={'class': 'point-scale'})]  # Can search div by inspecting elements\n",
    "\n",
    "    # Get the review tags\n",
    "    user_review_list = soup.find_all('a', attrs={'class':'title'})\n",
    "    ans = []\n",
    "    for i in range(5):\n",
    "        ans.append(user_review_list[random.randint(0, len(user_review_list) -1)])\n",
    "    links = [\"https://www.imdb.com\" + tag['href'] for tag in ans]\n",
    "    return links\n",
    "\n",
    "def getReviewText(review_url):\n",
    "    '''Returns the user review text given the review URL.'''\n",
    "    # Get the review URL's soup\n",
    "    soup = getSoup(review_url)\n",
    "    # Find div tags with class text show-more__control\n",
    "    tag = soup.find('div', attrs={'class': 'text show-more__control'})\n",
    "    return tag.getText()\n",
    "\n",
    "def getMovieTitle(review_url):\n",
    "    '''Returns the movie title from the review URL.'''\n",
    "    # Get the review URL's soup\n",
    "    soup = getSoup(review_url)\n",
    "    # Find h1 tag\n",
    "    tag = soup.find('h1')\n",
    "    return list(tag.children)[1].getText()\n",
    "\n",
    "def getNounChunks(user_review):\n",
    "    '''Extracts noun chunks from the provided user review.'''\n",
    "    # Create the doc object\n",
    "    doc = nlp(user_review)\n",
    "    # Get a list of noun_chunks\n",
    "    noun_chunks = list(doc.noun_chunks)\n",
    "    # Convert noun_chunks from span objects to strings\n",
    "    noun_chunks_strlist = [chunk.text for chunk in noun_chunks]\n",
    "    return noun_chunks_strlist\n",
    "\n",
    "# Get the soup for the URL\n",
    "movies_soup = getSoup(url)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filtering the movie tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are a total of 200 movie titles\n",
      "Displaying 10 titles\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['/title/tt1262416/', '/title/tt0491152/']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find all <a> tags in the movies_soup\n",
    "movie_tags = movies_soup.find_all('a', attrs={'class': None})\n",
    "\n",
    "# Filter the <a> tags to get just the titles\n",
    "movie_tags = [tag.attrs['href'] for tag in movie_tags \n",
    "              if tag.attrs['href'].startswith('/title') and tag.attrs['href'].endswith('/')]\n",
    "\n",
    "# Remove duplicate links\n",
    "movie_tags = list(dict.fromkeys(movie_tags))\n",
    "\n",
    "# Print the total number of movie titles\n",
    "print(\"There are a total of \" + str(len(movie_tags)) + \" movie titles\")\n",
    "print(\"Displaying 10 titles\")\n",
    "print(movie_tags[:2])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filtering the movie URL's"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are a total of 200 movie user reviews\n",
      "Displaying 20 user reviews links\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['https://www.imdb.com/title/tt1262416/reviews',\n",
       " 'https://www.imdb.com/title/tt0491152/reviews']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "base_url = \"https://www.imdb.com\"\n",
    "movie_links = [base_url + tag + 'reviews' for tag in movie_tags][:20]\n",
    "print(\"There are a total of \" + str(len(movie_links)) + \" movie user reviews\")\n",
    "\"print(\\\"Displaying 20 user reviews links\\\")\\n\",\n",
    "\n",
    "movie_links[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "movie_soups = [getSoup(link) for link in movie_links]\n",
    "\n",
    "# get all movie review links from the 200 listing\n",
    "movie_review_list = [getReviews(movie_soup) for movie_soup in movie_soups]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000\n",
      "There are a total of 1000 individual movie reviews\n",
      "Displaying 10 reviews\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['https://www.imdb.com/review/rw2503689/',\n",
       " 'https://www.imdb.com/review/rw6051819/',\n",
       " 'https://www.imdb.com/review/rw2416020/',\n",
       " 'https://www.imdb.com/review/rw5243143/',\n",
       " 'https://www.imdb.com/review/rw5243143/']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Checking how many movie review were able to filter.\n",
    "movie_review_list = list(itertools.chain(*movie_review_list))\n",
    "print(len(movie_review_list))\n",
    "\n",
    "print(\"There are a total of \" + str(len(movie_review_list)) + \" individual movie reviews\")\n",
    "print(\"Displaying 10 reviews\")\n",
    "movie_review_list[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Converting into the Pandas Data Frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get review texts for each movie URL in movie_review_list\n",
    "review_texts = [getReviewText(url) for url in movie_review_list]\n",
    "\n",
    "# Get movie names from the review links\n",
    "movie_titles = [getMovieTitle(url) for url in movie_review_list]\n",
    "\n",
    "# Construct a dataframe with user reviews and movie titles\n",
    "df = pd.DataFrame({\n",
    "    'movie_title': movie_titles,\n",
    "    'user_review': review_texts\n",
    "})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>That wasn't really I question for my review. B...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Despite there being no Scream movies for a dec...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>The \"Scream\" film franchise carved a memorable...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Fifteen years have passed since the original W...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fifteen years have passed since the original W...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         user_review\n",
       "0  That wasn't really I question for my review. B...\n",
       "1  Despite there being no Scream movies for a dec...\n",
       "2  The \"Scream\" film franchise carved a memorable...\n",
       "3  Fifteen years have passed since the original W...\n",
       "4  Fifteen years have passed since the original W..."
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(5) #displaying the resultant data frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The data frame need to remove index and filter the limit review length by 250 words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_list = [m for m in df['user_review']]\n",
    "#text_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_review</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>That wasn't really I question for my review. B...</td>\n",
       "      <td>208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Despite there being no Scream movies for a dec...</td>\n",
       "      <td>337</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>The \"Scream\" film franchise carved a memorable...</td>\n",
       "      <td>957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Fifteen years have passed since the original W...</td>\n",
       "      <td>352</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fifteen years have passed since the original W...</td>\n",
       "      <td>352</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>I watched Film 2009 with Jonathan Ross, and re...</td>\n",
       "      <td>316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996</th>\n",
       "      <td>Kind of a tame CGI fest Hebrew Exorcist regard...</td>\n",
       "      <td>506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>This is an astonishing story about a vengeful ...</td>\n",
       "      <td>430</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>998</th>\n",
       "      <td>Well it's now a few years since The Unborn was...</td>\n",
       "      <td>248</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999</th>\n",
       "      <td>The Unborn has come in for some criticism whic...</td>\n",
       "      <td>300</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           user_review  length\n",
       "0    That wasn't really I question for my review. B...     208\n",
       "1    Despite there being no Scream movies for a dec...     337\n",
       "2    The \"Scream\" film franchise carved a memorable...     957\n",
       "3    Fifteen years have passed since the original W...     352\n",
       "4    Fifteen years have passed since the original W...     352\n",
       "..                                                 ...     ...\n",
       "995  I watched Film 2009 with Jonathan Ross, and re...     316\n",
       "996  Kind of a tame CGI fest Hebrew Exorcist regard...     506\n",
       "997  This is an astonishing story about a vengeful ...     430\n",
       "998  Well it's now a few years since The Unborn was...     248\n",
       "999  The Unborn has come in for some criticism whic...     300\n",
       "\n",
       "[1000 rows x 2 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#calculating the length of the text\n",
    "text_list_length = [len(m.split()) for m in text_list]     \n",
    "df['length'] = text_list_length\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_review</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>That wasn't really I question for my review. B...</td>\n",
       "      <td>208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Rachel (Ginnifer Goodwin) and Darcy (Kate Huds...</td>\n",
       "      <td>169</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Overall I can see why this movie got bad revie...</td>\n",
       "      <td>135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Surprisingly not that bad. Very predictable an...</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>23 December 2016. Credit can be given to the p...</td>\n",
       "      <td>135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>984</th>\n",
       "      <td>Another Film that plays the Same Scenes again ...</td>\n",
       "      <td>154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>985</th>\n",
       "      <td>I would love to hate Richard Gere. He's too go...</td>\n",
       "      <td>243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>987</th>\n",
       "      <td>An Akita Inu puppy, transplanted from a Japane...</td>\n",
       "      <td>232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>989</th>\n",
       "      <td>Based on the true story of a faithful Akita, t...</td>\n",
       "      <td>198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>998</th>\n",
       "      <td>Well it's now a few years since The Unborn was...</td>\n",
       "      <td>248</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>455 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           user_review  length\n",
       "0    That wasn't really I question for my review. B...     208\n",
       "5    Rachel (Ginnifer Goodwin) and Darcy (Kate Huds...     169\n",
       "7    Overall I can see why this movie got bad revie...     135\n",
       "8    Surprisingly not that bad. Very predictable an...      11\n",
       "9    23 December 2016. Credit can be given to the p...     135\n",
       "..                                                 ...     ...\n",
       "984  Another Film that plays the Same Scenes again ...     154\n",
       "985  I would love to hate Richard Gere. He's too go...     243\n",
       "987  An Akita Inu puppy, transplanted from a Japane...     232\n",
       "989  Based on the true story of a faithful Akita, t...     198\n",
       "998  Well it's now a few years since The Unborn was...     248\n",
       "\n",
       "[455 rows x 2 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df[df['length'] < 250]  #limiting the df by 250 in length\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sanja\\anaconda3\\lib\\site-packages\\pandas\\core\\frame.py:4163: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  return super().drop(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>That wasn't really I question for my review. B...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Rachel (Ginnifer Goodwin) and Darcy (Kate Huds...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Overall I can see why this movie got bad revie...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Surprisingly not that bad. Very predictable an...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>23 December 2016. Credit can be given to the p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>984</th>\n",
       "      <td>Another Film that plays the Same Scenes again ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>985</th>\n",
       "      <td>I would love to hate Richard Gere. He's too go...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>987</th>\n",
       "      <td>An Akita Inu puppy, transplanted from a Japane...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>989</th>\n",
       "      <td>Based on the true story of a faithful Akita, t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>998</th>\n",
       "      <td>Well it's now a few years since The Unborn was...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>455 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           user_review\n",
       "0    That wasn't really I question for my review. B...\n",
       "5    Rachel (Ginnifer Goodwin) and Darcy (Kate Huds...\n",
       "7    Overall I can see why this movie got bad revie...\n",
       "8    Surprisingly not that bad. Very predictable an...\n",
       "9    23 December 2016. Credit can be given to the p...\n",
       "..                                                 ...\n",
       "984  Another Film that plays the Same Scenes again ...\n",
       "985  I would love to hate Richard Gere. He's too go...\n",
       "987  An Akita Inu puppy, transplanted from a Japane...\n",
       "989  Based on the true story of a faithful Akita, t...\n",
       "998  Well it's now a few years since The Unborn was...\n",
       "\n",
       "[455 rows x 1 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.drop('length', axis=1, inplace=True)\n",
    "df\n",
    "#dropping the len row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#converting only reviews to CSV & removing the index\n",
    "df.to_csv('data_scrapped/data.csv', index=False) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting the csv file to the indivitual text files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "with open(\"data_scrapped/data.csv\", \"r\") as f:\n",
    "        reader = csv.reader(f)\n",
    "        rownumber = 2639\n",
    "        for row in reader:\n",
    "             g=open(str(rownumber)+\".txt\",\"w\")\n",
    "             g.write(str(row))\n",
    "             rownumber = rownumber + 1\n",
    "             g.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <------------------------------------------------------EOL----------------------------------------------------------------->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final Dataset\n",
    "\n",
    "Here is the Link to Final Dataset: [Drive Link](https://drive.google.com/file/d/1sTNAeuy-99Hao0V5AOVznLXyDJC2zuFn/view?usp=sharing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "23784c1f7cb08f751fc8275aaeeb793366452f178ab646d6f5c554e80b94c083"
  },
  "kernelspec": {
   "display_name": "Python 3.8.5 64-bit ('base': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}