Project_Data_Gathering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Gathering Notebook\n",
    "\n",
    "This notebook consists of code that will:\n",
    "\n",
    "1. Search Yelp based on a search phrase and location.\n",
    "2. Scrape (usng Beautiful Soup) Yelp search results to store information for each business found.\n",
    "3. Scrape (usng Beautiful Soup) reviews for each business found in step 2.\n",
    "4. Store the cleaned results into two csv files for use in other notebooks: `csv_files/business_info.csv` and `csv_files/reviews.csv`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import json\n",
    "import requests\n",
    "import re\n",
    "from bs4 import BeautifulSoup\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "#Need to use a delay between page scrapes in order to limit getting blocked by Yelp\n",
    "from time import sleep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scrape Restaurant Information\n",
    "\n",
    "Get a list of businesses and their info:\n",
    "- Name\n",
    "- Address\n",
    "- Star Rating\n",
    "- Price Range\n",
    "- Number of Reviews\n",
    "- URL for business Yelp review page\n",
    "- URL for image used as business icon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Enter city and food type search terms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "#ENTER SEARCH TERMS BELOW:\n",
    "cuisine_type = \"Italian\"\n",
    "location = \"Albany, NY\"\n",
    "\n",
    "#Generate URL based on search terms\n",
    "base_url = \"https://www.yelp.com\"\n",
    "search_url = f\"{base_url}/search?find_desc={cuisine_type}&find_loc={location}\"\n",
    "\n",
    "#Or manually set search_url by copying directly from Yelp Page if desired\n",
    "#search_url = \"https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set class names that are used on the Yelp business search results page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "star_container_class = \"lemon--div__373c0__1mboc attribute__373c0__1hPI_ display--inline-block__373c0__2de_K u-space-r1 border-color--default__373c0__2oFDT\"\n",
    "price_range_class = \"lemon--span__373c0__3997G text__373c0__2pB8f priceRange__373c0__2DY87 text-color--normal__373c0__K_MKN text-align--left__373c0__2pnx_ text-bullet--after__373c0__1ZHaA\"\n",
    "review_count_class = \"lemon--span__373c0__3997G text__373c0__2pB8f reviewCount__373c0__2r4xT text-color--mid__373c0__3G312 text-align--left__373c0__2pnx_\"\n",
    "next_page_class = \"lemon--a__373c0__IEZFH link__373c0__29943 next-link navigation-button__373c0__1D3Ug link-color--blue-dark__373c0__1mhJo link-size--default__373c0__1skgq\"\n",
    "search_result_class = \"lemon--div__373c0__1mboc searchResult__373c0__1yggB border-color--default__373c0__2oFDT\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scrape data from each results page\n",
    "\n",
    "This loop will run continuously until there is no longer a \"next page\" button found on the bottom of the screen.\n",
    "\n",
    "In this loop:\n",
    "\n",
    "1. Request html page and lead into Beautiful Soup object.\n",
    "2. Loop through each result found on the page.\n",
    "3. For each result store desired business information: url, name, address, categories, star rating, price range, number of reviews.\n",
    "4. Find \"next page\" button and store url for next pass through the while loop. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Page 1, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY\n",
      "Page 2, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=10\n",
      "Page 3, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=20\n",
      "Page 4, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=30\n",
      "Page 5, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=40\n",
      "Page 6, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=50\n",
      "Page 7, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=60\n",
      "Page 8, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=70\n",
      "Page 9, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=80\n",
      "Page 10, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=90\n",
      "Page 11, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=100\n",
      "Page 12, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=110\n",
      "Page 13, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=120\n",
      "Page 14, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=130\n",
      "Page 15, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=140\n",
      "Page 16, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=150\n",
      "Page 17, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=160\n",
      "Page 18, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=170\n",
      "Page 19, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=180\n",
      "Page 20, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=190\n",
      "Page 21, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=200\n",
      "Page 22, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=210\n",
      "Page 23, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=220\n",
      "Page 24, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=230\n",
      "Page 25, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=240\n",
      "Page 26, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=250\n",
      "Page 27, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=260\n",
      "Page 28, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=270\n",
      "Page 29, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=280\n",
      "Page 30, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=290\n",
      "Page 31, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=300\n",
      "Page 32, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=310\n",
      "Page 33, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=320\n",
      "Page 34, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=330\n",
      "Page 35, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=340\n",
      "Page 36, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=350\n",
      "Page 37, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=360\n",
      "Page 38, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=370\n",
      "Page 39, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=380\n",
      "Page 40, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=390\n",
      "Page 41, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=400\n",
      "Page 42, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=410\n",
      "Page 43, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=420\n",
      "Page 44, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=430\n",
      "Page 45, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=440\n",
      "Page 46, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=450\n",
      "Page 47, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=460\n",
      "Page 48, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=470\n",
      "Page 49, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=480\n",
      "Page 50, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=490\n",
      "Page 51, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=500\n",
      "Page 52, 10 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=510\n",
      "Page 53, 8 results https://www.yelp.com/search?find_desc=burger&find_loc=Albany%2C%20NY&start=520\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1761"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next_page_url = search_url\n",
    "page_counter = 1\n",
    "business_list = []\n",
    "\n",
    "#Run continuously until there is no longer a \"next page\" url found.\n",
    "while next_page_url:\n",
    "    #Request HTML page and load into Beautiful Soup object\n",
    "    request = requests.get(next_page_url)\n",
    "    soup = BeautifulSoup(request.content,'html.parser')\n",
    "    \n",
    "    #Find search results container on page.\n",
    "    search_results = soup.findAll(class_=search_result_class)\n",
    "    print(f\"Page {page_counter}, {len(search_results)-1} results {next_page_url}\")\n",
    "    result_counter = 1\n",
    "    \n",
    "    #Loop through search results and store information for each business\n",
    "    for search_result in search_results:\n",
    "        business_info = {}\n",
    "        try:\n",
    "            business_name_url = search_result.findAll('a', href=True)[1]\n",
    "            business_info['url'] = f\"https://www.yelp.com{business_name_url['href']}\"\n",
    "            business_info['name'] = business_name_url['name']\n",
    "            business_info['biz_id'] = business_name_url['href'].split('/biz/')[1].split('?')[0]\n",
    "        except:\n",
    "            continue\n",
    "            \n",
    "        try:\n",
    "            business_info['address'] = search_result.find('address').text\n",
    "        except:\n",
    "            pass\n",
    "        try:\n",
    "            business_info['category'] = [category.text for category in search_result.findAll(\"a\",attrs={\"role\":\"link\"})]\n",
    "        except:\n",
    "            pass\n",
    "        try:\n",
    "            business_info['star_rating'] = float(re.findall(r\"[-+]?\\d*\\.\\d+|\\d+\", \n",
    "                                                      search_result.find(\n",
    "                                                          class_=star_container_class).find('div')['aria-label'] )[0] )\n",
    "        except:\n",
    "            pass\n",
    "        try:\n",
    "            business_info['price_range'] = search_result.find(class_=price_range_class).text\n",
    "        except:\n",
    "            pass\n",
    "        try:\n",
    "            business_info['num_reviews'] = int(re.findall(r\"[-+]?\\d*\\.\\d+|\\d+\",\n",
    "                                                      search_result.find(\n",
    "                                                          class_=review_count_class).text )[0] )\n",
    "        except:\n",
    "            pass\n",
    "        try:\n",
    "            business_info['image_shown'] = search_result.find('img')['src']\n",
    "        except:\n",
    "            pass\n",
    "        \n",
    "        #Append business information for each search result to a list containing all businesses.\n",
    "        if business_info:\n",
    "            business_list.append(business_info)\n",
    "            \n",
    "        result_counter+=1\n",
    "    \n",
    "    #Set url for next page. If not found, break out of loop.\n",
    "    if soup.find(class_=next_page_class):\n",
    "        next_page_url = base_url + soup.find(class_=next_page_class)['href']\n",
    "        page_counter+=1\n",
    "    else:\n",
    "        break\n",
    "    \n",
    "    #Random delay between 2 and 20 seconds to prevent getting blocked\n",
    "    sleep(np.random.randint(2,20))\n",
    "\n",
    "print(len(business_list), \"businesses scraped\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load results into a Pandas DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1081\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>address</th>\n",
       "      <th>biz_id</th>\n",
       "      <th>category</th>\n",
       "      <th>image_shown</th>\n",
       "      <th>name</th>\n",
       "      <th>num_reviews</th>\n",
       "      <th>price_range</th>\n",
       "      <th>star_rating</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1756</th>\n",
       "      <td>NaN</td>\n",
       "      <td>johnnys-to-go-schenectady</td>\n",
       "      <td>[Italian]</td>\n",
       "      <td>https://s3-media4.fl.yelpcdn.com/bphoto/dr6Msn...</td>\n",
       "      <td>Johnny’s To-Go</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>https://www.yelp.com/biz/johnnys-to-go-schenec...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1757</th>\n",
       "      <td>93 W Campbell Rd</td>\n",
       "      <td>burger-grill-and-middle-eastern-cuisine-schene...</td>\n",
       "      <td>[Imported Food]</td>\n",
       "      <td>https://s3-media1.fl.yelpcdn.com/assets/srv0/y...</td>\n",
       "      <td>Burger Grill &amp; Middle Eastern Cuisine</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>https://www.yelp.com/biz/burger-grill-and-midd...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1758</th>\n",
       "      <td>908 River St</td>\n",
       "      <td>kennedy-fried-chicken-and-pizza-troy</td>\n",
       "      <td>[Chicken Wings, Pizza, Halal]</td>\n",
       "      <td>https://s3-media1.fl.yelpcdn.com/bphoto/rvwZlp...</td>\n",
       "      <td>Kennedy Fried Chicken &amp; Pizza</td>\n",
       "      <td>26.0</td>\n",
       "      <td>$</td>\n",
       "      <td>2.0</td>\n",
       "      <td>https://www.yelp.com/biz/kennedy-fried-chicken...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1759</th>\n",
       "      <td>123 4th St</td>\n",
       "      <td>ginos-pizzeria-troy</td>\n",
       "      <td>[Pizza]</td>\n",
       "      <td>https://s3-media1.fl.yelpcdn.com/bphoto/Ie8eEW...</td>\n",
       "      <td>Gino’s Pizzeria</td>\n",
       "      <td>8.0</td>\n",
       "      <td>$</td>\n",
       "      <td>2.0</td>\n",
       "      <td>https://www.yelp.com/biz/ginos-pizzeria-troy?o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1760</th>\n",
       "      <td>441 State St</td>\n",
       "      <td>nicos-pizzeria-schenectady</td>\n",
       "      <td>[Pizza, Italian]</td>\n",
       "      <td>https://s3-media3.fl.yelpcdn.com/bphoto/XDhLzF...</td>\n",
       "      <td>Nico’s Pizzeria</td>\n",
       "      <td>45.0</td>\n",
       "      <td>$</td>\n",
       "      <td>3.0</td>\n",
       "      <td>https://www.yelp.com/biz/nicos-pizzeria-schene...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               address                                             biz_id  \\\n",
       "1756               NaN                          johnnys-to-go-schenectady   \n",
       "1757  93 W Campbell Rd  burger-grill-and-middle-eastern-cuisine-schene...   \n",
       "1758      908 River St               kennedy-fried-chicken-and-pizza-troy   \n",
       "1759        123 4th St                                ginos-pizzeria-troy   \n",
       "1760      441 State St                         nicos-pizzeria-schenectady   \n",
       "\n",
       "                           category  \\\n",
       "1756                      [Italian]   \n",
       "1757                [Imported Food]   \n",
       "1758  [Chicken Wings, Pizza, Halal]   \n",
       "1759                        [Pizza]   \n",
       "1760               [Pizza, Italian]   \n",
       "\n",
       "                                            image_shown  \\\n",
       "1756  https://s3-media4.fl.yelpcdn.com/bphoto/dr6Msn...   \n",
       "1757  https://s3-media1.fl.yelpcdn.com/assets/srv0/y...   \n",
       "1758  https://s3-media1.fl.yelpcdn.com/bphoto/rvwZlp...   \n",
       "1759  https://s3-media1.fl.yelpcdn.com/bphoto/Ie8eEW...   \n",
       "1760  https://s3-media3.fl.yelpcdn.com/bphoto/XDhLzF...   \n",
       "\n",
       "                                       name  num_reviews price_range  \\\n",
       "1756                         Johnny’s To-Go          5.0         NaN   \n",
       "1757  Burger Grill & Middle Eastern Cuisine          2.0         NaN   \n",
       "1758          Kennedy Fried Chicken & Pizza         26.0           $   \n",
       "1759                        Gino’s Pizzeria          8.0           $   \n",
       "1760                        Nico’s Pizzeria         45.0           $   \n",
       "\n",
       "      star_rating                                                url  \n",
       "1756          3.0  https://www.yelp.com/biz/johnnys-to-go-schenec...  \n",
       "1757          2.0  https://www.yelp.com/biz/burger-grill-and-midd...  \n",
       "1758          2.0  https://www.yelp.com/biz/kennedy-fried-chicken...  \n",
       "1759          2.0  https://www.yelp.com/biz/ginos-pizzeria-troy?o...  \n",
       "1760          3.0  https://www.yelp.com/biz/nicos-pizzeria-schene...  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "business_info_df = pd.DataFrame(business_list)\n",
    "#Drop businesses with no reviews\n",
    "business_info_df.dropna(subset=['num_reviews'], inplace=True)\n",
    "#Drop duplicates\n",
    "business_info_df.drop(business_info_df[business_info_df.biz_id.duplicated(keep='first')].index, inplace=True)\n",
    "print(len(business_info_df))\n",
    "business_info_df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#CLEAN UP CATEGORY VALUES - remove parenthesis\n",
    "business_info_categories = []\n",
    "for category in business_info_df.category:\n",
    "    cat_list = []\n",
    "    for cat in category:\n",
    "        cat = cat.replace('(',' ')\n",
    "        cat = cat.replace(')',' ')\n",
    "        cat = re.sub(' +',' ', cat).strip()\n",
    "        cat_list.append(cat)\n",
    "    business_info_categories.append(cat_list)\n",
    "business_info_df.category = business_info_categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#CLEAN UP BUSINESS NAMES\n",
    "business_names = []\n",
    "for name in business_info_df.name:\n",
    "    name = name.replace('â\\x80\\x99',\"\\'\")\n",
    "    business_names.append(name)\n",
    "business_info_df.name = business_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scrape Reviews Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately there are two versions of reviews pages that Yelp uses. Therefore, two versions of review page scrapers were needed in order to handle the different structures of the webpage. The following functions are used to scrape review data:\n",
    "\n",
    "`get_reviews()`: returns list of reviews for a business across all of the review pages. For each page, it will determine which version of the review page is being used, and call the appropriate review scraper (see below) to gather all review data for the page.\n",
    "\n",
    "`get_reviews_page_v1()`: Returns a list of review details for a single page of review results.\n",
    "\n",
    "`get_reviews_page_v2()`: Does the same thing as v1 but uses a different set of class names and a different structure of soup object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 264,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_reviews(business_name, business_index, yelp_business_url, verbose=False):\n",
    "    \"\"\"\n",
    "    This function will iterate through all of the review pages for a particular business and\n",
    "    return a list populated with all reviews found.\n",
    "    \n",
    "    INPUTS:\n",
    "    business_name     = The name of the business. It is contained in the results list records.\n",
    "    business_index    = The business index (unique identifier). It is contained in the results records.\n",
    "    yelp_business_url = The URL for the starting page of reviews for the business.\n",
    "    verbose           = Summary info is always printed, but with verbose validation of each page is printed.\n",
    "    \n",
    "    OUTPUT:\n",
    "    List of reviews. Each review is a dictionary containing desired review information.\n",
    "    \"\"\"\n",
    "    \n",
    "    #Class names used in Yelp Review pages.\n",
    "    #There are two flavors of page design that yelp uses\n",
    "    search_result_class_v1 = \"lemon--li__373c0__1r9wz u-space-b3 u-padding-b3 border--bottom__373c0__uPbXS border-color--default__373c0__2oFDT\"\n",
    "    search_result_class_v2 = \"review review--with-sidebar\"\n",
    "    \n",
    "    #Set starting page (first page of reviews)\n",
    "    next_page_url = yelp_business_url\n",
    "\n",
    "    reviews_list = []\n",
    "    page_counter=1\n",
    "\n",
    "    #Continue to loop through review pages until there is no longer a \"next\" link at the bottom.\n",
    "    while next_page_url:\n",
    "        if verbose:\n",
    "            #Print the page url being parsed\n",
    "            print(f\"Page {page_counter}, {next_page_url}\")\n",
    "\n",
    "        #Request html for page and load into BeautifulSoup object.\n",
    "        request = requests.get(next_page_url)\n",
    "        soup = BeautifulSoup(request.content,'html.parser')\n",
    "        \n",
    "        #Check which version of the page is being used. If neither is found, print error message.\n",
    "        if len(soup.findAll(class_=search_result_class_v1))!=0:\n",
    "            reviews_list.extend(get_reviews_page_v1(soup,business_name,business_index,verbose))\n",
    "        elif len(soup.findAll(class_=search_result_class_v2))!=0:\n",
    "            reviews_list.extend(get_reviews_page_v2(soup,business_name,business_index,verbose))\n",
    "        else:\n",
    "            print(\"Could not parse page: \", next_page_url)\n",
    "        \n",
    "        #Check for \"next\" page link - update next_page_url if found.\n",
    "        #Break from while loop if there is no next page.\n",
    "        if soup.find(\"link\", attrs={'rel':'next'}):\n",
    "            next_page_url = soup.find(\"link\", attrs={'rel':'next'})['href']\n",
    "            page_counter+=1\n",
    "        else:\n",
    "            break\n",
    "        \n",
    "        #Random delay between 1 and 4 seconds to prevent getting blocked\n",
    "        sleep(np.random.randint(1,3))\n",
    "    \n",
    "    return reviews_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_reviews_page_v1(soup, business_name, business_index, verbose=False):\n",
    "    \"\"\"\n",
    "    This function will extract reviews information from the BeautifulSoup object representing\n",
    "    version 1 of a Yelp review page.\n",
    "    \n",
    "    INPUTS:\n",
    "    soup           = BeautifulSoup object to traverse.\n",
    "    business_name  = The name of the business. It is contained in the results list records.\n",
    "    business_index = The business index (unique identifier). It is contained in the results records.\n",
    "    verbose        = If True, print status of review extraction.\n",
    "    \n",
    "    OUTPUT:\n",
    "    List of reviews. Each review is a dictionary containing desired review information.\n",
    "    \"\"\"\n",
    "    search_result_class = \"lemon--li__373c0__1r9wz u-space-b3 u-padding-b3 border--bottom__373c0__uPbXS border-color--default__373c0__2oFDT\"\n",
    "    star_container_class = \"lemon--div__373c0__1mboc arrange-unit__373c0__1piwO border-color--default__373c0__2oFDT\"\n",
    "    date_class = \"lemon--span__373c0__3997G text__373c0__2pB8f text-color--mid__373c0__3G312 text-align--left__373c0__2pnx_\"\n",
    "    pic_class = \"lemon--span__373c0__3997G photo-box-grid-item__373c0__2kFqV display--inline__373c0__1DbOG u-space-r2 u-space-b2 border-color--default__373c0__2oFDT\"\n",
    "    pic_url_class = \"lemon--img__373c0__3GQUb photo-box-img__373c0__O0tbt\"\n",
    "    \n",
    "    #Get each review block\n",
    "    reviews = soup.findAll(class_=search_result_class)\n",
    "    reviews_list=[]\n",
    "    skipped_review_counter=0\n",
    "    #Loop through each review and pull out pertinent information. Put into list of dictionaries.\n",
    "    for review in reviews:\n",
    "        try:\n",
    "            review_info = {}\n",
    "            review_info[\"business_name\"] = business_name\n",
    "            review_info[\"business_index\"] = business_index\n",
    "            review_info[\"date\"] = review.find(class_=date_class).text.strip()\n",
    "            #review_info[\"review\"] = review.find(\"span\", attrs={\"class\": \"lemon--span__373c0__3997G\", \"lang\": \"en\"}).text\n",
    "            review_info[\"review\"] = review.find(attrs={\"lang\": \"en\"}).text\n",
    "            review_info['star_rating'] = float(re.findall(r\"[-+]?\\d*\\.\\d+|\\d+\", \n",
    "                           review.find(class_=star_container_class).find('div')['aria-label'] )[0] )\n",
    "            review_info[\"pic_count\"] = len(review.find_all(class_=pic_class))\n",
    "            review_info[\"pic_urls\"] = [obj['src'] for obj in review.findAll(class_=pic_url_class)]\n",
    "\n",
    "            #Sometimes the user id is not being found\n",
    "            try:\n",
    "                review_info[\"user_id\"] = review.find('a')['href'].split('userid=')[1]\n",
    "            except:\n",
    "                None\n",
    "\n",
    "            reviews_list.append(review_info)\n",
    "        except:\n",
    "            skipped_review_counter+=1\n",
    "            \n",
    "    if verbose:\n",
    "        if skipped_review_counter!=0:\n",
    "            print(f\"Skipped {skipped_review_counter} reviews\")\n",
    "\n",
    "    return(reviews_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_reviews_page_v2(soup, business_name, business_index, verbose=False):\n",
    "    \"\"\"\n",
    "    This function will extract reviews information from the BeautifulSoup object representing\n",
    "    version 2 of a Yelp review page.\n",
    "    \n",
    "    INPUTS:\n",
    "    soup           = BeautifulSoup object to traverse.\n",
    "    business_name  = The name of the business. It is contained in the results list records.\n",
    "    business_index = The business index (unique identifier). It is contained in the results records.\n",
    "    verbose        = If True, print status of review extraction.\n",
    "    \n",
    "    OUTPUT:\n",
    "    List of reviews. Each review is a dictionary containing desired review information.\n",
    "    \"\"\"\n",
    "    \n",
    "    search_result_class = \"review review--with-sidebar\"\n",
    "    star_container_class = \"biz-rating__stars\"\n",
    "    date_class = \"rating-qualifier\"\n",
    "    review_photo_box_class = \"photo-box-grid clearfix js-content-expandable lightbox-media-parent\"\n",
    "    \n",
    "    #Get each review block\n",
    "    reviews = soup.findAll(class_=search_result_class)\n",
    "    reviews_list=[]\n",
    "    skipped_review_counter=0\n",
    "    #Loop through each review and pull out pertinent information. Put into list of dictionaries.\n",
    "    for review in reviews:\n",
    "        try:\n",
    "            review_info = {}\n",
    "            review_info[\"business_name\"] = business_name\n",
    "            review_info[\"business_index\"] = business_index\n",
    "            review_info[\"date\"] = review.find(class_=date_class).text.strip()\n",
    "            review_info[\"review\"] = review.find(attrs={\"lang\": \"en\"}).text\n",
    "            review_info['star_rating'] = float(re.findall(r\"[-+]?\\d*\\.\\d+|\\d+\", \n",
    "                           review.find(class_=star_container_class).find('div')['title'])[0] )\n",
    "            try:\n",
    "                pic_line_items = review.find(class_=review_photo_box_class).findAll('li')\n",
    "                review_info[\"pic_count\"] = len(pic_line_items)\n",
    "                review_info[\"pic_urls\"] = [obj.find('img')['src'] for obj in pic_line_items]\n",
    "            except:\n",
    "                review_info[\"pic_count\"] = 0\n",
    "                review_info[\"pic_urls\"] = []\n",
    "\n",
    "            #Sometimes the user id is not being found\n",
    "            try:\n",
    "                review_info[\"user_id\"] = review.find('a')['href'].split('userid=')[1]\n",
    "            except:\n",
    "                None\n",
    "\n",
    "            reviews_list.append(review_info)\n",
    "        except:\n",
    "            skipped_review_counter+=1\n",
    "\n",
    "    if verbose:\n",
    "        if skipped_review_counter!=0:\n",
    "            print(f\"Skipped {skipped_review_counter} reviews\")\n",
    "            \n",
    "    return(reviews_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Testing reviews scraper for a single business in business_info_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Page 1, https://www.yelp.com/biz/kuma-ani-albany-4?osq=restaurants\n",
      "Page 2, https://www.yelp.com/biz/kuma-ani-albany-4?osq=restaurants&start=20\n"
     ]
    }
   ],
   "source": [
    "#TESTING SCRAPER FOR A SINGLE BUSINESS\n",
    "index_num = 4\n",
    "business_url = business_info_df.url[index_num]\n",
    "business_name = business_info_df.name[index_num]\n",
    "business_index = business_info_df.biz_id[index_num]\n",
    "\n",
    "reviews_df = pd.DataFrame(get_reviews(business_name,business_index, business_url,verbose=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>business_index</th>\n",
       "      <th>business_name</th>\n",
       "      <th>date</th>\n",
       "      <th>pic_count</th>\n",
       "      <th>pic_urls</th>\n",
       "      <th>review</th>\n",
       "      <th>star_rating</th>\n",
       "      <th>user_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>kuma-ani-albany-4</td>\n",
       "      <td>Kuma Ani</td>\n",
       "      <td>9/1/2019</td>\n",
       "      <td>3</td>\n",
       "      <td>[https://s3-media2.fl.yelpcdn.com/bphoto/FLsrY...</td>\n",
       "      <td>I heard this place makes its own ramen noodles...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>6HOtEJ8wp9Q3XkpLVStKnA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>kuma-ani-albany-4</td>\n",
       "      <td>Kuma Ani</td>\n",
       "      <td>9/11/2019</td>\n",
       "      <td>3</td>\n",
       "      <td>[https://s3-media4.fl.yelpcdn.com/bphoto/IVQgv...</td>\n",
       "      <td>Alright, a ramen place in Colonie! It's a welc...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>JRSIPNLEpWnZkoFGGO-WDw</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>kuma-ani-albany-4</td>\n",
       "      <td>Kuma Ani</td>\n",
       "      <td>9/6/2019</td>\n",
       "      <td>3</td>\n",
       "      <td>[https://s3-media3.fl.yelpcdn.com/bphoto/Bf2jH...</td>\n",
       "      <td>As someone that loves japanese food when I hea...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>MVIlQinGxgwPH_XOUpoXrA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>kuma-ani-albany-4</td>\n",
       "      <td>Kuma Ani</td>\n",
       "      <td>8/26/2019</td>\n",
       "      <td>3</td>\n",
       "      <td>[https://s3-media2.fl.yelpcdn.com/bphoto/O8WK_...</td>\n",
       "      <td>I am very excited to have a ramen place in an ...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>YX927Jezj85xgM_XX6bVxw</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>kuma-ani-albany-4</td>\n",
       "      <td>Kuma Ani</td>\n",
       "      <td>8/25/2019</td>\n",
       "      <td>0</td>\n",
       "      <td>[]</td>\n",
       "      <td>So, like I said... my 1st visit would not be m...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>7MohwGwMyJaErZ_SRQQNdQ</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      business_index business_name       date  pic_count  \\\n",
       "0  kuma-ani-albany-4      Kuma Ani   9/1/2019          3   \n",
       "1  kuma-ani-albany-4      Kuma Ani  9/11/2019          3   \n",
       "2  kuma-ani-albany-4      Kuma Ani   9/6/2019          3   \n",
       "3  kuma-ani-albany-4      Kuma Ani  8/26/2019          3   \n",
       "4  kuma-ani-albany-4      Kuma Ani  8/25/2019          0   \n",
       "\n",
       "                                            pic_urls  \\\n",
       "0  [https://s3-media2.fl.yelpcdn.com/bphoto/FLsrY...   \n",
       "1  [https://s3-media4.fl.yelpcdn.com/bphoto/IVQgv...   \n",
       "2  [https://s3-media3.fl.yelpcdn.com/bphoto/Bf2jH...   \n",
       "3  [https://s3-media2.fl.yelpcdn.com/bphoto/O8WK_...   \n",
       "4                                                 []   \n",
       "\n",
       "                                              review  star_rating  \\\n",
       "0  I heard this place makes its own ramen noodles...          5.0   \n",
       "1  Alright, a ramen place in Colonie! It's a welc...          4.0   \n",
       "2  As someone that loves japanese food when I hea...          4.0   \n",
       "3  I am very excited to have a ramen place in an ...          5.0   \n",
       "4  So, like I said... my 1st visit would not be m...          5.0   \n",
       "\n",
       "                  user_id  \n",
       "0  6HOtEJ8wp9Q3XkpLVStKnA  \n",
       "1  JRSIPNLEpWnZkoFGGO-WDw  \n",
       "2  MVIlQinGxgwPH_XOUpoXrA  \n",
       "3  YX927Jezj85xgM_XX6bVxw  \n",
       "4  7MohwGwMyJaErZ_SRQQNdQ  "
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scrape ALL Reviews for each Business in initial search\n",
    "Get all reviews for businesses listed in `business_info_df`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a tricky process. It takes some time to scrape all of the pages, and it is likely that problems can occur if the website being scraped suspects that I am a bot. You may have to stop and continue running this code at different times.\n",
    "\n",
    "`all_reviews` will contain all of the reviews for all businesses. \n",
    "\n",
    "`last_index_completed` indicates the index that will be used as a starting point in `business_info_df`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set container\n",
    "last_index_completed = 0\n",
    "all_reviews = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the code below to set `last_index_completed` when you want to continue scraping reviews after stopping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Use call to set index starting point\n",
    "#last_index_completed = 828"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loop through restaurants and get reviews for each\n",
    "\n",
    "This loop will iterate for each business in `business_info_df`, then it will gather the reviews from all review pages for that business and append it to the `all_reviews` list.\n",
    "\n",
    "In this loop:\n",
    "\n",
    "1. For each business, get the url its first page of reviews from `business_info_df`. \n",
    "2. Pass the url into `get_reviews()` which will iteratively scrape all review pages for the business.\n",
    "3. Add results to master `all_reviews` list. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 493,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1076 5 reviews\t Johnny’s To-Go https://www.yelp.com/biz/johnnys-to-go-schenectady?osq=burger completed\n",
      "1077 2 reviews\t Burger Grill & Middle Eastern Cuisine https://www.yelp.com/biz/burger-grill-and-middle-eastern-cuisine-schenectady?osq=burger completed\n",
      "1078 26 reviews\t Kennedy Fried Chicken & Pizza https://www.yelp.com/biz/kennedy-fried-chicken-and-pizza-troy?osq=burger completed\n",
      "1079 8 reviews\t Gino’s Pizzeria https://www.yelp.com/biz/ginos-pizzeria-troy?osq=burger completed\n",
      "1080 45 reviews\t Nico’s Pizzeria https://www.yelp.com/biz/nicos-pizzeria-schenectady?osq=burger completed\n"
     ]
    }
   ],
   "source": [
    "while last_index_completed < len(business_info_df):\n",
    "    biz = business_info_df.iloc[last_index_completed]\n",
    "    \n",
    "    #Random delay between 1 and 5 seconds to try to prevent getting blocked\n",
    "    sleep(np.random.randint(2,5))\n",
    "    print(last_index_completed, int(biz['num_reviews']),\"reviews\\t\", biz['name'], biz['url'], end='')\n",
    "    \n",
    "    #Get all reviews for the business and add to list.\n",
    "    all_reviews.extend(get_reviews(biz['name'],biz['biz_id'],biz['url'])) \n",
    "\n",
    "    print(\" completed\")\n",
    "    last_index_completed+=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 526,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "59274"
      ]
     },
     "execution_count": 526,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_reviews_df = pd.DataFrame(all_reviews)\n",
    "len(all_reviews_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean the Date Field"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Noticed that some dates have \"updated\" after them in the string indicating that the review was updated. I want to strip this information from the date field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Scrub date strings and convert to datetime type\n",
    "scrubbed_dates = [re.sub(\"\\D\\D\\D\", \"\", date) for date in all_reviews_df.date]\n",
    "all_reviews_df.date = pd.to_datetime(scrubbed_dates)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Write dataframes to CSV files\n",
    "\n",
    "Currently commented out because I have scraped everything I need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#business_info_df.to_csv('csv_files/business_info_all.csv',index=False)\n",
    "#all_reviews_df.to_csv('csv_files/reviews_all.csv',index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:learn-env]",
   "language": "python",
   "name": "conda-env-learn-env-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}