Mini Search Engine (Study Case : News Question About COVID-19).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Mini Search Engine (Study Case : News Question About COVID-19).ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyMnMsYODlY2gyZuqDThO7SB",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hazmicokro/Mini-Search-Engine-Study-Case-News-Question-About-COVID-19/blob/main/Mini%20Search%20Engine%20(Study%20Case%20%3A%20News%20Question%20About%20COVID-19).ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FdbppSL9UKSh"
      },
      "source": [
        "**Nama : Mochammad Hazmi Cokro Mandiri**\n",
        "\n",
        "**NIM : 201710370311240**\n",
        "\n",
        "**Temu Kembali Informasi 7A**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qcb1NLh4UL6N"
      },
      "source": [
        "Dataset : [NEWS](https://www.kaggle.com/xhlulu/covidqa?select=news.csv) dari .csv File QnA Kaggle "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ni6nC3f42KrP"
      },
      "source": [
        "**NEWS QUESTION COVID-19 -  SEARCH ENGINE**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sv6hbtbr7zoU"
      },
      "source": [
        "Penilaian :\n",
        "1. Tahapan yang harus ada : crawling, preprocessing, indexing, retrieval, perankingan.\n",
        "2. Link URL minimal 10 links.\n",
        "3. Preprocessing : tokenizing, stemming, stopword removal, case folding.\n",
        "4. Indexing : term, dan ada pembobotan (bisa pakai tfidf atau metode pembobotan lain).\n",
        "5. Retrieval : menggunakan cosine similarity atau improvisasi metode lain.\n",
        "6. perankingan berdasarkan nilai similarity\n",
        "7. Output seperti search engine (ada daftar link yang bisa di klik)\n",
        "\n",
        "Poin plus :\n",
        "Jika ada improvisasi => bisa dari metode pembobotan atau retrieval, studi kasus/dokumen yang di retrieve, metode perankingan, atau bisa juga dari metode preprocessing."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XWIRt_O_fXiM"
      },
      "source": [
        "#[1] Download Dataset (Crawling)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ybNl1j688zoH"
      },
      "source": [
        "## Mount dan Lokasi Kerja"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UTiToahoucSx",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "03e4d6db-a184-4f71-e4fe-a2da22fa98dd"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vSBIqCpxwg2k",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c00b2364-7fef-4c16-de9b-65b945bd9d2f"
      },
      "source": [
        "%cd \"/content/drive/MyDrive/Tubes Search Engine - 240\""
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/MyDrive/Tubes Search Engine - 240\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QZKRu92y83fv"
      },
      "source": [
        "##Download Dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q4Z1xCg7ww38"
      },
      "source": [
        "import os\n",
        "os.environ['KAGGLE_CONFIG_DIR'] = \"/content/drive/MyDrive/Tubes Search Engine - 240\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zMr4HgNkw1nz",
        "outputId": "4b2cf52b-c94e-4bbe-8f48-b727e70b915c"
      },
      "source": [
        "from google.colab import files\n",
        "files.upload()\n",
        "print(\"Done Upload Kaggle.json\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Done Upload Kaggle.json\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TZ1R9TvLw6Sy",
        "outputId": "cce9bfd6-99cb-485c-f46a-33ef4b743540"
      },
      "source": [
        "!mkdir -p ~/.kaggle\n",
        "!cp kaggle.json ~/.kaggle/\n",
        "!ls ~/.kaggle\n",
        "\n",
        "!chmod 600 /root/.kaggle/kaggle.json"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "kaggle.json\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "s6oH6QbGkP7a",
        "outputId": "9c6aaa4c-b867-45c6-9113-46e5cdbbda22"
      },
      "source": [
        "!kaggle datasets download -d xhlulu/covidqa"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading covidqa.zip to /content\n",
            "100% 1.77G/1.78G [00:18<00:00, 114MB/s]\n",
            "100% 1.78G/1.78G [00:18<00:00, 105MB/s]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mXDxPOQZw50h",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2cac1758-37b4-450d-958d-3ae25b47dcda"
      },
      "source": [
        "!unzip covidqa.zip"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Archive:  covidqa.zip\n",
            "  inflating: news.csv                \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PrZJ79ihVD7U"
      },
      "source": [
        "##Load Dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 759
        },
        "id": "Q2tcb7WfJfCR",
        "outputId": "730c9caa-bc68-4e25-a1f2-ed82386fb771"
      },
      "source": [
        "import pandas as pd \n",
        "\n",
        "df = pd.read_csv(\"/content/drive/MyDrive/Tubes Search Engine - 240/news.csv\")\n",
        "with pd.option_context('display.max_rows',10):\n",
        "    display(df)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source</th>\n",
              "      <th>url</th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>wrong_answer</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>What are the symptoms?</td>\n",
              "      <td>Symptoms include fever, coughing, sore throat,...</td>\n",
              "      <td>As cases of coronavirus surge, health authorit...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>When should I get tested?</td>\n",
              "      <td>Your doctor will tell you if you need to get t...</td>\n",
              "      <td>If you have been in contact with a person with...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>What's the difference between physical distanc...</td>\n",
              "      <td>As cases of coronavirus surge, health authorit...</td>\n",
              "      <td>Remember, the vast majority of Australians are...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>How do I practice physical distancing?</td>\n",
              "      <td>If you have been in contact with a person with...</td>\n",
              "      <td>Other countries have closed schools and univer...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>What's closed?</td>\n",
              "      <td>Physical distancing is the reason the Federal ...</td>\n",
              "      <td>Symptoms include fever, coughing, sore throat,...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>476</th>\n",
              "      <td>USA Today</td>\n",
              "      <td>https://www.usatoday.com/story/news/health/202...</td>\n",
              "      <td>Is the presence of a fever always the first sy...</td>\n",
              "      <td>Symptoms of COVID-19 resemble the flu and incl...</td>\n",
              "      <td>No, for several reasons, says Peter Hotez, dea...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>477</th>\n",
              "      <td>USA Today</td>\n",
              "      <td>https://www.usatoday.com/story/news/health/202...</td>\n",
              "      <td>How can I know if I have the coronavirus and s...</td>\n",
              "      <td>If you have symptoms and want to get tested, t...</td>\n",
              "      <td>No. While shortness of breath is among the mos...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>478</th>\n",
              "      <td>USA Today</td>\n",
              "      <td>https://www.usatoday.com/story/news/health/202...</td>\n",
              "      <td>Does testing positive mean that I have the vir...</td>\n",
              "      <td>Yes, testing positive means that you have the ...</td>\n",
              "      <td>You should take precaution with any containers...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>479</th>\n",
              "      <td>USA Today</td>\n",
              "      <td>https://www.usatoday.com/story/news/health/202...</td>\n",
              "      <td>There has been a Hong Kong flu, a Swine flu, H...</td>\n",
              "      <td>The current outbreak is the world's first pand...</td>\n",
              "      <td>Vaccines against pneumonia, such as pneumococc...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>480</th>\n",
              "      <td>USA Today</td>\n",
              "      <td>https://www.usatoday.com/story/news/health/202...</td>\n",
              "      <td>How old was the youngest person to die from th...</td>\n",
              "      <td>A study of more than 72,000 cases in mainland ...</td>\n",
              "      <td>If you have symptoms and want to get tested, t...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>481 rows × 5 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "            source  ...                                       wrong_answer\n",
              "0    ABC Australia  ...  As cases of coronavirus surge, health authorit...\n",
              "1    ABC Australia  ...  If you have been in contact with a person with...\n",
              "2    ABC Australia  ...  Remember, the vast majority of Australians are...\n",
              "3    ABC Australia  ...  Other countries have closed schools and univer...\n",
              "4    ABC Australia  ...  Symptoms include fever, coughing, sore throat,...\n",
              "..             ...  ...                                                ...\n",
              "476      USA Today  ...  No, for several reasons, says Peter Hotez, dea...\n",
              "477      USA Today  ...  No. While shortness of breath is among the mos...\n",
              "478      USA Today  ...  You should take precaution with any containers...\n",
              "479      USA Today  ...  Vaccines against pneumonia, such as pneumococc...\n",
              "480      USA Today  ...  If you have symptoms and want to get tested, t...\n",
              "\n",
              "[481 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sAwQprr8XO-p"
      },
      "source": [
        "#[2] Preprocessing Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OXmF2a6KZOJD"
      },
      "source": [
        "**Drop Wrong Answer (Fitur yang kurang berkorelasi dengan fitur lain)**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        },
        "id": "2mJ-mdY0Sk0R",
        "outputId": "73c27c1c-1d7e-40bc-e706-e52c97401971"
      },
      "source": [
        "df.drop('wrong_answer', axis=1, inplace=True)\n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source</th>\n",
              "      <th>url</th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>What are the symptoms?</td>\n",
              "      <td>Symptoms include fever, coughing, sore throat,...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>When should I get tested?</td>\n",
              "      <td>Your doctor will tell you if you need to get t...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>What's the difference between physical distanc...</td>\n",
              "      <td>As cases of coronavirus surge, health authorit...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>How do I practice physical distancing?</td>\n",
              "      <td>If you have been in contact with a person with...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>What's closed?</td>\n",
              "      <td>Physical distancing is the reason the Federal ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "          source  ...                                             answer\n",
              "0  ABC Australia  ...  Symptoms include fever, coughing, sore throat,...\n",
              "1  ABC Australia  ...  Your doctor will tell you if you need to get t...\n",
              "2  ABC Australia  ...  As cases of coronavirus surge, health authorit...\n",
              "3  ABC Australia  ...  If you have been in contact with a person with...\n",
              "4  ABC Australia  ...  Physical distancing is the reason the Federal ...\n",
              "\n",
              "[5 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1FSuySKZXSen"
      },
      "source": [
        "**Drop NaN / na / null value**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ItGyeAfqT0pa",
        "outputId": "002ffd71-d8f1-4447-c014-238ad59731ae"
      },
      "source": [
        "df.dropna(inplace=True)\n",
        "df.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 479 entries, 0 to 480\n",
            "Data columns (total 4 columns):\n",
            " #   Column    Non-Null Count  Dtype \n",
            "---  ------    --------------  ----- \n",
            " 0   source    479 non-null    object\n",
            " 1   url       479 non-null    object\n",
            " 2   question  479 non-null    object\n",
            " 3   answer    479 non-null    object\n",
            "dtypes: object(4)\n",
            "memory usage: 18.7+ KB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3d_4DAfuZmnx"
      },
      "source": [
        "###Case Folding\n",
        "**Mengubah huruf Kapital menjadi lower_case (kecil)**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "id": "Y6xzsrF5X8F3",
        "outputId": "d830b42b-cf54-493b-db47-ce23f0791671"
      },
      "source": [
        "def lower_case(input_str):\n",
        "    input_str = input_str.lower()\n",
        "    return input_str\n",
        "\n",
        "df['question'] = df['question'].apply(lambda x: lower_case(x))\n",
        "df['answer'] = df['answer'].apply(lambda x: lower_case(x))\n",
        "df.head(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source</th>\n",
              "      <th>url</th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>what are the symptoms?</td>\n",
              "      <td>symptoms include fever, coughing, sore throat,...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>when should i get tested?</td>\n",
              "      <td>your doctor will tell you if you need to get t...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>what's the difference between physical distanc...</td>\n",
              "      <td>as cases of coronavirus surge, health authorit...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "          source  ...                                             answer\n",
              "0  ABC Australia  ...  symptoms include fever, coughing, sore throat,...\n",
              "1  ABC Australia  ...  your doctor will tell you if you need to get t...\n",
              "2  ABC Australia  ...  as cases of coronavirus surge, health authorit...\n",
              "\n",
              "[3 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5a81-WwIZcca"
      },
      "source": [
        "###Tokenizing \n",
        "**Menghapus Simbol dan Angka , Melakukan Word Tokenize**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "id": "3fZAX_hpU-5N",
        "outputId": "bbf70e41-7e9d-4813-a02d-17da5ccc286e"
      },
      "source": [
        "import re\n",
        "\n",
        "df['question'] = df['question'].apply(lambda x: re.sub('[^a-zA-z0-9\\s]','',x))\n",
        "df['answer'] = df['answer'].apply(lambda x: re.sub('[^a-zA-z0-9\\s]','',x))\n",
        "df.head(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source</th>\n",
              "      <th>url</th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>what are the symptoms</td>\n",
              "      <td>symptoms include fever coughing sore throat fa...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>when should i get tested</td>\n",
              "      <td>your doctor will tell you if you need to get t...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>whats the difference between physical distanci...</td>\n",
              "      <td>as cases of coronavirus surge health authoriti...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "          source  ...                                             answer\n",
              "0  ABC Australia  ...  symptoms include fever coughing sore throat fa...\n",
              "1  ABC Australia  ...  your doctor will tell you if you need to get t...\n",
              "2  ABC Australia  ...  as cases of coronavirus surge health authoriti...\n",
              "\n",
              "[3 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 229
        },
        "id": "LBuzKNLniiRm",
        "outputId": "cdb3c551-7f3a-4534-bbd6-fb5a23b7716c"
      },
      "source": [
        "import nltk\n",
        "nltk.download('punkt')\n",
        "\n",
        "df[\"question\"] = df.apply(lambda row: nltk.word_tokenize(row[\"question\"]), axis=1)\n",
        "df[\"answer\"] = df.apply(lambda row: nltk.word_tokenize(row[\"answer\"]), axis=1)\n",
        "\n",
        "df.head(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source</th>\n",
              "      <th>url</th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>[what, are, the, symptoms]</td>\n",
              "      <td>[symptoms, include, fever, coughing, sore, thr...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>[when, should, i, get, tested]</td>\n",
              "      <td>[your, doctor, will, tell, you, if, you, need,...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>[whats, the, difference, between, physical, di...</td>\n",
              "      <td>[as, cases, of, coronavirus, surge, health, au...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "          source  ...                                             answer\n",
              "0  ABC Australia  ...  [symptoms, include, fever, coughing, sore, thr...\n",
              "1  ABC Australia  ...  [your, doctor, will, tell, you, if, you, need,...\n",
              "2  ABC Australia  ...  [as, cases, of, coronavirus, surge, health, au...\n",
              "\n",
              "[3 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZifWRxL-ev6t"
      },
      "source": [
        "###Stemming \n",
        "**Mengubah Kata tidak baku menjadi Kata Baku**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "id": "cNR2B5miVraw",
        "outputId": "4a4647bc-d17c-4371-d8ec-1d468ec20ebb"
      },
      "source": [
        "from nltk.stem.snowball import SnowballStemmer\n",
        "\n",
        "stemmer = SnowballStemmer(\"english\")\n",
        "df['question'] = df['question'].apply(lambda x: [stemmer.stem(y) for y in x])\n",
        "df['answer'] = df['answer'].apply(lambda x: [stemmer.stem(y) for y in x])\n",
        "df.head(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source</th>\n",
              "      <th>url</th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>[what, are, the, symptom]</td>\n",
              "      <td>[symptom, includ, fever, cough, sore, throat, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>[when, should, i, get, test]</td>\n",
              "      <td>[your, doctor, will, tell, you, if, you, need,...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>[what, the, differ, between, physic, distanc, ...</td>\n",
              "      <td>[as, case, of, coronavirus, surg, health, auth...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "          source  ...                                             answer\n",
              "0  ABC Australia  ...  [symptom, includ, fever, cough, sore, throat, ...\n",
              "1  ABC Australia  ...  [your, doctor, will, tell, you, if, you, need,...\n",
              "2  ABC Australia  ...  [as, case, of, coronavirus, surg, health, auth...\n",
              "\n",
              "[3 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DI4YC_p8f5VZ"
      },
      "source": [
        "###Stop Words\n",
        "**Menghapus kata imbuhan**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 267
        },
        "id": "OXcDLZG1jb-8",
        "outputId": "b0653d50-be57-4dc9-d1fa-bf8a203f3ac5"
      },
      "source": [
        "import nltk\n",
        "from nltk.corpus import stopwords\n",
        "nltk.download('stopwords')\n",
        "\n",
        "stop = set(stopwords.words('english'))\n",
        "exclude = set(('what', 'where', 'when', 'who', 'why' ,'how' , 'i', 'we','us'))\n",
        "stop_words = stop.difference(exclude)\n",
        "print(stop_words)\n",
        "\n",
        "def remove_stops1(row):\n",
        "    my_list1 = row['question']\n",
        "    meaningful_words1 = [w for w in my_list1 if not w in stop_words]\n",
        "    return (meaningful_words1)\n",
        "\n",
        "def remove_stops2(row):\n",
        "    my_list2 = row['answer']\n",
        "    meaningful_words2 = [w for w in my_list2 if not w in stop_words]\n",
        "    return (meaningful_words2)\n",
        "\n",
        "df['question'] = df.apply(remove_stops1, axis=1)\n",
        "df['answer'] = df.apply(remove_stops2, axis=1)\n",
        "\n",
        "df.head(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "{\"weren't\", 'below', 'both', 'or', 't', 'ours', 'no', 'the', 'after', 'because', 'some', 'too', \"hasn't\", \"won't\", 'such', 'of', 'those', 'again', 'on', \"you've\", 'needn', 'itself', 'won', 'her', 'been', 'if', 'and', 'that', 'shan', 'hers', 'ma', 'into', 'he', 'as', 'wasn', 'himself', \"doesn't\", 'am', 'up', 'once', 'yourself', 'themselves', 'here', \"mightn't\", 'to', 'now', \"wasn't\", 'but', 'all', 'o', 'm', 'while', 'hadn', 'not', \"mustn't\", 'so', 'own', 'will', 'she', 'whom', 'you', 'll', \"she's\", 'hasn', 'be', 'his', 've', 'does', 'any', 'few', 'off', 'mustn', \"needn't\", \"you'll\", 'other', 'being', 'have', \"shouldn't\", 'further', 'between', 'nor', 'there', 'it', 'above', 'couldn', 'at', 'were', 'herself', \"wouldn't\", 'has', 'only', 'had', 'their', 'don', \"aren't\", 'doesn', \"haven't\", 'which', 'very', 'having', 'should', 'd', 'do', \"shan't\", 'did', 'before', 'mightn', 'same', 'isn', \"hadn't\", 'by', 'wouldn', 'myself', \"don't\", 'just', 'a', \"that'll\", 'each', 'they', 'its', 'doing', 'our', 'them', 'in', 'with', \"it's\", 'against', 's', 're', 'theirs', 'until', 'yours', \"you'd\", \"you're\", 'was', 'can', 'during', 'then', 'ain', 'for', \"didn't\", 'over', 'down', 'my', 'these', 'me', 'about', 'y', 'more', 'ourselves', 'out', \"couldn't\", 'aren', 'than', 'yourselves', \"should've\", 'an', \"isn't\", 'shouldn', 'weren', 'this', 'are', 'haven', 'through', 'most', 'your', 'from', 'him', 'under', 'is', 'didn'}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source</th>\n",
              "      <th>url</th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>[what, symptom]</td>\n",
              "      <td>[symptom, includ, fever, cough, sore, throat, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>[when, i, get, test]</td>\n",
              "      <td>[doctor, tell, need, get, test]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>[what, differ, physic, distanc, selfisol]</td>\n",
              "      <td>[case, coronavirus, surg, health, author, advi...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "          source  ...                                             answer\n",
              "0  ABC Australia  ...  [symptom, includ, fever, cough, sore, throat, ...\n",
              "1  ABC Australia  ...                    [doctor, tell, need, get, test]\n",
              "2  ABC Australia  ...  [case, coronavirus, surg, health, author, advi...\n",
              "\n",
              "[3 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KOAfGYoihjEA"
      },
      "source": [
        "###Re Join Words\n",
        "\n",
        "**Menggabungkan ulang kata yang telah di preprocessing ke kalimat awal**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "id": "G-4e6aXpgpN6",
        "outputId": "0ea4fae2-3cde-4459-f75f-4bd2d9abcaaf"
      },
      "source": [
        "def rejoin_words1(row):\n",
        "    my_list1 = row['question']\n",
        "    joined_words1 = ( \" \".join(my_list1))\n",
        "    return joined_words1\n",
        "\n",
        "def rejoin_words2(row):\n",
        "    my_list2 = row['answer']\n",
        "    joined_words2 = ( \" \".join(my_list2))\n",
        "    return joined_words2\n",
        "\n",
        "df['question'] = df.apply(rejoin_words1, axis=1)\n",
        "df['answer'] = df.apply(rejoin_words2, axis=1)\n",
        "\n",
        "df.head(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source</th>\n",
              "      <th>url</th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>what symptom</td>\n",
              "      <td>symptom includ fever cough sore throat fatigu ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>when i get test</td>\n",
              "      <td>doctor tell need get test</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>what differ physic distanc selfisol</td>\n",
              "      <td>case coronavirus surg health author advis aust...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "          source  ...                                             answer\n",
              "0  ABC Australia  ...  symptom includ fever cough sore throat fatigu ...\n",
              "1  ABC Australia  ...                          doctor tell need get test\n",
              "2  ABC Australia  ...  case coronavirus surg health author advis aust...\n",
              "\n",
              "[3 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "beLjw3KKvUNn"
      },
      "source": [
        "#[3] Indexing / Pembobotan TF-IDF\n",
        "**TF-IDF Vectorize Sklearn**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JoHXWgEwvTMI"
      },
      "source": [
        "# Library TF-IDF Vectorize Sklearn\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "from scipy.spatial import distance"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UEQDSWHoi6r0"
      },
      "source": [
        "### TF - IDF\n",
        "**Term Frequency - Inverse Document Frequency**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jc7WJtjJvYoP",
        "outputId": "4feb336e-e96e-4307-f5ca-95874ae19ee6"
      },
      "source": [
        "# Membuat Fungsi TF IDF menggunakan Library Sklearn\n",
        "vectorizer = TfidfVectorizer()\n",
        "X = vectorizer.fit_transform(df['question'])\n",
        "print(X.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(479, 739)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ygImA6PUioob"
      },
      "source": [
        "#[4] Retrieval / Cosine Similarity\n",
        "**Cosine Similarity antar query dan data**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 625
        },
        "id": "e_sl3AexvYWK",
        "outputId": "83172570-d464-4992-dfbe-fb4c58c5b98b"
      },
      "source": [
        "print('Masukkan Kata Kunci Pencarian : ')\n",
        "query = input()\n",
        "\n",
        "# Query Input di Vectorize dan dihitung nilai cosine similarity\n",
        "query_vec = vectorizer.transform([query]) \n",
        "results = cosine_similarity(X,query_vec).reshape(-1)\n",
        "\n",
        "df['cos_sim'] = cosine_similarity(X, query_vec).reshape(-1)\n",
        "with pd.option_context('display.max_rows',10):\n",
        "    display(df)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Masukkan Kata Kunci Pencarian : \n",
            "how to know coronavirus\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source</th>\n",
              "      <th>url</th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>cos_sim</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>what symptom</td>\n",
              "      <td>symptom includ fever cough sore throat fatigu ...</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>when i get test</td>\n",
              "      <td>doctor tell need get test</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>what differ physic distanc selfisol</td>\n",
              "      <td>case coronavirus surg health author advis aust...</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>how i practic physic distanc</td>\n",
              "      <td>contact person covid19 infect ill need isol 14...</td>\n",
              "      <td>0.086971</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ABC Australia</td>\n",
              "      <td>https://www.abc.net.au/triplej/programs/hack/c...</td>\n",
              "      <td>what close</td>\n",
              "      <td>physic distanc reason feder govern close pub i...</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>476</th>\n",
              "      <td>USA Today</td>\n",
              "      <td>https://www.usatoday.com/story/news/health/202...</td>\n",
              "      <td>presenc fever alway first symptom coronavirus ...</td>\n",
              "      <td>symptom covid19 resembl flu includ fever cough...</td>\n",
              "      <td>0.060106</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>477</th>\n",
              "      <td>USA Today</td>\n",
              "      <td>https://www.usatoday.com/story/news/health/202...</td>\n",
              "      <td>how i know i coronavirus go test</td>\n",
              "      <td>symptom want get test cdc recommend call state...</td>\n",
              "      <td>0.757903</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>478</th>\n",
              "      <td>USA Today</td>\n",
              "      <td>https://www.usatoday.com/story/news/health/202...</td>\n",
              "      <td>doe test posit mean i virus i develop symptom</td>\n",
              "      <td>yes test posit mean virus doe mean develop sym...</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>479</th>\n",
              "      <td>USA Today</td>\n",
              "      <td>https://www.usatoday.com/story/news/health/202...</td>\n",
              "      <td>hong kong flu swine flu h1n1 sar mer ebola zik...</td>\n",
              "      <td>current outbreak world first pandem caus coron...</td>\n",
              "      <td>0.086961</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>480</th>\n",
              "      <td>USA Today</td>\n",
              "      <td>https://www.usatoday.com/story/news/health/202...</td>\n",
              "      <td>how old youngest person die virus</td>\n",
              "      <td>studi 72000 case mainland china found least on...</td>\n",
              "      <td>0.082996</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>479 rows × 5 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "            source  ...   cos_sim\n",
              "0    ABC Australia  ...  0.000000\n",
              "1    ABC Australia  ...  0.000000\n",
              "2    ABC Australia  ...  0.000000\n",
              "3    ABC Australia  ...  0.086971\n",
              "4    ABC Australia  ...  0.000000\n",
              "..             ...  ...       ...\n",
              "476      USA Today  ...  0.060106\n",
              "477      USA Today  ...  0.757903\n",
              "478      USA Today  ...  0.000000\n",
              "479      USA Today  ...  0.086961\n",
              "480      USA Today  ...  0.082996\n",
              "\n",
              "[479 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gaTmCUoEUetT"
      },
      "source": [
        "#[5] Perankingan / TOP 10"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ePbKrIiuUdIn",
        "outputId": "cb368014-a42d-49f8-a75a-acd6564e107e"
      },
      "source": [
        "print('Kata Kunci Pencarian :  ' + query, \"\\n\")\n",
        "print(\"Top 10 - Pencarian yang serupa dengan kata kunci\")\n",
        "print(\"--------------------------------------------------------------------\")\n",
        "\n",
        "for i in results.argsort()[-10:][::-1]:\n",
        "    print(\"Hasil Pencarian \\t: \",df.iloc[i,2],\n",
        "              \"\\nCosine Similarity \\t: \",df.iloc[i,4],\n",
        "              \"\\nPortal Berita \\t\\t: \",df.iloc[i,0],\n",
        "              \"\\nLink Berita \\t\\t: \",df.iloc[i,1])\n",
        "    print(\" \")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Kata Kunci Pencarian :  how to know coronavirus \n",
            "\n",
            "Top 10 - Pencarian yang serupa dengan kata kunci\n",
            "--------------------------------------------------------------------\n",
            "Hasil Pencarian \t:  how i know i coronavirus \n",
            "Cosine Similarity \t:  1.0 \n",
            "Portal Berita \t\t:  ABC News \n",
            "Link Berita \t\t:  https://www.abc.net.au/news/2020-03-13/coronavirus-faq-covid-19-frequently-asked-questions/12042318\n",
            " \n",
            "Hasil Pencarian \t:  how i know i coronavirus go test \n",
            "Cosine Similarity \t:  0.7579033832948884 \n",
            "Portal Berita \t\t:  USA Today \n",
            "Link Berita \t\t:  https://www.usatoday.com/story/news/health/2020/03/12/coronavirus-reader-questions-answered-symptoms-testing-elderly/5012457002/\n",
            " \n",
            "Hasil Pencarian \t:  how i know i coronavirus go test \n",
            "Cosine Similarity \t:  0.7579033832948884 \n",
            "Portal Berita \t\t:  USA Today \n",
            "Link Berita \t\t:  https://www.usatoday.com/story/news/health/2020/03/12/coronavirus-reader-questions-answered-symptoms-testing-elderly/5012457002/\n",
            " \n",
            "Hasil Pencarian \t:  how know cold flu covid19 \n",
            "Cosine Similarity \t:  0.5257847110779313 \n",
            "Portal Berita \t\t:  CTV \n",
            "Link Berita \t\t:  https://www.ctvnews.ca/health/coronavirus/covid-19-q-a-transmission-testing-who-s-at-risk-and-more-1.4856659\n",
            " \n",
            "Hasil Pencarian \t:  how i know when bad enough get help \n",
            "Cosine Similarity \t:  0.43307570645603527 \n",
            "Portal Berita \t\t:  NY Times \n",
            "Link Berita \t\t:  https://www.nytimes.com/interactive/2020/world/coronavirus-tips-advice.html\n",
            " \n",
            "Hasil Pencarian \t:  how coronavirus test \n",
            "Cosine Similarity \t:  0.40008612561057 \n",
            "Portal Berita \t\t:  ABC News \n",
            "Link Berita \t\t:  https://www.abc.net.au/news/2020-03-13/coronavirus-faq-covid-19-frequently-asked-questions/12042318\n",
            " \n",
            "Hasil Pencarian \t:  how catch coronavirus \n",
            "Cosine Similarity \t:  0.3539623156383117 \n",
            "Portal Berita \t\t:  ABC News \n",
            "Link Berita \t\t:  https://www.abc.net.au/news/2020-03-13/coronavirus-faq-covid-19-frequently-asked-questions/12042318\n",
            " \n",
            "Hasil Pencarian \t:  how coronavirus transmit \n",
            "Cosine Similarity \t:  0.35020309276275946 \n",
            "Portal Berita \t\t:  The Australian \n",
            "Link Berita \t\t:  https://www.theaustralian.com.au/science/how-do-i-avoid-coronavirus-and-other-vital-questions/news-story/07d3552cd2acccc9c72108ab848642b3\n",
            " \n",
            "Hasil Pencarian \t:  how get test coronavirus \n",
            "Cosine Similarity \t:  0.3477835903904707 \n",
            "Portal Berita \t\t:  ABC News \n",
            "Link Berita \t\t:  https://www.abc.net.au/news/2020-03-13/coronavirus-faq-covid-19-frequently-asked-questions/12042318\n",
            " \n",
            "Hasil Pencarian \t:  how coronavirus start \n",
            "Cosine Similarity \t:  0.3199332453900374 \n",
            "Portal Berita \t\t:  ABC News \n",
            "Link Berita \t\t:  https://www.abc.net.au/news/2020-03-13/coronavirus-faq-covid-19-frequently-asked-questions/12042318\n",
            " \n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}