Module 1 - Difference between BM25 similarity and Semantic similarity.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c229af9c",
   "metadata": {},
   "source": [
    "# Module 1 : Exploring BM25 similarity and Semantic similarity\n",
    "\n",
    "Before we get started with Amazon OpenSearch and our search web app, let's explore some of the core concepts in search. Below, we'll demonstrate the different between algorithms for matching data using BM25 similarity (keyword matching) and Cosine similarity (sematnic vector matching)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae752e16",
   "metadata": {},
   "source": [
    "### 1. Upgrade PyTorch and restart Kernel\n",
    "\n",
    "Before we begin, we need to upgrade PyTorch and restart the notebook kernel. The following should take 2-3 minutes to complete, and you should see the following message::\"Successfully intalled torch-1.nn.n\".\n",
    "\n",
    "You may see a message with stating \"ERROR: pip's dependency resolver does not...\" - you can ignore this error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89acd366",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade torch==2.0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ab7c15c",
   "metadata": {},
   "source": [
    "Now we need to restart the kernel by running below cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f87b791b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display_html\n",
    "def restartkernel() :\n",
    "    display_html(\"<script>Jupyter.notebook.kernel.restart()</script>\",raw=True)\n",
    "restartkernel()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22739d58",
   "metadata": {},
   "source": [
    "Next, let's verify the version of Torch to ensure everything is up to date. The version should be 1.13.1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1dd8491",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "print(torch.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "311ff552",
   "metadata": {},
   "source": [
    "### 2. Install Pre-Requisites\n",
    "\n",
    "Before we can experiment with different searches, we need to install some required libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30e40333",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q transformers\n",
    "!pip install -U sentence-transformers rank_bm25\n",
    "\n",
    "from rank_bm25 import BM25Okapi\n",
    "from sklearn.feature_extraction import _stop_words\n",
    "import string\n",
    "from tqdm.autonotebook import tqdm\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72e83e3b",
   "metadata": {},
   "source": [
    "### 3. Create a sample dataset\n",
    "\n",
    "Let's now create a very simple dataset as an array of 4 questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52ac493e",
   "metadata": {},
   "outputs": [],
   "source": [
    "passages=[\"does this work with xbox?\",\n",
    "          \"Does the M70 work with Android phones?\", \n",
    "          \"does this work with iphone?\",\n",
    "          \"Can this work with an xbox \"\n",
    "         ]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09b8e8b6",
   "metadata": {},
   "source": [
    "### 4. Explore BM25 similarity \n",
    "\n",
    "Execute the following to explore BM25 similarity. First, we'll tokenize the data set, then use BM25 similarity to compare the phrase \"does this work with xbox?\" with our sample questions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af5409ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# split the sentence into words and remove stop words. For example setence \"does this work with xbox\" \n",
    "# will be converted into words \"does\", \"work\", \"xbox\"\n",
    "def bm25_tokenizer(text):\n",
    "    tokenized_doc = []\n",
    "    for token in text.lower().split():\n",
    "        token = token.strip(string.punctuation)\n",
    "\n",
    "        if len(token) > 0 and token not in _stop_words.ENGLISH_STOP_WORDS:\n",
    "            tokenized_doc.append(token)\n",
    "    return tokenized_doc\n",
    "\n",
    "\n",
    "tokenized_corpus = []\n",
    "for passage in tqdm(passages):\n",
    "    tokenized_corpus.append(bm25_tokenizer(passage))\n",
    "\n",
    "#get the BM25 score between the query \"does this work with xbox?\" and exiting 4 questions above. \n",
    "#If the score is high, it means BM25 think the two sentences are similiar.\n",
    "bm25 = BM25Okapi(tokenized_corpus)\n",
    "bm25_scores = bm25.get_scores(bm25_tokenizer(\"does this work with xbox?\"))\n",
    "\n",
    "all_sentence_combinations = []\n",
    "for i in range(len(bm25_scores)):\n",
    "    all_sentence_combinations.append([bm25_scores[i], i])\n",
    "\n",
    "#sort the score descending\n",
    "all_sentence_combinations = sorted(all_sentence_combinations, key=lambda x: x[0], reverse=True)\n",
    "\n",
    "# print the 4 sentences, tokens and the BM25 score with query \"does this work with xbox?\"\n",
    "# You can \"does this work with iphone?\" has high BM25 score with \"does this work with xbox?\" \n",
    "# even though the semantics meaning is different. While \"Can this work with an xbox\" has low BM25 score \n",
    "# with \"does this work with xbox?\" even though the two sentence has same semantic meaning. \n",
    "# This is the drawback of BM25.\n",
    "print(\"Top most similar pairs:\")\n",
    "for score, i in all_sentence_combinations[0:4]:\n",
    "    print(\"{} \\t {} \\t {:.4f}\".format(passages[i],bm25_tokenizer(passages[i]),bm25_scores[i]))\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ed5375b",
   "metadata": {},
   "source": [
    "### 5. Semantic Similarities\n",
    "\n",
    "\n",
    "Execute the following to explore semantic similarity with cosine similarity. In this code, we'll use the same dataset as above, but using cosine similarity. Compare the differences in how similarity is measured.\n",
    "\n",
    "The 'all-MiniLM-L6-v2' is a sentence transformer from HuggingFace that maps sentences & paragraphs to a 384 dimensional dense vector space. We'll use this library to translate our sample data set to a vector set. We'll then use util.cos_sim to provide a cosine similarity between each combination of records in the dataset. Finally, we'll print the cosine similarity score of the first record (\"does this work with xbox?\") with records in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17db6997",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sentence_transformers import SentenceTransformer, util\n",
    "model = SentenceTransformer('all-MiniLM-L6-v2')\n",
    "\n",
    "#Encode all sentences\n",
    "embeddings = model.encode(passages)\n",
    "\n",
    "#Compute cosine similarity between all pairs\n",
    "cos_sim = util.cos_sim(embeddings, embeddings)\n",
    "\n",
    "#cosine similarity score with query\n",
    "all_sentence_combinations = []\n",
    "for i in range(len(cos_sim)):\n",
    "    all_sentence_combinations.append([cos_sim[0][i], i])\n",
    "\n",
    "#Sort list by the highest cosine similarity score\n",
    "all_sentence_combinations = sorted(all_sentence_combinations, key=lambda x: x[0], reverse=True)\n",
    "\n",
    "# You see \"does this work with xbox?\" has the same meaning with query \"does this work with xbox?\", \n",
    "# so the semantic score is highest of course. \"Can this work with an xbox\" has similiar meaning \n",
    "# with \"does this work with xbox?\", semantic score ranked second.\n",
    "print(\"Top most similar pairs:\")\n",
    "for score, i in all_sentence_combinations[0:4]:\n",
    "    print(\"{} \\t {:.4f}\".format(passages[i],cos_sim[0][i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6c6e0a3",
   "metadata": {},
   "source": [
    "### 6. Compare the differences.\n",
    "\n",
    "As you can see, the similarity is significantly different, even with with a trivial data set. In particular, observe the differences between the text for \"Can this work with an xbox\" - with BM25 keyword search, the word \"can\" doesn't match the original question, and so it's given a low rating. Cosine similarity semantic search provides a much better match in this case.\n",
    "\n",
    "In this module, we've used fairly simple steps with a very small dataset to demonstrate the difference between BM25 and cosine similarity. In the following modules, we'll demonstrate these same concepts with using OpenSearch and a larger and more complex dataset."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_p310",
   "language": "python",
   "name": "conda_pytorch_p310"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}