UMAP_Instabiliity.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/drob-xx/UMAP_Instability/blob/main/UMAP_Instabiliity.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ael0M-9idukq"
      },
      "source": [
        "This notebook demonstrates the impact that UMAP instability can exert on BERTopic and shows that tuning HDBSCAN can both reduce that impact while improving BERTopic modeling overall.\n",
        "\n",
        "The steps below are:\n",
        "\n",
        "1. Setup environment\n",
        "1. Create ten TopicTuner models where each model has its own UMAP instance.\n",
        "1. Select two of these models for demonstration\n",
        "1. Generate BERTopic models based on the 1. 1. TopicTuner models using default BERTopic settings\n",
        "1. Apply tuned HDBSCAN settings to BERTopic Models\n",
        "\n",
        "The text used here is a 2000 article random sample taken from the [bbc-news](https://huggingface.co/datasets/SetFit/bbc-news) dataset, labeled for five categories - *business, entertainment, politics, sport* and *tech*.\n",
        "\n",
        "The tuning parameters were arrived at (elsewhere) by generating ten models using this data and using TopicTuner to search through hundreds of combinations of HDBSCAN's min_cluster_size and sample_size parameters to choose settings that produced similar results in all ten models for a given number of clusters.\n",
        "\n",
        "To ensure reproducability, UMAP's random_state parameter is set to the same value used for the models tuned in the previous step. Note, however that each of the UMAP instances is different from the others.\n",
        "\n",
        "This notebook assumes a colab environment. You can get a free colab account if you don't have one, or you can modify the notebook for your environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "F0XISFWvFB1t"
      },
      "outputs": [],
      "source": [
        "!pip install BERTopic"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HAq2rxZBdYN4"
      },
      "source": [
        "[TopicTuner](https://github.com/drob-xx/TopicTuner) is an HDBSCAN tuning solution for BERTopic. There is no current install but cloning the repository into the base directory will work."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WcVKG770Rhab"
      },
      "outputs": [],
      "source": [
        "!git clone https://github.com/drob-xx/TopicTuner.git /content/TopicTuner"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3DV31uix3Ej7"
      },
      "source": [
        "Then make sure it is on the path."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dMS57uFMrBJW"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "sys.path.insert(0,'/content/TopicTuner')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZpuWImOS6y7s"
      },
      "source": [
        "Clone the UMAP_Instability repo to access the needed data files."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6gQIUsBR5iEz"
      },
      "outputs": [],
      "source": [
        "!git clone https://github.com/drob-xx/UMAP_Instability.git /content/UMAP_Instability"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4PCSSoPDRqqR"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "from random import randrange\n",
        "import plotly.express as px\n",
        "import pickle\n",
        "from tqdm.notebook import tqdm\n",
        "\n",
        "from topictuner import TopicModelTuner as TMT\n",
        "\n",
        "from sklearn.metrics import adjusted_mutual_info_score\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from sklearn.feature_extraction import text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6w2AXQfX6_Wb"
      },
      "source": [
        "Load and save utility procedures."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mOy7RJe7c2OW"
      },
      "outputs": [],
      "source": [
        "def load(filepath):\n",
        "    with open(filepath, 'rb') as fp:\n",
        "        return pickle.load(fp)\n",
        "\n",
        "def save(var, filepath):\n",
        "    with open(filepath, 'wb') as fp:\n",
        "        return pickle.dump(var, fp)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TDax6-Oicotn"
      },
      "outputs": [],
      "source": [
        "BBC_ModelParamSettings = load('/content/UMAP_Instability/BBC_ModelParamSettings')\n",
        "BBC_UMAP_RandomStates = load('/content/UMAP_Instability/BBC_UMAP_RandomStates')\n",
        "bbcDataSets = load('/content/UMAP_Instability/bbcDataSets')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8R9csx2--ZrU"
      },
      "source": [
        "Create the embeddings. On a colab instance with gpu this should take less than two minutes. Longer without gpu."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J00EfIeSf1fQ"
      },
      "outputs": [],
      "source": [
        "bbcModels = {}\n",
        "# for ssize, aDF in tqdm(bbcDataSets.items()) :\n",
        "for _ in range(10) :\n",
        "  aModel = TMT(verbose=0)\n",
        "  aModel.createEmbeddings(bbcDataSets[2000]['text'].to_list())\n",
        "  bbcModels[2000] = aModel"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KUK4ybzd-vjA"
      },
      "source": [
        "Setup a container for the ten models."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eIs0_kSOweYe"
      },
      "outputs": [],
      "source": [
        "BBC_Not_Optimized_Models = {}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Oast4iX_-64t"
      },
      "source": [
        "Create ten TopicModelTuner models. \n",
        "\n",
        "1. Set UMAP's random state\n",
        "1. set the docs so we have an easy way to access them \n",
        "1. set the embeddings\n",
        "1. Call .reduce() which runs UMAP against the embeddings (takes about 2 minutes with gpu)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FrCJ1V54oI6A"
      },
      "outputs": [],
      "source": [
        "for sampleSize in tqdm([2000]) :\n",
        "  tmtmodels = []\n",
        "  for idx in tqdm(range(len(BBC_UMAP_RandomStates[sampleSize]))) :\n",
        "    tmtmodel = TMT(verbose=0)\n",
        "    tmtmodel.reducer_model.random_state = BBC_UMAP_RandomStates[sampleSize][idx]\n",
        "    tmtmodel.docs = bbcDataSets[sampleSize]['text'].to_list()\n",
        "    tmtmodel.embeddings = bbcModels[sampleSize]\n",
        "    tmtmodel.reduce()\n",
        "    tmtmodels.append(tmtmodel)\n",
        "  BBC_Not_Optimized_Models[sampleSize] = tmtmodels"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WhgkZtELARgB"
      },
      "source": [
        "We now have ten models generated against the same text and using ten different UMAP instances. Now we can compare each of the models to the others in the set to see what sort of variations occur due to UMAP instability.\n",
        "\n",
        "The following code will compare each pair of models in the set (100 comparisons) by running HDBSCAN against the UMAP reduction and comparing the resultant .labels_ - the classifications that BERTopic uses to create a topic model. The labels are compared using Sklearn's [adjusted_mutual_info_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_mutual_info_score.html) which gives a percentage match between two cluster classifications.\n",
        "\n",
        "We generate a heatmap to visualize the relationships."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GlQ4c5LDZTCO"
      },
      "outputs": [],
      "source": [
        "modelCompareResults = {}\n",
        "modelFigs = {}\n",
        "for setKey, modelSet in tqdm(BBC_Not_Optimized_Models.items()) :\n",
        "    csize = len(modelSet)\n",
        "    ComparedResults = np.zeros((csize,csize))\n",
        "    for row, model1 in enumerate(modelSet) :\n",
        "      for column, model2 in enumerate(modelSet) :\n",
        "        ComparedResults[row, column] = adjusted_mutual_info_score(model1.runHDBSCAN(),\n",
        "                                                                  model2.runHDBSCAN())\n",
        "    fig = px.imshow(ComparedResults, color_continuous_scale='RdBu_r', zmin=0, zmax=1)\n",
        "    fig.update_layout(\n",
        "            xaxis={'side': 'top'}, \n",
        "    )\n",
        "    modelFigs[setKey] = fig\n",
        "    modelCompareResults[setKey] = ComparedResults"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0emcOuVDFzyG"
      },
      "source": [
        "The heatmap shows how alike each model is from the others in the set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wdrDJ1Aaj6hb"
      },
      "outputs": [],
      "source": [
        "for fig in modelFigs.values() :\n",
        "  fig.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rDZLD35dZ94i"
      },
      "source": [
        "We can summarize these results:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6gLKLqkBfhDE"
      },
      "outputs": [],
      "source": [
        "txformedData = {}\n",
        "for idx, runData in modelCompareResults.items() :\n",
        "    vals = np.zeros(45)\n",
        "    count = 0\n",
        "    for x in range(10) :\n",
        "      for y in range(10) :\n",
        "        if (x <= y) :\n",
        "          continue\n",
        "        else :\n",
        "          vals[count] = runData[x,y]\n",
        "          count += 1\n",
        "    txformedData[idx] = vals\n",
        "\n",
        "pd.DataFrame(txformedData).describe()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TzxQPOpqY_2m"
      },
      "source": [
        "To provide a more detailed look on how UMAP instability effects these models we'll select two of the lesser correlated models - 4 and 9 - and see how these differences are reflected in BERTopic models.\n",
        "\n",
        "Each of the models has been created using BERTopic default settings. When we compare 4 and 9 we see that they are 81% correlated, below the 1st quartile cutoff of 83%."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Og5Kz61MKFd8"
      },
      "outputs": [],
      "source": [
        "model4 = BBC_Not_Optimized_Models[2000][4]\n",
        "model9 = BBC_Not_Optimized_Models[2000][9]\n",
        "adjusted_mutual_info_score(model4.runHDBSCAN(), model9.runHDBSCAN())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T90imXixase_"
      },
      "source": [
        "Now lets create BERTopic models using these parameters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RCu9gbHlygFO"
      },
      "outputs": [],
      "source": [
        "btmodels = {4:None, 9:None}\n",
        "custom_stop_words = ['mr', 'said']\n",
        "stop_words_list = text.ENGLISH_STOP_WORDS.union(custom_stop_words)\n",
        "\n",
        "vectorizer_model = CountVectorizer(ngram_range=(1, 2), stop_words=stop_words_list)\n",
        "\n",
        "for idx in btmodels.keys() :\n",
        "  btmodels[idx] = BBC_Not_Optimized_Models[2000][idx].getBERTopicModel(10, None)\n",
        "\n",
        "for idx, model in btmodels.items() :\n",
        "  model.umap_model = BBC_Not_Optimized_Models[2000][idx].reducer_model\n",
        "  model.vectorizer_model = vectorizer_model\n",
        "  model.fit_transform(bbcDataSets[2000]['text'].to_list(), embeddings=BBC_Not_Optimized_Models[2000][idx].embeddings)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8JyvNv8nbENa"
      },
      "source": [
        "Of course the TMT model and the BERTopic Models produce the same results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4QlnGaeULEKL"
      },
      "outputs": [],
      "source": [
        "adjusted_mutual_info_score(model4.runHDBSCAN(), btmodels[4].topics_)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "35Vu9I1ibMcf"
      },
      "source": [
        "Taking a look at the topics that BERTopic has determined."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_vbbKwqS26kS"
      },
      "outputs": [],
      "source": [
        "for model in btmodels.values() :\n",
        "  print(model.get_topic_info())\n",
        "  print('')\n",
        "  print('-----------------------------------------------')\n",
        "  print('')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rF9IeYYCbi3P"
      },
      "source": [
        "That's a lot of topics. Let's reduce them."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EcyIE4yg7bsi"
      },
      "outputs": [],
      "source": [
        "for idx, model in btmodels.items() :\n",
        "  model.reduce_topics(bbcDataSets[2000]['text'].to_list(), 5)\n",
        "  for topic in model.generate_topic_labels(nr_words=10, topic_prefix=False, separator=' ') :\n",
        "    print(topic)\n",
        "  print('')\n",
        "  print('----------------------------------')\n",
        "  print('')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BZosRyQIbmn0"
      },
      "source": [
        "More comprehensible. But now lets look at how correlated the models are."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dlwnZ7pvLjGs"
      },
      "outputs": [],
      "source": [
        "adjusted_mutual_info_score(btmodels[4].topics_,btmodels[9].topics_)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uyoMC4fxfl3I"
      },
      "source": [
        "Looking at the topic vocabularies we see that four of the five topics line up very closely to one another. However, the fifth - \n",
        "\n",
        "- growth economy bank year economic sales prices rates rise dollar\n",
        "\n",
        "and\n",
        "\n",
        "- people mobile phone technology broadband tv digital search phones games\n",
        "\n",
        "do not."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PXlYyq1lgkXy"
      },
      "source": [
        "We can also see the differences in the embeddings visualizations they produce."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "RPfoHmGK_1d5"
      },
      "outputs": [],
      "source": [
        "embedding_vizs = []\n",
        "for idx, model in tqdm(btmodels.items()) :\n",
        "  BBC_Not_Optimized_Models[2000][idx].createVizReduction()\n",
        "  embedding_vizs.append(model.visualize_documents(bbcDataSets[2000]['text'].to_list(),\n",
        "                            reduced_embeddings=BBC_Not_Optimized_Models[2000][idx].viz_reducer.embedding_))\n",
        "for viz in embedding_vizs :\n",
        "  viz.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1FX3qK2Thgm4"
      },
      "source": [
        "Let's see what happens when we use optimized UMAP settings (calculated elsewhere) and run summaries on their classifications."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QsSL5UyBIlWe"
      },
      "outputs": [],
      "source": [
        "tunedModels = {}\n",
        "for idx in [4, 9] :\n",
        "  tunedModels[idx] = (BBC_Not_Optimized_Models[2000][idx]) \n",
        "  tunedModels[idx].best_cs = BBC_ModelParamSettings[2000][idx][0]\n",
        "  tunedModels[idx].best_ss = BBC_ModelParamSettings[2000][idx][1]\n",
        "for model in tunedModels.values() :\n",
        "  print(model.best_cs, model.best_ss)\n",
        "  print(pd.Series(model.runHDBSCAN(model.best_cs, model.best_ss)).value_counts())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G2Hldi83h3_E"
      },
      "source": [
        "During the optimization process we don't just make the model more efficient (reducing -1 results) but we also can often choose a specific number of clusters.\n",
        "\n",
        "We can see our metric thinks these models are very aligned."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SqZf9HWDKLuU"
      },
      "outputs": [],
      "source": [
        "adjusted_mutual_info_score(tunedModels[4].runHDBSCAN(), tunedModels[9].runHDBSCAN())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zc8su7s5if0J"
      },
      "source": [
        "Next we can create BERTopic models based on these optimized results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fBtJCpfYSnft"
      },
      "outputs": [],
      "source": [
        "btmodels2 = {}\n",
        "\n",
        "for idx, model in tqdm(tunedModels.items()) :\n",
        "  btmodel = model.getBERTopicModel()\n",
        "  btmodel.umap_model = model.reducer_model\n",
        "  btmodel.vectorizer_model = vectorizer_model\n",
        "  btmodel.fit_transform(bbcDataSets[2000]['text'].to_list(), model.embeddings)\n",
        "  btmodels2[idx] = btmodel"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bXArYmLqjEBt"
      },
      "source": [
        "Let's look at some scored differences. The first is how the first two BERTopic models compare to one another. The second is how different the \"before\" models are from the \"after\" models and finally, how correlated the \"after\" models are to each other. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PgczSn1nYRpa"
      },
      "outputs": [],
      "source": [
        "print('BERTopic \"before\" models compared with each other')\n",
        "print(adjusted_mutual_info_score(btmodels[4].topics_, btmodels[9].topics_))\n",
        "print('')\n",
        "print('BERTopic \"after\" models vs. \"before\" models')\n",
        "print(adjusted_mutual_info_score(btmodels2[4].topics_, btmodels[4].topics_))\n",
        "print(adjusted_mutual_info_score(btmodels2[9].topics_, btmodels[9].topics_))\n",
        "print('')\n",
        "print('BERTopic \"after\" models compared with each other')\n",
        "print(adjusted_mutual_info_score(btmodels2[4].topics_, btmodels2[9].topics_))\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oIKfZAPJY8-z"
      },
      "outputs": [],
      "source": [
        "for model in btmodels2.values() :\n",
        "  for topic in model.generate_topic_labels(nr_words=10, topic_prefix=False, separator=' ') :\n",
        "    print(topic)\n",
        "  print('')\n",
        "  print('----------------------------------')\n",
        "  print('')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JLvvqqIwkDGH"
      },
      "source": [
        "Despite the fact that each of these models has a different UMAP instance they are almost the same.\n",
        "\n",
        "The takeaway is that UMAP instability can have a very pronounced effect on model consistency, unless the model is tuned. Tuning also has the added advantage of significantly reducing the number of uncategorized documents."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "HNBfgU7gTqP_"
      },
      "outputs": [],
      "source": [
        "docs = bbcDataSets[2000]['text'].to_list()\n",
        "viz_embeddings = tunedModels[4].viz_reducer.embedding_\n",
        "tuned_btmodel4 = btmodels2[4]\n",
        "tuned_btmodel4.visualize_documents(docs, reduced_embeddings=viz_embeddings).show()"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "provenance": [],
      "mount_file_id": "1TGu_pei8Y7OGYmohhKaLb2Yho1be-xWg",
      "authorship_tag": "ABX9TyPdFYhIwHaEozg6YuEPLmsS",
      "include_colab_link": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}