Monzo Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 1 - Summary of blog posts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We start this exploration with a summary of all of the blog posts on Monzo as of June 4th, 2019. We scraped them using a python script, the code for which is available at https://github.com/emielver"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of this section is to provide an insight into the nature of the blog posts, and hopefully to see some interesting results. We will look at when the blog posts were written, who wrote them, and in what capacity each author was working when they wrote them. If we see any ideas that we deem worth exploring, we will attempt to do so."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by defining some user-created classes, which will aid in our analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Post:\n",
    "    \n",
    "    def __init__(self, author, position, title, date, text):\n",
    "        self.author = author\n",
    "        self.position = position\n",
    "        self.title = title\n",
    "        self.date = date\n",
    "        self.text = text\n",
    "\n",
    "    def get_author(self):\n",
    "        return self.author\n",
    "\n",
    "    def get_position(self):\n",
    "        return self.position\n",
    "    \n",
    "    def get_title(self):\n",
    "        return self.title\n",
    "\n",
    "    def get_date(self):\n",
    "        return self.date\n",
    "\n",
    "    def get_text(self):\n",
    "        return self.text\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Author:\n",
    "\n",
    "    def __init__(self, name, position):\n",
    "        self.name = name\n",
    "        self.positions = [position]\n",
    "        self.gender = ''\n",
    "        self.blogs = []\n",
    "\n",
    "\n",
    "    def add_blog(self, blog):\n",
    "        self.blogs.append(blog)\n",
    "\n",
    "    def add_position(self, position):\n",
    "        self.positions.append(position)\n",
    "\n",
    "    def set_gender(self, gender):\n",
    "        self.gender = gender\n",
    "\n",
    "    def get_name(self):\n",
    "        return self.name\n",
    "\n",
    "    def get_cmp_name(self):\n",
    "        return \"\".join(self.name.lower().split())\n",
    "\n",
    "    def get_position(self):\n",
    "        return self.position[0]\n",
    "\n",
    "    def get_cmp_position(self):\n",
    "        return \"\".join(self.positions[0].lower().split())\n",
    "\n",
    "    def get_all_positions(self):\n",
    "        return self.positions\n",
    "\n",
    "    def get_blogs(self):\n",
    "        return self.blogs\n",
    "\n",
    "    def get_number_blogs(self):\n",
    "        return len(self.blogs)\n",
    "\n",
    "    def get_gender(self):\n",
    "        return self.gender\n",
    "\n",
    "    def cmp_strings(self, str1, str2):\n",
    "        return \"\".join(str1.lower()) == \"\".join(str2.lower())\n",
    "\n",
    "    def has_name(self, potential):\n",
    "        return self.cmp_strings(self.name, potential)\n",
    "\n",
    "    def has_position(self, potential):\n",
    "        for position in self.positions:\n",
    "            if self.cmp_strings(position, potential):\n",
    "                return True\n",
    "        return False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by importing the necessary libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "        <script type=\"text/javascript\">\n",
       "        window.PlotlyConfig = {MathJaxConfig: 'local'};\n",
       "        if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}\n",
       "        if (typeof require !== 'undefined') {\n",
       "        require.undef(\"plotly\");\n",
       "        requirejs.config({\n",
       "            paths: {\n",
       "                'plotly': ['https://cdn.plot.ly/plotly-latest.min']\n",
       "            }\n",
       "        });\n",
       "        require(['plotly'], function(Plotly) {\n",
       "            window._Plotly = Plotly;\n",
       "        });\n",
       "        }\n",
       "        </script>\n",
       "        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import glob\n",
    "import plotly\n",
    "import plotly.plotly as py\n",
    "import plotly.graph_objs as go\n",
    "import plotly.figure_factory as ff\n",
    "from scipy import stats\n",
    "import collections\n",
    "import numpy as np\n",
    "import datetime\n",
    "import colorlover as cl\n",
    "import pickle\n",
    "import operator\n",
    "import pandas as pd\n",
    "from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\n",
    "init_notebook_mode(connected=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "monzo_blue = 'rgb(20, 70, 115)'\n",
    "grey = 'rgb(204,204,204)'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We set up our API credentials for Plotly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# user, key = open('Secret/secret.txt', 'r').readlines()\n",
    "# plotly.tools.set_credentials_file(username=user.strip(), api_key=key.strip())\n",
    "# user = ''\n",
    "# key = ''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define our user defined functions (UDFs) which allow us to store all information regarding blogs and authors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_blogs():\n",
    "    blogs = []\n",
    "    file_names = glob.glob('Data/*.txt')\n",
    "    for file_name in file_names:\n",
    "        lines = open(file_name, 'r').readlines()\n",
    "        title = lines[0].split(\":\")[1].strip()\n",
    "        author = lines[1].split(\":\")[1].strip()\n",
    "        position = lines[2].split(\":\")[1].strip()\n",
    "        date = lines[3].split(\":\")[1].strip()\n",
    "        text = lines[4:]\n",
    "        if len(author.split(\" \")) > 3:\n",
    "            print(file_name)\n",
    "        new_blog = Post(author, position, title, date, text)\n",
    "        blogs.append(new_blog)\n",
    "##        print(\"Added %s to the list of blogs.\" % (title)\n",
    "##              + \"Written by %s, working as %s, on %s\"\n",
    "##              % (author, position, date))\n",
    "    return blogs\n",
    "\n",
    "def get_authors(blogs):\n",
    "    authors = []\n",
    "    for blog in blogs:\n",
    "        added = False\n",
    "        author_name = blog.get_author()\n",
    "        author_position = blog.get_position()\n",
    "        for author in authors:\n",
    "            if author.has_name(author_name):\n",
    "                if not author.has_position(author_position):\n",
    "                    author.add_position(author_position)\n",
    "                author.add_blog(blog)\n",
    "                added = True\n",
    "        if not added:\n",
    "            new_author = Author(author_name, author_position)\n",
    "            new_author.add_blog(blog)\n",
    "            authors.append(new_author)\n",
    "            \n",
    "            \n",
    "    return authors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now read in the blogs and the authors, and sort the authors based on how many blog posts they have written in descending order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "blogs = get_blogs()\n",
    "authors = get_authors(blogs)\n",
    "authors.sort(key = lambda x: x.get_number_blogs(), reverse = True)\n",
    "# for author in authors:\n",
    "#     print(\"Author %s wrote %d blogs\"\n",
    "#           % (author.get_name(), author.get_number_blogs()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We manually went through each other to determine their gender. We did this on the basis of their associated picture on their blog posts, however there may be some margin for error. Additionally, we gave Team Monzo a gender of 'None', as we do not know who on the Monzo team actually wrote the blog post."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'authors.pkl'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-8-e1c89498fd50>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mauthor_genders\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpickle\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'authors.pkl'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'rb'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mauthor\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mauthors\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgender\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mauthor_genders\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mauthor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhas_name\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m             \u001b[0mauthor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_gender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgender\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'authors.pkl'"
     ]
    }
   ],
   "source": [
    "author_genders = pickle.load(open('authors.pkl', 'rb'))\n",
    "for author in authors:\n",
    "    for name, gender in author_genders:\n",
    "        if author.has_name(name):\n",
    "            author.set_gender(gender)\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the data has been processed and prepared, we look into the first question we want to ask of our data set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How active have the blog posts been over the years?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.7/site-packages/IPython/core/display.py:689: UserWarning:\n",
      "\n",
      "Consider using IPython.display.IFrame instead\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/2.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "years = {}\n",
    "for blog in blogs:\n",
    "    year = blog.get_date().split(\" \")[2]\n",
    "    if year in years:\n",
    "        years[year] += 1\n",
    "    else:\n",
    "        years[year] = 1\n",
    "intermediate_years = sorted(years.items(), key=lambda kv: kv[0])\n",
    "years = collections.OrderedDict(intermediate_years)\n",
    "labels = []\n",
    "values = []\n",
    "for key, value in years.items():\n",
    "    labels.append(key)\n",
    "    values.append(value)\n",
    "\n",
    "\n",
    "data = [go.Bar(x=labels,y=values,\n",
    "               marker = {'color' : monzo_blue})]\n",
    "layout = go.Layout(\n",
    "    title=go.layout.Title(\n",
    "        text='Monzo blog posts per year',\n",
    "        x=0\n",
    "    ),\n",
    "    xaxis=go.layout.XAxis(\n",
    "        title=go.layout.xaxis.Title(\n",
    "            text='Year',\n",
    "        )\n",
    "    ),\n",
    "    yaxis=go.layout.YAxis(\n",
    "        title=go.layout.yaxis.Title(\n",
    "            text='Blog posts',\n",
    "        )\n",
    "    ))\n",
    "fig = go.Figure(data= data, layout=layout)\n",
    "py.iplot(fig, filename='basic-bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we obviously see an increasing trend in the number of blog posts per year. We also have to note that this analysis was done on all blog posts up to and including the 4th of June. We also know that the first blog post was made on the 1st of June, 2015, so the number of blog posts in 2015 is also not completely representative. This then begs the question..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How many blog posts do we expect in 2019?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### According to a simple linear trend"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "xi = [float(labels[i]) for i in range(4,1,-1)]\n",
    "y = [float(values[i]) for i in range(4,1,-1)]\n",
    "slope, intercept, r_value, p_value, std_err = stats.linregress(xi, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "According to a linear fit, we find a slope of 10.000000 and an intercept of -20031.333333, with an R^2 of 0.151509\n"
     ]
    }
   ],
   "source": [
    "print(\"According to a linear fit, we find a slope of %f and an intercept of %f, with an R^2 of %f\"\n",
    "     % (slope, intercept, r_value))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having all the relevant coefficients, we see what our prediction would be for 2019"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "158.66666666666788"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "intercept + slope * 2019"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That isn't very good, but we also only have an R^2 of 0.15, and we can see just by eyeballing the graph that a linear trend is not present. It looks much more like an exponential trend, or some sort of higher order polynomial. We first start with a quadratic trend, and then explore the explanatory power of an exponential trend."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### According to a quadratic trend"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "coefs = np.polyfit(xi, y, deg = 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "120.99999982118607"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coefs[0]* 2019 **2 + coefs[1] * 2019 + coefs[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Maybe a quadratic equation also doesn't help. Let's look at an exponential trend. One way we can do this is by taking the logarithm of the y values, and then fitting a linear trend. We can then reverse this transformation to get the expected number of blog posts in 2019 according to an exponential trend."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### According to an exponential trend"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "logged_y = np.log(y)\n",
    "slope, intercept, r_value, p_value, std_err = stats.linregress(xi, logged_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "According to an exponential fit, we find a slope of 0.090335 and an intercept of -177.355207, with an R^2 of 0.216316\n"
     ]
    }
   ],
   "source": [
    "print(\"According to an exponential fit, we find a slope of %f and an intercept of %f, with an R^2 of %f\"\n",
    "     % (slope, intercept, r_value))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plugging in 2019 and taking the exponential to this result yields the expected number of blog posts in 2019."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "153.1147110362509"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.exp(intercept + slope*2019)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we do not find a very realistic result, however this is more promising than previous results, suggesting that there is stronger evidence for an exponential trend. We can also chalk up the lacklustre results to a lack of data, since we are only looking over 3 complete years. Hopefully we find better results once we have a larger data set to look at. Therefore, the logical next step is to look at the blog posts per month."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Which month has the highest number of blog posts, and how does this vary over the months?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/2.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "months = {\"Jan\" : 0, \"Feb\" : 0, \"Mar\" : 0, \"Apr\" : 0, \n",
    "         \"May\" : 0, \"Jun\" : 0, \"Jul\" : 0, \"Aug\" : 0, \n",
    "         \"Sep\" : 0, \"Oct\" : 0, \"Nov\" : 0,\n",
    "         \"Dec\" : 0}\n",
    "for blog in blogs:\n",
    "    month = blog.get_date().split(\" \")[1]\n",
    "    months[month] += 1\n",
    "labels = []\n",
    "values = []\n",
    "for key, value in months.items():\n",
    "    labels.append(key)\n",
    "    values.append(value)\n",
    "\n",
    "\n",
    "data = [go.Bar(x=labels,y=values, marker = {'color' : monzo_blue})]\n",
    "layout = go.Layout(\n",
    "    title=go.layout.Title(\n",
    "        text='Monzo blog posts per month',\n",
    "        x=0\n",
    "    ),\n",
    "    xaxis=go.layout.XAxis(\n",
    "        title=go.layout.xaxis.Title(\n",
    "            text='Month',\n",
    "        )\n",
    "    ),\n",
    "    yaxis=go.layout.YAxis(\n",
    "        title=go.layout.yaxis.Title(\n",
    "            text='Blog posts',\n",
    "        )\n",
    "    ))\n",
    "fig = go.Figure(data= data, layout=layout)\n",
    "py.iplot(fig, filename='basic-bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we don't get many insights. We see that in May has the highest number of blog posts, and that the first 5 months on average have more blog posts than the last 7 months. An obvious explanation for this could be that this analysis was carried out at the beginning of June in 2019, and so we are still missing the data for June onwards in 2019. We can quickly remedy this by only looking up until December 2018, which yields the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/2.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "months = {\"Jan\" : 0, \"Feb\" : 0, \"Mar\" : 0, \"Apr\" : 0, \n",
    "         \"May\" : 0, \"Jun\" : 0, \"Jul\" : 0, \"Aug\" : 0, \n",
    "         \"Sep\" : 0, \"Oct\" : 0, \"Nov\" : 0,\n",
    "         \"Dec\" : 0}\n",
    "for blog in blogs:\n",
    "    day, month, year = blog.get_date().split(\" \")\n",
    "    if year != '2019':\n",
    "        months[month] += 1\n",
    "labels = []\n",
    "values = []\n",
    "for key, value in months.items():\n",
    "    labels.append(key)\n",
    "    values.append(value)\n",
    "\n",
    "\n",
    "data = [go.Bar(x=labels,y=values, marker = {'color' : monzo_blue})]\n",
    "layout = go.Layout(\n",
    "    title=go.layout.Title(\n",
    "        text='Monzo blog posts per month (excluding 2019)',\n",
    "        x=0\n",
    "    ),\n",
    "    xaxis=go.layout.XAxis(\n",
    "        title=go.layout.xaxis.Title(\n",
    "            text='Month',\n",
    "        )\n",
    "    ),\n",
    "    yaxis=go.layout.YAxis(\n",
    "        title=go.layout.yaxis.Title(\n",
    "            text='Blog posts',\n",
    "        )\n",
    "    ))\n",
    "fig = go.Figure(data= data, layout=layout)\n",
    "py.iplot(fig, filename='basic-bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks a little more like we expected. We now have a bias against the first 5 months, because Monzo's first blog post was in June 2015. If we wanted a more fair representation, we would only look at blog posts in full years, so 2016 until 2018. However, considering there were only 16 blog posts in 2015 in total, we do not expect much to change. Furthermore, it is quite straightforward to adapt the above code to exclude 2015 along with 2019.\n",
    "\n",
    "Instead of looking at the monthly figures aggregated over the years, we can instead create month/year pairings, and see the natural evolution over time a bit better. We do this in the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/2.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "month_names = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']\n",
    "names = month_names[:]\n",
    "months = {}\n",
    "for year in range(2015,2020):\n",
    "    if year == 2015:\n",
    "        names = month_names[5:]\n",
    "    elif year == 2019:\n",
    "        names = month_names[:5]\n",
    "    else:\n",
    "        names = month_names\n",
    "    for name in names:\n",
    "        month = name + \" \" + str(year)\n",
    "        months[month] = 0\n",
    "for blog in blogs:\n",
    "    month = \" \".join(blog.get_date().split(\" \")[1:])\n",
    "    if month not in months:\n",
    "        pass\n",
    "    else:\n",
    "        months[month] += 1\n",
    "\n",
    "labels = []\n",
    "values = []\n",
    "for key, value in months.items():\n",
    "    labels.append(key)\n",
    "    values.append(value)\n",
    "\n",
    "\n",
    "data = [go.Bar(x=labels,y=values, marker = {'color' : monzo_blue})]\n",
    "layout = go.Layout(\n",
    "    title=go.layout.Title(\n",
    "        text='Monzo blog posts per month/year combination',\n",
    "        x=0\n",
    "    ),\n",
    "    xaxis=go.layout.XAxis(\n",
    "        title=go.layout.xaxis.Title(\n",
    "            text='Month',\n",
    "        )\n",
    "    ),\n",
    "    yaxis=go.layout.YAxis(\n",
    "        title=go.layout.yaxis.Title(\n",
    "            text='Blog posts',\n",
    "        )\n",
    "    ))\n",
    "fig = go.Figure(data= data, layout=layout)\n",
    "py.iplot(fig, filename='basic-bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see a very clear progression. August 2018 does seem to peak sharply above the rest, with a reason for this being that August was the deadline for \"The Big List\", initially published in May of 2018 (https://monzo.com/blog/2018/05/22/making-monzo-better/). \"The Big List\" set up a list of features Monzo was determined to introduce within three months, and by August 2018 they had managed to implement 11 out of the 14 features (https://monzo.com/blog/2018/08/22/end-of-the-big-list/). It is fair to say this can largely explain the spike in the number of blogs posted in August, and so if Monzo wants to increase its blog posts again it may be helpful to introduce another wishlist of features with a deadline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What happens when we aggregate to quarters of the year? Do we find other interesting results?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/2.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "months = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']\n",
    "quarters = ['Q1', 'Q2', 'Q3', 'Q4']\n",
    "names = quarters[:]\n",
    "quarter_year = {}\n",
    "for year in range(2015,2020):\n",
    "    if year == 2015:\n",
    "        names = quarters[1:]\n",
    "    elif year == 2019:\n",
    "        names = quarters[:2]\n",
    "    else:\n",
    "        names = quarters\n",
    "    for quarter in names:\n",
    "        quarter = quarter + \" \" + str(year)\n",
    "        quarter_year[quarter] = 0\n",
    "quarter = \"\"\n",
    "for blog in blogs:\n",
    "    month, year = blog.get_date().split(\" \")[1:]\n",
    "    if month in months[:3]:\n",
    "        quarter = quarters[0]\n",
    "    elif month in months[3:6]:\n",
    "        quarter = quarters[1]\n",
    "    elif month in months[6:9]:\n",
    "        quarter = quarters[2]\n",
    "    else:\n",
    "        quarter = quarters[3]\n",
    "    name = quarter + \" \" + year\n",
    "    quarter_year[name] += 1\n",
    "\n",
    "labels = []\n",
    "values = []\n",
    "for key, value in quarter_year.items():\n",
    "    labels.append(key)\n",
    "    values.append(value)\n",
    "\n",
    "\n",
    "data = [go.Bar(x=labels,y=values, marker = {'color' : monzo_blue})]\n",
    "layout = go.Layout(\n",
    "    title=go.layout.Title(\n",
    "        text='Monzo blog posts per quarter/year combination',\n",
    "        x=0\n",
    "    ),\n",
    "    xaxis=go.layout.XAxis(\n",
    "        title=go.layout.xaxis.Title(\n",
    "            text='Quarter',\n",
    "        )\n",
    "    ),\n",
    "    yaxis=go.layout.YAxis(\n",
    "        title=go.layout.yaxis.Title(\n",
    "            text='Blog posts',\n",
    "        )\n",
    "    ))\n",
    "fig = go.Figure(data= data, layout=layout)\n",
    "py.iplot(fig, filename='basic-bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We obviously see the same general increasing pattern as we did using monthly data, but would not be able to immediately identify the spike in blog posts in Q3 of 2018, like we could using monthly data. We do, however, see a surge in blog post volume in 2015 as the company is starting up, with a fairly equal distribution in 2016. In 2017 we see a general uptick, with Q1 blog posts doubling in volume compared to the previous year. In 2018, we see a ramping up in Q2 and Q3, which is sustained in Q4. This could again be as a result of the introduction and follow-through of \"The Big List\" that Monzo introduced. \n",
    "\n",
    "Now we can also look at if there are distinct differences in the number of blog posts per day of the month. This should be (largely) unaffected by incomplete years, and so we should be able to run this analysis over the whole sample."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Looking at day of the month"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/2.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "days = {}\n",
    "for blog in blogs:\n",
    "    day = blog.get_date().split(\" \")[0]\n",
    "    if day in days:\n",
    "        days[day] += 1\n",
    "    else:\n",
    "        days[day] = 1\n",
    "intermediate_days = sorted(days.items(), key=lambda kv: int(kv[0]))\n",
    "days = collections.OrderedDict(intermediate_days)\n",
    "labels = []\n",
    "values = []\n",
    "for key, value in days.items():\n",
    "    labels.append(key)\n",
    "    values.append(value)\n",
    "\n",
    "\n",
    "data = [go.Bar(x=labels,y=values, marker = {'color' : monzo_blue})]\n",
    "layout = go.Layout(\n",
    "    title=go.layout.Title(\n",
    "        text='Monzo blog posts per day of the month',\n",
    "        x=0\n",
    "    ),\n",
    "    xaxis=go.layout.XAxis(\n",
    "        title=go.layout.xaxis.Title(\n",
    "            text='Day',\n",
    "        )\n",
    "    ),\n",
    "    yaxis=go.layout.YAxis(\n",
    "        title=go.layout.yaxis.Title(\n",
    "            text='Blog posts',\n",
    "        )\n",
    "    ))\n",
    "fig = go.Figure(data = data, layout=layout)\n",
    "py.iplot(fig, filename='basic-bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks pretty normal, but we can't infer too much from this. Obviously we expect the number of blog posts on the 31st to be low, since this is the least commonly occuring day of the month. However, it might be more interesting to look at the days of the week."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Which day of the week is most productive?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "How to manage money when you’re terrible with money was posted on Sunday, 3 Mar 2019.\n",
      "Why Monzo Cards Aren't Working Today was posted on Sunday, 5 Mar 2017.\n",
      "#mondohack was posted on Sunday, 22 Nov 2015.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/2.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "days_of_week = {\"Monday\" : 0, \"Tuesday\" : 0, \"Wednesday\" : 0,\n",
    "               \"Thursday\" : 0, \"Friday\" : 0, \"Saturday\" : 0, \n",
    "               \"Sunday\" : 0}\n",
    "for blog in blogs:\n",
    "    day = datetime.datetime.strptime(blog.get_date(), '%d %b %Y').strftime('%A')\n",
    "    days_of_week[day] += 1\n",
    "    if day == 'Sunday' or day == 'Saturday':\n",
    "        print(\"%s was posted on %s, %s.\" \n",
    "              % (blog.get_title(), day, blog.get_date()))\n",
    "labels = []\n",
    "values = []\n",
    "for key, value in days_of_week.items():\n",
    "    labels.append(key)\n",
    "    values.append(value)\n",
    "\n",
    "\n",
    "data = [go.Bar(x=labels,y=values, marker = {'color': monzo_blue})]\n",
    "layout = go.Layout(\n",
    "    title=go.layout.Title(\n",
    "        text='Monzo blog posts per day of the week',\n",
    "        x=0\n",
    "    ),\n",
    "    xaxis=go.layout.XAxis(\n",
    "        title=go.layout.xaxis.Title(\n",
    "            text='day',\n",
    "        )\n",
    "    ),\n",
    "    yaxis=go.layout.YAxis(\n",
    "        title=go.layout.yaxis.Title(\n",
    "            text='Blog posts',\n",
    "        )\n",
    "    ))\n",
    "fig = go.Figure(data= data, layout=layout)\n",
    "py.iplot(fig, filename='basic-bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Surprise, surprise, Monzonauts don't like to work on weekends\n",
    "If they have to then they post on Sundays. However, this only happens once every two years, and since it has already happened in 2019, no one should be posting on a Sunday until 2021."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# So who is actually writing all of these blogposts?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/2.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels = []\n",
    "values = []\n",
    "for author in authors[:10]:\n",
    "    labels.append(author.get_name())\n",
    "    values.append(author.get_number_blogs())\n",
    "labels.append(\"Other\")\n",
    "rest = 0\n",
    "for author in authors[10:]:\n",
    "    rest += author.get_number_blogs()\n",
    "values.append(rest)\n",
    "\n",
    "colours = [monzo_blue for x in range(10)]\n",
    "colours.append(grey)\n",
    "\n",
    "data = [go.Bar(x=labels,y=values, marker = {'color' : colours})]\n",
    "layout = go.Layout(\n",
    "    title=go.layout.Title(\n",
    "        text='Monzo blog posts per author',\n",
    "        x=0\n",
    "    ),\n",
    "    xaxis=go.layout.XAxis(\n",
    "        title=go.layout.xaxis.Title(\n",
    "            text='Author',\n",
    "        )\n",
    "    ),\n",
    "    yaxis=go.layout.YAxis(\n",
    "        title=go.layout.yaxis.Title(\n",
    "            text='Number of blog posts',\n",
    "        )\n",
    "    ))\n",
    "fig = go.Figure(data= data, layout=layout)\n",
    "py.iplot(fig, filename='basic-bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Look into gender balance of blog posts over time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/17.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gender_count = {'female' : 0, 'male' : 0, 'None' : 0}\n",
    "for author in authors:\n",
    "    gender_count[author.get_gender()] += author.get_number_blogs()\n",
    "    \n",
    "labels = []\n",
    "values = []\n",
    "for key, value in gender_count.items():\n",
    "    labels.append(key)\n",
    "    values.append(value)\n",
    "colours = [monzo_blue]\n",
    "colours.append(grey)\n",
    "trace = go.Bar(x = labels[:2], y = values[:2], \n",
    "               marker = {'color' : colours},)\n",
    "py.iplot([trace], filename = 'gender_bar_chart')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/15.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_labels = []\n",
    "all_values = []\n",
    "gender_count = {}\n",
    "for author in authors:\n",
    "    gender = author.get_gender()\n",
    "    authors_blogs = author.get_blogs()\n",
    "    for blog in authors_blogs:\n",
    "        year = blog.get_date().split(\" \")[2]\n",
    "        gender_year = gender + \" \" + str(year)\n",
    "        if gender_year in gender_count:\n",
    "            gender_count[gender_year] += 1\n",
    "        else:\n",
    "            gender_count[gender_year] = 1\n",
    "\n",
    "years = [2015, 2016, 2017, 2018, 2019]\n",
    "for year in years:\n",
    "    all_labels.append([key.split(\" \")[0] for key, value in gender_count.items() if key.split(\" \")[1] == str(year)])\n",
    "    all_values.append([value for key, value in gender_count.items() if key.split(\" \")[1] == str(year)])\n",
    "\n",
    "trace1 = go.Bar(\n",
    "    y=['2015', '2016', '2017', '2018', '2019'],\n",
    "    x=[all_values[0][0], all_values[1][0], all_values[2][1],\n",
    "       all_values[3][1], all_values[4][1]],\n",
    "    name='Males',\n",
    "    orientation = 'h',\n",
    "    marker = dict(\n",
    "        color = grey,\n",
    "        line = dict(\n",
    "            color = grey,\n",
    "            width = 3)\n",
    "    )\n",
    ")\n",
    "trace2 = go.Bar(\n",
    "    y=['2015', '2016', '2017', '2018', '2019'],\n",
    "    x=[all_values[0][1], all_values[1][1], all_values[2][0],\n",
    "       all_values[3][0], all_values[4][0]],\n",
    "    name='Females',\n",
    "    orientation = 'h',\n",
    "    marker = dict(\n",
    "        color = monzo_blue,\n",
    "        line = dict(\n",
    "            color = monzo_blue,\n",
    "            width = 3)\n",
    "    )\n",
    ")\n",
    "\n",
    "data = [trace1, trace2]\n",
    "layout = go.Layout(\n",
    "    barmode='stack',\n",
    "    title = \"Absolute Male/Female split of authorship\"\n",
    ")\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "py.iplot(fig, filename='marker-h-bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can see a definitive increase in the number of posts made by women over the last couple of years. To put things more into perspective, we can look at the change in the percentage of total blog posts contributed by women over the last five years."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/2.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "female_ratio = []\n",
    "for i in range(len(years)):\n",
    "    for j in range(len(all_labels[i])):\n",
    "        if all_labels[i][j] == 'female':\n",
    "            break\n",
    "    females = all_values[i][j]\n",
    "    total = sum([int(x) for x in all_values[i]])\n",
    "    female_ratio.append((females/total) * 100)\n",
    "    \n",
    "trace0 = go.Scatter(\n",
    "    x = years,\n",
    "    y = female_ratio,\n",
    "    mode = 'lines+markers',\n",
    "    name = 'lines'\n",
    ")\n",
    "data = [trace0]\n",
    "layout = go.Layout(\n",
    "    title=go.layout.Title(\n",
    "        text='Percentage of female authored blog posts per year',\n",
    "        x=0\n",
    "    ),\n",
    "    xaxis=go.layout.XAxis(\n",
    "        title=go.layout.xaxis.Title(\n",
    "            text='Year',\n",
    "        ),\n",
    "        dtick = 1.0\n",
    "    ),\n",
    "    yaxis=go.layout.YAxis(\n",
    "        title=go.layout.yaxis.Title(\n",
    "            text='% of Female authored blog posts')))\n",
    "fig = go.Figure(data = data, layout=layout)\n",
    "py.iplot(fig, filename='basic-bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see a huge increase in the percentage of women authoring blog posts after a consistently low authorship of around 22% for the first three years of Monzo's operation. In 2018, we see the balance is fairly even, with roughly 47% of posts being penned by women, a huge upward surge from the years prior. In 2019 this upwards trend continues, with 72% of all blog posts this year being authored by women."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Having looked at gender, we now look at the positions of the authors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(None, columns = ['Position', 'Count15', 'Count16', 'Count17',\n",
    "                                   'Count18', 'Count19'])\n",
    "position_count = {}\n",
    "for blog in blogs:\n",
    "    position = blog.get_position()\n",
    "    year = blog.get_date().split(\" \")[2]\n",
    "    year_index = (int(year) - 2015)\n",
    "    if position in position_count:\n",
    "        position_count[position][year_index] += 1\n",
    "    else:\n",
    "        position_count[position] = [0 for x in range(5)]\n",
    "        position_count[position][year_index] += 1\n",
    "for key, value in position_count.items():\n",
    "    df = df.append({'Position' : key, 'Count15' : value[0], 'Count16' : value[1],\n",
    "                   'Count17' : value[2], 'Count18' : value[3], 'Count19' : value[4]}, ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/13.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trace = go.Table(\n",
    "    header=dict(values=['Position', '2015', '2016', '2017', '2018', '2019'],\n",
    "                fill = dict(color='#C2D4FF'),\n",
    "                align = ['left'] * 5),\n",
    "    cells=dict(values=[df.Position, df.Count15, df.Count16, df.Count17,\n",
    "                       df.Count18, df.Count19],\n",
    "               fill = dict(color='#F5F8FF'),\n",
    "               align = ['left'] * 5))\n",
    "data = [trace] \n",
    "py.iplot(data, filename = 'pandas_table')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can infer the introduction of new roles and titles within Monzo, and we can also see that the official titles stated on blog posts are constantly changing as time goes on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 2 - Analysis of text data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goals we are setting out to achieve in this section are as follows:\n",
    "\n",
    "a) Topic modeling to see if we can create buckets of words which characterise certain blog posts\n",
    "\n",
    "b) Automated Document Summarisation to make summaries of blog posts\n",
    "\n",
    "c) Clustering of blog posts and seeing if we can identify the overarching theme in each cluster\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start with part 2a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by importing all the libraries we will need"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim\n",
    "from gensim import corpora, models, similarities\n",
    "from gensim.corpora import Dictionary, MmCorpus\n",
    "from gensim.models import NormModel\n",
    "from gensim.sklearn_api import Text2BowTransformer\n",
    "from gensim.models.wrappers import DtmModel\n",
    "from gensim.similarities import MatrixSimilarity\n",
    "import nltk\n",
    "import matplotlib.pyplot as plt\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "from operator import itemgetter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /Users/emielv/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n",
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     /Users/emielv/nltk_data...\n",
      "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
      "[nltk_data]       date!\n",
      "[nltk_data] Downloading package wordnet to /Users/emielv/nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n",
      "[nltk_data] Downloading package punkt to /Users/emielv/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('stopwords')\n",
    "nltk.download('averaged_perceptron_tagger')\n",
    "nltk.download('wordnet')\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### An important part of any data analysis, but especially when doing Machine Learning, is the preprocessing of the data. Luckily, we have functions in nltk and gensim that will aide us in this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "525\n"
     ]
    }
   ],
   "source": [
    "documents = []\n",
    "for blog in blogs:\n",
    "    temp_text = blog.get_text()\n",
    "    all_words = \" \".join(temp_text)\n",
    "    documents.append(all_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have loaded our documents into an array, we can begin preprocessing. We start by simplifying the penn tags that will be supplied to us by nltk. We use the coding of 'a' for adjective, 'r' for adverb, 'v' for verb, and 'n' for noun. Then we preprocess by removing stop words, getting rid of punctuation and making every word lower case. We also lemmatize every word, which means reducing it to its stem. This is why we have to classify every word into a noun, adjective, adverb, or verb."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "def simplify(penn_tag):\n",
    "    pre = penn_tag[0]\n",
    "    if (pre == 'J'):\n",
    "        return 'a'\n",
    "    elif (pre == 'R'):\n",
    "        return 'r'\n",
    "    elif (pre == 'V'):\n",
    "        return 'v'\n",
    "    else:\n",
    "        return 'n'\n",
    "\n",
    "def pre_process(text):\n",
    "    stop_words = stopwords.words('english')\n",
    "    tokens = gensim.utils.simple_preprocess(str(text), deacc=True)\n",
    "    lem = WordNetLemmatizer()\n",
    "    return [lem.lemmatize(token, simplify(pos)) for token, pos in nltk.pos_tag(tokens)\n",
    "           if token not in stop_words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "corpus = [pre_process(document) for document in documents]\n",
    "dictionary = Dictionary(corpus)\n",
    "bag_of_words = [dictionary.doc2bow(line) for line in corpus]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def flatten_corpus(corpus):\n",
    "    flat = []\n",
    "    for doc in corpus:\n",
    "        for word in doc:\n",
    "            flat.append(word)\n",
    "    return flat\n",
    "\n",
    "def compute_ngrams(sequence, n = 1):\n",
    "    return zip(*[sequence[index:] for index in range(n)])\n",
    "\n",
    "def get_top_ngrams(corpus, ngram_length = 1, limit = 10):\n",
    "    if len(corpus) < 1000:\n",
    "        corpus = flatten_corpus(corpus)\n",
    "\n",
    "    ngrams = compute_ngrams(corpus, ngram_length)\n",
    "    ngrams_freq_dist = nltk.FreqDist(ngrams)\n",
    "    sorted_ngrams_fd = sorted(ngrams_freq_dist.items(),\n",
    "                             key = itemgetter(1), reverse = True)\n",
    "    sorted_ngrams = sorted_ngrams_fd[0:limit]\n",
    "    sorted_ngrams = [(' '.join(text), freq)\n",
    "                    for text, freq in sorted_ngrams]\n",
    "    return sorted_ngrams\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('let u know', 118),\n",
       " ('monzo money tip', 67),\n",
       " ('current account switch', 59),\n",
       " ('recently update name', 54),\n",
       " ('update name monzo', 54),\n",
       " ('name monzo read', 54),\n",
       " ('account switch service', 52),\n",
       " ('pic twitter com', 48),\n",
       " ('u know think', 46),\n",
       " ('account tab app', 42),\n",
       " ('follow monzo money', 42),\n",
       " ('want make sure', 38),\n",
       " ('monzo current account', 38),\n",
       " ('customer support team', 36),\n",
       " ('love hear think', 35),\n",
       " ('discussion community forum', 35),\n",
       " ('high interest rate', 34),\n",
       " ('set direct debit', 34),\n",
       " ('set money aside', 33),\n",
       " ('head account tab', 33)]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_top_ngrams(corpus, 3, 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having found the most common sequences of words of whatever length we want, this allows us to get a quick overview of what's being written on Monzo blogs. Using the words we found an ngram length of 1, we get a pretty nice summary of what Monzo is all about. But maybe we can use extra information to see more about what specifically is being written about. We do this using the concept of weighted tag-based phrase extraction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_only_type(corpus, code):\n",
    "    tag = ''\n",
    "    if code.lower() == 'adjective':\n",
    "        tag = 'J'\n",
    "    elif code.lower() == 'verb':\n",
    "        tag = 'V'\n",
    "    elif code.lower() == 'adverb':\n",
    "        tag = 'R'\n",
    "    else:\n",
    "        tag = 'N'\n",
    "    flat = flatten_corpus(corpus)\n",
    "    all_tags = nltk.pos_tag(flat)\n",
    "    return [word for index, word in enumerate(flat)\n",
    "           if all_tags[index][1][0] == tag]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('monzo money tip', 55),\n",
       " ('account switch service', 52),\n",
       " ('pic twitter com', 37),\n",
       " ('customer support team', 36),\n",
       " ('debit stand order', 33),\n",
       " ('money work everyone', 28),\n",
       " ('join discussion community', 24),\n",
       " ('email email protect', 23),\n",
       " ('customer operation team', 23),\n",
       " ('sort code account', 23),\n",
       " ('money tip monzo', 23),\n",
       " ('account number sort', 22),\n",
       " ('number sort code', 22),\n",
       " ('code account number', 22),\n",
       " ('use monzo account', 21),\n",
       " ('support video tag', 21),\n",
       " ('guide tip money', 21),\n",
       " ('tip money monzo', 21),\n",
       " ('money monzo money', 21),\n",
       " ('tip monzo money', 21)]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "noun_corpus = get_only_type(corpus, 'Noun')\n",
    "get_top_ngrams(noun_corpus, 3, 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This should give a better idea of what the keyphrases are from Monzo's blog posts, and we can tweak this number to look at longer or shorter phrases. We can also look at what kinds of verbs are used, or whatever other kind of word we want."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now we move on to Topic Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we perform Latent Semantic Indexing (LSI) on the corpus. We let the model find up to 25 topics, although this is chosen quite arbitrarily. We then print out the top 10 topics, with the 10 most commonly occuring words. We then see if we can identify some sort of theme within the topics created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.284*\"monzo\" + 0.262*\"make\" + 0.228*\"account\" + 0.223*\"money\" + 0.197*\"use\" + 0.196*\"work\" + 0.184*\"get\" + 0.156*\"help\" + 0.150*\"people\" + 0.149*\"pay\"'),\n",
       " (1,\n",
       "  '-0.320*\"money\" + -0.258*\"interest\" + -0.251*\"rate\" + -0.244*\"pay\" + -0.198*\"saving\" + 0.182*\"team\" + 0.182*\"monzo\" + -0.174*\"isa\" + -0.171*\"credit\" + 0.160*\"customer\"'),\n",
       " (2,\n",
       "  '-0.572*\"account\" + -0.320*\"bank\" + -0.227*\"payment\" + -0.206*\"monzo\" + -0.186*\"card\" + 0.152*\"pension\" + -0.151*\"current\" + 0.137*\"team\" + -0.135*\"switch\" + 0.126*\"work\"'),\n",
       " (3,\n",
       "  '0.393*\"credit\" + 0.234*\"rate\" + -0.231*\"monzo\" + 0.225*\"interest\" + 0.217*\"use\" + -0.196*\"money\" + 0.193*\"card\" + -0.159*\"save\" + -0.158*\"pot\" + 0.158*\"score\"'),\n",
       " (4,\n",
       "  '0.340*\"rate\" + 0.293*\"interest\" + -0.293*\"money\" + 0.244*\"isa\" + -0.200*\"get\" + 0.169*\"people\" + 0.162*\"account\" + 0.147*\"team\" + 0.143*\"customer\" + -0.131*\"go\"'),\n",
       " (5,\n",
       "  '-0.343*\"isa\" + 0.263*\"credit\" + 0.201*\"pay\" + -0.187*\"saving\" + -0.180*\"use\" + 0.172*\"card\" + -0.171*\"investment\" + 0.160*\"monzo\" + 0.159*\"work\" + 0.139*\"people\"'),\n",
       " (6,\n",
       "  '0.657*\"pension\" + 0.254*\"tax\" + -0.176*\"isa\" + -0.170*\"interest\" + 0.162*\"employer\" + 0.153*\"contribution\" + -0.139*\"saving\" + 0.129*\"scheme\" + 0.126*\"relief\" + 0.121*\"get\"'),\n",
       " (7,\n",
       "  '0.335*\"bank\" + -0.264*\"monzo\" + 0.240*\"money\" + 0.214*\"customer\" + -0.189*\"pot\" + 0.189*\"account\" + 0.187*\"people\" + -0.181*\"app\" + -0.170*\"add\" + -0.167*\"interest\"'),\n",
       " (8,\n",
       "  '0.463*\"isa\" + 0.425*\"card\" + -0.197*\"interest\" + -0.160*\"bank\" + 0.160*\"buy\" + 0.157*\"allowance\" + -0.153*\"saving\" + -0.141*\"rate\" + -0.135*\"investment\" + -0.133*\"pot\"'),\n",
       " (9,\n",
       "  '0.307*\"investment\" + 0.241*\"pay\" + -0.238*\"customer\" + -0.208*\"help\" + -0.200*\"pension\" + 0.185*\"people\" + -0.173*\"saving\" + 0.161*\"invest\" + 0.151*\"crowdfunding\" + -0.149*\"save\"')]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lsi = models.lsimodel.LsiModel(corpus=bag_of_words, id2word=dictionary, num_topics=25)\n",
    "lsi.print_topics(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At a first glance, it is hard to distinguish between the topics presented. A large reason for this is the limited scope of our data set. Furthermore, LSI works much better on larger data sets, due to the fact that we are reducing the dimensionality of the data. The larger the data set and the more varied the sources of the data set, the easier it is to generate overarching topics or clusters of words, which again is not really the case for our data. However, we can see that the third topic has something to do with switching to Monzo as the primary current account of customers. This was a move that happened in the last year, and we can see evidence for it by the fact that words such as current, account, switch, work, and pension are included in this topic. Furthermore, we can definitely see that the fourth topic seems to be about savings, with words like credit, interest, isa (Individual Savings Account), and pot (the name for a savings account in Monzo terminology) being group together. Additionally, perhaps the last topic looked more at additional crowdfunding of customers, as this is how Monzo was initially funded. We can see this in the inclusion of words like investment, help, invest, and crowdfunding in the topic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since LSI is mostly for very large data sets,We repeat the procedure with a Latent Dirichlet Analysis (LDA). The difference in LSI and LDA is that in LDA we assume a generative process for the document, using Bayesian inference to make inference on the parameters. Keep in mind the difference in LSI vs LDA, where LSI uses maximum likelihood estimation and LDA uses Bayes estimation. Bayes estimation takes the priors for parameters into account, and if those priors are accurate then better parameter estimates are yielded. This is often of benefit with a small corpus, which we have. Therefore, we expect the results of the LDA to be more fitting than those of LSI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1,\n",
       "  '0.015*\"make\" + 0.013*\"use\" + 0.011*\"data\" + 0.011*\"api\" + 0.009*\"system\" + 0.009*\"time\" + 0.009*\"build\" + 0.009*\"framework\" + 0.008*\"could\" + 0.007*\"need\"'),\n",
       " (4,\n",
       "  '0.018*\"make\" + 0.018*\"term\" + 0.017*\"bank\" + 0.016*\"write\" + 0.013*\"condition\" + 0.011*\"europe\" + 0.010*\"c\" + 0.010*\"need\" + 0.009*\"sepa\" + 0.008*\"t\"'),\n",
       " (16,\n",
       "  '0.021*\"insurance\" + 0.020*\"office\" + 0.015*\"cost\" + 0.012*\"dog\" + 0.011*\"policy\" + 0.011*\"home\" + 0.010*\"cardiff\" + 0.010*\"may\" + 0.009*\"vega\" + 0.009*\"cover\"'),\n",
       " (2,\n",
       "  '0.024*\"money\" + 0.020*\"get\" + 0.013*\"go\" + 0.010*\"like\" + 0.009*\"work\" + 0.008*\"buy\" + 0.008*\"spend\" + 0.008*\"cost\" + 0.007*\"thing\" + 0.007*\"year\"'),\n",
       " (23,\n",
       "  '0.040*\"pot\" + 0.034*\"money\" + 0.025*\"set\" + 0.023*\"monzo\" + 0.022*\"save\" + 0.021*\"saving\" + 0.020*\"add\" + 0.020*\"payment\" + 0.018*\"month\" + 0.018*\"spending\"'),\n",
       " (17,\n",
       "  '0.021*\"node\" + 0.016*\"envoy\" + 0.013*\"etcd\" + 0.012*\"cluster\" + 0.011*\"peer\" + 0.010*\"use\" + 0.010*\"internal\" + 0.008*\"service\" + 0.007*\"new\" + 0.006*\"zone\"'),\n",
       " (18,\n",
       "  '0.024*\"use\" + 0.020*\"help\" + 0.019*\"u\" + 0.017*\"need\" + 0.015*\"make\" + 0.015*\"chat\" + 0.015*\"customer\" + 0.012*\"find\" + 0.011*\"information\" + 0.011*\"way\"'),\n",
       " (11,\n",
       "  '0.024*\"monzo\" + 0.021*\"work\" + 0.019*\"make\" + 0.017*\"u\" + 0.013*\"help\" + 0.011*\"customer\" + 0.011*\"people\" + 0.010*\"feature\" + 0.009*\"new\" + 0.009*\"team\"'),\n",
       " (7,\n",
       "  '0.022*\"pay\" + 0.021*\"money\" + 0.018*\"interest\" + 0.017*\"rate\" + 0.015*\"get\" + 0.013*\"saving\" + 0.011*\"may\" + 0.010*\"need\" + 0.010*\"account\" + 0.010*\"tax\"'),\n",
       " (3,\n",
       "  '0.027*\"team\" + 0.014*\"product\" + 0.013*\"work\" + 0.012*\"research\" + 0.011*\"engineer\" + 0.009*\"make\" + 0.009*\"company\" + 0.008*\"call\" + 0.008*\"need\" + 0.008*\"monzo\"')]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda = models.ldamodel.LdaModel(corpus=bag_of_words, id2word=dictionary,\n",
    "                               num_topics=25, update_every=1, passes=100)\n",
    "lda.print_topics(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = []\n",
    "values = []\n",
    "for number_topics in range(10,102):\n",
    "    lda = models.ldamodel.LdaModel(corpus=bag_of_words, id2word=dictionary, random_state = 12, \n",
    "                                   num_topics=number_topics, update_every=1, \n",
    "                                   passes=10)\n",
    "    labels.append(number_topics)\n",
    "    values.append(lda.log_perplexity(bag_of_words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.7/site-packages/IPython/core/display.py:689: UserWarning:\n",
      "\n",
      "Consider using IPython.display.IFrame instead\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/19.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trace0 = go.Scatter(\n",
    "    x = labels,\n",
    "    y = values,\n",
    "    mode = 'lines+markers',\n",
    "    name = 'lines'\n",
    ")\n",
    "data = [trace0]\n",
    "layout = go.Layout(\n",
    "    title=go.layout.Title(\n",
    "        text='log perplexity against number of topics',\n",
    "        x=0\n",
    "    ),\n",
    "    xaxis=go.layout.XAxis(\n",
    "        title=go.layout.xaxis.Title(\n",
    "            text='Number of topics',\n",
    "        ),\n",
    "        dtick = 1.0\n",
    "    ),\n",
    "    yaxis=go.layout.YAxis(\n",
    "        title=go.layout.yaxis.Title(\n",
    "            text='Log perplexity')))\n",
    "fig = go.Figure(data = data, layout=layout)\n",
    "py.iplot(fig, filename='perplexity-scatter')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The graph seems to suggest that we should take 96 as the number of topics, and so we do so. Then, we get the following 10 most significant topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(73,\n",
       "  '0.109*\"account\" + 0.042*\"switch\" + 0.037*\"bank\" + 0.026*\"monzo\" + 0.024*\"current\" + 0.016*\"money\" + 0.013*\"new\" + 0.013*\"open\" + 0.013*\"payment\" + 0.012*\"make\"'),\n",
       " (25,\n",
       "  '0.098*\"flight\" + 0.063*\"airline\" + 0.049*\"book\" + 0.034*\"travel\" + 0.024*\"get\" + 0.023*\"holiday\" + 0.022*\"back\" + 0.021*\"claim\" + 0.020*\"contact\" + 0.018*\"cancel\"'),\n",
       " (15,\n",
       "  '0.044*\"money\" + 0.026*\"get\" + 0.016*\"go\" + 0.014*\"like\" + 0.014*\"save\" + 0.013*\"spend\" + 0.011*\"want\" + 0.011*\"buy\" + 0.011*\"work\" + 0.010*\"think\"'),\n",
       " (88,\n",
       "  '0.024*\"bank\" + 0.022*\"use\" + 0.021*\"api\" + 0.018*\"data\" + 0.015*\"customer\" + 0.014*\"monzo\" + 0.012*\"make\" + 0.012*\"information\" + 0.010*\"developer\" + 0.009*\"fraud\"'),\n",
       " (18,\n",
       "  '0.178*\"name\" + 0.024*\"new\" + 0.024*\"mark\" + 0.024*\"trade\" + 0.018*\"idea\" + 0.018*\"register\" + 0.018*\"potential\" + 0.016*\"shortlist\" + 0.016*\"sound\" + 0.015*\"suggestion\"'),\n",
       " (95,\n",
       "  '0.032*\"customer\" + 0.025*\"card\" + 0.019*\"monzo\" + 0.017*\"use\" + 0.017*\"u\" + 0.016*\"ticketmaster\" + 0.014*\"accessibility\" + 0.011*\"breach\" + 0.011*\"share\" + 0.010*\"year\"'),\n",
       " (89,\n",
       "  '0.055*\"debt\" + 0.034*\"credit\" + 0.023*\"interest\" + 0.022*\"borrow\" + 0.020*\"loan\" + 0.017*\"pay\" + 0.014*\"card\" + 0.014*\"overdraft\" + 0.013*\"repayment\" + 0.012*\"use\"'),\n",
       " (55,\n",
       "  '0.032*\"u\" + 0.026*\"health\" + 0.025*\"mental\" + 0.024*\"help\" + 0.019*\"support\" + 0.017*\"people\" + 0.016*\"money\" + 0.016*\"way\" + 0.014*\"might\" + 0.014*\"tell\"'),\n",
       " (26,\n",
       "  '0.037*\"money\" + 0.021*\"people\" + 0.015*\"help\" + 0.014*\"financial\" + 0.014*\"company\" + 0.012*\"subscription\" + 0.011*\"get\" + 0.011*\"give\" + 0.010*\"crowdfunding\" + 0.009*\"advice\"'),\n",
       " (13,\n",
       "  '0.034*\"gamble\" + 0.023*\"block\" + 0.017*\"people\" + 0.015*\"transaction\" + 0.013*\"self\" + 0.013*\"turn\" + 0.011*\"problem\" + 0.011*\"someone\" + 0.010*\"support\" + 0.010*\"help\"')]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda = models.ldamodel.LdaModel(corpus=bag_of_words, id2word=dictionary, random_state=12, \n",
    "                               num_topics=96, update_every=1, passes=100)\n",
    "lda.print_topics(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One limitation of LDA is that we need to specify the number of topics beforehand, and the model will fit the data to these parameters. An obvious consequence of this is if you underspecify the number of topics, then unrelated terms may be grouped together, which could lead to false inference of the relatedness of terms. Additionally, overspecifying the number of topics could lead to very homogenous topics created by the model, and therefore leads to an inability to say anything about the results found. That is where the Hierarchical Dirichlet Process (HDP) comes in. Here we do not specify the number of topics, with this instead being derived from the data itself. For the HDP model function from gensim, we need to specify a bag-of-words model and a Dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "150\n"
     ]
    }
   ],
   "source": [
    "model = models.HdpModel(bag_of_words, id2word=dictionary)\n",
    "print(len(model.get_topics()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.014*monzo + 0.010*make + 0.010*use + 0.009*get + 0.009*account + 0.008*money + 0.007*card + 0.007*like + 0.007*pay + 0.006*work'),\n",
       " (1,\n",
       "  '0.017*monzo + 0.012*make + 0.011*account + 0.010*work + 0.008*pay + 0.008*get + 0.007*use + 0.007*month + 0.007*money + 0.007*payment'),\n",
       " (2,\n",
       "  '0.009*request + 0.009*crowdfunding + 0.009*make + 0.009*use + 0.008*u + 0.007*could + 0.006*app + 0.006*go + 0.006*time + 0.006*load'),\n",
       " (3,\n",
       "  '0.014*monzo + 0.010*account + 0.008*make + 0.007*card + 0.007*use + 0.006*work + 0.006*u + 0.006*bank + 0.006*customer + 0.006*new'),\n",
       " (4,\n",
       "  '0.011*monzo + 0.009*make + 0.007*april + 0.007*twitter + 0.007*pot + 0.007*work + 0.006*com + 0.006*help + 0.006*pic + 0.006*get'),\n",
       " (5,\n",
       "  '0.004*make + 0.004*use + 0.004*data + 0.004*monzo + 0.003*time + 0.003*team + 0.003*one + 0.002*build + 0.002*u + 0.002*work'),\n",
       " (6,\n",
       "  '0.005*monzo + 0.004*account + 0.004*money + 0.004*bank + 0.003*work + 0.003*get + 0.003*people + 0.003*u + 0.003*make + 0.003*help'),\n",
       " (7,\n",
       "  '0.004*account + 0.004*bank + 0.004*make + 0.004*monzo + 0.003*get + 0.003*work + 0.003*money + 0.003*people + 0.002*help + 0.002*need'),\n",
       " (8,\n",
       "  '0.004*u + 0.004*get + 0.003*monzo + 0.003*money + 0.003*product + 0.003*people + 0.003*health + 0.003*mental + 0.003*make + 0.002*share'),\n",
       " (9,\n",
       "  '0.008*isa + 0.007*money + 0.005*credit + 0.004*account + 0.004*pay + 0.004*use + 0.003*wage + 0.003*rate + 0.003*interest + 0.003*bank')]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.print_topics(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having looked at LSA, LDA, and HDP, we now look at Dynamic Topic Modeling. This will allow us to see if the topics change over time, as Monzo has grown as a company."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we can look at Guided LDA. In Guided LDA, we specify certain terms and give them topic numbers. Therefore, we can steer the direction that LDA will shift in by selecting certain keywords that we think will appear in different topics. To do so, we can simply pass on a dictionary of certain terms and their topic numbers in the 'eta' argument of the LDA model function that we were using previously. However, we should first create a function to generate this eta, which we do now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_eta(priors, dictionary, number_topics):\n",
    "    eta = np.full(shape=(number_topics, len(dictionary)), fill_value=1) # create a (ntopics, nterms) matrix and fill with 1\n",
    "    for word, topic in priors.items(): # for each word in the list of priors\n",
    "        key_index = [index for index,term in dictionary.items() if term==word] # look up the word in the dictionary\n",
    "        if len(key_index): # if it's in the dictionary\n",
    "            eta[topic,key_index[0]] = 1e10  # put a large number in there\n",
    "    eta = np.divide(eta, eta.sum(axis=0)) # normalize so that the probabilities sum to 1 over all topics\n",
    "    return eta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using this function, we can now create an eta and pass it along to our model creation function. It is important to note that if we simply pass along an empty eta, we will get the same output as with normal LDA, as we are not 'guiding' the process in any sense."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(66,\n",
       "  '0.176*\"community\" + 0.084*\"event\" + 0.068*\"host\" + 0.023*\"wider\" + 0.022*\"office\" + 0.022*\"new\" + 0.018*\"idea\" + 0.018*\"space\" + 0.016*\"involve\" + 0.015*\"look\"'),\n",
       " (96,\n",
       "  '0.030*\"monzo\" + 0.026*\"make\" + 0.016*\"feature\" + 0.015*\"work\" + 0.015*\"help\" + 0.015*\"u\" + 0.014*\"see\" + 0.014*\"app\" + 0.013*\"add\" + 0.012*\"month\"'),\n",
       " (28,\n",
       "  '0.029*\"android\" + 0.024*\"pay\" + 0.018*\"give\" + 0.018*\"top\" + 0.017*\"card\" + 0.014*\"app\" + 0.013*\"today\" + 0.011*\"store\" + 0.011*\"debit\" + 0.011*\"update\"'),\n",
       " (33,\n",
       "  '0.011*\"service\" + 0.011*\"cost\" + 0.009*\"wedding\" + 0.009*\"people\" + 0.008*\"option\" + 0.008*\"design\" + 0.008*\"old\" + 0.007*\"even\" + 0.007*\"year\" + 0.007*\"user\"'),\n",
       " (89,\n",
       "  '0.030*\"loan\" + 0.026*\"overdraft\" + 0.022*\"car\" + 0.019*\"unsecured\" + 0.018*\"secure\" + 0.016*\"use\" + 0.016*\"card\" + 0.016*\"credit\" + 0.014*\"borrow\" + 0.013*\"purchase\"'),\n",
       " (36,\n",
       "  '0.048*\"money\" + 0.017*\"get\" + 0.015*\"think\" + 0.015*\"couple\" + 0.014*\"go\" + 0.013*\"say\" + 0.013*\"would\" + 0.013*\"year\" + 0.012*\"pay\" + 0.011*\"u\"'),\n",
       " (13,\n",
       "  '0.042*\"gamble\" + 0.036*\"people\" + 0.028*\"block\" + 0.022*\"support\" + 0.018*\"help\" + 0.015*\"self\" + 0.014*\"problem\" + 0.012*\"could\" + 0.011*\"turn\" + 0.011*\"make\"'),\n",
       " (4,\n",
       "  '0.055*\"matt\" + 0.047*\"sian\" + 0.024*\"monzo\" + 0.022*\"next\" + 0.020*\"student\" + 0.015*\"go\" + 0.013*\"card\" + 0.013*\"user\" + 0.013*\"maxed\" + 0.013*\"wonder\"'),\n",
       " (98,\n",
       "  '0.077*\"framework\" + 0.034*\"progression\" + 0.024*\"people\" + 0.020*\"manager\" + 0.020*\"discipline\" + 0.017*\"make\" + 0.015*\"work\" + 0.014*\"use\" + 0.013*\"company\" + 0.013*\"level\"'),\n",
       " (30,\n",
       "  '0.035*\"charity\" + 0.031*\"may\" + 0.023*\"give\" + 0.019*\"people\" + 0.018*\"donate\" + 0.016*\"support\" + 0.014*\"year\" + 0.012*\"health\" + 0.011*\"month\" + 0.010*\"help\"')]"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prior = {'plus' : 0, 'pot' : 1, 'isa' : 1, 'android' : 2, 'ios' : 2, 'golden' : 3,\n",
    "        'crowdfunding' : 4, 'support' : 5,  'apple' : 6}\n",
    "number_topics = 100\n",
    "eta = create_eta(prior, dictionary, number_topics)\n",
    "guided_lda = models.ldamodel.LdaModel(corpus=bag_of_words, id2word=dictionary, random_state = 12,\n",
    "                               num_topics=number_topics, update_every=1, passes=100,\n",
    "                              eta=eta)\n",
    "guided_lda.print_topics(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.011*\"plus\" + 0.000*\"get\" + 0.000*\"money\" + 0.000*\"save\" + 0.000*\"work\" + 0.000*\"mortgage\" + 0.000*\"pay\" + 0.000*\"need\" + 0.000*\"year\" + 0.000*\"use\"'),\n",
       " (1,\n",
       "  '0.061*\"golden\" + 0.059*\"ticket\" + 0.034*\"wait\" + 0.033*\"friend\" + 0.032*\"list\" + 0.031*\"monzo\" + 0.028*\"alpha\" + 0.023*\"email\" + 0.021*\"invite\" + 0.018*\"vote\"'),\n",
       " (2,\n",
       "  '0.122*\"mondo\" + 0.024*\"name\" + 0.023*\"recently\" + 0.019*\"update\" + 0.018*\"transaction\" + 0.015*\"read\" + 0.015*\"every\" + 0.012*\"user\" + 0.011*\"student\" + 0.010*\"get\"'),\n",
       " (3,\n",
       "  '0.013*\"make\" + 0.012*\"need\" + 0.010*\"monzo\" + 0.008*\"one\" + 0.008*\"write\" + 0.008*\"use\" + 0.007*\"way\" + 0.007*\"thing\" + 0.007*\"u\" + 0.007*\"team\"'),\n",
       " (4,\n",
       "  '0.055*\"matt\" + 0.047*\"sian\" + 0.024*\"monzo\" + 0.022*\"next\" + 0.020*\"student\" + 0.015*\"go\" + 0.013*\"card\" + 0.013*\"user\" + 0.013*\"maxed\" + 0.013*\"wonder\"'),\n",
       " (5,\n",
       "  '0.079*\"investment\" + 0.072*\"invest\" + 0.068*\"crowdfunding\" + 0.048*\"round\" + 0.041*\"investor\" + 0.027*\"million\" + 0.023*\"monzo\" + 0.019*\"raise\" + 0.015*\"pre\" + 0.015*\"everyone\"'),\n",
       " (6,\n",
       "  '0.011*\"apple\" + 0.000*\"use\" + 0.000*\"make\" + 0.000*\"node\" + 0.000*\"insurance\" + 0.000*\"u\" + 0.000*\"want\" + 0.000*\"monzo\" + 0.000*\"need\" + 0.000*\"new\"'),\n",
       " (7,\n",
       "  '0.000*\"make\" + 0.000*\"mondo\" + 0.000*\"help\" + 0.000*\"monzo\" + 0.000*\"work\" + 0.000*\"money\" + 0.000*\"account\" + 0.000*\"use\" + 0.000*\"one\" + 0.000*\"payment\"'),\n",
       " (8,\n",
       "  '0.052*\"overdraft\" + 0.030*\"monzo\" + 0.024*\"account\" + 0.020*\"use\" + 0.016*\"make\" + 0.015*\"day\" + 0.015*\"charge\" + 0.014*\"pay\" + 0.012*\"u\" + 0.011*\"limit\"'),\n",
       " (9,\n",
       "  '0.035*\"therapy\" + 0.028*\"free\" + 0.020*\"offer\" + 0.017*\"session\" + 0.017*\"online\" + 0.016*\"fee\" + 0.013*\"one\" + 0.012*\"service\" + 0.012*\"psychotherapy\" + 0.012*\"organisation\"'),\n",
       " (10,\n",
       "  '0.035*\"get\" + 0.023*\"work\" + 0.019*\"go\" + 0.015*\"thing\" + 0.013*\"like\" + 0.011*\"alex\" + 0.010*\"rosie\" + 0.010*\"time\" + 0.010*\"one\" + 0.009*\"vic\"'),\n",
       " (11,\n",
       "  '0.000*\"pension\" + 0.000*\"monzo\" + 0.000*\"work\" + 0.000*\"get\" + 0.000*\"make\" + 0.000*\"week\" + 0.000*\"employee\" + 0.000*\"help\" + 0.000*\"mental\" + 0.000*\"take\"'),\n",
       " (12,\n",
       "  '0.013*\"really\" + 0.012*\"get\" + 0.012*\"thing\" + 0.011*\"work\" + 0.011*\"project\" + 0.011*\"build\" + 0.010*\"go\" + 0.008*\"monzo\" + 0.008*\"time\" + 0.007*\"learn\"'),\n",
       " (13,\n",
       "  '0.042*\"gamble\" + 0.036*\"people\" + 0.028*\"block\" + 0.022*\"support\" + 0.018*\"help\" + 0.015*\"self\" + 0.014*\"problem\" + 0.012*\"could\" + 0.011*\"turn\" + 0.011*\"make\"'),\n",
       " (14,\n",
       "  '0.046*\"engineer\" + 0.038*\"work\" + 0.032*\"make\" + 0.026*\"team\" + 0.021*\"u\" + 0.019*\"help\" + 0.014*\"android\" + 0.013*\"company\" + 0.013*\"feature\" + 0.012*\"mean\"'),\n",
       " (15,\n",
       "  '0.044*\"money\" + 0.024*\"get\" + 0.022*\"spend\" + 0.020*\"save\" + 0.017*\"like\" + 0.015*\"go\" + 0.015*\"make\" + 0.014*\"budget\" + 0.013*\"spending\" + 0.013*\"want\"'),\n",
       " (16,\n",
       "  '0.052*\"design\" + 0.028*\"mondo\" + 0.021*\"really\" + 0.019*\"apps\" + 0.017*\"follow\" + 0.016*\"recur\" + 0.015*\"good\" + 0.015*\"scratch\" + 0.013*\"charge\" + 0.013*\"bit\"'),\n",
       " (17,\n",
       "  '0.077*\"credit\" + 0.050*\"score\" + 0.024*\"lender\" + 0.020*\"may\" + 0.019*\"money\" + 0.015*\"use\" + 0.015*\"report\" + 0.014*\"get\" + 0.013*\"apply\" + 0.012*\"check\"'),\n",
       " (18,\n",
       "  '0.049*\"dip\" + 0.041*\"nudge\" + 0.034*\"reminder\" + 0.029*\"march\" + 0.024*\"envelope\" + 0.024*\"guilt\" + 0.023*\"pic\" + 0.022*\"com\" + 0.020*\"twitter\" + 0.019*\"likely\"'),\n",
       " (19,\n",
       "  '0.026*\"fingerprint\" + 0.021*\"use\" + 0.016*\"pay\" + 0.016*\"app\" + 0.016*\"platform\" + 0.016*\"android\" + 0.011*\"one\" + 0.011*\"pin\" + 0.011*\"always\" + 0.011*\"people\"'),\n",
       " (20,\n",
       "  '0.037*\"answer\" + 0.036*\"use\" + 0.035*\"help\" + 0.032*\"search\" + 0.031*\"screen\" + 0.028*\"find\" + 0.022*\"article\" + 0.020*\"file\" + 0.019*\"need\" + 0.017*\"make\"'),\n",
       " (21,\n",
       "  '0.040*\"service\" + 0.023*\"envoy\" + 0.020*\"kubernetes\" + 0.018*\"vpn\" + 0.017*\"network\" + 0.015*\"linkerd\" + 0.015*\"rpc\" + 0.015*\"pod\" + 0.014*\"use\" + 0.012*\"system\"'),\n",
       " (22,\n",
       "  '0.057*\"node\" + 0.037*\"etcd\" + 0.036*\"cluster\" + 0.022*\"peer\" + 0.016*\"use\" + 0.014*\"zone\" + 0.012*\"new\" + 0.012*\"requirement\" + 0.011*\"vol\" + 0.011*\"failure\"'),\n",
       " (23,\n",
       "  '0.097*\"payment\" + 0.066*\"direct\" + 0.064*\"bill\" + 0.061*\"debit\" + 0.038*\"pay\" + 0.033*\"month\" + 0.031*\"money\" + 0.027*\"regular\" + 0.025*\"like\" + 0.024*\"summary\"'),\n",
       " (24,\n",
       "  '0.039*\"pay\" + 0.025*\"monzo\" + 0.023*\"nick\" + 0.017*\"find\" + 0.016*\"need\" + 0.016*\"card\" + 0.016*\"wallet\" + 0.015*\"apple\" + 0.014*\"timeline\" + 0.014*\"add\"'),\n",
       " (25,\n",
       "  '0.079*\"urgent\" + 0.062*\"customer\" + 0.062*\"support\" + 0.052*\"question\" + 0.051*\"query\" + 0.050*\"minute\" + 0.035*\"average\" + 0.025*\"conversation\" + 0.024*\"hour\" + 0.020*\"non\"'),\n",
       " (26,\n",
       "  '0.025*\"research\" + 0.015*\"get\" + 0.014*\"help\" + 0.010*\"meditation\" + 0.009*\"researcher\" + 0.009*\"march\" + 0.008*\"feel\" + 0.008*\"app\" + 0.008*\"pic\" + 0.008*\"free\"'),\n",
       " (27,\n",
       "  '0.047*\"forum\" + 0.038*\"week\" + 0.021*\"read\" + 0.019*\"want\" + 0.019*\"question\" + 0.019*\"comment\" + 0.018*\"hear\" + 0.018*\"thread\" + 0.017*\"blog\" + 0.017*\"community\"'),\n",
       " (28,\n",
       "  '0.029*\"android\" + 0.024*\"pay\" + 0.018*\"give\" + 0.018*\"top\" + 0.017*\"card\" + 0.014*\"app\" + 0.013*\"today\" + 0.011*\"store\" + 0.011*\"debit\" + 0.011*\"update\"'),\n",
       " (29,\n",
       "  '0.022*\"family\" + 0.021*\"agent\" + 0.019*\"house\" + 0.018*\"cost\" + 0.016*\"mum\" + 0.016*\"want\" + 0.016*\"money\" + 0.014*\"student\" + 0.014*\"time\" + 0.013*\"sonya\"'),\n",
       " (30,\n",
       "  '0.035*\"charity\" + 0.031*\"may\" + 0.023*\"give\" + 0.019*\"people\" + 0.018*\"donate\" + 0.016*\"support\" + 0.014*\"year\" + 0.012*\"health\" + 0.011*\"month\" + 0.010*\"help\"'),\n",
       " (31,\n",
       "  '0.034*\"chat\" + 0.019*\"code\" + 0.018*\"new\" + 0.017*\"class\" + 0.010*\"use\" + 0.010*\"like\" + 0.010*\"work\" + 0.009*\"message\" + 0.009*\"implementation\" + 0.009*\"layer\"'),\n",
       " (32,\n",
       "  '0.000*\"account\" + 0.000*\"monzo\" + 0.000*\"money\" + 0.000*\"payment\" + 0.000*\"bank\" + 0.000*\"get\" + 0.000*\"overdraft\" + 0.000*\"make\" + 0.000*\"u\" + 0.000*\"pay\"'),\n",
       " (33,\n",
       "  '0.011*\"service\" + 0.011*\"cost\" + 0.009*\"wedding\" + 0.009*\"people\" + 0.008*\"option\" + 0.008*\"design\" + 0.008*\"old\" + 0.007*\"even\" + 0.007*\"year\" + 0.007*\"user\"'),\n",
       " (34,\n",
       "  '0.051*\"customer\" + 0.042*\"vulnerable\" + 0.039*\"people\" + 0.027*\"financial\" + 0.017*\"difficulty\" + 0.017*\"support\" + 0.014*\"help\" + 0.011*\"need\" + 0.010*\"money\" + 0.010*\"data\"'),\n",
       " (35,\n",
       "  '0.049*\"resource\" + 0.043*\"raw\" + 0.027*\"notification\" + 0.022*\"sound\" + 0.021*\"file\" + 0.016*\"co\" + 0.016*\"uk\" + 0.014*\"getmondo\" + 0.013*\"topup\" + 0.011*\"use\"'),\n",
       " (36,\n",
       "  '0.048*\"money\" + 0.017*\"get\" + 0.015*\"think\" + 0.015*\"couple\" + 0.014*\"go\" + 0.013*\"say\" + 0.013*\"would\" + 0.013*\"year\" + 0.012*\"pay\" + 0.011*\"u\"'),\n",
       " (37,\n",
       "  '0.512*\"business\" + 0.082*\"small\" + 0.020*\"maybe\" + 0.011*\"hub\" + 0.008*\"blomfield\" + 0.004*\"heard\" + 0.004*\"sophisticated\" + 0.004*\"connects\" + 0.000*\"account\" + 0.000*\"get\"'),\n",
       " (38,\n",
       "  '0.062*\"dan\" + 0.051*\"lauren\" + 0.032*\"spend\" + 0.032*\"week\" + 0.025*\"target\" + 0.017*\"per\" + 0.015*\"choice\" + 0.013*\"look\" + 0.012*\"budget\" + 0.012*\"challenge\"'),\n",
       " (39,\n",
       "  '0.028*\"scheme\" + 0.017*\"idea\" + 0.016*\"pyramid\" + 0.015*\"join\" + 0.014*\"people\" + 0.014*\"pay\" + 0.013*\"could\" + 0.012*\"u\" + 0.012*\"see\" + 0.011*\"share\"'),\n",
       " (40,\n",
       "  '0.039*\"woman\" + 0.035*\"gender\" + 0.029*\"diversity\" + 0.023*\"people\" + 0.023*\"senior\" + 0.022*\"role\" + 0.017*\"gap\" + 0.014*\"hotel\" + 0.013*\"identify\" + 0.013*\"diverse\"'),\n",
       " (41,\n",
       "  '0.055*\"payment\" + 0.044*\"account\" + 0.040*\"bank\" + 0.019*\"money\" + 0.015*\"make\" + 0.013*\"use\" + 0.013*\"send\" + 0.012*\"transfer\" + 0.011*\"card\" + 0.010*\"merchant\"'),\n",
       " (42,\n",
       "  '0.067*\"bank\" + 0.050*\"europe\" + 0.043*\"sepa\" + 0.040*\"around\" + 0.039*\"country\" + 0.038*\"send\" + 0.027*\"money\" + 0.024*\"world\" + 0.023*\"clearing\" + 0.021*\"different\"'),\n",
       " (43,\n",
       "  '0.074*\"card\" + 0.027*\"customer\" + 0.027*\"use\" + 0.026*\"cashback\" + 0.023*\"information\" + 0.021*\"monzo\" + 0.019*\"fraud\" + 0.018*\"website\" + 0.018*\"transaction\" + 0.016*\"steal\"'),\n",
       " (44,\n",
       "  '0.055*\"insurance\" + 0.021*\"flight\" + 0.020*\"cover\" + 0.018*\"home\" + 0.017*\"damage\" + 0.016*\"claim\" + 0.016*\"book\" + 0.016*\"building\" + 0.015*\"may\" + 0.013*\"content\"'),\n",
       " (45,\n",
       "  '0.071*\"energy\" + 0.052*\"wage\" + 0.037*\"living\" + 0.034*\"supplier\" + 0.028*\"switch\" + 0.026*\"tariff\" + 0.021*\"national\" + 0.016*\"minimum\" + 0.015*\"pay\" + 0.014*\"get\"'),\n",
       " (46,\n",
       "  '0.018*\"make\" + 0.015*\"test\" + 0.014*\"load\" + 0.014*\"system\" + 0.013*\"app\" + 0.013*\"request\" + 0.012*\"use\" + 0.012*\"call\" + 0.012*\"u\" + 0.010*\"could\"'),\n",
       " (47,\n",
       "  '0.066*\"pay\" + 0.043*\"tax\" + 0.023*\"employer\" + 0.019*\"income\" + 0.019*\"year\" + 0.018*\"earn\" + 0.017*\"much\" + 0.015*\"may\" + 0.014*\"money\" + 0.014*\"job\"'),\n",
       " (48,\n",
       "  '0.000*\"chat\" + 0.000*\"use\" + 0.000*\"make\" + 0.000*\"class\" + 0.000*\"information\" + 0.000*\"send\" + 0.000*\"bank\" + 0.000*\"app\" + 0.000*\"email\" + 0.000*\"monzo\"'),\n",
       " (49,\n",
       "  '0.129*\"ticketmaster\" + 0.045*\"april\" + 0.037*\"replace\" + 0.034*\"u\" + 0.028*\"breach\" + 0.028*\"four\" + 0.026*\"th\" + 0.023*\"share\" + 0.023*\"similar\" + 0.017*\"attempted\"'),\n",
       " (50,\n",
       "  '0.026*\"current\" + 0.023*\"work\" + 0.022*\"day\" + 0.020*\"rollout\" + 0.018*\"u\" + 0.018*\"get\" + 0.015*\"account\" + 0.013*\"thing\" + 0.012*\"news\" + 0.009*\"second\"'),\n",
       " (51,\n",
       "  '0.042*\"rate\" + 0.041*\"interest\" + 0.026*\"saving\" + 0.022*\"save\" + 0.021*\"money\" + 0.021*\"account\" + 0.018*\"pay\" + 0.016*\"may\" + 0.014*\"buy\" + 0.013*\"mortgage\"'),\n",
       " (52,\n",
       "  '0.083*\"crime\" + 0.061*\"financial\" + 0.033*\"criminal\" + 0.024*\"fincrime\" + 0.022*\"detect\" + 0.019*\"activity\" + 0.018*\"security\" + 0.013*\"suspect\" + 0.013*\"suspicious\" + 0.013*\"post\"'),\n",
       " (53,\n",
       "  '0.079*\"app\" + 0.052*\"feed\" + 0.050*\"performance\" + 0.046*\"time\" + 0.029*\"io\" + 0.028*\"scroll\" + 0.027*\"item\" + 0.027*\"squad\" + 0.024*\"happen\" + 0.022*\"fix\"'),\n",
       " (54,\n",
       "  '0.016*\"user\" + 0.016*\"research\" + 0.013*\"team\" + 0.012*\"make\" + 0.010*\"u\" + 0.009*\"product\" + 0.009*\"people\" + 0.008*\"assumption\" + 0.006*\"around\" + 0.006*\"whether\"'),\n",
       " (55,\n",
       "  '0.032*\"u\" + 0.030*\"health\" + 0.030*\"mental\" + 0.027*\"help\" + 0.019*\"support\" + 0.018*\"way\" + 0.016*\"share\" + 0.016*\"money\" + 0.015*\"might\" + 0.015*\"tell\"'),\n",
       " (56,\n",
       "  '0.025*\"refund\" + 0.021*\"product\" + 0.021*\"flight\" + 0.018*\"get\" + 0.018*\"right\" + 0.015*\"return\" + 0.015*\"seller\" + 0.015*\"airline\" + 0.014*\"may\" + 0.011*\"day\"'),\n",
       " (57,\n",
       "  '0.036*\"work\" + 0.034*\"office\" + 0.024*\"support\" + 0.021*\"vega\" + 0.019*\"move\" + 0.017*\"team\" + 0.017*\"time\" + 0.015*\"people\" + 0.015*\"uk\" + 0.015*\"hour\"'),\n",
       " (58,\n",
       "  '0.037*\"frank\" + 0.032*\"ticket\" + 0.025*\"charlotte\" + 0.022*\"train\" + 0.017*\"railcard\" + 0.016*\"make\" + 0.014*\"travel\" + 0.013*\"split\" + 0.013*\"get\" + 0.011*\"buy\"'),\n",
       " (59,\n",
       "  '0.029*\"make\" + 0.029*\"monzo\" + 0.025*\"company\" + 0.020*\"people\" + 0.019*\"share\" + 0.018*\"work\" + 0.016*\"goal\" + 0.013*\"team\" + 0.012*\"u\" + 0.012*\"transparency\"'),\n",
       " (60,\n",
       "  '0.053*\"bank\" + 0.046*\"name\" + 0.038*\"address\" + 0.033*\"account\" + 0.027*\"monzo\" + 0.023*\"uk\" + 0.022*\"new\" + 0.021*\"open\" + 0.020*\"card\" + 0.020*\"get\"'),\n",
       " (61,\n",
       "  '0.018*\"monzo\" + 0.014*\"pay\" + 0.013*\"issue\" + 0.012*\"teenager\" + 0.012*\"cash\" + 0.012*\"big\" + 0.011*\"young\" + 0.010*\"people\" + 0.010*\"find\" + 0.010*\"feature\"'),\n",
       " (62,\n",
       "  '0.034*\"api\" + 0.025*\"card\" + 0.014*\"make\" + 0.014*\"access\" + 0.013*\"use\" + 0.013*\"bank\" + 0.013*\"third\" + 0.012*\"company\" + 0.012*\"data\" + 0.012*\"party\"'),\n",
       " (63,\n",
       "  '0.048*\"meat\" + 0.044*\"vegan\" + 0.030*\"save\" + 0.025*\"food\" + 0.020*\"eat\" + 0.019*\"may\" + 0.018*\"expensive\" + 0.016*\"twitter\" + 0.014*\"april\" + 0.014*\"money\"'),\n",
       " (64,\n",
       "  '0.000*\"interest\" + 0.000*\"rate\" + 0.000*\"people\" + 0.000*\"monzo\" + 0.000*\"customer\" + 0.000*\"get\" + 0.000*\"work\" + 0.000*\"account\" + 0.000*\"pay\" + 0.000*\"use\"'),\n",
       " (65,\n",
       "  '0.058*\"statement\" + 0.037*\"export\" + 0.030*\"bank\" + 0.020*\"monzo\" + 0.020*\"graph\" + 0.016*\"download\" + 0.016*\"tab\" + 0.014*\"app\" + 0.014*\"top\" + 0.013*\"need\"'),\n",
       " (66,\n",
       "  '0.176*\"community\" + 0.084*\"event\" + 0.068*\"host\" + 0.023*\"wider\" + 0.022*\"office\" + 0.022*\"new\" + 0.018*\"idea\" + 0.018*\"space\" + 0.016*\"involve\" + 0.015*\"look\"'),\n",
       " (67,\n",
       "  '0.031*\"pay\" + 0.027*\"budget\" + 0.022*\"need\" + 0.014*\"month\" + 0.013*\"get\" + 0.012*\"money\" + 0.012*\"use\" + 0.012*\"income\" + 0.012*\"like\" + 0.011*\"work\"'),\n",
       " (68,\n",
       "  '0.037*\"bank\" + 0.031*\"money\" + 0.020*\"customer\" + 0.016*\"monzo\" + 0.014*\"account\" + 0.014*\"make\" + 0.011*\"people\" + 0.009*\"u\" + 0.008*\"get\" + 0.008*\"financial\"'),\n",
       " (69,\n",
       "  '0.018*\"make\" + 0.017*\"c\" + 0.015*\"club\" + 0.013*\"t\" + 0.013*\"give\" + 0.011*\"code\" + 0.009*\"customer\" + 0.009*\"use\" + 0.009*\"word\" + 0.008*\"term\"'),\n",
       " (70,\n",
       "  '0.077*\"credit\" + 0.063*\"debt\" + 0.035*\"pay\" + 0.032*\"interest\" + 0.029*\"card\" + 0.021*\"get\" + 0.019*\"borrow\" + 0.018*\"rate\" + 0.018*\"charge\" + 0.017*\"may\"'),\n",
       " (71,\n",
       "  '0.077*\"isa\" + 0.037*\"loan\" + 0.029*\"student\" + 0.024*\"allowance\" + 0.022*\"year\" + 0.021*\"isas\" + 0.017*\"help\" + 0.016*\"lifetime\" + 0.016*\"grant\" + 0.015*\"money\"'),\n",
       " (72,\n",
       "  '0.038*\"choose\" + 0.025*\"accessibility\" + 0.023*\"song\" + 0.016*\"cop\" + 0.011*\"get\" + 0.011*\"listen\" + 0.006*\"like\" + 0.006*\"customer\" + 0.005*\"music\" + 0.005*\"funk\"'),\n",
       " (73,\n",
       "  '0.132*\"account\" + 0.060*\"switch\" + 0.040*\"bank\" + 0.032*\"monzo\" + 0.025*\"current\" + 0.022*\"joint\" + 0.016*\"payment\" + 0.014*\"service\" + 0.012*\"old\" + 0.011*\"want\"'),\n",
       " (74,\n",
       "  '0.097*\"pension\" + 0.035*\"tax\" + 0.031*\"get\" + 0.024*\"contribution\" + 0.022*\"employer\" + 0.018*\"pay\" + 0.016*\"scheme\" + 0.015*\"relief\" + 0.014*\"money\" + 0.014*\"retirement\"'),\n",
       " (75,\n",
       "  '0.073*\"identity\" + 0.054*\"limit\" + 0.049*\"verification\" + 0.020*\"information\" + 0.020*\"money\" + 0.019*\"complete\" + 0.013*\"verify\" + 0.012*\"level\" + 0.011*\"need\" + 0.011*\"customer\"'),\n",
       " (76,\n",
       "  '0.067*\"pot\" + 0.047*\"save\" + 0.037*\"set\" + 0.035*\"saving\" + 0.034*\"money\" + 0.028*\"monzo\" + 0.022*\"start\" + 0.020*\"round\" + 0.018*\"aside\" + 0.016*\"day\"'),\n",
       " (77,\n",
       "  '0.018*\"prometheus\" + 0.016*\"server\" + 0.013*\"data\" + 0.011*\"thanos\" + 0.009*\"make\" + 0.008*\"alert\" + 0.008*\"run\" + 0.008*\"query\" + 0.008*\"everything\" + 0.008*\"component\"'),\n",
       " (78,\n",
       "  '0.074*\"monzo\" + 0.038*\"pay\" + 0.035*\"split\" + 0.031*\"bill\" + 0.027*\"share\" + 0.024*\"use\" + 0.023*\"friend\" + 0.023*\"tab\" + 0.021*\"tap\" + 0.020*\"payment\"'),\n",
       " (79,\n",
       "  '0.068*\"saving\" + 0.038*\"money\" + 0.037*\"pot\" + 0.027*\"interest\" + 0.025*\"account\" + 0.020*\"bank\" + 0.018*\"earn\" + 0.016*\"open\" + 0.015*\"offer\" + 0.015*\"monzo\"'),\n",
       " (80,\n",
       "  '0.074*\"lisa\" + 0.028*\"company\" + 0.015*\"figure\" + 0.015*\"solid\" + 0.015*\"meet\" + 0.014*\"criterion\" + 0.014*\"unlike\" + 0.012*\"vary\" + 0.007*\"employed\" + 0.007*\"young\"'),\n",
       " (81,\n",
       "  '0.023*\"name\" + 0.021*\"make\" + 0.020*\"card\" + 0.015*\"use\" + 0.012*\"monzo\" + 0.011*\"thing\" + 0.010*\"top\" + 0.010*\"financial\" + 0.010*\"fraud\" + 0.010*\"sound\"'),\n",
       " (82,\n",
       "  '0.036*\"card\" + 0.032*\"monzo\" + 0.024*\"account\" + 0.022*\"upgrade\" + 0.016*\"prepaid\" + 0.014*\"get\" + 0.014*\"use\" + 0.012*\"new\" + 0.011*\"debit\" + 0.011*\"u\"'),\n",
       " (83,\n",
       "  '0.022*\"ticket\" + 0.017*\"get\" + 0.016*\"delay\" + 0.015*\"make\" + 0.014*\"money\" + 0.014*\"sell\" + 0.014*\"item\" + 0.013*\"company\" + 0.013*\"train\" + 0.011*\"buyer\"'),\n",
       " (84,\n",
       "  '0.034*\"gift\" + 0.032*\"purchase\" + 0.032*\"return\" + 0.030*\"retailer\" + 0.030*\"buy\" + 0.028*\"receipt\" + 0.020*\"shop\" + 0.020*\"may\" + 0.019*\"christmas\" + 0.018*\"product\"'),\n",
       " (85,\n",
       "  '0.041*\"abroad\" + 0.040*\"charge\" + 0.030*\"card\" + 0.029*\"free\" + 0.028*\"withdrawal\" + 0.028*\"fee\" + 0.026*\"atm\" + 0.023*\"exchange\" + 0.023*\"rate\" + 0.021*\"make\"'),\n",
       " (86,\n",
       "  '0.024*\"team\" + 0.023*\"work\" + 0.021*\"monzo\" + 0.020*\"customer\" + 0.016*\"people\" + 0.015*\"u\" + 0.015*\"help\" + 0.015*\"make\" + 0.010*\"need\" + 0.009*\"also\"'),\n",
       " (87,\n",
       "  '0.023*\"inflation\" + 0.022*\"choose\" + 0.011*\"numeracy\" + 0.010*\"international\" + 0.008*\"national\" + 0.008*\"song\" + 0.008*\"math\" + 0.007*\"customer\" + 0.007*\"way\" + 0.007*\"make\"'),\n",
       " (88,\n",
       "  '0.029*\"use\" + 0.025*\"data\" + 0.015*\"time\" + 0.014*\"api\" + 0.012*\"model\" + 0.011*\"build\" + 0.010*\"developer\" + 0.009*\"event\" + 0.009*\"one\" + 0.008*\"want\"'),\n",
       " (89,\n",
       "  '0.030*\"loan\" + 0.026*\"overdraft\" + 0.022*\"car\" + 0.019*\"unsecured\" + 0.018*\"secure\" + 0.016*\"use\" + 0.016*\"card\" + 0.016*\"credit\" + 0.014*\"borrow\" + 0.013*\"purchase\"'),\n",
       " (90,\n",
       "  '0.019*\"money\" + 0.019*\"budget\" + 0.016*\"shawbrook\" + 0.016*\"kid\" + 0.015*\"child\" + 0.013*\"help\" + 0.013*\"find\" + 0.012*\"goal\" + 0.012*\"saving\" + 0.012*\"may\"'),\n",
       " (91,\n",
       "  '0.028*\"save\" + 0.020*\"baby\" + 0.016*\"use\" + 0.015*\"year\" + 0.013*\"money\" + 0.012*\"help\" + 0.011*\"week\" + 0.011*\"liverpool\" + 0.011*\"nappy\" + 0.009*\"work\"'),\n",
       " (92,\n",
       "  '0.013*\"model\" + 0.012*\"may\" + 0.009*\"use\" + 0.009*\"word\" + 0.009*\"question\" + 0.009*\"pot\" + 0.008*\"th\" + 0.008*\"total\" + 0.008*\"idea\" + 0.007*\"time\"'),\n",
       " (93,\n",
       "  '0.000*\"mondo\" + 0.000*\"use\" + 0.000*\"get\" + 0.000*\"card\" + 0.000*\"time\" + 0.000*\"build\" + 0.000*\"go\" + 0.000*\"money\" + 0.000*\"spend\" + 0.000*\"make\"'),\n",
       " (94,\n",
       "  '0.032*\"product\" + 0.024*\"internal\" + 0.022*\"customer\" + 0.021*\"work\" + 0.017*\"team\" + 0.014*\"support\" + 0.013*\"cop\" + 0.012*\"tool\" + 0.010*\"design\" + 0.009*\"user\"'),\n",
       " (95,\n",
       "  '0.020*\"rate\" + 0.018*\"price\" + 0.014*\"call\" + 0.014*\"money\" + 0.013*\"currency\" + 0.013*\"trade\" + 0.013*\"exchange\" + 0.012*\"order\" + 0.011*\"one\" + 0.011*\"market\"'),\n",
       " (96,\n",
       "  '0.030*\"monzo\" + 0.026*\"make\" + 0.016*\"feature\" + 0.015*\"work\" + 0.015*\"help\" + 0.015*\"u\" + 0.014*\"see\" + 0.014*\"app\" + 0.013*\"add\" + 0.012*\"month\"'),\n",
       " (97,\n",
       "  '0.025*\"account\" + 0.020*\"card\" + 0.019*\"monzo\" + 0.017*\"current\" + 0.014*\"work\" + 0.014*\"payment\" + 0.012*\"get\" + 0.012*\"debit\" + 0.011*\"pay\" + 0.010*\"app\"'),\n",
       " (98,\n",
       "  '0.077*\"framework\" + 0.034*\"progression\" + 0.024*\"people\" + 0.020*\"manager\" + 0.020*\"discipline\" + 0.017*\"make\" + 0.015*\"work\" + 0.014*\"use\" + 0.013*\"company\" + 0.013*\"level\"'),\n",
       " (99,\n",
       "  '0.065*\"app\" + 0.060*\"update\" + 0.047*\"release\" + 0.032*\"support\" + 0.031*\"store\" + 0.029*\"version\" + 0.026*\"bug\" + 0.024*\"feature\" + 0.023*\"new\" + 0.021*\"fix\"')]"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "guided_lda.print_topics(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, given the arbirtrary ordering of the blogs, as a result of the glob library used to read them all in, this does not allow us to easily verify if the results are accurate. Therefore, we take a concrete example. Let us see for example what the predicted topics are for the second blog post in our data set, titled ‘I put £1 in a Pot every time I swore during Ramadan’. Now obviously, we expect this document to be about savings, since the 'Pot' is Monzo terminology for a savings account. First, we need to find the document in our data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Block gambling transactions from your Monzo account\n"
     ]
    }
   ],
   "source": [
    "blog_index = -1\n",
    "for index, blog in enumerate(blogs):\n",
    "    if blog.get_title() == \"Block gambling transactions from your Monzo account\":\n",
    "        blog_index = index\n",
    "print(blogs[blog_index].get_title())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having found the correct blog, we now see what document topics our LDA believes it belongs to, which is relatively straightforward to do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(8, 0.090974025),\n",
       " (13, 0.5695669),\n",
       " (55, 0.22475727),\n",
       " (65, 0.06227863),\n",
       " (78, 0.05018204)]"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda.get_document_topics(bag_of_words[blog_index])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('monzo', 0.0607184),\n",
       " ('app', 0.025906827),\n",
       " ('pay', 0.023707861),\n",
       " ('feature', 0.023042062),\n",
       " ('android', 0.020099118),\n",
       " ('account', 0.016735727),\n",
       " ('share', 0.01527054),\n",
       " ('work', 0.014750406),\n",
       " ('add', 0.014287875),\n",
       " ('bill', 0.013862641)]"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda.show_topic(78)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Should I use a credit card for my holiday?\n"
     ]
    }
   ],
   "source": [
    "blog_index = -1\n",
    "for index, blog in enumerate(blogs):\n",
    "    if blog.get_title() == \"Should I use a credit card for my holiday?\":\n",
    "        blog_index = index\n",
    "print(blogs[blog_index].get_title())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(17, 0.10478089),\n",
       " (25, 0.06570986),\n",
       " (36, 0.04847568),\n",
       " (43, 0.18249317),\n",
       " (51, 0.29793718),\n",
       " (56, 0.028525414),\n",
       " (87, 0.099248886),\n",
       " (89, 0.1707452)]"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda.get_document_topics(bag_of_words[blog_index])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('bank', 0.035058785),\n",
       " ('charge', 0.02711987),\n",
       " ('abroad', 0.01717822),\n",
       " ('account', 0.016829802),\n",
       " ('fee', 0.016416285),\n",
       " ('free', 0.015912335),\n",
       " ('withdrawal', 0.014838539),\n",
       " ('make', 0.01450894),\n",
       " ('atm', 0.013768934),\n",
       " ('money', 0.013627977)]"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda.show_topic(87)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now, we look at Automated Document Summarization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by using the conventional summarisation function from gensim."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.summarization import summarize, keywords\n",
    "from scipy.sparse.linalg import svds\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "import re\n",
    "import networkx\n",
    "\n",
    "def text_summarization_gensim(text, summary_ratio=0.5):\n",
    "    summary = summarize(text, split=True, ratio=summary_ratio)\n",
    "    for sentence in summary:\n",
    "        print(sentence)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we want to see how this works in a more detailed manner, we start by creating our own summariser using Latent Semantic Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_document(document):\n",
    "    document = re.sub('\\n', ' ', document)\n",
    "    if isinstance(document, str):\n",
    "        document = document\n",
    "    elif isinstance(document, unicode):\n",
    "        return unicodedata.normalize('NFKD', document).encode('ascii',\n",
    "        'ignore')\n",
    "    else:\n",
    "        raise ValueError('Document is not string or unicode!')\n",
    "    document = document.strip()\n",
    "    sentences = nltk.sent_tokenize(document)\n",
    "    sentences = [sentence.strip() for sentence in sentences]\n",
    "    return sentences\n",
    "\n",
    "def low_rank_svd(matrix, singular_count=2):\n",
    "    u, s, vt = svds(matrix, k=singular_count)\n",
    "    return u, s, vt\n",
    "\n",
    "def build_feature_matrix(documents, feature_type='frequency'):\n",
    "    feature_type = feature_type.lower().strip()\n",
    "    if feature_type == 'binary':\n",
    "        vectorizer = CountVectorizer(binary=True, min_df=1,\n",
    "                                     ngram_range=(1, 1))\n",
    "    elif feature_type == 'frequency':\n",
    "        vectorizer = CountVectorizer(binary=False, min_df=1,\n",
    "                                     ngram_range=(1, 1))\n",
    "    elif feature_type == 'tfidf':\n",
    "        vectorizer = TfidfVectorizer(min_df=1,\n",
    "                                     ngram_range=(1, 1))\n",
    "    else:\n",
    "        pass\n",
    "    feature_matrix = vectorizer.fit_transform(documents).astype(float)\n",
    "    return vectorizer, feature_matrix\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lsa_text_summarizer(documents, num_sentences=2,\n",
    "                        num_topics=2, feature_type='frequency',\n",
    "                        sv_threshold=0.5):\n",
    "    vec, dt_matrix = build_feature_matrix(documents,\n",
    "                                          feature_type=feature_type)\n",
    "    td_matrix = dt_matrix.transpose()\n",
    "    td_matrix = td_matrix.multiply(td_matrix > 0)\n",
    "    u, s, vt = low_rank_svd(td_matrix, singular_count=num_topics)\n",
    "    min_sigma_value = max(s) * sv_threshold\n",
    "    s[s < min_sigma_value] = 0\n",
    "    salience_scores = np.sqrt(np.dot(np.square(s), np.square(vt)))\n",
    "    top_sentence_indices = (salience_scores.argsort()\n",
    "                            [-num_sentences:][::-1])\n",
    "    top_sentence_indices.sort()\n",
    "    for index in top_sentence_indices:\n",
    "        print(documents[index])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Should I use a credit card for my holiday?\n",
      "There are three main reasons why you might use a credit card for your holiday: to borrow money, protect your purchases, and access your money abroad.\n",
      "Try estimating costs by:    Tracking flight prices with Google Flights  Using an app to predict flight and hotel prices, such as Hopper    Browsing credit deals on comparison sites – just remember that you may not get the advertised rate (the representative APR)    How much interest will you pay?\n",
      "Even though you could stand to save money by taking advantage of deals when you see them, once you factor in the interest you’ll pay for borrowing money, you could actually end up spending more than you would if you just saved up and waited.\n",
      "Holidays aren’t cheap, so it’s important to get your money back if something goes wrong – like if your flight is cancelled or your hotel isn’t up to scratch.\n"
     ]
    }
   ],
   "source": [
    "print(blogs[blog_index].get_title())\n",
    "lsa_text_summarizer(parse_document(documents[blog_index]), num_sentences = 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Should I use a credit card for my holiday?\n",
      "There are three main reasons why you might use a credit card for your holiday: to borrow money, protect your purchases, and access your money abroad.\n",
      "Should I use a credit card to pay for my holiday?\n",
      "Most debit and credit cards charge a fee for withdrawing or spending money abroad.\n"
     ]
    }
   ],
   "source": [
    "print(blogs[blog_index].get_title())\n",
    "summary = text_summarization_gensim(documents[blog_index], summary_ratio = 0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The detecting of new sentences could do with some work, but for the rest the results are quite impressive. However, I would say the gensim function works better. You can adapt it using the summary ratio parameter, just like with our function and the number of sentences and topics parameters. Lastly, we look at an algorithm based on Google's TextRank algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "def textrank_text_summarizer(document, num_sentences=None,\n",
    "                             feature_type='frequency',\n",
    "                            summary_ratio = None):\n",
    "    if not (num_sentences or summary_ratio):\n",
    "        print(\"Enter either a maximum number of sentences or a summary ratio\")\n",
    "        return\n",
    "    elif not num_sentences:\n",
    "        num_sentences = int(summary_ratio * len(document))\n",
    "    vec, dt_matrix = build_feature_matrix(document,\n",
    "                                      feature_type='tfidf')\n",
    "    similarity_matrix = (dt_matrix * dt_matrix.T)\n",
    "    similarity_graph = networkx.from_scipy_sparse_matrix(\n",
    "        similarity_matrix)\n",
    "    scores = networkx.pagerank(similarity_graph)\n",
    "    ranked_sentences = sorted(((score, index)\n",
    "                                for index, score\n",
    "                                in scores.items()),\n",
    "                              reverse=True)\n",
    "    top_sentence_indices = [ranked_sentences[index][1]\n",
    "                            for index in range(num_sentences)]\n",
    "    top_sentence_indices.sort()\n",
    "    for index in top_sentence_indices:\n",
    "        print(document[index])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Should I use a credit card for my holiday?\n",
      "There are three main reasons why you might use a credit card for your holiday: to borrow money, protect your purchases, and access your money abroad.\n",
      "It depends on how fast the price of your holiday goes up, how much by, and what you pay to use credit.\n",
      "Even though you could stand to save money by taking advantage of deals when you see them, once you factor in the interest you’ll pay for borrowing money, you could actually end up spending more than you would if you just saved up and waited.\n",
      "Only use credit to buy your holiday if you’re sure you can repay it.\n",
      "The difference is they aren’t a form of credit, so you won’t be borrowing money to use them.\n"
     ]
    }
   ],
   "source": [
    "print(blogs[blog_index].get_title())\n",
    "textrank_text_summarizer(parse_document(documents[blog_index]),\n",
    "                         summary_ratio = 0.10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These results are very impressive! This is a very good summary of the blog post, and answers the question in the title of the post."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Here we start clustering the documents, but to first get an overview we compute the cosine similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cosine_similarity(bag_of_words, num_features):\n",
    "    index = MatrixSimilarity(bag_of_words, num_features = num_features)\n",
    "    cosine_similarity = []\n",
    "    for document in bag_of_words:\n",
    "        cosine_similarity.append(index[document])\n",
    "    return cosine_similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/21.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cosine_similarity = get_cosine_similarity(bag_of_words, num_features=len(dictionary))\n",
    "\n",
    "trace = go.Heatmap(z=cosine_similarity)\n",
    "data=[trace]\n",
    "layout = go.Layout(\n",
    "    title='Frequency cosine similarity heatmap',\n",
    ")\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "py.iplot(fig, filename='basic-heatmap')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The logical progression is now to look at the cosine similarity of all documents, now using the TF-IDF representation rather than the simple BOW representation. Luckily, there are functions that allow us to convert between the two fairly quickly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/23.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_bag = TfidfVectorizer().fit_transform(documents)\n",
    "cosine_similarity = get_cosine_similarity(tfidf_bag.todense(), tfidf_bag.shape[1])\n",
    "\n",
    "trace = go.Heatmap(z=cosine_similarity)\n",
    "data=[trace]\n",
    "layout = go.Layout(\n",
    "    title='TF-IDF cosine similarity Heatmap',\n",
    ")\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "py.iplot(fig, filename='tfidf-heatmap')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see a much sharper and more refined image here, but this is to be expected as a result of using the TF-IDF instead of simple counts. TF-IDF highlights the differences between documents by putting extra weight on those terms which are common in the document but not common in the rest of the corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now we look at clustering methods for these documents\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans, AffinityPropagation\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "def k_means_cluster(matrix, num_clusters=10):\n",
    "    k_means = KMeans(n_clusters = num_clusters,\n",
    "                     random_state = 12,\n",
    "                    max_iter = 1000000)\n",
    "    k_means.fit(matrix)\n",
    "    clusters = k_means.labels_\n",
    "    return k_means, clusters\n",
    "\n",
    "\n",
    "def affinity_propagation_cluster(matrix):\n",
    "    sim = matrix * matrix.T\n",
    "    sim = sim.todense()\n",
    "    ap = AffinityPropagation()\n",
    "    ap.fit(sim)\n",
    "    clusters = ap.labels_\n",
    "    return ap, clusters\n",
    "\n",
    "def print_clusters(documents, clusters, num_clusters):\n",
    "    for cluster_number in range(num_clusters):\n",
    "        print('~~~~~~~Cluster Number %d~~~~~' % cluster_number)\n",
    "        for index, cluster in enumerate(clusters):\n",
    "            if cluster == cluster_number:\n",
    "                print(documents[index].get_title(), documents[index].get_date())\n",
    "            \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "~~~~~~~Cluster Number 0~~~~~\n",
      "See your Year in Monzo! 8 Jan 2019\n",
      "Keep track of what you’re owed with Shared Tabs 13 Dec 2018\n",
      "New Monzo Update 12 Sep 2016\n",
      "Monzo Monthly Update 6 Feb 2018\n",
      "Monzo Monthly Update 5 Jul 2018\n",
      "See your receipts in Monzo with Flux 31 Jan 2019\n",
      "How Much?! Introducing  Your Spending 22 Jan 2016\n",
      "Monzo Monthly Update 28 Feb 2018\n",
      "Monzo Monthly Update 9 Jan 2019\n",
      "Pulse Comes to Android 14 Feb 2018\n",
      "Monzo with Friends 14 Nov 2016\n",
      "How We Build the Monzo App 2 Mar 2018\n",
      "Custom Monzo app icons on iOS 3 Jul 2018\n",
      "Monzo Monthly Update 6 Mar 2019\n",
      "Pay on the go using Monzo with Apple Pay 17 May 2018\n",
      "No more waiting list! 24 May 2018\n",
      "We’re halfway through The Big List! An update 3 Jul 2018\n",
      "Testing Marketplace Beta 1 Mar 2018\n",
      "Spending on Android! New Update Out Now 14 Nov 2016\n",
      "Give the gift of Monzo this Christmas 🎁 11 Dec 2018\n",
      "Save together with Pots on joint accounts! 10 Dec 2018\n",
      "Payments on Android! 6 Dec 2016\n",
      "Half a Million Current Accounts! 19 Mar 2018\n",
      "Mondo on Android 25 May 2016\n",
      "Mondo Version 1.5 13 Jun 2016\n",
      "Help us build a bank! 4 Jun 2015\n",
      "Monzo Monthly Update 5 Apr 2019\n",
      "We’re testing energy switching 27 Feb 2019\n",
      "Helping every customer 24 Apr 2018\n",
      "Terms and Conditions 20 Mar 2018\n",
      "Launching Monzo.me! 7 Dec 2016\n",
      "Custom Monzo app icons on Android 21 Dec 2018\n",
      "Monzo Monthly Update 11 Dec 2018\n",
      "Monzo App Shortcuts on Android 16 Jan 2017\n",
      "Our Festive FAQs 12 Dec 2016\n",
      "Joining Monzo as a backend engineer 28 Jun 2018\n",
      "Try out joint accounts in Monzo Labs! 13 Jul 2018\n",
      "Getting a Monzo bank account as a recent immigrant to the UK 29 May 2018\n",
      "Getting a Monzo bank account as a refugee 30 May 2018\n",
      "We're Ready 30 Oct 2015\n",
      "Announcing Mondo v1.4 9 May 2016\n",
      "New Mondo Update 18 Jul 2016\n",
      "Split The Bill Using Monzo.me on Android 15 Mar 2017\n",
      "Golden Tickets Are Back! 🎫 10 Jan 2018\n",
      "Launching Settings on Android! 9 Feb 2017\n",
      "Announcing the Monzo Overdrafts Preview 24 Aug 2017\n",
      "Bill splitting made simple 9 Oct 2018\n",
      "Upgraded Monzo Accounts for Everyone! 6 Dec 2017\n",
      "Why Upgrade? 18 Dec 2017\n",
      "An Apology from Monzo 4 Aug 2017\n",
      "New Android Update 1 Aug 2017\n",
      "New Android Update 28 Mar 2017\n",
      "Split The Bill Using Monzo.me 22 Feb 2017\n",
      "Pots Are Now on Android! 7 Dec 2017\n",
      "Monzo Monthly Update 1 Oct 2018\n",
      "Launching Monzo.me on Android! 24 Feb 2017\n",
      "New iOS Update 5 Apr 2017\n",
      "Receiving international payments with Monzo 24 Jul 2018\n",
      "The Current Account Preview 23 Aug 2017\n",
      "The Year Ahead, 2018 11 Jan 2018\n",
      "Our 2017 To-Do List 23 Nov 2017\n",
      "Start spending immediately after signup with Google Pay 19 Nov 2018\n",
      "Our Plan to Close the Monzo Prepaid Cards 18 Jan 2018\n",
      "Top ups are back! 1 Dec 2017\n",
      "The Next Step 10 Oct 2017\n",
      "We're making it easier to manage your bills 8 Aug 2018\n",
      "An update on PIN lock 17 Aug 2018\n",
      "The Monzo Current Account Preview comes to Android! 7 Aug 2017\n",
      "ATM Fees Abroad 17 Oct 2017\n",
      "More Beautiful Direct Debits in Your Monzo Account 19 Jan 2018\n",
      "Introducing Monzo Chat 2 Nov 2018\n",
      "Launching Android Pay Top Ups! 22 May 2017\n",
      "A Current Account Rollout Update 25 Oct 2017\n",
      "The end of The Big List! 22 Aug 2018\n",
      "Delays Sending Out Cards 5 Oct 2017\n",
      "Monzo-to-Monzo Payments Come to the Current Account 30 Oct 2017\n",
      "Monzo Monthly Update 12 Nov 2018\n",
      "Searching for Answers on Android 27 Sep 2017\n",
      "Android Pay is Here! 24 Oct 2017\n",
      "Launching 3D Secure! 22 Aug 2018\n",
      "Try splitting the bill in Monzo Labs on iOS! 15 Aug 2018\n",
      "You can now switch energy supplier through Monzo! 8 May 2019\n",
      "An Android Progress Report 25 Sep 2017\n",
      "Start spending immediately after signup with Apple Pay 5 Nov 2018\n",
      "Making Monzo a better current account 16 Oct 2018\n",
      "Terminal decline 5 May 2017\n",
      "Making Monzo Signup Simple 3 Nov 2017\n",
      "Launching Machine Learning Powered Help on iOS 18 Jul 2017\n",
      "Introducing Monzo Plus 23 Apr 2019\n",
      "The Monzo Current Account Preview! 17 Jul 2017\n",
      "Pay in 15 seconds with Android Pay on Monzo.me! 8 May 2017\n",
      "We’re Building Monzo into a Marketplace 16 Nov 2017\n",
      "Testing Energy Partnerships 21 Sep 2017\n",
      "Manage your money together with joint accounts! 14 Aug 2018\n",
      "Monzo Monthly Update 4 Sep 2018\n",
      "Monzo Monthly Update 3 Aug 2018\n",
      "Monzo Plus 15 May 2019\n",
      "Help us test 3D Secure in Monzo Labs! 13 Aug 2018\n",
      "ATM Fees Abroad 15 Dec 2017\n",
      "Tap to Activate Your Debit Card 14 Nov 2017\n",
      "What we’ll build in 2019 29 Jan 2019\n",
      "View Your PIN 3 Jan 2017\n",
      "Launching The Bank 15 Aug 2016\n",
      "Welcome to Mondo! 1 Jun 2015\n",
      "Introducing Sneak Peeks 9 May 2018\n",
      "We’re launching the first 100 Monzo Business Accounts! 11 Feb 2019\n",
      "You've saved over £1,000,000 with Coin Jar! 11 May 2018\n",
      "New Monzo Update 17 Oct 2016\n",
      "How we hire product designers 17 Jan 2019\n",
      "An Update on Overdrafts 23 Jan 2018\n",
      "#TagYourTransactions 12 Apr 2018\n",
      "Introducing Monzo Labs 27 Apr 2018\n",
      "Monzo Time 26 Jun 2018\n",
      "Block gambling transactions from your Monzo account 19 Jun 2018\n",
      "Access your account in an emergency with Monzo Web 21 Jun 2018\n",
      "New Monzo Update 10 Oct 2016\n",
      "New Mondo Update 8 Aug 2016\n",
      "Monzo Does Bank Statements 25 Jan 2018\n",
      "The Year in Monzo 2018 17 Jan 2019\n",
      "Monzo Monthly Update 5 Apr 2018\n",
      "We've added some new categories 9 May 2018\n",
      "Getting Students On Board 23 Sep 2016\n",
      "Send money abroad with Monzo and TransferWise! 25 Jun 2018\n",
      "Pay people with your fingerprint on Android! 5 Jul 2018\n",
      "Pay anyone, with just a link 4 Feb 2019\n",
      "Making recurring payments on Monzo even better 4 Jul 2018\n",
      "The Next Step 17 Mar 2016\n",
      "Better Transaction Information on Android 15 Feb 2018\n",
      "Nearby Friends 8 May 2018\n",
      "Launching Early Access to Travel Reports 16 Aug 2016\n",
      "Android is Here! 29 Sep 2016\n",
      "New Monzo Update 26 Oct 2016\n",
      "Monzo Monthly Update 4 Jun 2018\n",
      "Our plans to help you pay in cheques, just by taking a picture 21 Jun 2018\n",
      "Making Monzo Better 22 May 2018\n",
      "The Monzo Prepaid Beta Has Ended 4 Apr 2018\n",
      "Our Approach to Overdrafts 7 Sep 2016\n",
      "Monzo Monthly Update 4 May 2018\n",
      "Using Monzo over Christmas 🎄 13 Dec 2018\n",
      "~~~~~~~Cluster Number 1~~~~~\n",
      "Flight delays and cancellations – know your rights 18 Dec 2018\n",
      "Upgrading Explained 23 Feb 2018\n",
      "How to ask for a pay rise (and how to spend it when you get one!) 8 Mar 2019\n",
      "How to sell stuff online 13 Mar 2019\n",
      "Set a monthly spending budget with Summary 5 Jul 2018\n",
      "Upgrading Explained 7 Feb 2018\n",
      "How to look after your mental health for free 20 Dec 2018\n",
      "Christmas shopping and sales – know your rights 12 Dec 2018\n",
      "​​Getting a bank account with no fixed address 1 Jun 2018\n",
      "How to use Monzo to manage money with your housemates 15 Mar 2019\n",
      "Monzo 2.0 23 May 2018\n",
      "​​Banking Aboard 31 May 2018\n",
      "Bringing Monzo.me to Everyone! 12 Feb 2018\n",
      "Returning purchases – know your rights 14 Dec 2018\n",
      "Upgrading Explained 1 Feb 2018\n",
      "Say thanks with emoji! 18 Apr 2018\n",
      "How to manage money when you’re getting out of debt 26 Feb 2019\n",
      "What to do if your card's been lost or stolen 11 Jun 2018\n",
      "Just Landed 25 May 2018\n",
      "How do card payments work? 13 Jun 2018\n",
      "Train delays and cancellations – know your rights 22 Jan 2019\n",
      "Unlock the Android App with your Fingerprint! 15 Mar 2018\n",
      "Ending the Monzo Prepaid Beta 31 Jan 2018\n",
      "How to use Monzo as a 5-a-side football team 26 Feb 2019\n",
      "How to protect your money from authorised push payment fraud 7 Feb 2019\n",
      "7 ways to save money on train tickets 22 Jan 2019\n",
      "Why we’re lending 20 Nov 2018\n",
      "Share with us 25 Sep 2018\n",
      "How to budget and save money as a student 11 Sep 2018\n",
      "How to protect yourself against APP fraud 23 May 2019\n",
      "Introducing Monzo for 16-17 year olds! 6 Aug 2018\n",
      "Student loans in Northern Ireland explained 13 Sep 2018\n",
      "Understanding your payslip 26 Sep 2018\n",
      "We’re making Summary more flexible 20 Nov 2018\n",
      "You can now pay cash into your Monzo account! 21 Nov 2018\n",
      "How to save for a wedding 28 Nov 2018\n",
      "How to rent student accommodation 10 Sep 2018\n",
      "Dear Monzo, buying a house feels out of reach 22 May 2019\n",
      "ATM Fees Abroad 4 Jan 2018\n",
      "How to protect yourself against identity theft and fraud 21 Sep 2018\n",
      "A guide to PAYE forms 10 Oct 2018\n",
      "Money tips for moving to the UK 7 Sep 2018\n",
      "How to open a UK bank account 7 Sep 2018\n",
      "Asking Some Customers To Do Additional Identity Verification 13 Jun 2017\n",
      "New to the UK? Open a bank account in minutes with Monzo 7 Sep 2018\n",
      "What is FSCS protection? 22 Oct 2018\n",
      "How to get a bank statement from Monzo 5 Sep 2018\n",
      "How to manage your bills with Monzo 15 Aug 2018\n",
      "11 reasons you should definitely get Monzo 10 May 2019\n",
      "How to get your salary paid into your Monzo account 4 Sep 2018\n",
      "Should you share your finances? 14 Aug 2018\n",
      "How to budget with Monzo 21 Aug 2018\n",
      "Search Launches on Android! 11 Apr 2017\n",
      "How to save for a house deposit 🏠 23 Nov 2018\n",
      "The 7-Day Switch! What happens when you switch to Monzo? 23 Aug 2018\n",
      "Earning money? Here's what you need to know 10 Aug 2018\n",
      "Student loans explained 10 Aug 2018\n",
      "How to manage your bills 30 Aug 2018\n",
      "What is crowdfunding? 23 Nov 2018\n",
      "Is Monzo free? 3 Sep 2018\n",
      "How to get fit on a budget 21 Jan 2019\n",
      "Can I have more than one bank account? 26 Apr 2018\n",
      "Is Monzo a bank? 7 Jun 2018\n",
      "Is Monzo safe? 6 Feb 2019\n",
      "You can now switch your joint accounts to Monzo! 7 Feb 2019\n",
      "Dear Monzo, I'm £7,000 in debt and I’m struggling to clear it 9 Apr 2019\n",
      "Bill Tracker tells you when a regular bill changes 14 Jun 2018\n",
      "We’re seeing more pyramid schemes on social media. Here’s how to protect yourself. 2 Apr 2019\n",
      "Summary Update - Predicting Committed Spending 7 Jun 2018\n",
      "Overdrafts are coming! 9 Apr 2018\n",
      "Switch to Monzo with the Current Account Switch Service! 25 Apr 2018\n",
      "How to budget (and actually stick to it!) 18 Jan 2019\n",
      "How cashback works 28 Jan 2019\n",
      "How to manage money when you have kids 👶 22 Jan 2019\n",
      "Piggy banking 21 Feb 2019\n",
      "How to use a zero-sum budget 6 Feb 2019\n",
      "How to use the 50/20/30 budget rule 6 Feb 2019\n",
      "How do I add money to my Monzo account? 8 Jun 2018\n",
      "Using a Monzo overdraft 18 Apr 2018\n",
      "What’s a Golden Ticket? 24 Jan 2019\n",
      "How much does a dog cost? 30 Jan 2019\n",
      "How to get a mortgage 17 Apr 2019\n",
      "Monzo Monthly Update 13 Feb 2019\n",
      "How to use Monzo to split bills 1 Feb 2019\n",
      "How to budget if you don’t get paid regularly 6 Mar 2019\n",
      "We added Travel Reports to Android 6 Jun 2018\n",
      "12 things to say to your friends to convince them to get Monzo 30 Jan 2019\n",
      "How much does it cost to have a baby? 30 Jan 2019\n",
      "How Monzo helps you plan ahead 16 Jan 2019\n",
      "How to throw a Christmas party without going broke 13 Dec 2018\n",
      "Can I switch bank accounts if I have an overdraft? 30 Apr 2018\n",
      "Switch to Monzo and Help us Test the Current Account Switch Service 3 Apr 2018\n",
      "5 Monzo features you might not know about 11 Apr 2019\n",
      "Home insurance – do you really need it? 19 Feb 2019\n",
      "How to save when you have no money 14 Feb 2019\n",
      "How to give back when you’re broke 4 Feb 2019\n",
      "Why doesn’t Monzo have branches? 22 Jun 2018\n",
      "Icelandic airline WOW Air has gone bust. Have you been affected? 28 Mar 2019\n",
      "Launching Targets 3 Oct 2016\n",
      "How to save money and the planet at the same time 30 Jan 2019\n",
      "~~~~~~~Cluster Number 2~~~~~\n",
      "How to save for retirement 27 Nov 2018\n",
      "The big guide to ISAs 14 Nov 2018\n",
      "An update on getting interest paid on savings 21 Aug 2018\n",
      "Flexible Savings Pots are back - with a new provider! 🎉 2 May 2019\n",
      "Choosing between different types of savings accounts 24 Oct 2018\n",
      "Inflation 30 Oct 2018\n",
      "Our vision to help people earn more from their savings 26 Apr 2019\n",
      "The big guide to interest 25 Oct 2018\n",
      "The big guide to pensions 7 Nov 2018\n",
      "You can now open an ISA through your Monzo app 🎉 4 Apr 2019\n",
      "What’s the exchange rate on Monzo? 5 Jun 2018\n",
      "~~~~~~~Cluster Number 3~~~~~\n",
      "Should I use a credit card for my holiday? 4 Mar 2019\n",
      "Upgrading Explained 26 Jan 2018\n",
      "Should you borrow money? 9 Aug 2018\n",
      "Should I get a credit card? 17 Apr 2019\n",
      "What kind of credit should I use? 9 Aug 2018\n",
      "How to improve your credit score 8 Aug 2018\n",
      "Secured and unsecured credit - what's the difference? 5 Oct 2018\n",
      "What is a credit score? 9 Aug 2018\n",
      "What is a guarantor? 4 Oct 2018\n",
      "Good debt vs. bad debt 24 Apr 2019\n",
      "What is a 'good' credit score? 12 Oct 2018\n",
      "4 money mistakes you might be making 24 Apr 2019\n",
      "How to check your credit score and report 11 Oct 2018\n",
      "~~~~~~~Cluster Number 4~~~~~\n",
      "Micro-saving is the trend that’s making it easier to save 30 May 2019\n",
      "How to manage money when you’re terrible with money 3 Mar 2019\n",
      "Stick to your savings goals by locking your Pots 13 Dec 2018\n",
      "Schedule payments into Pots! 16 Jul 2018\n",
      "Round up your Monzo purchases automatically! 18 Jul 2018\n",
      "Try this one trick to stop yourself spending your savings 18 Mar 2019\n",
      "You can now add your own images to Pots, and you've been taking full advantage 18 Mar 2019\n",
      "Save Spare Change with Coin Jar 7 Mar 2018\n",
      "Set goals for your Pots! 2 Aug 2018\n",
      "Get interest paid on your savings! 1 Nov 2018\n",
      "Everything you need to know about saving money that you didn't learn at school 17 May 2019\n",
      "How to save money with Monzo 7 May 2019\n",
      "3 ways to save 'little and often' with Monzo 1 May 2019\n",
      "Move money out of Pots automatically! 23 Aug 2018\n",
      "How to automate your financial life with Monzo 16 Aug 2018\n",
      "Try setting goals for Pots in Monzo Labs! 19 Jul 2018\n",
      "Launching Pots! 20 Nov 2017\n",
      "Save £1,378 a year with the 52 Week Savings Challenge 24 Apr 2019\n",
      "Have you left it too late to start saving? 9 Apr 2019\n",
      "What's your saving style? 6 Dec 2018\n",
      "Automate your financial life with Monzo and IFTTT 7 Jun 2018\n",
      "Save £667.95 a year with the 1p Saving Challenge 10 Apr 2019\n",
      "Why you can't create new Savings Pots – for now 5 Feb 2019\n",
      "Could rounding-up your spending help you afford your next holiday? 23 May 2019\n",
      "What is nudge theory? And can it help you manage your money? 23 May 2019\n",
      "You can now add your own images to Pots on both Android and iOS 11 Apr 2019\n",
      "~~~~~~~Cluster Number 5~~~~~\n",
      "We're Looking for Campus Insiders 1 Jul 2016\n",
      "Publishing our 2018 Annual Report! 2 Jul 2018\n",
      "Welcome to Monzo 2.0 21 May 2018\n",
      "Monzo 2.0 25 May 2018\n",
      "The new Monzo mission 2 Jul 2018\n",
      "An Update on our Name Change 23 Jun 2016\n",
      "Fighting Financial Crime the Monzo Way 31 Jan 2017\n",
      "Why we made a mistake with customer support (and how we fixed it) 17 Dec 2018\n",
      "Introducing the Monzo Tone of Voice 27 Mar 2018\n",
      "Introducing Our New Community Page 26 Jan 2017\n",
      "We're pausing our plans to let you pay in cheques with a picture 22 Mar 2019\n",
      "Making Monzo's quarterly goals public 10 Jul 2018\n",
      "Where in the World Has Mondo Been? 8 Jul 2016\n",
      "100,000 Users! 24 Jan 2017\n",
      "An update on diversity and inclusion at Monzo, March 2018 22 Mar 2018\n",
      "2016 In Numbers 6 Jan 2017\n",
      "Why we sometimes freeze or block Monzo accounts 4 Apr 2019\n",
      "Financial Inclusion 29 May 2018\n",
      "We suspect some data has been compromised in the Typeform breach. All money is safe. 29 Jun 2018\n",
      "An update on diversity and inclusion at Monzo, March 2019 21 Mar 2019\n",
      "Goodbye Mondo, Hello M… 13 Jun 2016\n",
      "Introducing progression.monzo.com! 7 Jan 2019\n",
      "Providing customer support from Las Vegas! 🇺🇸 11 Dec 2018\n",
      "A Quick Update on the New Name, via Snapchat 22 Jul 2016\n",
      "Ask Us Anything -  Join the Conversation on Our Forum 3 Jun 2016\n",
      "Where do Mondo users spend? 29 Apr 2016\n",
      "Getting to know each other  a little better 27 Aug 2015\n",
      "Say hello to the Coral Crew! 16 May 2018\n",
      "How we’ll work towards our quarterly goals 11 Jul 2018\n",
      "What Makes A Bank Ethical? 7 Jun 2016\n",
      "Launching the Monzo Extraordinary Ideas Board 13 Jan 2017\n",
      "Monzo in 2019 4 Jan 2019\n",
      "Releasing Monzo's 2017 Annual Report! 4 Jul 2017\n",
      "£12 Million Later 14 Mar 2017\n",
      "2017 In Numbers 5 Jan 2018\n",
      "We’re crowdfunding up to £20,000,000 and we’d love you to be part of it! 26 Nov 2018\n",
      "1,000,000 people are now using Monzo! 24 Sep 2018\n",
      "An update on our API 11 May 2017\n",
      "The Class of 2017 13 Jul 2017\n",
      "We’re sharing the vision and principles for our internal product 3 Aug 2018\n",
      "How we’re gearing up to support one billion Monzo customers! 30 Jul 2018\n",
      "Have you seen our ads on TV? 22 May 2019\n",
      "Viva Las Vegas! Providing customer support from the States 31 Jul 2018\n",
      "Measuring customer happiness 😃 3 Aug 2018\n",
      "Setting the record straight on our crowdfunding round 30 Nov 2018\n",
      "We’re planning to launch a business current account 22 Nov 2018\n",
      "An update on the Women in Finance Charter 20 Nov 2018\n",
      "You’ve invested £20,000,000 in Monzo! 5 Dec 2018\n",
      "What we're doing to keep Monzo accessible 16 May 2019\n",
      "Diversity and Inclusion at Monzo 9 Mar 2017\n",
      "Making our company goals public 3 Oct 2018\n",
      "Big news 31 Oct 2018\n",
      "Crowdfunding is open to new investors! 5 Dec 2018\n",
      "Introducing the Monzo Transparency Dashboard 6 Mar 2017\n",
      "Turn That Frown Upside Down 9 Aug 2017\n",
      "The Monzo Reliability Report 25 Jul 2018\n",
      "2,000,000 people are now using Monzo! 20 May 2019\n",
      "Announcing Our £22m Investment Round and Crowdfunding! 23 Feb 2017\n",
      "Unleashing the Dragons 1 Dec 2017\n",
      "Welcome to Monzo Bank! 5 Apr 2017\n",
      "Working towards our quarterly goals 18 Jul 2018\n",
      "How our community helps us build a better bank 5 Nov 2018\n",
      "Introducing the Monzo naming system 6 Nov 2018\n",
      "Why Monzo Cards Aren't Working Today 5 Mar 2017\n",
      "Crowdfunding in Numbers 20 Apr 2017\n",
      "Announcing Our Public Hiring Referral Bonus! 7 Jun 2017\n",
      "Host a Monzo Meetup! 8 Jun 2017\n",
      "We've signed the Women in Finance Charter 31 May 2017\n",
      "Invest in Monzo 28 Feb 2017\n",
      "Putting transparency into practice 8 Nov 2018\n",
      "Announcing our New Investment Round of £71 Million 🎉 7 Nov 2017\n",
      "Making Monzo 7 Nov 2018\n",
      "An update on our gender pay gap, April 2019 15 May 2019\n",
      "Tell us when your question's urgent 2 Aug 2018\n",
      "Welcome to The Drawing Board! 27 Apr 2017\n",
      "200,000 Customers and £250,000,000 Spent! 16 May 2017\n",
      "Crowdfunding in numbers 6 Dec 2018\n",
      "Monzo’s gender pay gap 12 Apr 2018\n",
      "We deployed Envoy Proxy to make Monzo faster 3 Apr 2019\n",
      "Fastest Crowdfunding Ever 3 Mar 2016\n",
      "We Have Temporarily Paused our Crowdcube Round 29 Feb 2016\n",
      "We Are Now a Bank 11 Aug 2016\n",
      "Monzo for Alexa – A Community Hack 21 Dec 2016\n",
      "Over £20 Million Spent Through Mondo! 5 Aug 2016\n",
      "Launching the Monzo Shop 👕 6 Oct 2016\n",
      "A Mondo Update...In 96 Seconds 1 Jun 2016\n",
      "Invest in Mondo 15 Feb 2016\n",
      "£1,000,000 spent through Mondo! 26 Feb 2016\n",
      "The Monzo Transparent Product Roadmap 27 May 2016\n",
      "Get £2,500 for Referring a Backend Engineer 27 Apr 2018\n",
      "Mondo is now Monzo! 25 Aug 2016\n",
      "Three Years of Monzo 19 Feb 2018\n",
      "An Update on our Alpha Rollout 14 Jan 2016\n",
      "Big news about small print 22 May 2018\n",
      "The #BringMeMondo Results 15 Jan 2016\n",
      "Celebrating Living Wage Week 4 Nov 2016\n",
      "How we prepared our systems for our £20 million crowdfunding round 16 Jan 2019\n",
      "~~~~~~~Cluster Number 6~~~~~\n",
      "Money in Love 28 Feb 2019\n",
      "Disclosing mental health to your bank 18 May 2018\n",
      "How we manage mental health 30 Jan 2019\n",
      "Supporting Mindfulness and Wellbeing at Monzo 18 May 2018\n",
      "Money in Love 14 Mar 2019\n",
      "Celebrating Pride 🏳️‍🌈 6 Jul 2018\n",
      "Money in Love 22 Mar 2019\n",
      "How to make therapy affordable 20 Dec 2018\n",
      "Meet the Vulnerable Customers Team 15 May 2018\n",
      "How to be vegan and stay within budget 3 Jun 2019\n",
      "‘I put £1 in a Pot every time I swore during Ramadan’ 3 Jun 2019\n",
      "Mental Health Awareness Week at Monzo 14 May 2018\n",
      "The dog ate my Mondo card... 17 May 2016\n",
      "Supporting people to self-exclude from gambling 16 May 2018\n",
      "Are your friends' weddings costing you a fortune? Here's how to cut the costs 4 Jun 2019\n",
      "Introducing 21 Feb 2019\n",
      "Money in Love 18 Apr 2019\n",
      "Things we wish we’d known about money 7 Aug 2018\n",
      "‘I saved up enough money to go to Madrid for the Champions League final’ 22 May 2019\n",
      "Budgeting, Money Management and Planning Ahead 16 Feb 2017\n",
      "Could eating less meat help you cut your costs? 16 May 2019\n",
      "Understanding how young people manage their money 7 Aug 2018\n",
      "Take the National Numeracy Challenge! 15 May 2019\n",
      "​​Do you donate? You explain how you give to charity 9 May 2019\n",
      "‘Impulsivity is one of the symptoms of ADHD’ 16 May 2019\n",
      "Money in Love 8 May 2019\n",
      "Money in Love 25 Apr 2019\n",
      "Meet the Vulnerable Customers Team 13 May 2019\n",
      "An update on how we're supporting mental health 15 May 2019\n",
      "'I give my mum money so she can buy whatever she wants' 10 Apr 2019\n",
      "Money in Love 4 Apr 2019\n",
      "Money in Love 21 Feb 2019\n",
      "How to get a job as an engineer without a computer science degree 6 Mar 2019\n",
      "Money in Love 14 Feb 2019\n",
      "Money in Love 29 Mar 2019\n",
      "Money in Love 11 Apr 2019\n",
      "~~~~~~~Cluster Number 7~~~~~\n",
      "Designing a product with mental health issues in mind 27 Jan 2017\n",
      "Laying the Foundation for a Data Team 30 Nov 2016\n",
      "Crowdsourcing Merchant Data 11 Apr 2016\n",
      "How do bank payments  actually work? 20 Jan 2016\n",
      "Meet Tracy—our Chief  Marketing Officer 15 Oct 2015\n",
      "We’re experiencing problems with some bank transfers 30 May 2019\n",
      "Making Monzo Work in More Places 25 May 2018\n",
      "The Hidden Costs of Free Banking 22 Oct 2015\n",
      "Meet Hugo Cornejo 8 Apr 2016\n",
      "Monzo x Old Street – Our First Shopping Event 5 Dec 2016\n",
      "Building a Modern Bank Backend 19 Sep 2016\n",
      "#mondohack IV 21 Apr 2016\n",
      "Making Our Mark 8 Oct 2015\n",
      "Why everyone at Monzo does research 30 May 2019\n",
      "How we accidentally broke our Android notification sounds (and fixed them!) 28 May 2019\n",
      "The First Mondo Hackathon 1 Oct 2015\n",
      "Automated Testing &  Accessibility on iOS 26 Apr 2016\n",
      "How we improved our staff VPN 19 Mar 2019\n",
      "Engineering Principles at Monzo 29 Jun 2018\n",
      "Mondo at the Bank of England 3 Jul 2015\n",
      "Can You Marry An App? Customer Support Quotables 11 Jan 2017\n",
      "Understanding our Users 2 Feb 2018\n",
      "Why API-Driven Banking Matters 5 Nov 2015\n",
      "How we put together our progression frameworks 7 Feb 2019\n",
      "Meet Paul—Our Chief Risk Officer 4 Aug 2015\n",
      "Protecting customers from the Ticketmaster breach 28 Jun 2018\n",
      "Our Old Street Shopping Guide 8 Dec 2016\n",
      "Why you should care about keeping your design files clean 11 Dec 2018\n",
      "Open Office August 1 Sep 2017\n",
      "Meet Jordan 12 Dec 2017\n",
      "Meet Will - Business Operations 25 Jul 2017\n",
      "The Monzo Mixtape Vol.1 3 Jul 2017\n",
      "Using data to build a better help screen 1 Aug 2018\n",
      "Introducing Monzo Insider! 27 Jul 2017\n",
      "The Monzo Mixtape Vol.2 8 Dec 2017\n",
      "Meet Irina 8 Feb 2017\n",
      "Meet Venkat 22 Mar 2017\n",
      "Working with backend services on iOS 7 Jul 2017\n",
      "COps 15 May 2017\n",
      "The British Airways data breach 7 Sep 2018\n",
      "What we learned from moving Monzo 📦 6 Sep 2018\n",
      "Very Robust etcd 29 Nov 2017\n",
      "Transparent By Default 10 Mar 2017\n",
      "An Update on the Monzo API 12 Jan 2018\n",
      "New Beginnings at Monzo 23 Jun 2017\n",
      "The Help Search Algorithm 22 Aug 2017\n",
      "How international payments work 24 Jul 2018\n",
      "Tried and Tested! 30 Jun 2017\n",
      "Designers’ Show and Tell 31 Mar 2017\n",
      "Monzo's Response to Cloudbleed 24 Feb 2017\n",
      "Live updates 30 Nov 2018\n",
      "Meet Josh 22 Feb 2017\n",
      "Meet Gary - Chief Financial Officer 22 Jun 2017\n",
      "Writing customer-friendly terms and conditions 3 May 2019\n",
      "How we monitor Monzo 27 Jul 2018\n",
      "Meet Emma 9 Mar 2017\n",
      "We’re working with the Financial Capability Lab 15 Nov 2018\n",
      "The Refactathon 4 Dec 2018\n",
      "Share Options 20 Nov 2017\n",
      "Working on Android 29 Sep 2017\n",
      "Meet Dan 6 Apr 2017\n",
      "To a Billion and Beyond 26 May 2017\n",
      "From desktop design to your doorstep 9 Jun 2017\n",
      "Code Club Came to Monzo 20 Oct 2017\n",
      "Sharing our social media guidelines 9 Nov 2018\n",
      "Meet Phil – Head of Partnerships 4 Sep 2017\n",
      "Night COps 18 May 2017\n",
      "Meet Natasha 1 Feb 2017\n",
      "Fighting Fraud with Machine Learning 3 Feb 2017\n",
      "The Future of Customer Operations (\"COps\") 19 May 2017\n",
      "Splitting the bill with Monzo and Deliveroo 8 Sep 2017\n",
      "The national minimum wage, the national living wage and the real living wage 9 Nov 2018\n",
      "Come and join our Customer Operations team! 18 Oct 2018\n",
      "We’re working to improve the performance of our iOS app 18 Sep 2018\n",
      "Meet Tapas from Customer Operations 17 May 2017\n",
      "Practical Machine Learning With Event Streaming 3 May 2017\n",
      "ATM Fees Abroad 13 Sep 2017\n",
      "How we do on-call at Monzo 20 Sep 2018\n",
      "Target Practice 19 Oct 2016\n",
      "3 Second Sandwich 2 Dec 2015\n",
      "What is Open Banking? 14 Jun 2018\n",
      "How we built Monzo Chat on Android 11 Feb 2019\n",
      "#mondohack 30 Nov 2015\n",
      "Diversity Debt 16 Nov 2015\n",
      "We use this tool to turn user research into insight very quickly 11 Feb 2019\n",
      "How we design our internal product 11 Feb 2019\n",
      "#mondohack 22 Nov 2015\n",
      "Organising teams and managing engineers 27 Jun 2018\n",
      "How we load tested our bank before our £20 million crowdfunding round 15 Jan 2019\n",
      "Meet Jonas Huckestein 10 Mar 2016\n",
      "The Mondo API 3 Feb 2016\n",
      "We’re helping The Big Issue sellers get more out of each magazine 1 Apr 2019\n",
      "How Does the Wholesale Foreign Exchange Market Work? 8 Jan 2016\n",
      "Why are foreign card  payments such a mess? 15 Dec 2015\n",
      "#mondohack III 28 Jan 2016\n",
      "We’ve published our Engineering Progression Framework 25 Jun 2018\n",
      "#MonzoJournals - Philippines 20 Sep 2016\n",
      "A technical look at how Monzo-to-Monzo payments work 5 Apr 2018\n",
      "How we built a backend for our £20 million crowdfunding round 14 Jan 2019\n",
      "How We Picked 'Monzo' 26 Aug 2016\n",
      "Happy birthday to the Monzo tone of voice 27 Mar 2019\n",
      "The bank of the future will be a marketplace 4 Feb 2016\n",
      "Meet Leah! 9 Feb 2016\n"
     ]
    }
   ],
   "source": [
    "num_km_clusters = 8\n",
    "km, km_clusters = k_means_cluster(tfidf_bag, num_km_clusters)\n",
    "km_cluster_count_placeholder = Counter(km_clusters)\n",
    "km_cluster_count = [0 for x in km_cluster_count_placeholder.keys()]\n",
    "for key, value in km_cluster_count_placeholder.items():\n",
    "    km_cluster_count[int(key)] = value\n",
    "\n",
    "print_clusters(blogs, km_clusters, num_km_clusters)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.7/site-packages/IPython/core/display.py:689: UserWarning:\n",
      "\n",
      "Consider using IPython.display.IFrame instead\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/27.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = [go.Bar(\n",
    "            x=[i for i in range(len(km_cluster_count))],\n",
    "            y=km_cluster_count,\n",
    "            text=km_cluster_count,\n",
    "            textposition='auto'\n",
    "    )]\n",
    "layout = go.Layout(\n",
    "    title='Number of articles per cluster (K-means Propagation)',\n",
    ")\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "py.iplot(fig, filename='cluster-size-bar')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.7/site-packages/IPython/core/display.py:689: UserWarning:\n",
      "\n",
      "Consider using IPython.display.IFrame instead\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/27.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ap, ap_clusters = affinity_propagation_cluster(tfidf_bag)\n",
    "ap_cluster_count_placeholder = Counter(ap_clusters)\n",
    "num_ap_clusters = len(ap_cluster_count_placeholder)\n",
    "ap_cluster_count = [0 for x in ap_cluster_count_placeholder.keys()]\n",
    "for key, value in ap_cluster_count_placeholder.items():\n",
    "    ap_cluster_count[int(key)] = value\n",
    "\n",
    "data = [go.Bar(\n",
    "            x=[i for i in range(len(ap_cluster_count))],\n",
    "            y=ap_cluster_count,\n",
    "            text=ap_cluster_count,\n",
    "            textposition='auto'\n",
    "    )]\n",
    "layout = go.Layout(\n",
    "    title='Number of articles per cluster (Affinity Propagation)',\n",
    ")\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "py.iplot(fig, filename='cluster-size-bar')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "~~~~~~~Cluster Number 0~~~~~\n",
      "Flight delays and cancellations – know your rights 18 Dec 2018\n",
      "How to be vegan and stay within budget 3 Jun 2019\n",
      "Just Landed 25 May 2018\n",
      "How do card payments work? 13 Jun 2018\n",
      "Train delays and cancellations – know your rights 22 Jan 2019\n",
      "7 ways to save money on train tickets 22 Jan 2019\n",
      "What is a guarantor? 4 Oct 2018\n",
      "Launching Android Pay Top Ups! 22 May 2017\n",
      "​​Do you donate? You explain how you give to charity 9 May 2019\n",
      "Inflation 30 Oct 2018\n",
      "Search Launches on Android! 11 Apr 2017\n",
      "Home insurance – do you really need it? 19 Feb 2019\n",
      "Icelandic airline WOW Air has gone bust. Have you been affected? 28 Mar 2019\n",
      "~~~~~~~Cluster Number 1~~~~~\n",
      "How to sell stuff online 13 Mar 2019\n",
      "How to look after your mental health for free 20 Dec 2018\n",
      "Returning purchases – know your rights 14 Dec 2018\n",
      "What to do if your card's been lost or stolen 11 Jun 2018\n",
      "How to protect yourself against APP fraud 23 May 2019\n",
      "Things we wish we’d known about money 7 Aug 2018\n",
      "How to rent student accommodation 10 Sep 2018\n",
      "How to protect yourself against identity theft and fraud 21 Sep 2018\n",
      "Earning money? Here's what you need to know 10 Aug 2018\n",
      "Save £1,378 a year with the 52 Week Savings Challenge 24 Apr 2019\n",
      "Is Monzo free? 3 Sep 2018\n",
      "We’re seeing more pyramid schemes on social media. Here’s how to protect yourself. 2 Apr 2019\n",
      "Piggy banking 21 Feb 2019\n",
      "How to use a zero-sum budget 6 Feb 2019\n",
      "How to throw a Christmas party without going broke 13 Dec 2018\n",
      "How to save money and the planet at the same time 30 Jan 2019\n",
      "~~~~~~~Cluster Number 2~~~~~\n",
      "Upgrading Explained 23 Feb 2018\n",
      "Upgrading Explained 7 Feb 2018\n",
      "Upgrading Explained 1 Feb 2018\n",
      "How to open a UK bank account 7 Sep 2018\n",
      "Can I have more than one bank account? 26 Apr 2018\n",
      "Overdrafts are coming! 9 Apr 2018\n",
      "Using a Monzo overdraft 18 Apr 2018\n",
      "~~~~~~~Cluster Number 3~~~~~\n",
      "How to manage money when you’re terrible with money 3 Mar 2019\n",
      "Monzo Monthly Update 9 Jan 2019\n",
      "Pulse Comes to Android 14 Feb 2018\n",
      "How to use Monzo to manage money with your housemates 15 Mar 2019\n",
      "Bringing Monzo.me to Everyone! 12 Feb 2018\n",
      "How to use Monzo as a 5-a-side football team 26 Feb 2019\n",
      "How to manage your bills with Monzo 15 Aug 2018\n",
      "How to automate your financial life with Monzo 16 Aug 2018\n",
      "Launching Pots! 20 Nov 2017\n",
      "How to budget with Monzo 21 Aug 2018\n",
      "How do I add money to my Monzo account? 8 Jun 2018\n",
      "12 things to say to your friends to convince them to get Monzo 30 Jan 2019\n",
      "5 Monzo features you might not know about 11 Apr 2019\n",
      "What is nudge theory? And can it help you manage your money? 23 May 2019\n",
      "How to save when you have no money 14 Feb 2019\n",
      "~~~~~~~Cluster Number 4~~~~~\n",
      "Crowdsourcing Merchant Data 11 Apr 2016\n",
      "Meet Tracy—our Chief  Marketing Officer 15 Oct 2015\n",
      "We’re experiencing problems with some bank transfers 30 May 2019\n",
      "#mondohack IV 21 Apr 2016\n",
      "Making Our Mark 8 Oct 2015\n",
      "The First Mondo Hackathon 1 Oct 2015\n",
      "Automated Testing &  Accessibility on iOS 26 Apr 2016\n",
      "2016 In Numbers 6 Jan 2017\n",
      "How we improved our staff VPN 19 Mar 2019\n",
      "Mondo at the Bank of England 3 Jul 2015\n",
      "Goodbye Mondo, Hello M… 13 Jun 2016\n",
      "Where do Mondo users spend? 29 Apr 2016\n",
      "Getting to know each other  a little better 27 Aug 2015\n",
      "The Help Search Algorithm 22 Aug 2017\n",
      "Designers’ Show and Tell 31 Mar 2017\n",
      "#mondohack 30 Nov 2015\n",
      "Diversity Debt 16 Nov 2015\n",
      "#mondohack 22 Nov 2015\n",
      "Meet Jonas Huckestein 10 Mar 2016\n",
      "The Mondo API 3 Feb 2016\n",
      "#mondohack III 28 Jan 2016\n",
      "~~~~~~~Cluster Number 5~~~~~\n",
      "Pay on the go using Monzo with Apple Pay 17 May 2018\n",
      "Unlock the Android App with your Fingerprint! 15 Mar 2018\n",
      "Golden Tickets Are Back! 🎫 10 Jan 2018\n",
      "Start spending immediately after signup with Google Pay 19 Nov 2018\n",
      "Start spending immediately after signup with Apple Pay 5 Nov 2018\n",
      "Pay in 15 seconds with Android Pay on Monzo.me! 8 May 2017\n",
      "Help us test 3D Secure in Monzo Labs! 13 Aug 2018\n",
      "Tap to Activate Your Debit Card 14 Nov 2017\n",
      "#TagYourTransactions 12 Apr 2018\n",
      "Access your account in an emergency with Monzo Web 21 Jun 2018\n",
      "Monzo Does Bank Statements 25 Jan 2018\n",
      "What’s a Golden Ticket? 24 Jan 2019\n",
      "Send money abroad with Monzo and TransferWise! 25 Jun 2018\n",
      "Monzo Monthly Update 13 Feb 2019\n",
      "Pay anyone, with just a link 4 Feb 2019\n",
      "Nearby Friends 8 May 2018\n",
      "New Monzo Update 26 Oct 2016\n",
      "Using Monzo over Christmas 🎄 13 Dec 2018\n",
      "~~~~~~~Cluster Number 6~~~~~\n",
      "How we accidentally broke our Android notification sounds (and fixed them!) 28 May 2019\n",
      "Where in the World Has Mondo Been? 8 Jul 2016\n",
      "100,000 Users! 24 Jan 2017\n",
      "How to make therapy affordable 20 Dec 2018\n",
      "The dog ate my Mondo card... 17 May 2016\n",
      "A Quick Update on the New Name, via Snapchat 22 Jul 2016\n",
      "Protecting customers from the Ticketmaster breach 28 Jun 2018\n",
      "The Monzo Mixtape Vol.2 8 Dec 2017\n",
      "You’ve invested £20,000,000 in Monzo! 5 Dec 2018\n",
      "Live updates 30 Nov 2018\n",
      "2,000,000 people are now using Monzo! 20 May 2019\n",
      "Take the National Numeracy Challenge! 15 May 2019\n",
      "An update on our gender pay gap, April 2019 15 May 2019\n",
      "Over £20 Million Spent Through Mondo! 5 Aug 2016\n",
      "£1,000,000 spent through Mondo! 26 Feb 2016\n",
      "The #BringMeMondo Results 15 Jan 2016\n",
      "~~~~~~~Cluster Number 7~~~~~\n",
      "Monzo Monthly Update 6 Mar 2019\n",
      "We’re halfway through The Big List! An update 3 Jul 2018\n",
      "Testing Marketplace Beta 1 Mar 2018\n",
      "Monzo Monthly Update 5 Apr 2019\n",
      "Helping every customer 24 Apr 2018\n",
      "Joining Monzo as a backend engineer 28 Jun 2018\n",
      "How we’ll work towards our quarterly goals 11 Jul 2018\n",
      "Share with us 25 Sep 2018\n",
      "Our 2017 To-Do List 23 Nov 2017\n",
      "Making our company goals public 3 Oct 2018\n",
      "We’re Building Monzo into a Marketplace 16 Nov 2017\n",
      "Testing Energy Partnerships 21 Sep 2017\n",
      "What we’ll build in 2019 29 Jan 2019\n",
      "Three Years of Monzo 19 Feb 2018\n",
      "Android is Here! 29 Sep 2016\n",
      "~~~~~~~Cluster Number 8~~~~~\n",
      "We're Looking for Campus Insiders 1 Jul 2016\n",
      "Money in Love 28 Feb 2019\n",
      "Monzo 2.0 25 May 2018\n",
      "How we manage mental health 30 Jan 2019\n",
      "We're pausing our plans to let you pay in cheques with a picture 22 Mar 2019\n",
      "Why everyone at Monzo does research 30 May 2019\n",
      "No more waiting list! 24 May 2018\n",
      "Can You Marry An App? Customer Support Quotables 11 Jan 2017\n",
      "Providing customer support from Las Vegas! 🇺🇸 11 Dec 2018\n",
      "Why API-Driven Banking Matters 5 Nov 2015\n",
      "Why you should care about keeping your design files clean 11 Dec 2018\n",
      "Introducing Monzo Insider! 27 Jul 2017\n",
      "What we're doing to keep Monzo accessible 16 May 2019\n",
      "Writing customer-friendly terms and conditions 3 May 2019\n",
      "More Beautiful Direct Debits in Your Monzo Account 19 Jan 2018\n",
      "We’re working with the Financial Capability Lab 15 Nov 2018\n",
      "The Refactathon 4 Dec 2018\n",
      "Share Options 20 Nov 2017\n",
      "Introducing the Monzo naming system 6 Nov 2018\n",
      "Why Monzo Cards Aren't Working Today 5 Mar 2017\n",
      "Searching for Answers on Android 27 Sep 2017\n",
      "Monzo for Alexa – A Community Hack 21 Dec 2016\n",
      "Launching the Monzo Shop 👕 6 Oct 2016\n",
      "Organising teams and managing engineers 27 Jun 2018\n",
      "New Mondo Update 8 Aug 2016\n",
      "How we load tested our bank before our £20 million crowdfunding round 15 Jan 2019\n",
      "Mondo is now Monzo! 25 Aug 2016\n",
      "Money in Love 29 Mar 2019\n",
      "An Update on our Alpha Rollout 14 Jan 2016\n",
      "Big news about small print 22 May 2018\n",
      "~~~~~~~Cluster Number 9~~~~~\n",
      "See your Year in Monzo! 8 Jan 2019\n",
      "Monzo App Shortcuts on Android 16 Jan 2017\n",
      "Announcing Mondo v1.4 9 May 2016\n",
      "Launching Settings on Android! 9 Feb 2017\n",
      "New Android Update 1 Aug 2017\n",
      "ATM Fees Abroad 4 Jan 2018\n",
      "New Android Update 28 Mar 2017\n",
      "Asking Some Customers To Do Additional Identity Verification 13 Jun 2017\n",
      "Terminal decline 5 May 2017\n",
      "Launching Machine Learning Powered Help on iOS 18 Jul 2017\n",
      "View Your PIN 3 Jan 2017\n",
      "New Monzo Update 17 Oct 2016\n",
      "New Monzo Update 10 Oct 2016\n",
      "Pay people with your fingerprint on Android! 5 Jul 2018\n",
      "Our plans to help you pay in cheques, just by taking a picture 21 Jun 2018\n",
      "~~~~~~~Cluster Number 10~~~~~\n",
      "Laying the Foundation for a Data Team 30 Nov 2016\n",
      "Meet Hugo Cornejo 8 Apr 2016\n",
      "Celebrating Pride 🏳️‍🌈 6 Jul 2018\n",
      "Meet Paul—Our Chief Risk Officer 4 Aug 2015\n",
      "Meet Will - Business Operations 25 Jul 2017\n",
      "New Beginnings at Monzo 23 Jun 2017\n",
      "Meet Gary - Chief Financial Officer 22 Jun 2017\n",
      "Meet Emma 9 Mar 2017\n",
      "Meet Dan 6 Apr 2017\n",
      "From desktop design to your doorstep 9 Jun 2017\n",
      "Sharing our social media guidelines 9 Nov 2018\n",
      "Meet Phil – Head of Partnerships 4 Sep 2017\n",
      "Meet Tapas from Customer Operations 17 May 2017\n",
      "How we do on-call at Monzo 20 Sep 2018\n",
      "3 Second Sandwich 2 Dec 2015\n",
      "We use this tool to turn user research into insight very quickly 11 Feb 2019\n",
      "Why are foreign card  payments such a mess? 15 Dec 2015\n",
      "A technical look at how Monzo-to-Monzo payments work 5 Apr 2018\n",
      "Happy birthday to the Monzo tone of voice 27 Mar 2019\n",
      "Meet Leah! 9 Feb 2016\n",
      "~~~~~~~Cluster Number 11~~~~~\n",
      "Payments on Android! 6 Dec 2016\n",
      "Launching Monzo.me! 7 Dec 2016\n",
      "Split The Bill Using Monzo.me on Android 15 Mar 2017\n",
      "Bill splitting made simple 9 Oct 2018\n",
      "Split The Bill Using Monzo.me 22 Feb 2017\n",
      "Launching Monzo.me on Android! 24 Feb 2017\n",
      "Try splitting the bill in Monzo Labs on iOS! 15 Aug 2018\n",
      "~~~~~~~Cluster Number 12~~~~~\n",
      "Designing a product with mental health issues in mind 27 Jan 2017\n",
      "How We Build the Monzo App 2 Mar 2018\n",
      "Making Monzo's quarterly goals public 10 Jul 2018\n",
      "An update on diversity and inclusion at Monzo, March 2019 21 Mar 2019\n",
      "Understanding our Users 2 Feb 2018\n",
      "Meet Jordan 12 Dec 2017\n",
      "How we’re gearing up to support one billion Monzo customers! 30 Jul 2018\n",
      "COps 15 May 2017\n",
      "Tried and Tested! 30 Jun 2017\n",
      "The Monzo Reliability Report 25 Jul 2018\n",
      "Working on Android 29 Sep 2017\n",
      "To a Billion and Beyond 26 May 2017\n",
      "How our community helps us build a better bank 5 Nov 2018\n",
      "Putting transparency into practice 8 Nov 2018\n",
      "The Future of Customer Operations (\"COps\") 19 May 2017\n",
      "Come and join our Customer Operations team! 18 Oct 2018\n",
      "We’re working to improve the performance of our iOS app 18 Sep 2018\n",
      "ATM Fees Abroad 13 Sep 2017\n",
      "Monzo Time 26 Jun 2018\n",
      "~~~~~~~Cluster Number 13~~~~~\n",
      "Building a Modern Bank Backend 19 Sep 2016\n",
      "Why we made a mistake with customer support (and how we fixed it) 17 Dec 2018\n",
      "Introducing Our New Community Page 26 Jan 2017\n",
      "An update on diversity and inclusion at Monzo, March 2018 22 Mar 2018\n",
      "What Makes A Bank Ethical? 7 Jun 2016\n",
      "An update on our API 11 May 2017\n",
      "We’re sharing the vision and principles for our internal product 3 Aug 2018\n",
      "Viva Las Vegas! Providing customer support from the States 31 Jul 2018\n",
      "Setting the record straight on our crowdfunding round 30 Nov 2018\n",
      "Transparent By Default 10 Mar 2017\n",
      "Diversity and Inclusion at Monzo 9 Mar 2017\n",
      "Introducing the Monzo Transparency Dashboard 6 Mar 2017\n",
      "Meet the Vulnerable Customers Team 13 May 2019\n",
      "We Are Now a Bank 11 Aug 2016\n",
      "A Mondo Update...In 96 Seconds 1 Jun 2016\n",
      "Invest in Mondo 15 Feb 2016\n",
      "The Monzo Transparent Product Roadmap 27 May 2016\n",
      "How we built a backend for our £20 million crowdfunding round 14 Jan 2019\n",
      "How We Picked 'Monzo' 26 Aug 2016\n",
      "How we prepared our systems for our £20 million crowdfunding round 16 Jan 2019\n",
      "~~~~~~~Cluster Number 14~~~~~\n",
      "Half a Million Current Accounts! 19 Mar 2018\n",
      "Try out joint accounts in Monzo Labs! 13 Jul 2018\n",
      "Getting a Monzo bank account as a recent immigrant to the UK 29 May 2018\n",
      "Getting a Monzo bank account as a refugee 30 May 2018\n",
      "Introducing Monzo for 16-17 year olds! 6 Aug 2018\n",
      "Why Upgrade? 18 Dec 2017\n",
      "Top ups are back! 1 Dec 2017\n",
      "New to the UK? Open a bank account in minutes with Monzo 7 Sep 2018\n",
      "How to get your salary paid into your Monzo account 4 Sep 2018\n",
      "Android Pay is Here! 24 Oct 2017\n",
      "Manage your money together with joint accounts! 14 Aug 2018\n",
      "Is Monzo a bank? 7 Jun 2018\n",
      "Switch to Monzo and Help us Test the Current Account Switch Service 3 Apr 2018\n",
      "~~~~~~~Cluster Number 15~~~~~\n",
      "Ending the Monzo Prepaid Beta 31 Jan 2018\n",
      "Upgraded Monzo Accounts for Everyone! 6 Dec 2017\n",
      "Our Plan to Close the Monzo Prepaid Cards 18 Jan 2018\n",
      "The Next Step 10 Oct 2017\n",
      "Monzo-to-Monzo Payments Come to the Current Account 30 Oct 2017\n",
      "The Monzo Current Account Preview! 17 Jul 2017\n",
      "The Monzo Prepaid Beta Has Ended 4 Apr 2018\n",
      "~~~~~~~Cluster Number 16~~~~~\n",
      "Publishing our 2018 Annual Report! 2 Jul 2018\n",
      "Making Monzo Work in More Places 25 May 2018\n",
      "An Update on our Name Change 23 Jun 2016\n",
      "Introducing the Monzo Tone of Voice 27 Mar 2018\n",
      "Supporting Mindfulness and Wellbeing at Monzo 18 May 2018\n",
      "Why we sometimes freeze or block Monzo accounts 4 Apr 2019\n",
      "Financial Inclusion 29 May 2018\n",
      "Meet the Vulnerable Customers Team 15 May 2018\n",
      "Supporting people to self-exclude from gambling 16 May 2018\n",
      "Launching the Monzo Extraordinary Ideas Board 13 Jan 2017\n",
      "An Apology from Monzo 4 Aug 2017\n",
      "Measuring customer happiness 😃 3 Aug 2018\n",
      "We’re planning to launch a business current account 22 Nov 2018\n",
      "An Update on the Monzo API 12 Jan 2018\n",
      "Receiving international payments with Monzo 24 Jul 2018\n",
      "Understanding how young people manage their money 7 Aug 2018\n",
      "Turn That Frown Upside Down 9 Aug 2017\n",
      "Working towards our quarterly goals 18 Jul 2018\n",
      "Announcing Our Public Hiring Referral Bonus! 7 Jun 2017\n",
      "Host a Monzo Meetup! 8 Jun 2017\n",
      "Our vision to help people earn more from their savings 26 Apr 2019\n",
      "Practical Machine Learning With Event Streaming 3 May 2017\n",
      "Tell us when your question's urgent 2 Aug 2018\n",
      "Monzo’s gender pay gap 12 Apr 2018\n",
      "Getting Students On Board 23 Sep 2016\n",
      "We’ve published our Engineering Progression Framework 25 Jun 2018\n",
      "Money in Love 11 Apr 2019\n",
      "~~~~~~~Cluster Number 17~~~~~\n",
      "Micro-saving is the trend that’s making it easier to save 30 May 2019\n",
      "Stick to your savings goals by locking your Pots 13 Dec 2018\n",
      "Save Spare Change with Coin Jar 7 Mar 2018\n",
      "Get interest paid on your savings! 1 Nov 2018\n",
      "Everything you need to know about saving money that you didn't learn at school 17 May 2019\n",
      "How to save money with Monzo 7 May 2019\n",
      "3 ways to save 'little and often' with Monzo 1 May 2019\n",
      "~~~~~~~Cluster Number 18~~~~~\n",
      "New Monzo Update 12 Sep 2016\n",
      "See your receipts in Monzo with Flux 31 Jan 2019\n",
      "How Much?! Introducing  Your Spending 22 Jan 2016\n",
      "Monzo with Friends 14 Nov 2016\n",
      "Spending on Android! New Update Out Now 14 Nov 2016\n",
      "Engineering Principles at Monzo 29 Jun 2018\n",
      "We’re testing energy switching 27 Feb 2019\n",
      "Our Festive FAQs 12 Dec 2016\n",
      "New Mondo Update 18 Jul 2016\n",
      "Using data to build a better help screen 1 Aug 2018\n",
      "The Class of 2017 13 Jul 2017\n",
      "What we learned from moving Monzo 📦 6 Sep 2018\n",
      "We're making it easier to manage your bills 8 Aug 2018\n",
      "An update on PIN lock 17 Aug 2018\n",
      "ATM Fees Abroad 17 Oct 2017\n",
      "Introducing Monzo Chat 2 Nov 2018\n",
      "Launching 3D Secure! 22 Aug 2018\n",
      "An Android Progress Report 25 Sep 2017\n",
      "Invest in Monzo 28 Feb 2017\n",
      "An update on how we're supporting mental health 15 May 2019\n",
      "ATM Fees Abroad 15 Dec 2017\n",
      "Welcome to The Drawing Board! 27 Apr 2017\n",
      "Introducing Sneak Peeks 9 May 2018\n",
      "How we built Monzo Chat on Android 11 Feb 2019\n",
      "Block gambling transactions from your Monzo account 19 Jun 2018\n",
      "Get £2,500 for Referring a Backend Engineer 27 Apr 2018\n",
      "We've added some new categories 9 May 2018\n",
      "Better Transaction Information on Android 15 Feb 2018\n",
      "Our Approach to Overdrafts 7 Sep 2016\n",
      "~~~~~~~Cluster Number 19~~~~~\n",
      "The Hidden Costs of Free Banking 22 Oct 2015\n",
      "Mental Health Awareness Week at Monzo 14 May 2018\n",
      "We suspect some data has been compromised in the Typeform breach. All money is safe. 29 Jun 2018\n",
      "Custom Monzo app icons on Android 21 Dec 2018\n",
      "Introducing progression.monzo.com! 7 Jan 2019\n",
      "How we put together our progression frameworks 7 Feb 2019\n",
      "Say hello to the Coral Crew! 16 May 2018\n",
      "Introducing 21 Feb 2019\n",
      "The Monzo Mixtape Vol.1 3 Jul 2017\n",
      "The British Airways data breach 7 Sep 2018\n",
      "An update on the Women in Finance Charter 20 Nov 2018\n",
      "Very Robust etcd 29 Nov 2017\n",
      "Monzo's Response to Cloudbleed 24 Feb 2017\n",
      "How we monitor Monzo 27 Jul 2018\n",
      "Unleashing the Dragons 1 Dec 2017\n",
      "Code Club Came to Monzo 20 Oct 2017\n",
      "We've signed the Women in Finance Charter 31 May 2017\n",
      "We deployed Envoy Proxy to make Monzo faster 3 Apr 2019\n",
      "~~~~~~~Cluster Number 20~~~~~\n",
      "How to ask for a pay rise (and how to spend it when you get one!) 8 Mar 2019\n",
      "Set a monthly spending budget with Summary 5 Jul 2018\n",
      "How to budget and save money as a student 11 Sep 2018\n",
      "Should you borrow money? 9 Aug 2018\n",
      "Dear Monzo, buying a house feels out of reach 22 May 2019\n",
      "Money tips for moving to the UK 7 Sep 2018\n",
      "Should you share your finances? 14 Aug 2018\n",
      "How to save for a house deposit 🏠 23 Nov 2018\n",
      "How to manage your bills 30 Aug 2018\n",
      "Have you left it too late to start saving? 9 Apr 2019\n",
      "How to budget (and actually stick to it!) 18 Jan 2019\n",
      "What's your saving style? 6 Dec 2018\n",
      "How to manage money when you have kids 👶 22 Jan 2019\n",
      "How to budget if you don’t get paid regularly 6 Mar 2019\n",
      "~~~~~~~Cluster Number 21~~~~~\n",
      "Disclosing mental health to your bank 18 May 2018\n",
      "The new Monzo mission 2 Jul 2018\n",
      "Terms and Conditions 20 Mar 2018\n",
      "Monzo Monthly Update 11 Dec 2018\n",
      "How to protect your money from authorised push payment fraud 7 Feb 2019\n",
      "Why we’re lending 20 Nov 2018\n",
      "Announcing the Monzo Overdrafts Preview 24 Aug 2017\n",
      "11 reasons you should definitely get Monzo 10 May 2019\n",
      "Making Monzo a better current account 16 Oct 2018\n",
      "Making Monzo Signup Simple 3 Nov 2017\n",
      "Introducing Monzo Plus 23 Apr 2019\n",
      "Monzo Plus 15 May 2019\n",
      "What is Open Banking? 14 Jun 2018\n",
      "Is Monzo safe? 6 Feb 2019\n",
      "We’re launching the first 100 Monzo Business Accounts! 11 Feb 2019\n",
      "How we hire product designers 17 Jan 2019\n",
      "An Update on Overdrafts 23 Jan 2018\n",
      "Introducing Monzo Labs 27 Apr 2018\n",
      "Why doesn’t Monzo have branches? 22 Jun 2018\n",
      "~~~~~~~Cluster Number 22~~~~~\n",
      "How do bank payments  actually work? 20 Jan 2016\n",
      "Fighting Financial Crime the Monzo Way 31 Jan 2017\n",
      "Open Office August 1 Sep 2017\n",
      "Meet Irina 8 Feb 2017\n",
      "Budgeting, Money Management and Planning Ahead 16 Feb 2017\n",
      "Meet Venkat 22 Mar 2017\n",
      "Working with backend services on iOS 7 Jul 2017\n",
      "Meet Josh 22 Feb 2017\n",
      "Night COps 18 May 2017\n",
      "Meet Natasha 1 Feb 2017\n",
      "Fighting Fraud with Machine Learning 3 Feb 2017\n",
      "How we design our internal product 11 Feb 2019\n",
      "How to get a job as an engineer without a computer science degree 6 Mar 2019\n",
      "The bank of the future will be a marketplace 4 Feb 2016\n",
      "Celebrating Living Wage Week 4 Nov 2016\n",
      "~~~~~~~Cluster Number 23~~~~~\n",
      "Should I use a credit card for my holiday? 4 Mar 2019\n",
      "How to manage money when you’re getting out of debt 26 Feb 2019\n",
      "What kind of credit should I use? 9 Aug 2018\n",
      "Good debt vs. bad debt 24 Apr 2019\n",
      "4 money mistakes you might be making 24 Apr 2019\n",
      "Dear Monzo, I'm £7,000 in debt and I’m struggling to clear it 9 Apr 2019\n",
      "How to use the 50/20/30 budget rule 6 Feb 2019\n",
      "How to get a mortgage 17 Apr 2019\n",
      "~~~~~~~Cluster Number 24~~~~~\n",
      "£12 Million Later 14 Mar 2017\n",
      "2017 In Numbers 5 Jan 2018\n",
      "We’re crowdfunding up to £20,000,000 and we’d love you to be part of it! 26 Nov 2018\n",
      "1,000,000 people are now using Monzo! 24 Sep 2018\n",
      "Big news 31 Oct 2018\n",
      "Crowdfunding is open to new investors! 5 Dec 2018\n",
      "Announcing Our £22m Investment Round and Crowdfunding! 23 Feb 2017\n",
      "Crowdfunding in Numbers 20 Apr 2017\n",
      "Announcing our New Investment Round of £71 Million 🎉 7 Nov 2017\n",
      "Fastest Crowdfunding Ever 3 Mar 2016\n",
      "We Have Temporarily Paused our Crowdcube Round 29 Feb 2016\n",
      "~~~~~~~Cluster Number 25~~~~~\n",
      "Upgrading Explained 26 Jan 2018\n",
      "Should I get a credit card? 17 Apr 2019\n",
      "How to improve your credit score 8 Aug 2018\n",
      "Secured and unsecured credit - what's the difference? 5 Oct 2018\n",
      "What is a credit score? 9 Aug 2018\n",
      "The big guide to interest 25 Oct 2018\n",
      "What is a 'good' credit score? 12 Oct 2018\n",
      "How to check your credit score and report 11 Oct 2018\n",
      "~~~~~~~Cluster Number 26~~~~~\n",
      "How to save for retirement 27 Nov 2018\n",
      "Understanding your payslip 26 Sep 2018\n",
      "A guide to PAYE forms 10 Oct 2018\n",
      "The big guide to ISAs 14 Nov 2018\n",
      "The big guide to pensions 7 Nov 2018\n",
      "~~~~~~~Cluster Number 27~~~~~\n",
      "Welcome to Monzo 2.0 21 May 2018\n",
      "Mondo on Android 25 May 2016\n",
      "Help us build a bank! 4 Jun 2015\n",
      "Ask Us Anything -  Join the Conversation on Our Forum 3 Jun 2016\n",
      "We're Ready 30 Oct 2015\n",
      "Monzo in 2019 4 Jan 2019\n",
      "The Current Account Preview 23 Aug 2017\n",
      "The Monzo Current Account Preview comes to Android! 7 Aug 2017\n",
      "A Current Account Rollout Update 25 Oct 2017\n",
      "Welcome to Monzo Bank! 5 Apr 2017\n",
      "Delays Sending Out Cards 5 Oct 2017\n",
      "Making Monzo 7 Nov 2018\n",
      "Launching The Bank 15 Aug 2016\n",
      "Welcome to Mondo! 1 Jun 2015\n",
      "The Next Step 17 Mar 2016\n",
      "~~~~~~~Cluster Number 28~~~~~\n",
      "The 7-Day Switch! What happens when you switch to Monzo? 23 Aug 2018\n",
      "You can now switch your joint accounts to Monzo! 7 Feb 2019\n",
      "Switch to Monzo with the Current Account Switch Service! 25 Apr 2018\n",
      "Can I switch bank accounts if I have an overdraft? 30 Apr 2018\n",
      "~~~~~~~Cluster Number 29~~~~~\n",
      "Money in Love 14 Mar 2019\n",
      "Money in Love 22 Mar 2019\n",
      "Are your friends' weddings costing you a fortune? Here's how to cut the costs 4 Jun 2019\n",
      "Money in Love 18 Apr 2019\n",
      "‘I saved up enough money to go to Madrid for the Champions League final’ 22 May 2019\n",
      "‘Impulsivity is one of the symptoms of ADHD’ 16 May 2019\n",
      "Money in Love 8 May 2019\n",
      "Money in Love 25 Apr 2019\n",
      "'I give my mum money so she can buy whatever she wants' 10 Apr 2019\n",
      "Money in Love 4 Apr 2019\n",
      "Money in Love 21 Feb 2019\n",
      "Money in Love 14 Feb 2019\n",
      "~~~~~~~Cluster Number 30~~~~~\n",
      "Christmas shopping and sales – know your rights 12 Dec 2018\n",
      "​​Getting a bank account with no fixed address 1 Jun 2018\n",
      "​​Banking Aboard 31 May 2018\n",
      "Student loans in Northern Ireland explained 13 Sep 2018\n",
      "How to save for a wedding 28 Nov 2018\n",
      "What is FSCS protection? 22 Oct 2018\n",
      "How to get a bank statement from Monzo 5 Sep 2018\n",
      "You can now switch energy supplier through Monzo! 8 May 2019\n",
      "Student loans explained 10 Aug 2018\n",
      "What is crowdfunding? 23 Nov 2018\n",
      "How to get fit on a budget 21 Jan 2019\n",
      "Save £667.95 a year with the 1p Saving Challenge 10 Apr 2019\n",
      "How much does a dog cost? 30 Jan 2019\n",
      "How to use Monzo to split bills 1 Feb 2019\n",
      "How much does it cost to have a baby? 30 Jan 2019\n",
      "Could rounding-up your spending help you afford your next holiday? 23 May 2019\n",
      "How to give back when you’re broke 4 Feb 2019\n",
      "~~~~~~~Cluster Number 31~~~~~\n",
      "Monzo x Old Street – Our First Shopping Event 5 Dec 2016\n",
      "Custom Monzo app icons on iOS 3 Jul 2018\n",
      "Give the gift of Monzo this Christmas 🎁 11 Dec 2018\n",
      "Mondo Version 1.5 13 Jun 2016\n",
      "Our Old Street Shopping Guide 8 Dec 2016\n",
      "Releasing Monzo's 2017 Annual Report! 4 Jul 2017\n",
      "Have you seen our ads on TV? 22 May 2019\n",
      "How international payments work 24 Jul 2018\n",
      "Splitting the bill with Monzo and Deliveroo 8 Sep 2017\n",
      "200,000 Customers and £250,000,000 Spent! 16 May 2017\n",
      "Target Practice 19 Oct 2016\n",
      "You've saved over £1,000,000 with Coin Jar! 11 May 2018\n",
      "The Year in Monzo 2018 17 Jan 2019\n",
      "We’re helping The Big Issue sellers get more out of each magazine 1 Apr 2019\n",
      "How Does the Wholesale Foreign Exchange Market Work? 8 Jan 2016\n",
      "#MonzoJournals - Philippines 20 Sep 2016\n",
      "Launching Early Access to Travel Reports 16 Aug 2016\n",
      "What’s the exchange rate on Monzo? 5 Jun 2018\n",
      "~~~~~~~Cluster Number 32~~~~~\n",
      "Keep track of what you’re owed with Shared Tabs 13 Dec 2018\n",
      "Monzo 2.0 23 May 2018\n",
      "Say thanks with emoji! 18 Apr 2018\n",
      "Schedule payments into Pots! 16 Jul 2018\n",
      "Round up your Monzo purchases automatically! 18 Jul 2018\n",
      "We’re making Summary more flexible 20 Nov 2018\n",
      "You can now pay cash into your Monzo account! 21 Nov 2018\n",
      "Pots Are Now on Android! 7 Dec 2017\n",
      "New iOS Update 5 Apr 2017\n",
      "Move money out of Pots automatically! 23 Aug 2018\n",
      "Try setting goals for Pots in Monzo Labs! 19 Jul 2018\n",
      "Bill Tracker tells you when a regular bill changes 14 Jun 2018\n",
      "Summary Update - Predicting Committed Spending 7 Jun 2018\n",
      "Automate your financial life with Monzo and IFTTT 7 Jun 2018\n",
      "Making recurring payments on Monzo even better 4 Jul 2018\n",
      "We added Travel Reports to Android 6 Jun 2018\n",
      "How Monzo helps you plan ahead 16 Jan 2019\n",
      "Launching Targets 3 Oct 2016\n",
      "~~~~~~~Cluster Number 33~~~~~\n",
      "Save together with Pots on joint accounts! 10 Dec 2018\n",
      "Set goals for your Pots! 2 Aug 2018\n",
      "An update on getting interest paid on savings 21 Aug 2018\n",
      "Flexible Savings Pots are back - with a new provider! 🎉 2 May 2019\n",
      "Choosing between different types of savings accounts 24 Oct 2018\n",
      "You can now open an ISA through your Monzo app 🎉 4 Apr 2019\n",
      "Why you can't create new Savings Pots – for now 5 Feb 2019\n",
      "~~~~~~~Cluster Number 34~~~~~\n",
      "‘I put £1 in a Pot every time I swore during Ramadan’ 3 Jun 2019\n",
      "Try this one trick to stop yourself spending your savings 18 Mar 2019\n",
      "You can now add your own images to Pots, and you've been taking full advantage 18 Mar 2019\n",
      "Could eating less meat help you cut your costs? 16 May 2019\n",
      "The national minimum wage, the national living wage and the real living wage 9 Nov 2018\n",
      "Crowdfunding in numbers 6 Dec 2018\n",
      "How cashback works 28 Jan 2019\n",
      "You can now add your own images to Pots on both Android and iOS 11 Apr 2019\n",
      "~~~~~~~Cluster Number 35~~~~~\n",
      "Monzo Monthly Update 6 Feb 2018\n",
      "Monzo Monthly Update 5 Jul 2018\n",
      "Monzo Monthly Update 28 Feb 2018\n",
      "Monzo Monthly Update 1 Oct 2018\n",
      "The Year Ahead, 2018 11 Jan 2018\n",
      "The end of The Big List! 22 Aug 2018\n",
      "Monzo Monthly Update 12 Nov 2018\n",
      "Monzo Monthly Update 4 Sep 2018\n",
      "Monzo Monthly Update 3 Aug 2018\n",
      "Monzo Monthly Update 5 Apr 2018\n",
      "Monzo Monthly Update 4 Jun 2018\n",
      "Making Monzo Better 22 May 2018\n",
      "Monzo Monthly Update 4 May 2018\n"
     ]
    }
   ],
   "source": [
    "print_clusters(blogs, ap_clusters, len(ap_cluster_count))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have clustered the blog posts, let us look at topic modeling and cosine similairty of the clusters. First, we look at the cosine similarity. If we reorder the blogs according to cluster membership and replot the heatmap of cosine similarity, do we see an increase in the level of structure in the map? We will compare the clusterings we found using affinity propagation and using k-means"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "def order_documents(cluster, documents, n_clusters):\n",
    "    ordered_documents = []\n",
    "    switch_indices = []\n",
    "    for i in range(n_clusters):\n",
    "        for ind, document in enumerate(documents):\n",
    "            if cluster[ind] == i:\n",
    "                ordered_documents.append(document)\n",
    "        switch_indices.append(len(ordered_documents))\n",
    "    return switch_indices, ordered_documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/23.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "km_indices, km_ordered_documents = order_documents(km_clusters, documents, num_km_clusters)\n",
    "tfidf_bag = TfidfVectorizer().fit_transform(km_ordered_documents)\n",
    "index = MatrixSimilarity(tfidf_bag.todense(), num_features= tfidf_bag.shape[1])\n",
    "cosine_similarity  = []\n",
    "for document in tfidf_bag:\n",
    "    cosine_similarity.append(index[document])\n",
    "\n",
    "\n",
    "trace = go.Heatmap(z=cosine_similarity)\n",
    "data=[trace]\n",
    "layout = go.Layout(\n",
    "    title='TF-IDF K-means ordered cosine similarity heatmap',\n",
    ")\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "py.iplot(fig, filename='tfidf-heatmap')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/23.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ap_indices, ap_ordered_documents = order_documents(ap_clusters, documents, num_ap_clusters)\n",
    "tfidf_bag = TfidfVectorizer().fit_transform(ap_ordered_documents)\n",
    "index = MatrixSimilarity(tfidf_bag.todense(), num_features= tfidf_bag.shape[1])\n",
    "cosine_similarity  = []\n",
    "for document in tfidf_bag:\n",
    "    cosine_similarity.append(index[document])\n",
    "\n",
    "\n",
    "trace = go.Heatmap(z=cosine_similarity)\n",
    "data=[trace]\n",
    "layout = go.Layout(\n",
    "    title= 'TF-IDF Affinity Propogation ordered cosine similarity heatmap',\n",
    "    shapes = [\n",
    "        # Line Vertical\n",
    "        {\n",
    "            'type': 'line',\n",
    "            'x0': 23,\n",
    "            'y0': 0,\n",
    "            'x1': 23,\n",
    "            'y1': 525,\n",
    "            'line': {\n",
    "                'color': 'rgb(0, 0, 0)',\n",
    "                'width': 1.5,\n",
    "            },\n",
    "        }])\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "py.iplot(fig, filename='tfidf-heatmap')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While there is more structure, it is still not very clear. What if we looked at the cosine similarty within and between the clusters. For the sake of convenience, let us only analyse this for the case of affinity propogation, but our functions will also work when clustering using k-means."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_documents_in_cluster(clusters, documents, cluster_number):\n",
    "    documents_list = []\n",
    "    for document, cluster in zip(documents, clusters):\n",
    "        if cluster == cluster_number:\n",
    "            documents_list.append(document)\n",
    "    return documents_list\n",
    "\n",
    "def get_all_clusters(clusters, documents, num_clusters):\n",
    "    cluster_master_list = []\n",
    "    for i in range(num_clusters):\n",
    "        cluster_master_list.append(get_documents_in_cluster(clusters, documents, i))\n",
    "    return cluster_master_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_corpus = []\n",
    "for doc in corpus:\n",
    "    new_corpus.append(' '.join(doc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster_list = get_all_clusters(km_clusters, new_corpus, num_km_clusters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "cosine_within = []\n",
    "for cluster in cluster_list:\n",
    "    tfidf_bag = TfidfVectorizer().fit_transform(cluster)\n",
    "    cosine_similarity = get_cosine_similarity(tfidf_bag.todense(), tfidf_bag.shape[1])\n",
    "    cosine_within.append(cosine_similarity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/29.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster_number = 2\n",
    "trace = go.Heatmap(z=cosine_within[cluster_number])\n",
    "data=[trace]\n",
    "layout = go.Layout(\n",
    "    title='TF-IDF K-means ordered cosine similarity heatmap within cluster number ' + str(cluster_number),\n",
    ")\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "py.iplot(fig, filename='tfidf-heatmap-cluster-1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can play around with this to see the different clusters and their heatmaps, and we can do the same with the clusters generated by Affinity Propagation. Now, we move on to looking at the clustering between clusters. To do this, we aggregate all text in all documents within a cluster, and treat this aggregated data as one document. Then, we compute the cosine similarity between all the documents. We should have as many documents as there are clusters, and so now we expect 8 documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "def aggregate_clusters(clusters, documents, num_clusters,):\n",
    "    new_documents = []\n",
    "    cluster_master_list = get_all_clusters(clusters, documents, num_clusters)\n",
    "    for cluster in cluster_master_list:\n",
    "        new_doc = ''\n",
    "        for document in cluster:\n",
    "            new_doc += document\n",
    "        new_documents.append(new_doc)\n",
    "    return new_documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_documents = aggregate_clusters(km_clusters, new_corpus, num_km_clusters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/31.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_bag = TfidfVectorizer().fit_transform(new_documents)\n",
    "cosine_similarity = get_cosine_similarity(tfidf_bag.todense(), tfidf_bag.shape[1])\n",
    "trace = go.Heatmap(z=cosine_similarity)\n",
    "data=[trace]\n",
    "layout = go.Layout(\n",
    "    title='TF-IDF K-means cosine similarity heatmap between clusters',\n",
    ")\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "py.iplot(fig, filename='tfidf-heatmap-between-clusters')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So as we can see, the cosine similarity is quite high between clusters. Note that in this scale, the lowest cosine-similarity is about 0.30, in stark contrast to the cosine similarities between individual documents which reaches about 0.02. In the end, all blog posts are talking about Monzo so this is also not a very surprising result. Lastly, we look at if we can find differences when performing topic modeling within and between clusters, and again we look at only using clusters that we found using K-means, although the AP approach is analogous. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda_list = []\n",
    "for cluster in cluster_list:\n",
    "    new_corp = [elem.split(\" \") for elem in cluster]\n",
    "    new_dic = Dictionary(new_corp)\n",
    "    new_bag = [new_dic.doc2bow(line) for line in new_corp]\n",
    "    lda = models.ldamodel.LdaModel(corpus=new_bag, \n",
    "                               id2word=new_dic,\n",
    "                               random_state=12, num_topics=10, update_every=1, passes=100)\n",
    "    lda_list.append(lda)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(4,\n",
       "  '0.019*\"monzo\" + 0.017*\"make\" + 0.017*\"feature\" + 0.017*\"work\" + 0.016*\"app\" + 0.015*\"u\" + 0.013*\"get\" + 0.010*\"new\" + 0.010*\"help\" + 0.009*\"team\"'),\n",
       " (6,\n",
       "  '0.023*\"account\" + 0.016*\"monzo\" + 0.015*\"work\" + 0.014*\"current\" + 0.012*\"card\" + 0.010*\"get\" + 0.009*\"make\" + 0.008*\"u\" + 0.007*\"people\" + 0.007*\"register\"'),\n",
       " (3,\n",
       "  '0.032*\"energy\" + 0.020*\"supplier\" + 0.019*\"apple\" + 0.013*\"wallet\" + 0.013*\"switch\" + 0.012*\"pay\" + 0.009*\"card\" + 0.008*\"option\" + 0.007*\"instruction\" + 0.007*\"ovo\"'),\n",
       " (0,\n",
       "  '0.016*\"update\" + 0.014*\"app\" + 0.013*\"new\" + 0.013*\"search\" + 0.012*\"transaction\" + 0.011*\"screen\" + 0.011*\"answer\" + 0.010*\"support\" + 0.010*\"card\" + 0.009*\"user\"'),\n",
       " (9,\n",
       "  '0.020*\"make\" + 0.012*\"see\" + 0.012*\"pin\" + 0.011*\"purchase\" + 0.011*\"u\" + 0.011*\"transaction\" + 0.010*\"monzo\" + 0.009*\"information\" + 0.009*\"work\" + 0.008*\"app\"')]"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_list[0].print_topics(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(7,\n",
       "  '0.032*\"return\" + 0.023*\"product\" + 0.017*\"buy\" + 0.016*\"get\" + 0.016*\"may\" + 0.015*\"right\" + 0.014*\"day\" + 0.014*\"seller\" + 0.011*\"refund\" + 0.011*\"item\"'),\n",
       " (5,\n",
       "  '0.036*\"pay\" + 0.021*\"debt\" + 0.020*\"loan\" + 0.019*\"tax\" + 0.018*\"income\" + 0.016*\"get\" + 0.015*\"much\" + 0.014*\"earn\" + 0.014*\"money\" + 0.013*\"year\"'),\n",
       " (1,\n",
       "  '0.020*\"insurance\" + 0.011*\"get\" + 0.011*\"free\" + 0.010*\"damage\" + 0.008*\"may\" + 0.008*\"home\" + 0.008*\"help\" + 0.008*\"building\" + 0.008*\"cover\" + 0.007*\"offer\"'),\n",
       " (6,\n",
       "  '0.022*\"bank\" + 0.016*\"fraud\" + 0.015*\"monzo\" + 0.012*\"make\" + 0.011*\"number\" + 0.011*\"money\" + 0.011*\"get\" + 0.008*\"password\" + 0.008*\"customer\" + 0.008*\"someone\"'),\n",
       " (4,\n",
       "  '0.025*\"monzo\" + 0.020*\"money\" + 0.017*\"pay\" + 0.015*\"payment\" + 0.014*\"use\" + 0.013*\"month\" + 0.013*\"budget\" + 0.012*\"bill\" + 0.012*\"like\" + 0.012*\"get\"')]"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_list[1].print_topics(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(2,\n",
       "  '0.080*\"interest\" + 0.075*\"rate\" + 0.024*\"may\" + 0.023*\"account\" + 0.022*\"pay\" + 0.019*\"saving\" + 0.015*\"credit\" + 0.014*\"earn\" + 0.014*\"borrow\" + 0.014*\"good\"'),\n",
       " (6,\n",
       "  '0.083*\"inflation\" + 0.035*\"price\" + 0.030*\"rate\" + 0.019*\"bank\" + 0.015*\"rise\" + 0.013*\"student\" + 0.013*\"rpi\" + 0.013*\"cpi\" + 0.013*\"base\" + 0.012*\"loan\"'),\n",
       " (1,\n",
       "  '0.001*\"rate\" + 0.001*\"interest\" + 0.001*\"saving\" + 0.001*\"pension\" + 0.001*\"pay\" + 0.001*\"money\" + 0.001*\"get\" + 0.001*\"base\" + 0.001*\"account\" + 0.001*\"isa\"'),\n",
       " (9,\n",
       "  '0.001*\"pension\" + 0.001*\"pay\" + 0.001*\"rate\" + 0.001*\"saving\" + 0.001*\"tax\" + 0.001*\"money\" + 0.001*\"interest\" + 0.001*\"get\" + 0.001*\"may\" + 0.001*\"account\"'),\n",
       " (8,\n",
       "  '0.001*\"pension\" + 0.001*\"rate\" + 0.001*\"saving\" + 0.001*\"money\" + 0.001*\"get\" + 0.001*\"isa\" + 0.001*\"tax\" + 0.001*\"pay\" + 0.001*\"account\" + 0.001*\"may\"')]"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_list[2].print_topics(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~emver/31.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<chart_studio.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_documents = aggregate_clusters(ap_clusters, new_corpus, num_ap_clusters)\n",
    "tfidf_bag = TfidfVectorizer().fit_transform(new_documents)\n",
    "cosine_similarity = get_cosine_similarity(tfidf_bag.todense(), tfidf_bag.shape[1])\n",
    "trace = go.Heatmap(z=cosine_similarity)\n",
    "data=[trace]\n",
    "layout = go.Layout(\n",
    "    title='TF-IDF AP cosine similarity heatmap between clusters',\n",
    ")\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "py.iplot(fig, filename='tfidf-heatmap-between-clusters')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster_list = get_all_clusters(ap_clusters, new_corpus, num_ap_clusters)\n",
    "lda_list = []\n",
    "for cluster in cluster_list:\n",
    "    new_corp = [elem.split(\" \") for elem in cluster]\n",
    "    new_dic = Dictionary(new_corp)\n",
    "    new_bag = [new_dic.doc2bow(line) for line in new_corp]\n",
    "    lda = models.ldamodel.LdaModel(corpus=new_bag, \n",
    "                               id2word=new_dic,\n",
    "                               random_state=12, num_topics=10, update_every=1, passes=100)\n",
    "    lda_list.append(lda)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(2,\n",
       "  '0.052*\"insurance\" + 0.025*\"damage\" + 0.020*\"cover\" + 0.020*\"home\" + 0.020*\"may\" + 0.020*\"building\" + 0.015*\"claim\" + 0.015*\"content\" + 0.015*\"policy\" + 0.014*\"pay\"'),\n",
       " (7,\n",
       "  '0.001*\"may\" + 0.001*\"ticket\" + 0.001*\"train\" + 0.001*\"give\" + 0.001*\"get\" + 0.001*\"money\" + 0.001*\"charity\" + 0.001*\"grimshaw\" + 0.001*\"goshcharity\" + 0.001*\"gmfa_uk\"'),\n",
       " (4,\n",
       "  '0.036*\"flight\" + 0.023*\"travel\" + 0.023*\"book\" + 0.021*\"get\" + 0.021*\"airline\" + 0.018*\"back\" + 0.018*\"contact\" + 0.018*\"wow\" + 0.016*\"air\" + 0.014*\"atol\"'),\n",
       " (9,\n",
       "  '0.049*\"inflation\" + 0.045*\"rate\" + 0.024*\"interest\" + 0.021*\"price\" + 0.015*\"bank\" + 0.014*\"money\" + 0.012*\"may\" + 0.011*\"use\" + 0.011*\"base\" + 0.011*\"go\"'),\n",
       " (3,\n",
       "  '0.034*\"may\" + 0.021*\"charity\" + 0.020*\"give\" + 0.013*\"flight\" + 0.012*\"support\" + 0.012*\"get\" + 0.011*\"month\" + 0.011*\"donate\" + 0.011*\"airline\" + 0.010*\"money\"')]"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_list[0].print_topics(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.034*\"product\" + 0.028*\"return\" + 0.022*\"seller\" + 0.019*\"right\" + 0.018*\"get\" + 0.017*\"day\" + 0.017*\"refund\" + 0.016*\"may\" + 0.015*\"buy\" + 0.013*\"store\"'),\n",
       " (6,\n",
       "  '0.018*\"challenge\" + 0.017*\"saving\" + 0.017*\"week\" + 0.014*\"set\" + 0.013*\"get\" + 0.012*\"help\" + 0.012*\"save\" + 0.011*\"one\" + 0.011*\"bank\" + 0.011*\"piggy\"'),\n",
       " (5,\n",
       "  '0.031*\"budget\" + 0.026*\"monzo\" + 0.020*\"use\" + 0.017*\"money\" + 0.017*\"expense\" + 0.016*\"need\" + 0.015*\"pay\" + 0.015*\"bill\" + 0.014*\"account\" + 0.014*\"zero\"'),\n",
       " (8,\n",
       "  '0.001*\"monzo\" + 0.001*\"pay\" + 0.001*\"free\" + 0.001*\"need\" + 0.001*\"money\" + 0.001*\"get\" + 0.001*\"charge\" + 0.001*\"like\" + 0.001*\"budget\" + 0.001*\"use\"'),\n",
       " (4,\n",
       "  '0.008*\"go\" + 0.008*\"make\" + 0.008*\"save\" + 0.008*\"get\" + 0.007*\"try\" + 0.007*\"less\" + 0.007*\"reduce\" + 0.007*\"food\" + 0.007*\"year\" + 0.007*\"could\"')]"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_list[1].print_topics(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(5,\n",
       "  '0.077*\"account\" + 0.034*\"bank\" + 0.026*\"money\" + 0.026*\"open\" + 0.026*\"new\" + 0.022*\"switch\" + 0.017*\"payment\" + 0.015*\"want\" + 0.015*\"one\" + 0.014*\"monzo\"'),\n",
       " (1,\n",
       "  '0.002*\"account\" + 0.002*\"card\" + 0.002*\"money\" + 0.002*\"overdraft\" + 0.002*\"bank\" + 0.002*\"use\" + 0.002*\"utility\" + 0.002*\"yet\" + 0.002*\"water\" + 0.002*\"walk\"'),\n",
       " (2,\n",
       "  '0.069*\"account\" + 0.061*\"bank\" + 0.015*\"open\" + 0.015*\"money\" + 0.013*\"monzo\" + 0.013*\"credit\" + 0.013*\"one\" + 0.011*\"get\" + 0.011*\"take\" + 0.011*\"want\"'),\n",
       " (9,\n",
       "  '0.075*\"overdraft\" + 0.024*\"credit\" + 0.020*\"limit\" + 0.020*\"account\" + 0.018*\"charge\" + 0.018*\"use\" + 0.017*\"app\" + 0.013*\"want\" + 0.013*\"check\" + 0.013*\"switch\"'),\n",
       " (8,\n",
       "  '0.002*\"overdraft\" + 0.002*\"account\" + 0.002*\"monzo\" + 0.002*\"make\" + 0.002*\"like\" + 0.002*\"limit\" + 0.002*\"know\" + 0.002*\"need\" + 0.002*\"money\" + 0.002*\"find\"')]"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_list[2].print_topics(5)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}