plot_03-CountsCollection.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Tutorial 03: Counts Collection\n",
    "\n",
    "Collecting term co-occurrence data from the scientific literature.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Term Co-occurrence\n",
    "\n",
    "The 'Counts' approach, or term co-occurrence, searches the literature for\n",
    "how often terms of interest appear together.\n",
    "\n",
    "This type of analysis can be used to infer associations between terms.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the Counts object, which is used for co-occurrence analysis\n",
    "from lisc import Counts\n",
    "\n",
    "# Import the SCDB object, which defines a database structure for saved data\n",
    "from lisc.utils.db import SCDB\n",
    "\n",
    "# Import a utility function for saving out collected data\n",
    "from lisc.utils.io import save_object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Counts Object\n",
    "\n",
    "The :class:`~.Counts` object is used to handle term co-occurrence collection and analyses.\n",
    "\n",
    "This object is based on the :class:`~.Base` object introduced in the prior tutorial,\n",
    "meaning we can define and add terms to it like introduced there.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Counts: Single List of Terms\n",
    "\n",
    "For the first example of running a counts analysis, we will use a single list of terms.\n",
    "\n",
    "When a single list of terms is provided, the word co-occurrence is collected as the\n",
    "co-occurrence of each term with every other term in the list.\n",
    "\n",
    "Let's start with an example using different parts of the brain, and examine\n",
    "how often these brain regions are talked about together.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up some terms to search for\n",
    "terms = [['roundworm'], ['whipworm'], ['hookworm']]\n",
    "\n",
    "# Initialize counts object & add the terms that we want to collect co-occurrences for\n",
    "counts = Counts()\n",
    "counts.add_terms(terms)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running counts for:  roundworm\n",
      "Running counts for:  whipworm\n",
      "Running counts for:  hookworm\n"
     ]
    }
   ],
   "source": [
    "# Collect co-occurrence data\n",
    "counts.run_collection(verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have now collected some literature data!\n",
    "\n",
    "The :class:`~.Counts` object will now contain data on term co-occurrence in the literature.\n",
    "\n",
    "This is stored in the `counts` attribute of the object.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  0  92  99]\n",
      " [ 92   0 107]\n",
      " [ 99 107   0]]\n"
     ]
    }
   ],
   "source": [
    "# Check out the raw count data\n",
    "print(counts.counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The :class:`~.Counts` object also has methods for exploring the collected data.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of documents found for each search term is:\n",
      "  'roundworm'   -    952\n",
      "  'whipworm'    -    410\n",
      "  'hookworm'    -   4667\n"
     ]
    }
   ],
   "source": [
    "# Check how many articles were found for each search term\n",
    "counts.check_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The most studied term is  'hookworm'  with  4667  articles.\n"
     ]
    }
   ],
   "source": [
    "# Check the most studied term\n",
    "counts.check_top()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Counts: Two Terms Lists\n",
    "\n",
    "In the first example above, we provided a single list of terms.\n",
    "\n",
    "Now let's explore using two different sets of terms. When providing two lists of terms,\n",
    "each term from each list is compared to each term of the other list.\n",
    "\n",
    "In this example, we will keep our list of brain regions, and explore how they\n",
    "relate to different sensory systems.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define some new terms\n",
    "terms_a = [['frontal lobe'], ['temporal lobe'], ['parietal lobe'], ['occipital lobe']]\n",
    "terms_b = [['vision'], ['audition', 'auditory'], ['somatosensory'], ['olfaction', 'smell'],\n",
    "           ['gustation', 'taste'], ['proprioception'], ['nociception', 'pain']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have multiple term lists, we have to keep track of and specify which is which.\n",
    "\n",
    "This is done with the `dim` argument (for 'dimension'), and encoded as either 'A' or 'B'.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set terms lists, indicating they are separate lists with the 'A' and 'B' labels\n",
    "counts.add_terms(terms_a, dim='A')\n",
    "counts.add_terms(terms_b, dim='B')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Collect co-occurrence data\n",
    "counts.run_collection()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From here you can use all the same methods to explore the data we just collected.\n",
    "\n",
    "In the next tutorial, we will explore analyzing our collected counts data.\n",
    "\n",
    "For now, let's save out our collected counts data, using the LISC utility to save the object.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save out the counts object\n",
    "save_object(counts, 'tutorial_counts', directory=SCDB('lisc_db'))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}