04-CellTypes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Allen Cell Types Database\n",
    "\n",
    "This notebook will serve as an introduction to the Allen Cell Types database. We'll work with the AllenSDK to see what information we can learn about the electrophysiology of various cells.\n",
    "\n",
    "First, we'll `import` the CellTypesCache module. This module provides tools to allow us to get information from the cell types database. We're giving it a manifest filename as well -- `CellTypesCache` will create this manifest file, which contains metadata about the cache. If you're curious, you can look under cell_types in your directory, and take a look at the file.\n",
    "\n",
    "(See the full documentation for the core package <a href=\"https://allensdk.readthedocs.io/en/latest/allensdk.core.html\">here</a>.)\n",
    "\n",
    "## Step Zero: Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Import the \"Cell Types Cache\" from the AllenSDK core package\n",
    "from allensdk.core.cell_types_cache import CellTypesCache\n",
    "\n",
    "#Import CellTypesApi, which will allow us to query the database.\n",
    "from allensdk.api.queries.cell_types_api import CellTypesApi\n",
    "\n",
    "# We'll then initialize the cache as 'ctc' (cell types cache)\n",
    "ctc = CellTypesCache(manifest_file='cell_types/manifest.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step One: Get Cells & Manipulate Dataframe\n",
    "\n",
    "Look through <a href=\"https://allensdk.readthedocs.io/en/latest/allensdk.core.cell_types_cache.html\">the documentation for the CellTypesCache</a> for information on the `get_cells` method.\n",
    "\n",
    "<div class=\"alert alert-success\"><b>Task</b>: Use the <code>get_cells</code> method in the cell below to get information about all of the human cells in the database. Assign the output of this to `human_cells`, and look at the output when it is done.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "human_cells = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chances are, your output looks a bit messy. This is where pandas can really come in handy!\n",
    "\n",
    "<div class=\"alert alert-success\"><b>Task</b>: Convert <code>human_cells</code> into a Pandas Dataframe by:\n",
    "    \n",
    "1. Importing `pandas` as `pd`\n",
    "2. Creating a dataframe with `pd.DataFrame()`\n",
    "3. Assigning that dataframe to `human_df`\n",
    "4. Showing the first five rows of the df using the `.head()` method.\n",
    "    \n",
    "</div>\n",
    "\n",
    "Note: If you're having trouble with Pandas, it can help to look at <a href=\"https://pandas.pydata.org/pandas-docs/stable/user_guide/\">the user guide</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create your dataframe here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get some information about our cells. We can use `len()` on a dataframe to get the number of rows. Alternatively, we can use the `count()` method on our dataframe to get detailed information for each column. For our purposes today, let's just get the number of rows, which is equivalent to the number of observations. \n",
    "\n",
    "<div class=\"alert alert-success\"><b>Task</b>: Get information about the dataset by:\n",
    "\n",
    "1. Using the `count()` method to see how many observations there are in each of the columns. Why might some be missing?\n",
    "2. Using `len()` to see the length of the whole dataframe and assign the output to `n_human_cells`.\n",
    "    \n",
    "   </div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check count and len here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the moment, our rows don't have any useful information -- they're simply a list of indices. We can reassign the row values by using the method `set_index`. \n",
    "\n",
    "<div class=\"alert alert-success\"><b>Task</b>: Execute the <code>set_index</code> method to set the 'id' column as the index, and reassign your dataframe as `human_df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reassign the index here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It would help to know what information is in our dataset. In other words, what is across the columns at the top? We can get a list by accessing the attribute `.columns`. Assign the output of this method to `human_df_columns`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "human_df_columns = human_df.columns\n",
    "print(human_df_columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access individual columns with the notation `dataframe['column name']`. \n",
    "\n",
    "<div class=\"alert alert-success\"><b>Task</b>: Check out the <code>dendrite_type</code> column by using this notation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at dendrite_type column\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like numpy arrays, we can use boolean indexing to filter our pandas dataframe. Our dataframe has data on two different dendrite types. Filter your dataframe by using the following syntax:\n",
    "```\n",
    "new_df = original_df[original_df['Column of Interest'] == 'Desired Value']\n",
    "```\n",
    "In plain english: save a dataframe from the original dataframe, where the original dataframe values in my Column of Interest are equal to my Desired Value.\n",
    "\n",
    "<div class=\"alert alert-success\"><b>Task</b>: Subset your dataframe by:\n",
    "\n",
    "1. Assigning your new dataframe and give it a reasonable name (e.g., `spiny_df`)\n",
    "2. Creating a second dataframe for aspiny.\n",
    "    \n",
    "    </div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spiny_df = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step Two: Get Electrophysiology Data\n",
    "\n",
    "At this point, you might have realized that this dataframe doesn't contain any data about the electrophysiology -- it's just metadata about the cells. In order to get information about the electrophysiological properties of these cells, we need to use the `get_ephys_features()` method on our instance of the cell types cache.\n",
    "\n",
    "1. Execute the `get_ephys_features` method on our cell types instance and assign the output of this to `ephys_features`.\n",
    "2. Convert `ephys_features` into a pandas dataframe.\n",
    "3. Re-assign the index to be the column labeled 'specimen_id' of the cell (and reassign to `ephys_features`). 'specimen_id' is the label that can link this dataframe to our metadata (human_df) dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ephys_features = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Join our dataframes\n",
    "\n",
    "Now we have two dataframes, one with the metadata for human cells (indexed by id) and another with the electrophysiology data for all cells, also indexed by id. Usefully, these ids are unique to each cell, meaning we can match them across dataframes.\n",
    "\n",
    "We can use either the `merge` or `join` pandas methods in order to pull all of this data into one dataframe. \n",
    "\n",
    "![](http://www.datasciencemadesimple.com/wp-content/uploads/2017/09/join-or-merge-in-python-pandas-1.png)\n",
    "\n",
    "There are different types of joins/merges you can do in pandas, illustrated <a href=\"http://www.datasciencemadesimple.com/join-merge-data-frames-pandas-python/\">above</a>. Here, we want to do an **inner** merge, where we're only keeping entries with indices that are in both dataframes. We could do this merge based on columns, alternatively.\n",
    "\n",
    "**Inner** is the default kind of join, so we do not need to specify it. And by default, join will use the 'left' dataframe, in other words, the dataframe that is executing the `join` method.\n",
    "\n",
    "If you need more information, look at the <a href=\"https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.join.html\">join</a> and <a href=\"https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.merge.html\">merge</a> documentation: you can use either of these to unite your dataframes, though join will be simpler!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "human_ephys_features = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step Three: Confirm the data and take a look!\n",
    "\n",
    "As a result, you should have a dataframe called `human_ephys_features` that contains metadata about your cells, as well as their electrophysiological properties.\n",
    "\n",
    "<div class=\"alert alert-success\"><b>Task</b>: Confirm and inspect your data.\n",
    "\n",
    "1. Confirm that you have right amount of data by checking it's length using a Boolean to test whether it is equal to `n_human_cells` that you assigned above.\n",
    "2. Confirm that you have all of the columns from *both* the human_df and ephys_features dataframes programmatically. Remember that you can get the columns by accessing the `columns` attribute, and that you already assigned the human_df columns to a variable above. There are a few different ways to do this!\n",
    "3. Confirm that the only 'species' in your `human_ephys_features` dataframe is 'Homo Sapiens'. You can use the `unique()` method to show unique values in a column.\n",
    "    </div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check your data structure here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Finally*, let's take a look at the data. You can use the `describe()` method to show the basic statistics for your cells. We'll start plotting these metrics next week!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect the data here"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}