CNS_TASK_3_AASHAY_GONDALIA.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "CNS_TASK_3_AASHAY_GONDALIA.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T4eSNDCDYQAs"
      },
      "source": [
        "Fall Student Projects 2021: Skills Tests\n",
        "---\n",
        " Task 3 -> Retrieve CellxGene Data\n",
        "\n",
        "---\n",
        "by Aashay Gondalia (aagond@iu.edu)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g3v-c1B8hPHU"
      },
      "source": [
        "\n",
        "In this workbook, I have implemented a data fetching function that can scrap the cellxgene website data, download\n",
        "and read the datasets from all the different collections in the required format mentioned in the [google doc](https://docs.google.com/document/d/1YncjOGbgKKRJw2M0fPt5bZqdCgm9EbwyTR9DoJmZtBs/edit#). \n",
        "\n",
        "A table is prepared after inputting the required information from all the incoming data. \n",
        "\n",
        "For future work, this task can be parallelized and the RAM usage can be maintained using lazy reads ([Dask](https://dask.org/)).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aJGfYpoPfxOD"
      },
      "source": [
        "Installing required packages\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        " -> Scanpy\n",
        "\n",
        " Scanpy is used in this workbook to read the h5 format datasets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O8CtozhNglNs",
        "outputId": "11358d20-c35e-4bd5-abb4-04fb9a65f612"
      },
      "source": [
        "!pip install scanpy"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: scanpy in /usr/local/lib/python3.7/dist-packages (1.8.1)\n",
            "Requirement already satisfied: networkx>=2.3 in /usr/local/lib/python3.7/dist-packages (from scanpy) (2.6.2)\n",
            "Requirement already satisfied: importlib_metadata>=0.7 in /usr/local/lib/python3.7/dist-packages (from scanpy) (4.6.4)\n",
            "Requirement already satisfied: pandas>=0.21 in /usr/local/lib/python3.7/dist-packages (from scanpy) (1.1.5)\n",
            "Requirement already satisfied: anndata>=0.7.4 in /usr/local/lib/python3.7/dist-packages (from scanpy) (0.7.6)\n",
            "Requirement already satisfied: seaborn in /usr/local/lib/python3.7/dist-packages (from scanpy) (0.11.1)\n",
            "Requirement already satisfied: sinfo in /usr/local/lib/python3.7/dist-packages (from scanpy) (0.3.4)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from scanpy) (4.62.0)\n",
            "Requirement already satisfied: patsy in /usr/local/lib/python3.7/dist-packages (from scanpy) (0.5.1)\n",
            "Requirement already satisfied: scikit-learn>=0.22 in /usr/local/lib/python3.7/dist-packages (from scanpy) (0.22.2.post1)\n",
            "Requirement already satisfied: natsort in /usr/local/lib/python3.7/dist-packages (from scanpy) (5.5.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from scanpy) (21.0)\n",
            "Requirement already satisfied: h5py>=2.10.0 in /usr/local/lib/python3.7/dist-packages (from scanpy) (3.1.0)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from scanpy) (1.0.1)\n",
            "Requirement already satisfied: statsmodels>=0.10.0rc2 in /usr/local/lib/python3.7/dist-packages (from scanpy) (0.10.2)\n",
            "Requirement already satisfied: scipy>=1.4 in /usr/local/lib/python3.7/dist-packages (from scanpy) (1.4.1)\n",
            "Requirement already satisfied: tables in /usr/local/lib/python3.7/dist-packages (from scanpy) (3.4.4)\n",
            "Requirement already satisfied: numba>=0.41.0 in /usr/local/lib/python3.7/dist-packages (from scanpy) (0.51.2)\n",
            "Requirement already satisfied: umap-learn>=0.3.10 in /usr/local/lib/python3.7/dist-packages (from scanpy) (0.5.1)\n",
            "Requirement already satisfied: matplotlib>=3.1.2 in /usr/local/lib/python3.7/dist-packages (from scanpy) (3.2.2)\n",
            "Requirement already satisfied: numpy>=1.17.0 in /usr/local/lib/python3.7/dist-packages (from scanpy) (1.19.5)\n",
            "Requirement already satisfied: xlrd<2.0 in /usr/local/lib/python3.7/dist-packages (from anndata>=0.7.4->scanpy) (1.1.0)\n",
            "Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py>=2.10.0->scanpy) (1.5.2)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib_metadata>=0.7->scanpy) (3.7.4.3)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib_metadata>=0.7->scanpy) (3.5.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.1.2->scanpy) (0.10.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.1.2->scanpy) (2.4.7)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.1.2->scanpy) (1.3.1)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.1.2->scanpy) (2.8.2)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from cycler>=0.10->matplotlib>=3.1.2->scanpy) (1.15.0)\n",
            "Requirement already satisfied: llvmlite<0.35,>=0.34.0.dev0 in /usr/local/lib/python3.7/dist-packages (from numba>=0.41.0->scanpy) (0.34.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from numba>=0.41.0->scanpy) (57.4.0)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.21->scanpy) (2018.9)\n",
            "Requirement already satisfied: pynndescent>=0.5 in /usr/local/lib/python3.7/dist-packages (from umap-learn>=0.3.10->scanpy) (0.5.4)\n",
            "Requirement already satisfied: stdlib-list in /usr/local/lib/python3.7/dist-packages (from sinfo->scanpy) (0.8.0)\n",
            "Requirement already satisfied: numexpr>=2.5.2 in /usr/local/lib/python3.7/dist-packages (from tables->scanpy) (2.7.3)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZECUYQTfX79o"
      },
      "source": [
        "1. Importing necessary packages\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_EIOBswzYC_d"
      },
      "source": [
        "import datetime\n",
        "\n",
        "import requests\n",
        "from requests.adapters import HTTPAdapter\n",
        "import json\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import scanpy as sc\n",
        "import os"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3p4zlWf3YkyW"
      },
      "source": [
        "2. Function to Fetch Collection Data from the https://cellxgene.cziscience.com/ website"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NTAQNyLmZQYb"
      },
      "source": [
        "def fetchCollectionData():\n",
        "  ## HTTP Adapter Setup\n",
        "  adapter = HTTPAdapter(max_retries=3)  #Hard-coded 3 Max Retries\n",
        "  https = requests.Session()\n",
        "  https.mount(\"https://\", adapter)\n",
        "\n",
        "  ## URL Elements\n",
        "  CELLXGENE_PRODUCTION_ENDPOINT = 'https://api.cellxgene.cziscience.com'\n",
        "  COLLECTIONS = CELLXGENE_PRODUCTION_ENDPOINT + \"/dp/v1/collections/\"\n",
        "  DATASETS = CELLXGENE_PRODUCTION_ENDPOINT + \"/dp/v1/datasets/\"\n",
        "\n",
        "  ## Fetch collection data\n",
        "  r = https.get(COLLECTIONS)\n",
        "  r.raise_for_status()\n",
        "\n",
        "  collections = sorted(r.json()['collections'], key= lambda key :key['created_at'], reverse=True)\n",
        "  print('Collection Fetch Complete.')\n",
        "  return collections, https, CELLXGENE_PRODUCTION_ENDPOINT, COLLECTIONS, DATASETS"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VPWRz-LmZe2-"
      },
      "source": [
        "3. Function to filter dataset - Applied Filters : {'Disease': 'Normal', 'Species': 'Homo Sapiens'}"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UGIQ6EHyZfTo"
      },
      "source": [
        "def filter_Dataset_Homo_Sapien_Normal(all_collections):\n",
        "  only_normal_homo_sapiens_ids = []\n",
        "  for metadata in all_collections:\n",
        "    collection_cell_counter = 0\n",
        "    for dataset in metadata['datasets']:\n",
        "      diseases = dataset['disease']\n",
        "      id = dataset['id']\n",
        "      for disease in diseases:\n",
        "        if (str(disease['label']).lower() == 'normal' and str(dataset['organism']['label']).lower() == 'homo sapiens'):\n",
        "          #Disease = disease['label']\n",
        "          #Assay = dataset['assay']\n",
        "          #Tissue = dataset['tissue']\n",
        "          #Dataset_Name = dataset['name']\n",
        "          try:\n",
        "            collection_cell_counter += dataset['cell_count']\n",
        "          except:\n",
        "            pass\n",
        "          only_normal_homo_sapiens_ids.append(id)\n",
        "  print('Dataset Filters Applied.')\n",
        "  return only_normal_homo_sapiens_ids"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S_KoAaLyap3g"
      },
      "source": [
        "4. Initializing the output table. As mentioned in the google document, the dataframe column names are set accordingly. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GzhheyCkapOS"
      },
      "source": [
        "def initializeTable():\n",
        "  table = pd.DataFrame({\n",
        "    'Organ/Tissue Type' : [], \n",
        "    'Cell Type CL ID' : [],\n",
        "    'HGNC/ENSEMBL Gene IDs' : [],\n",
        "    'No. of Cells of this type' : [],\n",
        "    'Disease' : [],\n",
        "    'Assay' : [],\n",
        "    'Tissue' : [],\n",
        "    'Dataset Name' : [],\n",
        "    })\n",
        "  print('Table Initialization Complete.')\n",
        "  return table\n"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Utse59s1teZD"
      },
      "source": [
        "Gene data is fetched from the X(embeddings) matrix and all the non-zero values of GENEs for the corresponding cells are fetched."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZDB27LKftYwS"
      },
      "source": [
        "def fetch_and_include_GENE_data(table, dataset, list_ex):\n",
        "  print('Introducting GENE data into the table.')\n",
        "  rows,cols = dataset.X.nonzero()\n",
        "  #print('Rows - ', rows , len(rows))\n",
        "  #print('Cols - ', cols, len(cols))\n",
        "\n",
        "  diction = {}\n",
        "  for i in list_ex:\n",
        "    diction[i] = []\n",
        "\n",
        "  #print(diction)\n",
        "\n",
        "  for j in range(len(cols)):\n",
        "    if (rows[j] in diction.keys()):\n",
        "      diction[rows[j]].append(cols[j])\n",
        "\n",
        "  gene_list = []\n",
        "  #dataset_rows = len(diction.keys())\n",
        "  for i in diction.keys():\n",
        "    genes = ''\n",
        "    for position in diction[i]:\n",
        "      genes = genes + dataset.var_names[position] + ';'\n",
        "    gene_list.append(genes)\n",
        "  #print(len(gene_list))\n",
        "  #print(gene_list)\n",
        "  #for g in gene_list:\n",
        "  #  print(g, '\\n')\n",
        "  for i in range(table.shape[0]):\n",
        "    table['HGNC/ENSEMBL Gene IDs'][i] = gene_list[i]\n",
        "  print('Gene Data Succesfully entered.')"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ORpS6AOSa-iM"
      },
      "source": [
        "This function is used to read the downloaded data from the cellxgene website, fetch the required information and enter it into the table. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bxM9Wloya8ay"
      },
      "source": [
        "def enter_Details_into_Table(download_name, Disease, Assay, Tissue, Dataset_Name):\n",
        "  '''try:\n",
        "    table = pd.read_csv('dataTable.csv', sep='|')\n",
        "    print('Table already exists -> Imported Data')\n",
        "  except:'''\n",
        "  table = initializeTable()\n",
        "\n",
        "  print('Adding data to Table....')\n",
        "  dataset = sc.read_h5ad(download_name)\n",
        "  print('Dataset Imported in Scanpy Successfully.')\n",
        "  os.remove(download_name)\n",
        "  print('Removed dataset to aid program execution.')\n",
        "\n",
        "  #print('Dataset Reading Complete')\n",
        "  # 'Organ/Tissue Type', 'Cell Type CL ID', 'HGNC/ENSEMBL Gene IDs',\n",
        "  # 'Cells of this type', 'Disease', 'Assay', 'Tissue', 'Dataset Name'\n",
        "  \n",
        "  # Gene IDs Aggregation into a single field.\n",
        "  table_cell_ids = []\n",
        "  table_row_ids = []\n",
        "  for i in range(dataset.shape[0]):\n",
        "\n",
        "    if (dataset.obs['cell_type_ontology_term_id'][i] not in table_cell_ids):\n",
        "      table_row_ids.append(i)\n",
        "      \n",
        "      num_cells_float = dataset.obs.cell_type_ontology_term_id.value_counts()[dataset.obs['cell_type_ontology_term_id'][i]] \n",
        "      no_of_cells_of_same_type = int (num_cells_float)\n",
        "      \n",
        "      table.loc[len(table.index)] = [\n",
        "                              dataset.obs['tissue'][i],\n",
        "                              dataset.obs['cell_type_ontology_term_id'][i],\n",
        "                              '_to_be_filled_',\n",
        "                              no_of_cells_of_same_type,\n",
        "                              Disease, \n",
        "                              dataset.obs['assay'][i], \n",
        "                              Tissue, \n",
        "                              Dataset_Name\n",
        "                              ]\n",
        "      \n",
        "      table_cell_ids.append(dataset.obs['cell_type_ontology_term_id'][i])\n",
        "\n",
        "  fetch_and_include_GENE_data(table, dataset, table_row_ids)\n",
        "  print('Data successfully added to the Table.')\n",
        "  print(table)\n",
        "\n",
        "  return table"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7evPdS5dzRc4"
      },
      "source": [
        "Total checker function to match the total cell_count in the dataset and the cell_count mentioned on the website."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yt26ZuGUxz0i"
      },
      "source": [
        "def check_total_cell_count(cell_count_total_website, table):\n",
        "  print('Website Cell Count : ', cell_count_total_website)\n",
        "  table_total = int(sum(table['No. of Cells of this type']))\n",
        "  print('Total Cell Counts in the Table : ', table_total)\n",
        "  try:\n",
        "    if (table_total == cell_count_total_website):\n",
        "      print('Cell count in the prepared table matches the cell count data on the website!!')\n",
        "  except:\n",
        "    pass\n",
        "\n"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qeVwgDwkcQZV"
      },
      "source": [
        "Master Function is the main executable function. It calls all the above mentioned functions and saves the table in the required 'pipe-seperated' values format. \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2HN9Oapna5Yn"
      },
      "source": [
        "table_dataholder = None\n",
        "def masterFunction():\n",
        "  \n",
        "  collections, https, CELLXGENE_PRODUCTION_ENDPOINT, COLLECTIONS, DATASETS = fetchCollectionData()\n",
        "  all_collections = []\n",
        "\n",
        "  ## INITIAL METADATA FETCH\n",
        "  for collection in collections:\n",
        "    r1 = https.get(COLLECTIONS + collection['id'], timeout=5)\n",
        "    collection_metadata = r1.json()\n",
        "    all_collections.append(collection_metadata)\n",
        "  \n",
        "  ## Populating only_normal_homo_sapiens_ids list with all the filtered dataset ids.\n",
        "  only_normal_homo_sapiens_ids = filter_Dataset_Homo_Sapien_Normal(all_collections)\n",
        "          \n",
        "  \n",
        "  for collection in all_collections:\n",
        "    for dataset in collection['datasets']:\n",
        "      try:\n",
        "        cell_count_total_website = dataset['cell_count']\n",
        "      except:\n",
        "        cell_count_total_website = None\n",
        "\n",
        "      for asset in dataset['dataset_assets']:\n",
        "\n",
        "        # Using the H5 format for less overload and compatibility\n",
        "        # Faced some issues with the RDS formatting. \n",
        "        #High overload on the python wrapper to read RDS files.\n",
        "        if ((asset['filetype'] == 'H5AD') and (asset['dataset_id'] in only_normal_homo_sapiens_ids)):\n",
        "          DATASET_REQUEST = DATASETS + asset['dataset_id']  +\"/asset/\"+  asset['id']\n",
        "          \n",
        "          r2 = requests.post(DATASET_REQUEST)\n",
        "          r2.raise_for_status()\n",
        "          presigned_url = r2.json()['presigned_url']\n",
        "          \n",
        "          headers = {'range': 'bytes=0-0'}\n",
        "          r3 = https.get(presigned_url, headers=headers)\n",
        "          print('\\nDataset -> ', dataset['name'], '\\nURL -> ', presigned_url)\n",
        "          \n",
        "          if (r3.status_code == requests.codes.partial):\n",
        "            download_name = dataset['name'] + '.h5ad'\n",
        "            print('Dataset Download Started.')\n",
        "            r3 = https.get(presigned_url, timeout=10)\n",
        "            r3.raise_for_status()\n",
        "            open(download_name, 'wb').write(r3.content)\n",
        "            print('Dataset Download Complete.')\n",
        "            table = enter_Details_into_Table(download_name, dataset['disease'], dataset['assay'], dataset['tissue'], dataset['name'])\n",
        "\n",
        "            ## SAVE TABLE \n",
        "            print('Saving Table....')\n",
        "            os.mkdir(collection['name'])\n",
        "            filepath = collection['name'] + '/' + dataset['name']\n",
        "            table.to_csv(filepath, sep='|')\n",
        "            print('Table Saved Successfully.')\n",
        "            table_dataholder = table\n",
        "            check_total_cell_count(cell_count_total_website, table)\n",
        "\n",
        "            ## To effectively use the Google COLAB RAM. \n",
        "            table = None\n",
        "            del table\n",
        "            print('Local Copy of table removed from RAM')\n",
        "\n"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ma6OQrfYjKm1"
      },
      "source": [
        "Master Function currently fetches collection data and downloads the dataset in a serial manner, which can be parallelized.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3odARkVijKLE",
        "outputId": "d4fe9566-727b-4ad7-e577-cfcced26e3f8"
      },
      "source": [
        "masterFunction()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collection Fetch Complete.\n",
            "Dataset Filters Applied.\n",
            "\n",
            "Dataset ->  Tabula Sapiens - Endothelial \n",
            "URL ->  https://corpora-data-prod.s3.amazonaws.com/5a11f879-d1ef-458a-910c-9b0bdfca5ebf/local.h5ad?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIATLYQ5N5XTAJYHK5X%2F20210823%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20210823T071432Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjENb%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJHMEUCIQDFohp83KeOaO%2BRApWcOVfbeNumiFZW5r2McxMncVYSYAIgO1x2%2FfUu6lJcAmFc%2FjFoLOj4qqDZXF6WgGO9%2Bgwg4hgq9AMI%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARABGgwyMzE0MjY4NDY1NzUiDLdD1SDWLB1zJxRv9CrIA66gFdNCQf4Ya9IwRA3wwRrftrp1m%2Bt0jN1yAV0Dd%2F3qlW6EylGydxf5MS%2BZvzwOzMsY2Fzd6DBn8Q%2BqYX014JFOiY1XkVPrMGECN39R%2FRfLYvrJpLG%2FQkDrqw7sOPE8YHWtcY2dNfx0PPgFchyUpQ1mcTSaIqJQhlxt8mXZAz70kYDkopQwr79tTYOoOGhJ4b4PIUkVVt94Hfs2u%2BUuTYf92K1ZpoBy6Lx0t0%2F%2FMm4WC7Cjbpa4dw0FYu5i%2FwC5%2FwEWcE15ahDme%2BipkTese2m%2B1scF1Bc%2FN6DCtuSo1ha36nFtNittmGpFfIzPIHl%2FHSu%2B8lFmv5TZt7ok4fBQw3mkucB3ub0knKSbH2U4S5F%2FvS3FKb3iH4SGHUgW7Sndae02c3sa%2B9BKT9u4bINsOjxST16vxE%2FmYoxM1sVhMCn%2B8l0Hp%2Fi%2FdM3VXfzLO4Qyx3quCX8qlwrAPxiWXqdSaX3xd19haeJ5kxQNFPR9Iex5RuKvIRkhgQ7OkuodpLbqFX1EFHhibFeJg99IOInbr62QovM2bj4%2BDD%2Bts5pRbC1dEEsbmLXmVhyDZhURGakQGy4G3YZupkGRd97UY9RPCSibn5g4n%2BJJEDCu8IyJBjqlARCCBxz44YeqaolVF9LA%2Fa7Y8qalmDJwH8ACFnmQEnor%2Blgz1zODilDxZEvmf9DWOBU6A6kOW0tnOmr578HMZAtMpc5XZeIa6noA46%2BowOoSQpXNfeuI%2FBlf4FpJD86Kv1oSqecaEPS8w7c1zC03Ko2zamGMr5UmzqsxbAKariXUHe8f%2BuZ%2BobbYG7h7pMcI1612ohckMv8K5e%2BnDbEnU65GicuBeQ%3D%3D&X-Amz-Signature=d2ebb37681a7638120b169663b5652117f14f9f8a3e8066f8523acc894f6c25b\n",
            "Dataset Download Started.\n",
            "Dataset Download Complete.\n",
            "Table Initialization Complete.\n",
            "Adding data to Table....\n",
            "Dataset Imported in Scanpy Successfully.\n",
            "Removed dataset to aid program execution.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
            "  return array(a, dtype, copy=False, order=order)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Introducting GENE data into the table.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:29: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Gene Data Succesfully entered.\n",
            "Data successfully added to the Table.\n",
            "         Organ/Tissue Type  ...                  Dataset Name\n",
            "0                    liver  ...  Tabula Sapiens - Endothelial\n",
            "1                  trachea  ...  Tabula Sapiens - Endothelial\n",
            "2   saliva-secreting gland  ...  Tabula Sapiens - Endothelial\n",
            "3                   tongue  ...  Tabula Sapiens - Endothelial\n",
            "4                   tongue  ...  Tabula Sapiens - Endothelial\n",
            "5                   tongue  ...  Tabula Sapiens - Endothelial\n",
            "6                      eye  ...  Tabula Sapiens - Endothelial\n",
            "7                    heart  ...  Tabula Sapiens - Endothelial\n",
            "8            muscle tissue  ...  Tabula Sapiens - Endothelial\n",
            "9          large intestine  ...  Tabula Sapiens - Endothelial\n",
            "10                    lung  ...  Tabula Sapiens - Endothelial\n",
            "11                    lung  ...  Tabula Sapiens - Endothelial\n",
            "\n",
            "[12 rows x 8 columns]\n",
            "Saving Table....\n",
            "Table Saved Successfully.\n",
            "Website Cell Count :  32701\n",
            "Total Cell Counts in the Table :  32701\n",
            "Cell count in the prepared table matches the cell count data on the website!!\n",
            "Local Copy of table removed from RAM\n",
            "\n",
            "Dataset ->  Tabula Sapiens - Immune \n",
            "URL ->  https://corpora-data-prod.s3.amazonaws.com/c5d88abe-f23a-45fa-a534-788985e93dad/local.h5ad?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIATLYQ5N5XTAJYHK5X%2F20210823%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20210823T071538Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjENb%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJHMEUCIQDFohp83KeOaO%2BRApWcOVfbeNumiFZW5r2McxMncVYSYAIgO1x2%2FfUu6lJcAmFc%2FjFoLOj4qqDZXF6WgGO9%2Bgwg4hgq9AMI%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARABGgwyMzE0MjY4NDY1NzUiDLdD1SDWLB1zJxRv9CrIA66gFdNCQf4Ya9IwRA3wwRrftrp1m%2Bt0jN1yAV0Dd%2F3qlW6EylGydxf5MS%2BZvzwOzMsY2Fzd6DBn8Q%2BqYX014JFOiY1XkVPrMGECN39R%2FRfLYvrJpLG%2FQkDrqw7sOPE8YHWtcY2dNfx0PPgFchyUpQ1mcTSaIqJQhlxt8mXZAz70kYDkopQwr79tTYOoOGhJ4b4PIUkVVt94Hfs2u%2BUuTYf92K1ZpoBy6Lx0t0%2F%2FMm4WC7Cjbpa4dw0FYu5i%2FwC5%2FwEWcE15ahDme%2BipkTese2m%2B1scF1Bc%2FN6DCtuSo1ha36nFtNittmGpFfIzPIHl%2FHSu%2B8lFmv5TZt7ok4fBQw3mkucB3ub0knKSbH2U4S5F%2FvS3FKb3iH4SGHUgW7Sndae02c3sa%2B9BKT9u4bINsOjxST16vxE%2FmYoxM1sVhMCn%2B8l0Hp%2Fi%2FdM3VXfzLO4Qyx3quCX8qlwrAPxiWXqdSaX3xd19haeJ5kxQNFPR9Iex5RuKvIRkhgQ7OkuodpLbqFX1EFHhibFeJg99IOInbr62QovM2bj4%2BDD%2Bts5pRbC1dEEsbmLXmVhyDZhURGakQGy4G3YZupkGRd97UY9RPCSibn5g4n%2BJJEDCu8IyJBjqlARCCBxz44YeqaolVF9LA%2Fa7Y8qalmDJwH8ACFnmQEnor%2Blgz1zODilDxZEvmf9DWOBU6A6kOW0tnOmr578HMZAtMpc5XZeIa6noA46%2BowOoSQpXNfeuI%2FBlf4FpJD86Kv1oSqecaEPS8w7c1zC03Ko2zamGMr5UmzqsxbAKariXUHe8f%2BuZ%2BobbYG7h7pMcI1612ohckMv8K5e%2BnDbEnU65GicuBeQ%3D%3D&X-Amz-Signature=262e60ca3fde1c09e495f7d8a4327df6ae7dd2760aef513a8a4d94a472f475a9\n",
            "Dataset Download Started.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FNPlExUGEpGI"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}