04_cnn_prediction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "20f337bb-9b72-46e7-a409-a8958997b434",
   "metadata": {},
   "source": [
    "## Use the CNN to detect Rana sierrae vocalizations\n",
    "\n",
    "This script uses the CNN trained in 03_train_cnn.py to detect Rana sierrae vocalizations in audio recordings. You can alternatively use the the original model object trained and used in the manuscript, which is included in the subfolder `./resources/rana_seirrae_cnn.model`. \n",
    "\n",
    "Note that the github repository does not contain the full (very large) table of CNN outputs nor the full original audio\n",
    "dataset, and thus does not contain the files needed to reproduce the outputs of this script. \n",
    "\n",
    "Instead, this script demonstrates the prediction process by generating CNN scores on the validation set.\n",
    "\n",
    "Researchers interested in accessing the full field dataset can contact `sam.lapp@pitt.edu`. \n",
    "\n",
    "This notebook is part of a series of notebooks and scripts in the [repository](https://github.com/kitzeslab/rana-sierrae-cnn):\n",
    "\n",
    "- `01_explore_annotated_data.ipynb` Explore annotated dataset of Rana sierrae call types\n",
    "\n",
    "- `02_prep_training_data.ipynb` Prepare annotated files for training a CNN machine learning model\n",
    "\n",
    "- `03_train_cnn.ipynb` Train a CNN to recognize Rana sierrae vocaliztaions\n",
    "\n",
    "- `04_cnn_prediction.ipynb` Use the cnn to detect Rana sierrae in audio recordings\n",
    "\n",
    "- `05_cnn_validation.ipynb` Analyze the accuracy and performance of the CNN\n",
    "\n",
    "- `06_aggregate_scores.py` Aggregate scores from CNN prediction across dates and times of day\n",
    "\n",
    "- `07_explore_results.ipynb` Analyze temporal patterns of vocal activity using the CNN detections\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f70da859-782e-4daf-b4ed-11dace20a327",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b09a2b15-6a87-4af1-af7f-570480c9a6ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "from opensoundscape.torch.models.cnn import load_model\n",
    "import wandb\n",
    "from pathlib import Path\n",
    "from datetime import datetime\n",
    "import pandas as pd\n",
    "from torch import softmax, tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3e90ef6-644c-4f36-a25f-db4e9e993d77",
   "metadata": {},
   "source": [
    "### Predict on validation set audio files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db78fce1-e8a3-41a1-bc13-21d958ba556e",
   "metadata": {},
   "source": [
    "load model and files to predict on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "285eb911-3ed4-4b70-a7ce-a7096a7606b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# choose output location for saving .csv files of CNN output scores\n",
    "score_save_dir = './resources/'\n",
    "\n",
    "# load the OpenSoundscape CNN model object\n",
    "# this model was trained and used in the original manuscript\n",
    "# you could replace this path with the path to a model you trained in the previous notebook\n",
    "model = load_model('./resources/rana_sierrae_cnn.model')\n",
    "\n",
    "# choose audio files to run prediction on\n",
    "validation_df = pd.read_csv('./resources/validation_set.csv').set_index(['file','start_time','end_time'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a71b3021-8d47-43d5-90e3-21ab9fbcb5f8",
   "metadata": {},
   "source": [
    "### Run prediction\n",
    "\n",
    "This cell takes approximately 1 minute to run without accelerating the model using a GPU. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e2bd6eda-7a54-4600-8063-12331b943755",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/SML161/miniconda3/envs/opso080/lib/python3.9/site-packages/opensoundscape/torch/models/cnn.py:886: UserWarning: The columns of input samples df differ from `model.classes`.\n",
      "  warnings.warn(\n",
      "/Users/SML161/miniconda3/envs/opso080/lib/python3.9/site-packages/torch/utils/data/dataloader.py:554: UserWarning: This DataLoader will create 12 worker processes in total. Our suggested max number of worker in current system is 10 (`cpuset` is not taken into account), which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n",
      "  warnings.warn(_create_warning_msg(\n",
      "/Users/SML161/miniconda3/envs/opso080/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp06z8kwz8wandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso080/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmprbqvwm8gwandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso080/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpchnz9m0nwandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso080/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpoxv33obnwandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso080/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpxn17a9arwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso080/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpwmhl_eo4wandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso080/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpkugdp7x2wandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso080/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp711dpsfdwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n"
     ]
    }
   ],
   "source": [
    "# generate predictions using batch size 1024\n",
    "# select smaller batch size if you have memory issues\n",
    "scores = model.predict(validation_df,num_workers=12,batch_size=1024,wandb_session=None)#wandb_session)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4413e1d1-f0ce-40e2-84dd-dd22d68e1f23",
   "metadata": {},
   "source": [
    "### Save validation scores along with labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1dee5d75-586d-4e64-89cf-188e4ddb7137",
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute the softmax score across the two classes\n",
    "scores['softmax']=softmax(tensor(scores[['rana_sierrae','negative']].values),1)[:,0].numpy()\n",
    "\n",
    "# save scores to file along with labels\n",
    "validation_df['score']=scores['softmax']\n",
    "validation_df[['rana_sierrae','score']].to_csv('./resources/validation_labels_and_scores.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c254f3ce-0494-41a4-97aa-bcd9b4ca2e28",
   "metadata": {},
   "source": [
    "Check validation metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5cf08f08-c70a-4ab2-b297-b61488d66650",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_recall_curve, average_precision_score, roc_auc_score, auc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "98ad1915-8a50-4407-9d5e-be97cc127791",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average precision: 0.9187549944446871\n",
      "Area under ROC curve: 0.951881297624947\n",
      "Area under P-R curve: 0.9186412082012038\n"
     ]
    }
   ],
   "source": [
    "p,r,t = precision_recall_curve(validation_df['rana_sierrae'],validation_df['score'])\n",
    "print(f\"Average precision: {average_precision_score(validation_df['rana_sierrae'],validation_df['score'])}\")\n",
    "print(f\"Area under ROC curve: {roc_auc_score(validation_df['rana_sierrae'],validation_df['score'])}\")\n",
    "print(f\"Area under P-R curve: {auc(r,p)}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "opso080",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}