speedup/analyze_profiled_runs.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "from tbp.monty.frameworks.utils.profile_utils import (\n",
    "    bar_chart_cumtime,\n",
    "    bar_chart_tottime,\n",
    "    get_data_from_df,\n",
    "    print_top_k_functions,\n",
    "    sort_by_cumtime,\n",
    "    sort_by_tottime\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_path = os.path.expanduser(\"~/tbp/results/monty/projects/monty_runs/\")\n",
    "fpt_path = os.path.join(results_path, \"feature_pred_tests_dev/profile\")\n",
    "fpt_eval = os.path.join(fpt_path, \"profile-evaluate.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Feature pred tests dev, full evaluation profile.\n",
    "\n",
    "This gives big picture information about the call to `evaluate` as opposed to smaller scale in a single episode which we will see later. Cumtime stands for cumulative time and means time taken by a function call, including all functions called inside the original function. Tottime stands for total time and means time taken by a function call, excluding time taken by all functions called within the original. My understanding is tottime = cumtime - sum(cumtime(f_i) for f_i in functions called by original)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(fpt_eval)\n",
    "bar_chart_cumtime(df, n_functions=25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I chopped off the filenames associated with function calls so they fit on the screen. I printed them below with the full filename for clarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df, func_names, cumtimes = get_data_from_df(df)\n",
    "print_top_k_functions(func_names, k=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(fpt_eval)\n",
    "bar_chart_tottime(df, n_functions=25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df, func_names, cumtimes = get_data_from_df(df, sortby=\"tottime\")\n",
    "print_top_k_functions(func_names, k=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpretation\n",
    "\n",
    "Swapping out objects in habitat (remove_all_objects, add_object) is the most expensive operation. Logging is next."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample episode\n",
    "\n",
    "Now let's look at a sample episode to see what calls inside an episode are the most costly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fpt_eval = os.path.join(fpt_path, \"profile-eval_epoch_0_episode_3.csv\")\n",
    "df = pd.read_csv(fpt_eval)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bar_chart_cumtime(df, n_functions=25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df, func_names, cumtimes = get_data_from_df(df)\n",
    "print_top_k_functions(func_names, k=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bar_chart_tottime(df, n_functions=25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df, func_names, cumtimes = get_data_from_df(df, sortby=\"tottime\")\n",
    "print_top_k_functions(func_names, k=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpretation\n",
    "\n",
    "The most expensive operations are `get_node_features`, cloning torch tensors, and logging are the most expensive parts of the code. Next is stuff like compute_possible_matches."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.13 ('tbp.monty')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "c58d2ad1eef270dd28e8febff09dccc29cd6c14ad8ae8485c3c8d4e991b6e982"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}