getting_started/1_Grid2opFramework.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This notebook present the most basic use of Grid2Op"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Objectives**\n",
    "\n",
    "This notebook will cover some basic raw functionalities at first. It will then show how these raw functionalities are encapsulated in easy-to-use functions.\n",
    "\n",
    "The recommended way to use these is to through the Runner, and not by going through the instanciation of the different classes one after the other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import grid2op"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Impossible to automatically add a menu / table of content to this notebook.\n",
      "You can download \"jyquickhelper\" package with: \n",
      "\"pip install jyquickhelper\"\n"
     ]
    }
   ],
   "source": [
    "res = None\n",
    "try:\n",
    "    from jyquickhelper import add_notebook_menu\n",
    "    res = add_notebook_menu()\n",
    "except ModuleNotFoundError:\n",
    "    print(\"Impossible to automatically add a menu / table of content to this notebook.\\nYou can download \\\"jyquickhelper\\\" package with: \\n\\\"pip install jyquickhelper\\\"\")\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## I) Creating an Environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I.A) Default settings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We provide a function that will handle the creation of the Environment with default values in a single call of the function.\n",
    "\n",
    "In this example we will use the `rte_case14_redisp` environment, in test mode.\n",
    "\n",
    "To define/create it, we can call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/benjamin/Documents/grid2op_dev/getting_started/grid2op/MakeEnv/Make.py:240: UserWarning: You are using a development environment. This environment is not intended for training agents.\n",
      "  warnings.warn(_MAKE_DEV_ENV_WARN)\n"
     ]
    }
   ],
   "source": [
    "env = grid2op.make(\"rte_case14_redisp\", test=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NB** By setting \"test=True\" in the above call, we only use the data for 2 different months for our environment. If you remove it, grid2op.make will attempt to download more data. By default, the data corresponding to this environment will be downloaded to your \"home\" directory, which corresponds to the location returned by this script:\n",
    "\n",
    "```python\n",
    "import os\n",
    "print(f\"grid2op dataset will be downloaded in {os.path.expanduser('~/data_grid2op')}\")\n",
    "```\n",
    "\n",
    "If you want to set another default download path, you can add a .grid2opconfig.json file and specify where to download the data.\n",
    "\n",
    "Only 4 environments are available locally when you install grid2op:\n",
    "- `rte_case5_example`, only available locally. Its main goal is to provide examples on a really small system that people can really study manually so that they can better understand the idea of the underlying mechanics.\n",
    "- `rte_case14_test`, only available locally. Its main goal is to be used inside the unit tests of grid2op. It is **NOT** recommended to use it.\n",
    "- `rte_case14_redisp`, an environment based on the IEEE case14 files and which introduces the redispatching actions. Only 2 datasets are available for this environment. More can be downloaded automatically by specifying \"local=False\" [default value of argument \"local\"].\n",
    "- `rte_case14_realistic`, a \"realistic\" environment based on the same grid and which has been adapted to be closer to the real grid. More data can be downloaded automatically by specifying \"local=False\" [default value of argument \"local\"].\n",
    "\n",
    "Other environments can be used and are available through the \"make\" command. To get a list of the available environments, you can do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['l2rpn_2019',\n",
       " 'l2rpn_case14_sandbox',\n",
       " 'rte_case14_realistic',\n",
       " 'rte_case14_redisp',\n",
       " 'wcci_test']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid2op.list_available_remote_env()  # this only works if you have an internet connection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to list the environments that you have already downloaded (if any). **NB** We remind that downloading is automatic and is done on the first time that you call `make` with an environment that has not been already locally downloaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['wcci_test', 'ieee118_R2subgrid_wcci_v2', 'l2rpn_case14_sandbox']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid2op.list_available_local_env()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I.B) Custom settings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `make` function, you can pass additional arguments to customize the environment (this is useful for training):\n",
    " - `param`: The parameters used for the Environment. See `grid2op.Parameters.Parameters`.\n",
    " - `backend` : The backend to use for the computation. If provided, it must be an instance of the class `grid2op.Backend.Backend`.\n",
    " - `action_class`: The type of BaseAction that the BaseAgent will be able to perform. If provided, it must be a subclass of `grid2op.BaseAction.BaseAction`.\n",
    " - `observation_class`: The type of BaseObservation that the BaseAgent will receive. If provided, It must be a subclass of `grid2op.BaseAction.BaseObservation`.\n",
    " - `reward_class`: The type of reward signal that the BaseAgent will receive. If provided, It must be a subclass of `grid2op.BaseReward.BaseReward`.\n",
    " - `gamerules_class`: The type of \"Rules\" that the BaseAgent will need to comply with. Rules are here to model some operational constraints. If provided, it must be a subclass of `grid2op.RulesChecker.BaseRules`.\n",
    " - `data_feeding_kwargs`: A dictionnary that is used to build the `data_feeding` (chronics) objects.\n",
    " - `chronics_class`: The type of chronics that represents the dynamics of the created Environment. Usually they come from different folders.\n",
    " - `data_feeding`: The type of chronics handler you want to use.\n",
    " - `volagecontroler_class`: The type of `grid2op.VoltageControler.VoltageControler` to use.\n",
    " - `chronics_path`: The path where to look for the chronics dataset (optional).\n",
    " - `grid_path`: The path where the powergrid is located. If provided, it must be a string and point to a valid file on the hard drive.\n",
    " \n",
    "For example, to set the number of maximum allowed substation changes per step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/benjamin/Documents/grid2op_dev/getting_started/grid2op/MakeEnv/Make.py:240: UserWarning: You are using a development environment. This environment is not intended for training agents.\n",
      "  warnings.warn(_MAKE_DEV_ENV_WARN)\n"
     ]
    }
   ],
   "source": [
    "from grid2op.Parameters import Parameters\n",
    "\n",
    "custom_params = Parameters()\n",
    "custom_params.MAX_SUB_CHANGED = 1\n",
    "env = grid2op.make(\"rte_case14_redisp\", param=custom_params, test=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NB** The `make` function is highly customizable. For example, you can change the reward that you are using:\n",
    "\n",
    "```python\n",
    "from grid2op.Reward import L2RPNReward\n",
    "env = grid2op.make(reward_class=L2RPNReward)\n",
    "```\n",
    "\n",
    "We also give the possibility to assess different rewards. This can be done with the following code:\n",
    "\n",
    "```python\n",
    "\n",
    "from grid2op.Reward import L2RPNReward, FlatReward\n",
    "env = grid2op.make(reward_class=L2RPNReward,\n",
    "                   other_rewards={\"other_reward\" : FlatReward })\n",
    "```\n",
    "The results for these rewards can be accessed with the `info` object returned by `env.step` (`info` is the 4th object returned by `env.step`, as you can see below). See the official reward documentation [here](https://grid2op.readthedocs.io/en/latest/reward.html) for more information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## II) Creating an Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An *Agent* is the name given to the \"operator\" / \"bot\" / \"algorithm\" that will perform some modifications of the powergrid when it faces some \"observation\".\n",
    "\n",
    "Some examples of Agents are provided in the file [Agent.py](grid2op/Agent/Agent.py).\n",
    "\n",
    "A deeper look at the different provided Agents can be found in the [4_StudyYourAgent](4_StudyYourAgent.ipynb.ipynb) notebook. We suppose here that we use the most simple Agent, the one that does nothing (`DoNothingAgent`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from grid2op.Agent import DoNothingAgent\n",
    "my_agent = DoNothingAgent(env.helper_action_player)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## III) Assess how the Agent is performing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The performance of an Agent is assessed with the cumulated reward it receives over time. In this example, the cumulated reward is a *FlatReward* that simply computes how many time steps the *Agent* has sucessfully managed before breaking any rules. For more control over this reward, it is recommended to use look at the documentation of the Environment class.\n",
    "\n",
    "More examples of rewards are also available on the official documentation or [here](https://grid2op.readthedocs.io/en/latest/reward.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "done = False\n",
    "time_step = int(0)\n",
    "cum_reward = 0.\n",
    "obs = env.reset()\n",
    "reward = env.reward_range[0]\n",
    "max_iter = 10\n",
    "while not done:\n",
    "    act = my_agent.act(obs, reward, done) # chose an action to do, in this case \"do nothing\"\n",
    "    obs, reward, done, info = env.step(act) # implement this action on the powergrid\n",
    "    cum_reward += reward\n",
    "    time_step += 1\n",
    "    if time_step >= max_iter:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now evaluate how well this *agent* is performing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This agent managed to survive 10 timesteps\n",
      "It's final cumulated reward is 11096.61962890625\n"
     ]
    }
   ],
   "source": [
    "print(\"This agent managed to survive {} timesteps\".format(time_step))\n",
    "print(\"It's final cumulated reward is {}\".format(cum_reward))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IV) More convenient ways to assess the performance of an agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the steps above have been detailed as a \"quick start\", to give an example of the main classes of the Grid2Op package. Having to code all of the above can be quite tedious, but offers a lot of flexibility.\n",
    "\n",
    "Implementing all this before starting to evaluate an agent can be tiring. What we show here is a much shorter way to perfom all this. In this section we will exhibit 2 ways:\n",
    "* The quickest way, using the grid2op.main API, most suited when basic computations need to be carried out.\n",
    "* The recommended way using a *Runner*. This gives more flexibility than the grid2op.main API but can be harder to configure.\n",
    "\n",
    "In this section, we assume the same as before:\n",
    "* The Agent is the \"Do Nothing\" agent\n",
    "* The Environment is the default Environment\n",
    "* PandaPower is used as the backend\n",
    "* The chronics comes from the files included in this package\n",
    "* etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IV.A) Using the grid2op.runner API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When only simple assessments need to be performed, the grid2op.main API is perfectly suited. This API can also be accessed with the command line:\n",
    "```bash\n",
    "python3 -m grid2op.main\n",
    "```\n",
    "\n",
    "We detail here its usage as an API, to assess the performance of a given Agent.\n",
    "\n",
    "As opposed to building en environment from scratch (see the previous section), this requires much less effort: we don't need to initialize (instanciate) anything. Everything is carried out inside the Runner called by the `main` function.\n",
    "\n",
    "We simulate 1 episode here (eg. we play one scenario until: either the agent does a game over, or the scenario ends), but this method would work too if we wanted to simulate more episodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from grid2op.Runner import Runner\n",
    "runner = Runner(**env.get_params_for_runner(), agentClass=DoNothingAgent)\n",
    "res = runner.run(nb_episode=1, max_iter=max_iter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A call of the single 2 lines above will:\n",
    "* Create a valid environment\n",
    "* Create a valid agent\n",
    "* Assess how well an agent performs on one episode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The results are:\n",
      "\tFor chronics located at /home/benjamin/Documents/grid2op_dev/getting_started/grid2op/data/rte_case14_redisp/chronics/0\n",
      "\t\t - cumulative reward: 10948.05\n",
      "\t\t - number of time steps completed: 10 / 10\n"
     ]
    }
   ],
   "source": [
    "print(\"The results are:\")\n",
    "for chron_name, _, cum_reward, nb_time_step, max_ts in res:\n",
    "    msg_tmp = \"\\tFor chronics located at {}\\n\".format(chron_name)\n",
    "    msg_tmp += \"\\t\\t - cumulative reward: {:.2f}\\n\".format(cum_reward)\n",
    "    msg_tmp += \"\\t\\t - number of time steps completed: {:.0f} / {:.0f}\".format(nb_time_step, max_ts)\n",
    "    print(msg_tmp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is particularly suited for evaluating different agents. For example, we can quickly evaluate a second agent. In the example below, we can import an agent class *PowerLineSwitch* whose job is to connect and disconnect the power lines in the power network. This *PowerLineSwitch* Agent will **simulate** the effect of disconnecting a powerline for each powerline in the powergrid, and take the best action found ie the one whose simulated effect is the best (its execution can take a long time, depending on the scenario and the amount of powerlines in the grid). **The execution of the code below can take a little time**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The results are:\n",
      "\tFor chronics located at /home/benjamin/Documents/grid2op_dev/getting_started/grid2op/data/rte_case14_redisp/chronics/0\n",
      "\t\t - cumulative reward: 10950.26\n",
      "\t\t - number of time steps completed: 10 / 10\n"
     ]
    }
   ],
   "source": [
    "from grid2op.Agent import PowerLineSwitch\n",
    "runner = Runner(**env.get_params_for_runner(), agentClass=PowerLineSwitch)\n",
    "res = runner.run(nb_episode=1, max_iter=max_iter)\n",
    "print(\"The results are:\")\n",
    "for chron_name, _, cum_reward, nb_time_step, max_ts in res:\n",
    "    msg_tmp = \"\\tFor chronics located at {}\\n\".format(chron_name)\n",
    "    msg_tmp += \"\\t\\t - cumulative reward: {:.2f}\\n\".format(cum_reward)\n",
    "    msg_tmp += \"\\t\\t - number of time steps completed: {:.0f} / {:.0f}\".format(nb_time_step, max_ts)\n",
    "    print(msg_tmp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible using this API to store the results for a detailed examination of the actions taken by the Agent. Note that writing on the hard drive has an overhead on the computation time.\n",
    "\n",
    "To do this, only a simple argument needs to be added to the *main* function call (`path_save`, which indicates where the outcome of the experiment will be stored). An example can be found below :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The results are:\n",
      "\tFor chronics located at /home/benjamin/Documents/grid2op_dev/getting_started/grid2op/data/rte_case14_redisp/chronics/0\n",
      "\t\t - cumulative reward: 10950.26\n",
      "\t\t - number of time steps completed: 10 / 10\n"
     ]
    }
   ],
   "source": [
    "runner = Runner(**env.get_params_for_runner(),\n",
    "                agentClass=PowerLineSwitch\n",
    "               )\n",
    "path_save_expe = os.path.abspath(\"saved_experiment_donothing\")\n",
    "res = runner.run(nb_episode=1, max_iter=max_iter, path_save=path_save_expe)\n",
    "print(\"The results are:\")\n",
    "for chron_name, _, cum_reward, nb_time_step, max_ts in res:\n",
    "    msg_tmp = \"\\tFor chronics located at {}\\n\".format(chron_name)\n",
    "    msg_tmp += \"\\t\\t - cumulative reward: {:.2f}\\n\".format(cum_reward)\n",
    "    msg_tmp += \"\\t\\t - number of time steps completed: {:.0f} / {:.0f}\".format(nb_time_step, max_ts)\n",
    "    print(msg_tmp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['episode_meta.json',\n",
       " 'disc_lines_cascading_failure.npz',\n",
       " 'actions.npz',\n",
       " 'rewards.npz',\n",
       " 'observations.npz',\n",
       " 'agent_exec_times.npz',\n",
       " 'episode_times.json',\n",
       " '_parameters.json',\n",
       " 'other_rewards.json',\n",
       " 'env_modifications.npz']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(os.path.join(path_save_expe, \"0\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the outcomes of the experiment are shown above. For more information, please don't hesitate to read the documentation of [Runner](https://grid2op.readthedocs.io/en/latest/runner.html) (if compiled locally) or consult the [Runner.py](../grid2op/Runner.py) file.\n",
    "\n",
    "**NB** A lot more of information about *Actions* is provided in the [2_Action_GridManipulation](2_Action_GridManipulation.ipynb) notebook. In the [3_TrainingAnAgent](3_TrainingAnAgent.ipynb) (last section), there is an quick example of how to read / write an action from a saved repository.\n",
    "\n",
    "In the notebook 7, more details will be given concerning the advantages of the runner, especially for analyzing the agent's performances."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `make` and `Runner` makes it easy to assess the performance of a trained agent. Besides, the `Runner` has been particularly integrated with other tools and makes it easy to replay and analyse an episode after it is finished. It is the recommended method to use in grid2op for the evaluation."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}