Merge pull request #40 from rte-france/bd-dev

version 0.6.0

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,49 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: bug
+assignees: ''
+
+---
+
+## System information
+ - Grid2op version: `1.x.x`
+ - l2rpn-baselines version: `1.x.x`
+ - System: `windows, osx, ubuntu16.04, ...`
+ - Baseline concerned: *eg* `PPO_SB3` or `ExpertAgent`
+ - *Additional system information*
+
+## Bug description
+<!--A clear and concise description of what the bug is.-->
+
+## How to reproduce
+<!--Explain in detail how to reproduce your issue. The easier it will be for us to
+reproduce it, the faster we will be able to work on this.-->
+
+### Command line (if any)
+<!--Ideally, if we execute the following command, the bug will directly be reproduced. 
+Here put the command line we have to execute-->
+```bash
+# command line used if any 
+```
+
+### Code snippet (if any)
+<!--Expose the python code you want us to test-->
+```python
+import grid2op
+
+... # Some code 
+```
+
+## Current output
+<!--Describe the output you have-->
+```
+The output of the code snippet above
+```
+
+## Expected output
+<!--Describe the output you desire-->
+```
+The expected output and/or expected behavior description
+```

.github/ISSUE_TEMPLATE/documentation.md

@@ -0,0 +1,27 @@
+---
+name: Documentation
+about: Anything related to the documentation
+title: ''
+labels: documentation
+assignees: ''
+
+---
+
+## Documentation issue description
+<!--A description of what the problem/suggestion is.-->
+
+## Suggested modifications
+<!--Be as concise and clear as possible.
+Ideally we could directly copy paste this code in grid2op documentation / notebook etc.
+-->
+
+```
+Please note:
+
+Documentation issues are low priority. 
+Please provide your suggested modifications to increase processing speed.
+Thanks for your understanding.
+```
+
+## Additional context
+<!--Add any other context here.-->

.github/ISSUE_TEMPLATE/external_contrib.md

@@ -0,0 +1,47 @@
+---
+name: External contribution
+about: Add an external contribution that many people will be able to use
+title: ''
+labels: external_contrib
+assignees: ''
+
+---
+
+## Algorithm implementation
+<!--Provide us with the code where your model is implemented-->
+
+Ex: https://github.com/blablabla/myawesomemodel.git
+
+Check your model follows the template code for l2rpn-baselines:
+
+- [ ] it is freely available on github / gitlab
+- [ ] it exports a function call exactly `evaluate` 
+- [ ] it exports a class named like your github / gitlab (for example, if it is    
+  hosted at https://github.com/YourName/MyAwesomeModel) then you need to be able to do something like `from MyAwesomeModel import MyAwesomeModel`). Once 
+  included into l2rpn-baselines, it will be imported with `from l2rpn_baselines.MyAwesomeModel import MyAwesomeModel`
+
+## License
+<!-- Detail here the license that is applicable to your code. It will also be written in the documentation. -->
+
+ex. I adopted for the MPL v2.0 license
+
+## Short description
+<!--This description will appear on the documentation-->
+
+
+## Required packages (optional)
+<!-- Write here the required packages need for your baselines.
+
+If provided, it will be used to allow the use of your baseline easily by installing with
+`pip install l2rpn-baselines[MyAwesomeModel]`
+(eg. `pip install l2rpn-baselines[PPO_SB3]`)
+-->
+
+- grid2op version xx.yy.zz
+- numpy version xx.yy.zz
+- etc.
+
+## Extra references (optional)
+<!--You can link acamemic papers, experiment results with your contribution  -->
+<!--You can also say on which environment your model has been tested and / or
+on which it does not work. -->

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,51 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: enhancement
+assignees: ''
+
+---
+
+## Is your feature request related to a problem? Please describe.
+<!--A clear and concise description of what the problem is.-->
+<!--Ex. I'm always frustrated when [...]-->
+<!-- Do not forget to include code you are currently running that is frustrating-->
+
+Ex. This is what i do:
+
+```python
+import l2rpn_baselines
+...
+
+```
+
+## Is your feature request related to a new RL agorithm? Please describe.
+<!--If you would like to see a new algorithm in this repository let us know and
+describe it as best you can.
+
+Do not hesistate to put links and anything else related that can help
+us implement it
+-->
+
+- [ ] yes
+- [ ] no
+
+## Describe the solution you'd like
+<!--A clear and concise description of what you want to happen.-->
+<!--Don't forget to include the "code of your dream" -->
+
+Ex. This is how i would like it to be done:
+```python
+import l2rpn_baselines
+...
+
+# give an example on how your awesome new feature would behave
+```
+
+## Describe alternatives you've considered
+<!--A clear and concise description of any 
+alternative solutions or features you've considered.-->
+
+## Additional context
+<!--Add any other context about the feature request here.-->

.gitignore

@@ -166,3 +166,30 @@ l2rpn_baselines/LeapNetEncoded/tf_logs/
 l2rpn_baselines/LeapNetEncoded/tf_logs_test/
 l2rpn_baselines/LeapNetEncoded/model_test/
 l2rpn_baselines/SACOld/trained_models/
+l2rpn_baselines/ppo_stablebaselines/logs
+l2rpn_baselines/ppo_stablebaselines/saved_baseline
+l2rpn_baselines/ppo_stablebaselines/saved_model
+test_Eva.py
+test_box_act.py
+test_eva_dn.py
+test_import_pposb.py
+test_make_gym_env.py
+test_multifolderwithcache.py
+l2rpn_baselines/PPO_SB3/saved_model/**
+l2rpn_baselines/PPO_SB3/logs/**
+l2rpn_baselines/PPO_RLLIB/logs/**
+l2rpn_baselines/PPO_RLLIB/saved_model/**
+test_jsonpickle.json
+examples/ppo_stable_baselines/saved_model/**
+examples/ppo_stable_baselines/logs/**
+.vscode/
+logs/
+saved_agents/
+test_train_DuelQSimple.py
+line_act.json
+tensorboard/
+test_sac/
+documentation/
+test_issue_glop.py
+examples/ppo_stable_baselines/preprocess_act.json
+examples/ppo_stable_baselines/preprocess_obs.json

.readthedocs.yml

@@ -1,7 +1,7 @@
 version: 2
 
 python:
-    version: 3.6
+    version: 3.8
     install:
        - method: pip
          path: .

CHANGELOG.rst

@@ -1,15 +1,39 @@
 Change Log
 ===========
+
 [TODO]
 --------
-- stack multiple states in `utils/DeepQAgent`
+- in the "examples" folder, make some examples for possible "submissions"
+  usable in the competition for PPO_SB3 and PPO_RLLIB
+- add a vectorized environment for PPO in stable baselines for example
+  (ie map a MultiEnvironment into the proper stuff)
+- code a baseline example using mazerl
+- code a baseline using deepmind acme
+- code a baseline with a GNN somewhere
+- show an example on how to use some "heuristic" in training / evaluation of trained agents
+- show an example of model based RL agent
+- train sowhere a working baseline (that does better than do nothing)
+- show an example of a baseline that uses a GNN
+
+[0.6.0] - 2022-06-07
+--------------------
+- [BREAKING] name of the file inside the submodule are now lowercase (PEP 8 compliance)
+  Use `from l2rpn_baselines.[BASELINENAME] import [BASELINENAME]` by replacing 
+  `[BASELINENAME]` with ... the baseline name (*eg* `from l2rpn_baselines.DoNothing import DoNothing`)
+- [FIXED] clean the documentation
+- [FIXED] some bugs (especially in the type of actions) for some agents
+- [ADDED] a code example to use stable baselines 3 (see `l2rpn_baselines.PPO_SB3`)
+- [ADDED] a code example to use RLLIB (see `l2rpn_baselines.PPO_RLLIB`)
+- [ADDED] an optimizer (see `l2rpn_baselines.OptimCVXPY`)
+- [ADDED] some issue templates
+- [ADDED] some examples in the "examples" folder
 
 [0.5.1] - 2021-04-09
 ---------------------
 - [FIXED] issue with grid2op version >= 1.2.3 for some baselines
 - [FIXED] `Issue 26 <https://github.com/rte-france/l2rpn-baselines/issues/26>`_ : package can be installed even
   if the requirement for some baselines is not met.
-- [UPDATED] Kaist baselines
+- [UPDATED] `Kaist` baselines
 - [ADDED] The expert agent
 
 [0.5.0] - 2020-08-18

Inspect_scenario_timestep.py

@@ -34,4 +34,4 @@
 #plot observation
 plot_helper = PlotMatplot(env.observation_space)
 fig_obs = plot_helper.plot_obs(new_obs)
-fig_obs.show()
+fig_obs.show()

Makefile

@@ -18,3 +18,5 @@ help:
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %: Makefile
 	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+#cmd windows: > sphinx-build -M html docs _doc_built

docs/conf.py

@@ -22,8 +22,8 @@
 author = 'Benjamin DONNOT'
 
 # The full version, including alpha/beta/rc tags
-release = '0.5.1'
-version = '0.5'
+release = '0.6.0'
+version = '0.6'
 
 # -- General configuration ---------------------------------------------------
 
@@ -74,6 +74,6 @@
 html_static_path = ['_static']
 
 def setup(app):
-  app.add_javascript('custom.js')
+  # app.add_javascript('custom.js')
   if app.config.language == 'ja':
         app.config.intersphinx_mapping['py'] = ('https://docs.python.org/ja/3', None)

docs/DeepQSimple.rst → docs/deepqsimple.rst

@@ -1,3 +1,5 @@
+.. currentmodule:: l2rpn_baselines.DeepQSimple
+
 DeepQSimple: A simple implementation of the Deep Q Learning
 ===========================================================
 
@@ -8,6 +10,14 @@ baseline. Don't expect to obtain state of the art method with this simple method
 
 An example to train this model is available in the train function :ref:`Example-deepqsimple`
 
+.. warning::
+    This baseline recodes entire the RL training procedure. You can use it if you
+    want to have a deeper look at Deep Q Learning algorithm and a possible (non 
+    optimized, slow, etc. implementation ).
+
+    For a much better implementation, you can reuse the code of :class:`l2rpn_baselines.PPO_RLLIB` 
+    or the :class:`l2rpn_baselines.PPO_SB3` baseline.
+
 Exported class
 --------------
 You can use this class with:
@@ -23,11 +33,12 @@ You can use this class with:
 Other non exported class
 ------------------------
 These classes need to be imported, if you want to import them with (non exhaustive list):
+
 .. code-block:: python
 
     from l2rpn_baselines.DeepQSimple.DeepQ_NN import DeepQ_NN
 
 
-.. autoclass:: l2rpn_baselines.DeepQSimple.DeepQ_NN.DeepQ_NN
+.. autoclass:: l2rpn_baselines.DeepQSimple.deepQ_NN.DeepQ_NN
     :members:
     :autosummary:

docs/DoubleDuelingDQN.rst → docs/doubleduelingdqn.rst

@@ -1,3 +1,5 @@
+.. currentmodule:: l2rpn_baselines.DoubleDuelingDQN
+
 DoubleDuelingDQN: A example implementation of Double Duelling Deep Q Network
 ============================================================================
 
@@ -8,7 +10,14 @@ This baseline is of type Double Duelling Deep Q Network, as in Duelling Q Networ
 
 It's main purpose is to provide an example of this network type running with Grid2Op. However, don't expect to obtain state of the art results.
 
-
+.. warning::
+    This baseline recodes entire the RL training procedure. You can use it if you
+    want to have a deeper look at Deep Q Learning algorithm and a possible (non 
+    optimized, slow, etc. implementation ).
+
+    For a much better implementation, you can reuse the code of :class:`l2rpn_baselines.PPO_RLLIB` 
+    or the :class:`l2rpn_baselines.PPO_SB3` baseline.
+
 Agent class
 ------------------------
 You can use this class with:
@@ -19,7 +28,7 @@ You can use this class with:
     from l2rpn_baselines.DoubleDuelingDQN import train
     from l2rpn_baselines.DoubleDuelingDQN import evaluate
 
-.. automodule:: l2rpn_baselines.DoubleDuelingDQN.DoubleDuelingDQN
+.. automodule:: l2rpn_baselines.DoubleDuelingDQN
     :members:
     :autosummary:
 
@@ -37,7 +46,7 @@ Training a model requires tweaking many hyperparameters, these can be found in a
     DoubleDuelingDQNConfig.FINAL_EPSILON = 0.001
     DoubleDuelingDQNConfig.DECAY_EPSILON = 10000
 
-.. automodule:: l2rpn_baselines.DoubleDuelingDQN.DoubleDuelingDQNConfig
+.. automodule:: l2rpn_baselines.DoubleDuelingDQN.doubleDuelingDQNConfig.DoubleDuelingDQNConfig
     :members:
     :undoc-members:
 
@@ -48,9 +57,9 @@ You may want to import it manually:
 
 .. code-block:: python
 
-    from l2rpn_baselines.DoubleDuelingDQN.DoubleDuelingDQN_NN import DoubleDuelingDQN_NN
+    from l2rpn_baselines.DoubleDuelingDQN.doubleDuelingDQN_NN import DoubleDuelingDQN_NN
 
 
-.. autoclass:: l2rpn_baselines.DoubleDuelingDQN.DoubleDuelingDQN_NN.DoubleDuelingDQN_NN
+.. autoclass:: l2rpn_baselines.DoubleDuelingDQN.doubleDuelingDQN_NN.DoubleDuelingDQN_NN
     :members:
     :autosummary: