Skip to content

Commit

Permalink
Merge pull request #31 from aivarsoo/simmer_saute_refactor
Browse files Browse the repository at this point in the history
Simmer saute refactor
  • Loading branch information
AntGro authored Nov 14, 2022
2 parents cce61a5 + ba04ece commit f9b7f57
Show file tree
Hide file tree
Showing 56 changed files with 252 additions and 4,864 deletions.
31 changes: 6 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,7 @@ Huawei, Noah's Ark Lab.
- [Bayesian Optimisation with Compositional Optimisers](./CompBO)
- [AntBO: Antibody Design with Combinatorial Bayesian Optimisation](./AntBO)
- Reinforcement Learning Research
- [Sauté RL: Almost Surely Safe RL Using State Augmentation](./SAUTE)
- [SIMMER - Enhancing Safe Exploration Using Safety State Augmentation](./SIMMER)
- [Sauté RL and Simmer RL: Safe Reinforcement Learning Using Safety State Augmentation ](./SIMMER)
- [Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief](./PMDB)

Further instructions are provided in the README files associated to each project.
Expand Down Expand Up @@ -119,27 +118,11 @@ in vitro experimentation.

# Reinforcement Learning Research

## [Sauté RL: Almost Surely Safe RL Using State Augmentation](./SAUTE/)
## [Sauté RL and Simmer RL: Safe Reinforcement Learning Using Safety State Augmentation](./SIMMER)

### Sautéing a safe environment
Codebase associated to: [Sauté RL: Almost Surely Safe RL Using State Augmentation](https://arxiv.org/pdf/2202.06558.pdf) and [Enhancing Safe Exploration Using Safety State Augmentation](https://arxiv.org/pdf/2206.02675.pdf).

Safety state augmentation (sautéing) is done in a straightforward manner. Assume a safe environment is defined in
a class `MySafeEnv`. The sautéed environment is defined using a decorator `saute_env`, which contains all the
required definitions. Custom and overloaded functions can be defined in the class body.

```python
from envs.common.saute_env import saute_env


@saute_env
class MySautedEnv(MySafeEnv):
"""New sauteed class."""
```

Codebase associated to: [Sauté RL: Almost Surely Safe RL Using State Augmentation](https://arxiv.org/pdf/2202.06558.pdf).
.

##### Abstract
##### Abstract for Sauté RL: Almost Surely Safe RL Using State Augmentation (ICML 2022)

Satisfying safety constraints almost surely (or with probability one) can be critical for deployment of Reinforcement
Learning (RL) in real-life applications. For example, plane landing and take-off should ideally occur with probability
Expand All @@ -151,12 +134,9 @@ approach has a plug-and-play nature, i.e., any RL algorithm can be "sauteed". Ad
for policy generalization across safety constraints. We finally show that Saute RL algorithms can outperform their
state-of-the-art counterparts when constraint satisfaction is of high importance.

## [SIMMER](./SIMMER)


Codebase associated to: [Enhancing Safe Exploration Using Safety State Augmentation](https://arxiv.org/pdf/2206.02675.pdf).

##### Abstract
##### Abstract for Effects of Safety State Augmentation on Safe Exploration (NeurIPS 2022)
Safe exploration is a challenging and important problem in model-free reinforcement learning (RL). Often the safety cost
is sparse and unknown, which unavoidably leads to constraint violations -- a phenomenon ideally to be avoided in
safety-critical applications. We tackle this problem by augmenting the state-space with a safety state, which is
Expand All @@ -168,6 +148,7 @@ Safe exploration is a challenging and important problem in model-free reinforcem
that simmering a safe algorithm can improve safety during training for both settings. We further show that Simmer can
stabilize training and improve the performance of safe RL with average constraints.


## [Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief](./PMDB)

Code associdated to: [Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief](https://nips.cc/Conferences/2022/Schedule?showEvent=54842) accepted
Expand Down
140 changes: 0 additions & 140 deletions SAUTE/.gitignore

This file was deleted.

114 changes: 0 additions & 114 deletions SAUTE/README.md

This file was deleted.

Empty file removed SAUTE/common/__init__.py
Empty file.
18 changes: 0 additions & 18 deletions SAUTE/common/argument_parser.py

This file was deleted.

Loading

0 comments on commit f9b7f57

Please sign in to comment.