Retraining the policy on real-world setup

[famora2]

Hello,

I have seen your script "environment.py" in the below discussion which gives a rough baseline for evaluating the trained policy in the real world setup. I would like to ask whether there is a way to extend this script so that the trained policy can be retrained in the real world setup in order to minimize the existing sim2real gap.

Discussed in #10

^{Originally posted by AntonBock May 2, 2022}
Hello,

We have trained a policy that we would like to test on a real-world setup. Does SKRL have any built-in support for this, or do you have any recommended method of doing this?

-Anton

[Toni-SM]

Hi @famora2

It is possible to use any environment to retrain the policy in the real world. The only constraint is that the environment follows one of the supported environment interfaces: OpenAI Gym, DeepMind or Isaac Gym. In addition, it is recommended that the real-world environment should, as far as possible, produce the same state/observation and reward.

Also, it's a good idea to store all the involved models during the training in simulation (not only the policy). Then, you can load the checkpoints and continue the training in the real world.

Let me know any comments and we will continue the discussion :)

[famora2]

The environment is built in Isaac Gym. Using your "realworld_joints_eval.py" and "environment.py", the trained agent can be evaluated. However, when I change the line 77 of "realworld_joints_eval.py", from trainer.eval() to trainer.train(), I get following error:

Traceback (most recent call last):
  File "realworld_eval.py", line 77, in <module>
    trainer.train()
  File "/home/user/skrl/skrl/trainers/torch/sequential.py", line 54, in train
    self.single_agent_train()
  File "/home/user/skrl/skrl/trainers/torch/base.py", line 226, in single_agent_train
    self.agents.post_interaction(timestep=timestep, timesteps=self.timesteps)
  File "/home/user/skrl/skrl/agents/torch/ppo/ppo.py", line 254, in post_interaction
    self._update(timestep, timesteps)
  File "/home/user/skrl/skrl/agents/torch/ppo/ppo.py", line 310, in _update
    last_values, _, _ = self.value.act(states=self._current_next_states.float() \
AttributeError: 'NoneType' object has no attribute 'act'

Did I miss something?

[Toni-SM]

Hi @famora2

You are getting the error because, in this script, only the policy (under "policy" key) is instantiated.

    # Instantiate the agent's models (function approximators).
    # PPO requires 2 models, visit its documentation for more details
    # https://skrl.readthedocs.io/en/latest/modules/skrl.agents.ppo.html#spaces-and-models
    models_ppo = {"policy": Policy(env.observation_space, env.action_space, device, clip_actions=True)}

    # load checkpoint
    models_ppo["policy"].load("./132500_policy.pt")

It is okay for evaluation because only the trained policy is required. However, for training, all the models are required ("policy" and "value"). That's why I mentioned that point in the previous thread:

Also, it's a good idea to store all the involved models during the training in simulation (not only the policy). Then, you can load the checkpoints and continue the training in the real world.

You can check the PPO models here: https://skrl.readthedocs.io/en/latest/modules/skrl.agents.ppo.html#spaces-and-models
Also, you can visit https://skrl.readthedocs.io/en/latest/intro/examples.html to see some examples