Evaluating policy on real-world setup

[AntonBock]

Hello,

We have trained a policy that we would like to test on a real-world setup. Does SKRL have any built-in support for this, or do you have any recommended method of doing this?

-Anton

[Toni-SM]

Hi @AntonBock

To test the policy in simulation or real-world you only need to compose the observation/state space with information from the environment (sensors, etc.) and apply the actions taken by the policy to the environment (controllers)...

So, it all depends on the environment you have...
Could you please provide more information about your simulated environment and the real one? Are you planning to use a middleware to control the objective in your environment, for example, ROS?

[AntonBock]

Hi again,

Thanks for the quick response!

We use ROS to control a Franka Panda arm.
Using your ppo_Franka_Cabinet.py example, we have trained a Franka robot in Isaac Gym.

How could we modify the franka_cabinet example to use observations we get from ROS, instead of wrapping Gym and getting the observations from there?

[Toni-SM]

Hi @AntonBock

I think the following questions are relevant for testing in real-world:

• Are you able to get, using ROS all the data required to build the observation space (according to the FrankaCabinet task it is self.cfg["env"]["numObservations"] = 23)? How do you pretend to control the robot: setting the joints directly using a ROS topic or with MoveIt?
• How important is it to return the instantaneous reward at each time step in the real-world test environment? Are you able to compute it using the information from the real world?
• Is your real-world system capable of working (retrieving sensor information and controlling the robot using ROS) at the frequency at which the agent was trained in the simulation (dt: 0.0166 # 1/60 seconds)?

Well... I think creating a separate environment, for testing in real-world, that uses ROS to build the observation space and control the robot, is the best solution... Currently, I am creating a testing environment (in the real world) using ROS... I think we can discuss its implementation here the day after tomorrow...

[AntonBock]

Hi @Toni-SM
Those are some of the same considerations we have made, and we should be able to get all the required information and rewards at the correct frequency.

We look forward to hearing about your ROS environment tomorrow

[Toni-SM]

Hi @AntonBock

Unfortunately, we (in our lab) had problems with the robot controller (iiwa) in the last days...
Then, I created a simulation environment in Isaac SIm to show the idea we have to test some trained policies in the real world using ROS.
The exemplified environment is a 3D reaching task

Here are some files:

• .usd stage for Isaac Sim (compressed): ros_iiwa14.zip
• ROS workspace (src folder): src.zip

sim_joints.mp4

The environment follows the OpenAI Gym interface and has the following spaces:

• observation space (18):
  • 1: normalized episode progress
  • 3: sphere position
  • 7: robot's joint positions
  • 7: robot's joint velocities
• action space (7)
  • delta robot's joint positions (-1 to 1 degrees)

The environment (src/rl/environment.py) has the following ROS components:

• a publisher to control the robot using joint position: "/iiwa/joint_command" (sensor_msgs/JointState)
• a subscriber to get the joint positions and velocities: "/iiwa/joint_states" (sensor_msgs/JointState)
• a TransformListener to get the sphere and the robot's end-effector position
• a service to reset (teleport) the sphere

A policy (a bad trained policy: src/rl/132500_policy.pt) controls the robot at 20Hz...
Learning to control the robot using joints, with our custom environment in Isaac Gym, is not giving us good results at the moment... that is why the checkpoint included is not able to execute the task successfully

---

Actually, I trained the same task, but this time, controlling the robot in the Cartesian space...
The results are better than controlling using joints... but until we solve the problem of the real robot controller we will not be able to test it with the real robot... there is always the simulation in Isaac Sim, for example 😅

Things we want to test: use MoveIt to control the robot... although we are not sure if the time constraints (control at 20Hz) can be met with MoveIt

gym_cartesian.mp4

[famora2]

Hi @AntonBock

Unfortunately, we (in our lab) had problems with the robot controller (iiwa) in the last days... Then, I created a simulation environment in Isaac SIm to show the idea we have to test some trained policies in the real world using ROS. The exemplified environment is a 3D reaching task

Here are some files:

* .usd stage for Isaac Sim (compressed): [ros_iiwa14.zip](https://github.com/Toni-SM/skrl/files/8629878/ros_iiwa14.zip)

* ROS workspace (src folder): [src.zip](https://github.com/Toni-SM/skrl/files/8629879/src.zip)

sim_joints.mp4

The environment follows the OpenAI Gym interface and has the following spaces:

* observation space (18):
  
  * 1: normalized episode progress
  * 3: sphere position
  * 7: robot's joint positions
  * 7: robot's joint velocities

* action space (7)
  
  * delta robot's joint positions (-1 to 1 degrees)

The environment (src/rl/environment.py) has the following ROS components:

* a publisher to control the robot using joint position: `"/iiwa/joint_command" (sensor_msgs/JointState)`

* a subscriber to get the joint positions and velocities: `"/iiwa/joint_states" (sensor_msgs/JointState)`

* a TransformListener to get the sphere and the robot's end-effector position

* a service to reset (teleport) the sphere

A policy (a bad trained policy: src/rl/132500_policy.pt) controls the robot at 20Hz... Learning to control the robot using joints, with our custom environment in Isaac Gym, is not giving us good results at the moment... that is why the checkpoint included is not able to execute the task successfully

Actually, I trained the same task, but this time, controlling the robot in the Cartesian space... The results are better than controlling using joints... but until we solve the problem of the real robot controller we will not be able to test it with the real robot... there is always the simulation in Isaac Sim, for example sweat_smile

Things we want to test: use MoveIt to control the robot... although we are not sure if the time constraints (control at 20Hz) can be met with MoveIt
gym_cartesian.mp4

Hi, I am testing the environment.py script in /src/rl. Is there a reason that reset(self) function in there is executed every step? Is this also in your case?

[Toni-SM]

Hi @famora2

I'll run some tests later and let you know

[AntonBock]

Hi @Toni-SM

Thank you! We will look at it and see if it can help us with our implementation. I will get back to you, if we get some interesting results.