Need some advice for GAIL implement

[nicholas0717]

Hi @Toni-SM

I'm trying to use skrl library to implement GAIL to perform the task FactoryTaskNutBoltPick in Isaac Gym. But it works badly. I need some advice.

Firstly, my idea is to use PPO as the generator for GAIL , so define the GAIL class inheriting the PPO class. And I define the model discriminator in GAIL and set related configurations. The memory_exp is the expert's demonstration which size is 300000.

The policy and value model is the same as PPO and discriminator model is define as follow:

So, the _update() in GAIL will include two update function. First is '_update_disc' which is used to update the discriminator. Its inputs are sampled states and actions from policy and demonstrations. After _update_disc, calculate disc_reward from discriminator and update the PPO super()._update(timestep, timesteps) with disc_reward (not using policy's initial rewards). And the process of super()._update(timestep, timesteps) is basically consistent with the process of _update() in PPO. Use disc_reward to calculate the GAE, sample and compute the loss of policy, value and entropy(same as PPO).

I‘m not sure if the GAIL I implemented is correct. But I used it to train FactoryTaskNutBoltPick and performed badly. Maybe related parameters are wrong? I need your advice.

[nicholas0717]

By the way, I want to know the differences between act and compute in Shared model. I'm puzzled about their using.

[Toni-SM]

Hi @nicholas0717

I hope this first iteration helps in some way.
Don't be afraid to reply and continue the discussion...

---

I would like to recommend the following:

1. Don't inherit from another agent, instead, duplicate the code of some agent in a new folder and start from there. This will allow you to have control over all the agent code and not have to navigate between base classes to inspect any implementation.
2. It would be better to use AMP as a starting agent. This agent is also based on PPO and makes use of a discriminator to update the reward.
3. Experiment with a simpler and faster environment such as Cartpole, BallBalance, or Ant

If it's ok with you, and you can make the changes (particularly points 1 and 2) it would be great if you could share the code to try to help with it.

---

Regarding the distinction between act and compute in shared (and not shared) models:

The compute method is responsible for calling the artificial neural network (defined by the user) to process the input data (observations and optionally actions). The artificial neural network, generally speaking, is ultimately a deterministic function (for the same input values it returns the same output values).

The act method, on the other hand, contains the implementation of the model (deterministic, Gaussian, categorical, etc.) which is nothing more than a post-processing stage of the neural network output to generate the desired output (deterministic/stochastic, discrete/continuous). It calls the compute method to get the output of the network as a first step.

[nicholas0717]

Hi @Toni-SM

Thank you for you reply!

After I changed the discriminator's Linear() units, it seemed to work, but not perfect. Maybe my related parameters still need to be tuned.

I want to get some of your experience when you need to tune parameters in different tasks. They played an important role in PPO, GAIL and AMP, especially like rollouts, learning_epochs, mini_batches and number of network's hidden units.

Thank you so much. You help me a lot!

[Toni-SM]

Hi @nicholas0717

Good to hear it works.

Yes, some algorithms require hyperparameter fine-tuning to work better.
There is currently no information related to hyperparameter tuning in the library documentation.
However, integration with a hyperparameter optimizer (such as optuna) is on the roadmap.
It should be easy to implement, but unfortunately, I have not had the time to do so.
It would be nice if you can try it and share your experiences...

Hope to do it at some free time :)