AlphaZero implementation with TorchRL

[dtsaras]

Hello, I am working on an implementation of AlphaZero with torchrl and I was hoping to get some hints on how to better utilize the tools provided. Currently, I have MCTS coded from scratch and I insert the experience later in a replay buffer. I wonder how I could utilize the data collector and rollout methods to do this more efficiently. Any suggestions/comments are appreciated.

[vmoens]

Amazing!
We'd be excited to help you implement that!

Some datapoints: We have an open PR to get a working version of MCTS with tensordict #629. As you can see, it isn't young and hasn't progressed much but we want to push for it in the next 6 months. I do think that since tensordict is a good primitive to represent tree structures, it makes so much sense to use it to represent decision trees.

RE your questions of data collector / rollout, I think I'd need a bit more context to be helpful.

If you want to integrate your code as an example this is also something we can talk about.

Happy to discuss this offline if you'd like. For that feel free to reach out on our slack or discord channel :)

[dtsaras]

Thanks for your quick reply I took a quick look over the MCTS code and it looks pretty nice. I am not sure if I missed this bit, but I did not see an action mask to prevent expanding actions that would be illegal. Also, I can't find the discord server. Do you mind sharing the link?

[vmoens]

Here it is: https://discord.gg/cZs26Qq3Dd

[vmoens]

@mjlaali maybe you'd like to join this conversation :)