Introductory Questions

[morrow0233]

Hi, OpenSpiel team. I am interested in exploring the potential of OpenSpiel as a tool for partially-automated playtesting of new game designs. I am relatively new to AI, but I have been a professional C++ programmer for about 20 years. I am running OpenSpiel on a Ryzen Threadripper system running Ubuntu via the Windows Subsystem for Linux. I managed to install the software and train up a competent tic-tac-toe agent, so now I have questions. :-)

1. I am currently attempting to train a chess agent via the libtorch AlphaZero implementation. So far, it's not going well. Could you please suggest parameters to use and tell me how long you'd expect it to take on my computer to train to a roughly 1600 Elo level of play? I feel like I'm doing something wrong, but it could also be that my training speed expectations are way out of whack.

2. I hope to start implementing some new games soon. The OpenSpiel documentation says that checkers is currently missing. Is that because no one has gotten around to it yet or because there are issues with checkers that make it deceptively difficult to implement?

3. I hope to eventually implement several multiplayer games that include random elements and have complexity levels ranging from fairly simple to quite complex (potentially more complex than chess). What algorithms should I be leaning toward for such games, and what parameter ranges should I use?

4. Is my 10k-foot goal feasible? Is the technology to the point where I could use OpenSpiel to create agents for new games at a) a basic human competence level, b) in an acceptable timescale, c) on a single high-end workstation?

Thanks for any help you can provide.

[lanctot]

Hi @morrow0233,

Apologies, I had somehow completely missed this!

I'm a bit surprised by the struggle to reach 1600 ELO in Chess. How long did you wait? How many games were played? Did you use convolutional res nets?

Checkers: there's no deeper reason other than nobody has done it yet -- would be a great contribution!

I would say if your games are two-player perfect information, AlphaZero is indeed the best place to start. The parameter ranges would depend on the game(s) and could take some experimenting with, but we could help with suggestions.

I think if you want higher-performing AI in a reasonable amount of time, you will need a distributed setup. The OpenSpiel algorithm implementations were not designed for scale. There are distributed training frameworks designed for scale that support OpenSpiel games, such as Mava and Ray/RLlib. I'm not sure either of them have AlphaZero implementations, though, as they are not designed around the games use case. So you might want to use e.g. Launchpad to make your own (although check to see if one already exists). It would be quite a great contribution, of course. There are also some MuZero implementations as well such as muzero-general that support OpenSpiel games via an adaptor (and also Ray).

Hope this helps!

[morrow0233]

Thanks for the reply. My first chess training was:

./build/examples/alpha_zero_torch_example --game=chess --actors=6 --checkpoint_freq=10 --max_simulations=100 --path=/home/jmorrow/chess_agent --train_batch_size=256

About three days later the graphs looked pretty flat, so I tried something that I thought would be faster:

./build/examples/alpha_zero_torch_example --game=chess --checkpoint_freq=1 --max_simulations=20 --path=/home/jmorrow/chess_agent_2

About six days after that the graphs were looking pretty much like a random walk, so a few days ago I decided to start a new run just using default parameters:

./build/examples/alpha_zero_torch_example --game=chess --checkpoint_freq=1 --path=/home/jmorrow/chess_agent_3

Any suggestions would be greatly appreciated.

[lanctot]

Hi @morrow0233, 100 simultations is really too low for Chess. We used 800 in the original paper and mini-batch sizes of 4096. If you take a look at the AlphaZero paper, Figure 1: we were able to get >= 2000 Elo after 25k - 50k training steps with this configuration. However, it was using a highly-optimized distributed framework. What size network are you using?

There are also a few other small differences that could affect performance (ego-centric observations, and stacked observations about the last K moves), but it's unclear by how much. Nonetheless I would be surprised if you would not be able to get at least above 1600 after 50k training steps using those values and a big enough network.

Perhaps the LeelaZero project members might have some helpful advice here too, as they've reproduced a high-performance AlphaZero player for chess.

[morrow0233]

Please forgive my basic-level questions. I have very little experience with AI, so my response to "what size network are you using" is "I have no idea". Also, when you say 25k-50k training steps, about how long do you think that would take on my setup?

The training that I'm currently running (the last command in my previous post) reached checkpoint 1 after about 2 days. Is that normal?

Also, for training, is it OK to use the alpha_zero_torch_example or should I be attempting to write some kind of trainer of my own?

Sorry for my increasingly-basic questions. I'm just trying to get a feel for what kind of training times are normal and which are not. Thanks.

[morrow0233]

Hi again. My training has been running for about two weeks now, and it's been consistently using about 40-50% of my processors since then. I'm getting graphs that look like this:

Is this normal? I'm hoping that seeing these graphs will allow someone with more experience to point me in the right direction.

Also, I'm running a libtorch-based AZ training, and when I try to play a game against the agent with this command:

python3 open_spiel/python/examples/mcts.py --game=chess --player1=az --az_path=/home/jmorrow/chess_agent_3/checkpoint-9.pt --player2=human

I get the error:

Traceback (most recent call last):
File "open_spiel/python/examples/mcts.py", line 223, in
app.run(main)
File "/home/jmorrow/.local/lib/python3.8/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/home/jmorrow/.local/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "open_spiel/python/examples/mcts.py", line 197, in main
_init_bot(FLAGS.player1, game, 0),
File "open_spiel/python/examples/mcts.py", line 96, in _init_bot
model = az_model.Model.from_checkpoint(FLAGS.az_path)
File "/home/jmorrow/open_spiel/open_spiel/python/algorithms/alpha_zero/model.py", line 188, in from_checkpoint
model = cls.from_graph(checkpoint, path)
File "/home/jmorrow/open_spiel/open_spiel/python/algorithms/alpha_zero/model.py", line 201, in from_graph
saver = tf.train.import_meta_graph(metagraph)
File "/home/jmorrow/.local/lib/python3.8/site-packages/tensorflow/python/training/saver.py", line 1565, in import_meta_graph
return _import_meta_graph_with_return_elements(meta_graph_or_file,
File "/home/jmorrow/.local/lib/python3.8/site-packages/tensorflow/python/training/saver.py", line 1581, in _import_meta_graph_with_return_elements
meta_graph_def = meta_graph.read_meta_graph_file(meta_graph_or_file)
File "/home/jmorrow/.local/lib/python3.8/site-packages/tensorflow/python/framework/meta_graph.py", line 646, in read_meta_graph_file
text_format.Merge(file_content.decode("utf-8"), meta_graph_def)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 70: invalid start byte

Does mcts.py assume the tensorflow implementation?

Thanks again!

[nsvemuri]

How does one get these graphs mentioned in post on April 2022?
I am running alpha zero training for a game. I don't see any signals around progress.
Can someone point me how to do this?