Unexpected behavior of eval_every argument for ppo agent

[david-lindner]

After changing the way episodes are tracked for the ppo agent (#58), the eval_every argument does not longer work as expected. If I run for example:

python main.py -E 10000 -V 50 -EE 10 -C -L runs/way/ppo-cnn/cheat way ppo-cnn -e 10 -r 200 -l 1e-3

I get no evaluations at all. The problem is two-fold:

1. -EE is being compared to history["episodes"] which for the ppo agent now measures episodes as determined by the -E argument times number of rollouts. In this case, I run 10000 'episodes' but for every episode history["episodes"] is incremented by 200. This means to get the desired behavior, I have to set -EE 2000. This is very unexpected behavior and inconsistent with the other agents.

2. Even when setting -EE 2000, I still get no evaluations. This seems to be caused by the check that is performed to determine whether we should evaluate:

if history["episode"] % args.eval_every == args.eval_every - 1:
        eval_next = True

This does not work anymore because history["episode"] does no longer increase in increments of 1 between these checks. So I will have the following:

...
history["episode"] = 1800     history["episode"] % args.eval_every = 1800
history["episode"] = 2000     history["episode"] % args.eval_every = 0
history["episode"] = 2200     history["episode"] % args.eval_every = 200
...

and eval_next will always be False. I now changed the check to the following as a quick fix:

if history["episode"] % args.eval_every == 0 and history["episode"] > 0:
        eval_next = True

This works for the experiments, I want to run. However, I am not sure if this is a good solution in general and we should discuss what to do here.