Understanding the Significance of Loss Change in Conservative Q-Learning Training

[constellation39]

Hello Cryolite,

Is the change in loss during Conservative Q-Learning training of any reference value?
In my attempts at training, the loss in CQL is always increasing. It remains very stable until 1 million training steps, after which it starts to increase linearly.

Thank you.

[constellation39]

I understand that there is an error in the reward function I wrote.