the problem occurred during training

[XJLXuJiaLi]

Traceback (most recent call last):
File "tools/train.py", line 272, in
trainer.train()
File "tools/train.py", line 163, in train
losses.backward()
File "/home/xjl/.conda/envs/env_torch/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/xjl/.conda/envs/env_torch/lib/python3.6/site-packages/torch/autograd/init.py", line 93, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: cuda runtime error (11) : invalid argument at /pytorch/aten/src/THC/THCGeneral.cpp:383

the problem occurred during training.how to slove it?thank you