I have tried training multiple times, but NaN keeps occurring consistently.

[rudxo9251]

I continuously train multiple times, but NaN occurs. I haven't changed the training parameters, but NaN appears in the loss during training. What could be the problem?

[rudxo9251]

i solve

[xuanyuzhang21]

I met the same problem. Can you tell me how to solve this issue? Thanks a lot.

[Lanjiong-Li]

I met the same problem. Can you tell me how to solve this issue? Thanks a lot.

set the lr to 1e-6 may help.

[rudxo9251]

change adam_epsilon to 1e-4.

[all1new]

I met the same problem. Can you tell me how to solve this issue? Thanks a lot.

set the lr to 1e-6 may help.

Is there any better training effect? ​​The training results here are not as good as the original weights.

[awais-nayyar]

I want to train this code on my own dataset. What is the minimum GPU and Ram requirements???
can anybody help to guide me?

[TnoobT]

I want to train this code on my own dataset. What is the minimum GPU and Ram requirements??? can anybody help to guide me?

58G gpu