Does it Work on FP16 with latest nvidia amp？

[578123043]

No description provided.

[578123043]

In FP16 , output is nan , and in mid layer(maybe 17th or 18th) in the bert , I found that the attention is -3000，and softmax result is nan

[mandarjoshi90]

I will need more information to help you debug. Could you please include what command you're running?

[578123043]

YES ， I use {https://dl.fbaipublicfiles.com/fairseq/models/spanbert_hf.tar.gz} to runing Huggingface`s run_squad , it Failed
Does it work with the latest amp？

[mandarjoshi90]

We haven't tested this with the new HF code. ICYMI, there's a run_squad.py in this repo. I'd recommend using that.

[marcos0318]

When I tried to finetune spanbert large on my own task using fp16 with the "amp" module from the apex, I meet a similar error.

I am using the code from huggingface and tried both using "Spanbert/spanbert-large-cased" and the model binaries provided in this repo. Both gives the identical result, which kept telling me gradient overflow and rescaling the loss.

Interestingly, when I tried to replace spanbert-large with BERT base/large, and spanbert-base model, these models work perfectly and achieved expected results.

Also, the spanbert-large work very well when I turn off the fp16 training.

Here I found this guy cannot run fp16 with spanbert on another task.

In a conclusion, I guess that the spanbert-large model may work with the new Nvidia amp

[YuxianMeng]

Met same issue here, is there any solution yet?