NFNet Training

[greeneggsandyaml]

Hello, thanks for your great work!

I'm impressed at how quickly you implemented NFNet. I saw you posted that you had some successful training runs, and I had a couple of questions. One, would you be able to share the commands you used to train those NFNets? Also, have you done any experiments with Adaptive Gradient Clipping?

Best,
greeneggsandyaml

[rwightman]

I'm testing AGC right now. But takes awhile to make sure it's doing what it's supposed to. It is possible to train with normal gradient clipping right now --clip-grad 1.0. I use a global norm of 1.0 for the NF- Resnet and seemed to be working well for NFNet-F0

[greeneggsandyaml]

Great, thanks for the quick reply! Awesome that you're currently testing ACG -- looking forward to seeing the results. If you need someone to test it at large batch sizes (moderately large, not insanely large), I might be able to help, as I have access to a node of GPUs at the moment.

Are you able to share the command you used for the nf_resnet50? More generally, is there a good way of seeing what command you used to train each of the pretrained models available in the repo? I did not see one, but I figured I would ask in case it's there and I missed it.