Skip to content

intent of LOSS_SCALE #453

Answered by Tom94
hturki asked this question in Q&A
Discussion options

You must be logged in to vote

Hi there, yes, it is a magic number that helps make better use of fp16 numbers during backpropagation through the neural network. There's otherwise no effect on training (the loss is multiplied by LOSS_SCALE before backpropagation, and the gradients are afterwards divided by LOSS_SCALE to compensate).

See this documentation for details.

Cheers!

Replies: 1 comment

Comment options

You must be logged in to vote
0 replies
Answer selected by hturki
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
2 participants