8-bit C-Optim optimizers

[odusseys]

Feature request

Implementing C-optim (https://github.com/kyleliang919/C-Optim) for the supported optimizers, especially Adam and AdamW, optionally or as separate optimizers.

Motivation

C-optim is a new development in optimizer research, which very slightly tweaks any optimizer update rule with significant convergence improvements, both theoretical and empirical.

Your contribution

unfortunately, very unfamiliar with C development so will have a hard time implementing this myself