More optimizer updates, add MARS, LaProp, add Adopt fix and more#2347

Merged

rwightman merged 10 commits intomainfrom opt_mars_more

Nov 26, 2024

+714-180

Commits on Nov 26, 2024

Update Adopt to include clipping for stability, separate wd so no param decay if update not taken on first step
rwightman
committed
Add MARS and LaProp impl, simplified from originals
rwightman
committed
Replace radam & nadam impl with torch.optim ver, rename legacy adamw, nadam, radam impl in timm. Update optim factory & tests.
rwightman
committed
Update Adan with newer impl (from original source) that includes multi-tensor fn
rwightman
committed
Reduce tolerance on model inference 'owl' test, pillow output varies a lot, was failing locally
rwightman
committed
Add guard around 'somewhat' newer torch RAdam / NAdam imports
rwightman
committed
Update optimizers.mdx
rwightman
committed
Update tests, need handling for radamw with older PyTorch, need to back-off basic test LR in mars?
rwightman
committed
Remove rogue import, thanks IDE :/
rwightman
committed
Keep basic optim test LR range closer to before w/ updated code
rwightman
committed