Prodigy + ScheduleFree

[hablaba]

Hello! How might I use this optimizer? https://github.com/LoganBooker/prodigy-plus-schedule-free

Would I use the schedulefree wrapper with it somehow?

[rockerBOO]

Probably need the schedule-free-opt branch. Then add the optimizer module from pip pip install git+http://github.com/LoganBooker/prodigy-plus-schedule-free. Then

--optimizer_type "prodigyplus.ProdigyPlusScheduleFree"

https://github.com/LoganBooker/prodigy-plus-schedule-free/blob/main/prodigyplus/prodigy_plus_schedulefree.py#L47C1-L99C28

And the options can be passed in optimizer_args

--optimizer_args "prodigy_steps=1000"

It should output the optimizer it's using in the logs to confirm.

Edit: Updated pip install

[gesen2egee]

Did it work successfully?

[rockerBOO]

I tested it and needed some fixes. I made #1799 for that. Seems to work at least.

[rockerBOO]

This version also supports split groups, so you can set the LR (LR effectively a multiplier of the dynamic LR) differently for the text encoder(s) and UNet. Also might work better since it's separate of the UNet so you won't have that overtraining issue as much. Will do this split by default and there is no constraints to limit it to be equal.

[hablaba]

Oh awesome, appreciate your work here. Yeah when I had tried it I got that error that it wasn’t supported and then I just had other things I was working on. I do think there’s a ton of potential though using schedule free prodigy

[bghira]

that learning rate doesn't look very "dynamic", it never goes back down?

[kohya-ss]

Supported by #1811.