Support new scheduler free optimizer

[kohya-ss]

original #1250

[gesen2egee]

In the requirements.txt, schedulefree is missing.

[yuwonlee-prodifi]

accelerate now officially supports schedulefree since 0.30.0 and you can directly call optimizer.train() and optimizer.eval().
huggingface/accelerate#2631
https://github.com/huggingface/accelerate/releases/tag/v0.30.0

[kohya-ss]

accelerate now officially supports schedulefree since 0.30.0 and you can directly call optimizer.train() and optimizer.eval(). huggingface/accelerate#2631 https://github.com/huggingface/accelerate/releases/tag/v0.30.0

Thank you for letting me know! This is really nice. I will check it and update this PR.

[yuwonlee-prodifi]

any update on this?

[67372a]

Hello @kohya-ss ,

I was wondering, is there any reason scheduler-free-opt is not yet merged into main?

From my testing using schedule-free with Flux, it appears to work as expected. It would be great if this could be merged into main so that it can be made more widely available, as schedule-free works well and simplifies the training process.

Thank you.

[kohya-ss]

I was wondering, is there any reason scheduler-free-opt is not yet merged into main?

Because there are so many changes, we have to test all scripts with and without it to make sure that existing training still works. It's good news that it works with FLUX. I'll work on it as soon as I have time, so please wait a little longer.

[67372a]

I was wondering, is there any reason scheduler-free-opt is not yet merged into main?

Because there are so many changes, we have to test all scripts with and without it to make sure that existing training still works. It's good news that it works with FLUX. I'll work on it as soon as I have time, so please wait a little longer.

Hello @kohya-ss ,

I understand and appreciate all the work you are doing. There is definitely interest in schedule free and expanding on it with other schedule free optimizer variants, pending the completion of your implementation and testing.

Thank you.