Scheduler lr_noise questions.

[ayasyrev]

Looking into Scheduler code and got several questions about lr_noise.
Why used torch.Generator instead of python random - we just need float here.
Why on every epoch doing manual_seed instead of one seed at start?
It is noise_std argument but it never used - supposed some functionality?
P.S. by the way - what t (in t_initial, warmup_t just t etc) stands for? See what it same as epoch number...

[rwightman]

@ayasyrev the random number for this needs to be separate from the python and other (global) library random generators and seeded consistently across all distributed workers in distributed training scenarios so that all wokers are using the same learning rate

t is for time as it could be epochs or steps, I have (as yet unralized) plans to transition the schedulers to step based for a variety of reasons, mostly fine grained warmup (per step)

the noise idea had some tweaks/improvements to explore that I didn't end up doing, hence that extra arg floating around

I'd be curious to know if you find any improvements for find the feature useful. It's (as far as I know) unpublished and unique to timm. Please cite if you do build on it and publish any work. Thanks!

[ayasyrev]

Thanks for you answer.
I had no experience in distributed training. I thought as Lr updates at the end of epoch , than Lr will be synced throughout devices. And anyway same seed on different devices got different generators. I got it!
I will try use noise, but can do it with just imagewoof now.

Right now I preparing nbs for timmdocs. Started with scheduler- add document noise, changes and new schedulers from last commits,