[Sigmas] Keep sigmas on CPU

[patrickvonplaten]

What does this PR do?

After a second round of testing, it seems like keeping the sigmas always on CPU is actually not a bad idea because there is no real slow-down for the "default" mode while the speed-up for torch.compile is significantly higher when leaving sigmas on CPU.
cc @sayakpaul

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

thanks!

[sayakpaul]

Thank you!

[sunhs]

I'm curious about how it works. self.sigmas.to("cpu") does return a Tensor on cpu but it's not used, whereas self. sigmas is still on gpu. I must have misunderstood something. Could you show me some references to make it clear?

[sayakpaul]

to is inplace.

[sunhs]

Can this demonstrate my uncertainty?

[sayakpaul]

Cc: @patrickvonplaten here. This still doesn't affect the results in the PyTorch blog post as it was done from a separate branch with another way of doing the device placement.

[yiyixuxu]

ahh does this means our pipeline.to() behave differently from the pytorch to()? - ours would modify in place and return a copy

[sunhs]

ahh does this means our pipeline.to() behave differently from the pytorch to()? - ours would modify in place and return a copy

This should not be a problem, since nn.Module.to indeed is in place while Tensor.to isn't.

[patrickvonplaten]

Nice catch @sunhs - correcting it here: #6708

It shouldn't change anything in practice since self.sigmas are always kept on CPU anyways for Euler and DPM. I'll keep an eye on the slow tests though (cc @DN6)