Add from single file to StableDiffusionUpscalePipeline and StableDiffusionLatentUpscalePipeline

[DN6]

What does this PR do?

Add from single file mixin to StableDiffusionUpscalePipeline and StableDiffusionLatentUpscalePipeline

Fixes # (issue)
#5150

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[DN6]

@patrickvonplaten Added the mixins to the upscale pipelines. Little confused on how to test this.

Is the expectation that the user is supposed to provide the appropriate yaml config when using from_single_file? Is this the case even if a single file version of the model is present on the hub? Or are we expected to fetch the config somehow if it is present on the hub?

e.g
https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler/blob/main/x4-upscaler-ema.safetensors

[patrickvonplaten]

The following should work:

wget https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler/resolve/main/x4-upscaler-ema.safetensors

And then doing:

from diffusers import StableDiffusionUpscalePipeline

pipeline = StableDiffusionUpscalePipeline.from_single_file("./x4-upscaler-ema.safetensors")

without giving any config.

No need to test the latent upscaler, I don't think there is even an official single file for this

[DN6]

The following should work:

wget https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler/resolve/main/x4-upscaler-ema.safetensors

And then doing:

from diffusers import StableDiffusionUpscalePipeline

pipeline = StableDiffusionUpscalePipeline.from_single_file("./x4-upscaler-ema.safetensors")

without giving any config.

No need to test the latent upscaler, I don't think there is even an official single file for this

So that snippet will fail since there is a mismatch in the layer shapes in the UNet.

ValueError: Trying to set a tensor of shape torch.Size([1024, 256]) in "weight" (which has shape torch.Size([1280, 320])), this look incorrect.

Without a config from_single_file creates a default Unet config that doesn't match the one in the checkpoint. I think it would apply to other components too.

[patrickvonplaten]

The following should work:

wget https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler/resolve/main/x4-upscaler-ema.safetensors

And then doing:

from diffusers import StableDiffusionUpscalePipeline

pipeline = StableDiffusionUpscalePipeline.from_single_file("./x4-upscaler-ema.safetensors")

without giving any config.
No need to test the latent upscaler, I don't think there is even an official single file for this

So that snippet will fail since there is a mismatch in the layer shapes in the UNet.

ValueError: Trying to set a tensor of shape torch.Size([1024, 256]) in "weight" (which has shape torch.Size([1280, 320])), this look incorrect.

Without a config from_single_file creates a default Unet config that doesn't match the one in the checkpoint. I think it would apply to other components too.

Then we should add a new if statement that automatically detects when the input model is a upsampling model just from the checkpoint and then set the correct configs

[DN6]

@patrickvonplaten this is ready for another review. There's a failing ShapE test that's unrelated. Rest of the CI is green.

[patrickvonplaten]

Very cool! Nice job

[xiaoyong-z]

ldm unet model can not load after this pr, exception.
File "/root/miniconda3/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 450, in convert_ldm_unet_checkpoint
new_checkpoint["class_embedding.weight"] = unet_state_dict["label_emb.weight"]
KeyError: 'label_emb.weight'

(in 0.21.0, ldm model can be loaded correctly)

[patrickvonplaten]

@xiaoyong-z could you open a new PR? cc @DN6 here as well for viz

[xiaoyong-z]

#5917 @patrickvonplaten