StableDiffusionXLInpaintPipeline.from_single_file for refiner returns back Img2ImgPipeline

[darshats]

Describe the bug

When using this call:
StableDiffusionXLInpaintPipeline.from_single_file(/sd_xl_refiner_1.0.safetensors), we get an instance of Img2ImgPipeline rather than Inpainting pipeline.
This seems related to 4186, but in this case Img2ImgPipeline doesnt throw an error when mask_image parameter is passed, it just silently ignores it.

Reproduction

from diffusers import StableDiffusionXLInpaintPipeline
from diffusers.utils import load_image

refiner = StableDiffusionXLInpaintPipeline.from_single_file(
"./sd_xl_refiner_1.0.safetensors",
torch_dtype=torch.float16,
use_safetensors=True,
variant="fp16",
)
refiner.to("cuda")

Logs

No response

System Info

Diffusers 0.23

Who can help?

No response

[sayakpaul]

Could you provide a fuller reproducible code snippet for us to understand the issue better?

Cc: @DN6

[darshats]

The intent is to get a refiner for inpainting like described here

However if I run the code below, StableDiffusionXLInpaintPipeline.from_single_file call returns an Img2Img pipeline. The AutoPipelineForInpainting.from_pipe returns an Inpainting pipeline. The single file call looks like a bug?

 pipe = StableDiffusionXLInpaintPipeline.from_single_file(
    "[https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors"](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors%22),
    torch_dtype=torch.float16,
    variant="fp16",
    use_safetensors=True
    )
pipe.to("cuda")

### this returns in Img2Img pipeline that ignores the mask parameter
refiner = StableDiffusionXLInpaintPipeline.from_single_file(
    "[https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0/blob/main/sd_xl_refiner_1.0_0.9vae.safetensors"](https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0/blob/main/sd_xl_refiner_1.0_0.9vae.safetensors%22),
    text_encoder_2=pipe.text_encoder_2,
    vae=pipe.vae,
    torch_dtype=torch.float16,
    variant="fp16",
    use_safetensors=True
)
refiner.to('cuda')

## this returns an inpainting pipeline
refiner2 = AutoPipelineForInpainting.from_pipe(
    refiner,
    text_encoder_2=pipe.text_encoder_2,
    torch_dtype=torch.float16,
    variant="fp16",
    use_safetensors=True
    )
refiner2.to('cuda')

[DN6]

@yiyixuxu Could you take a look here please.

[patrickvonplaten]

The issue is quite easy to reproduce:

from diffusers import StableDiffusionXLInpaintPipeline
import torch

pipe = StableDiffusionXLInpaintPipeline.from_single_file(
    "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors",
    torch_dtype=torch.float16,
    variant="fp16",
    use_safetensors=True
)
refiner = StableDiffusionXLInpaintPipeline.from_single_file(
    "https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0/blob/main/sd_xl_refiner_1.0_0.9vae.safetensors",
    text_encoder_2=pipe.text_encoder_2,
    vae=pipe.vae,
    torch_dtype=torch.float16,
    variant="fp16",
    use_safetensors=True
)


print("Refiner", refiner.__class__.__name__)   # prints incorrect class

@DN6 it would be nice if you can take a look here since we want to refactor the from_single_file function anyways

[darshats]

There is another similar issue - on diffusers 0.22.3
The call
pipe = StableDiffusionXLInpaintPipeline.from_single_file( "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors", torch_dtype=torch.float16, variant="fp16", use_safetensors=True )
returns back an object of type inpaint pipeline, but the unet config only has 4 channels.

If I use:
pipe = AutoPipelineForInpainting.from_pretrained( "diffusers/stable-diffusion-xl-1.0-inpainting-0.1", torch_dtype=torch.float16, variant="fp16" )
it correctly returns back in inpaint pipeline instance, with unet having 9 channels.

Some docs have the first way of calling (in conjunction with refiner) and some docs have the second (autopipeline) way of calling. But the second one is correct. First one config looks suspect, and its not really inpainting due to unet config only 4 channels.

[DN6]

@darshats should be fixed with #6147