Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The addition of 'load_ip_adapter' to the StableDiffusionXLAdapterPipeline #6167

Closed
jquintanilla4 opened this issue Dec 14, 2023 · 2 comments
Closed

Comments

@jquintanilla4
Copy link
Contributor

jquintanilla4 commented Dec 14, 2023

What API design would you like to have changed or added to the library? Why?
It would be great to have IP-adapters work with t2i-adapter as they're usually faster and less heavy on the image generation process than controlnet.

What use case would this enable or better enable? Can you give us a code example?

import torch
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, LCMScheduler
from diffusers.utils import load_image

model_id = 'stabilityai/stable-diffusion-xl-base-1.0'
lcm_lora_id = 'latent-consistency/lcm-lora-sdxl'
t2i_adapter_id = 'TencentARC/t2i-adapter-depth-midas-sdxl-1.0'

adapter = T2IAdapter.from_pretrained(t2i_adapter_id, torch_dtype=torch.float16, variant='fp16')

pipe = StableDiffusionXLAdapterPipeline.from_pretrained(model_id, adapter=adapter, torch_dtype=torch.float16, variant='fp16')
pipe.load_ip_adapter('h94/IP-Adapter', subfolder='sdxl_models', weight_name='ip-adapter_sdxl.safetensors')
pipe.load_lora_weights(lcm_lora_id)
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()

Current error:

AttributeError Traceback (most recent call last)
Cell In[6], line 12
9 adapter = T2IAdapter.from_pretrained(t2i_adapter_id, torch_dtype=torch.float16, variant='fp16').to('cuda')
11 pipe = StableDiffusionXLAdapterPipeline.from_pretrained(model_id, adapter=adapter, torch_dtype=torch.float16, variant='fp16')
---> 12 pipe.load_ip_adapter('h94/IP-Adapter', subfolder='sdxl_models', weight_name='ip-adapter_sdxl.safetensors')
13 pipe.load_ip_adapter('h94/IP-Adapter', subfolder='sdxl_models', weight_name='ip-adapter_sdxl.safetensors')
14 pipe.load_lora_weights(lcm_lora_id)

File ~/anaconda3/envs/diffusers2/lib/python3.9/site-packages/diffusers/configuration_utils.py:138, in ConfigMixin.getattr(self, name)
135 deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False)
136 return self._internal_dict[name]
--> 138 raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")

AttributeError: 'StableDiffusionXLAdapterPipeline' object has no attribute 'load_ip_adapter'

@sayakpaul
Copy link
Member

I think it should be fairly easy to do so. If you check this PR (#5915), it shows you how to add load_ip_adapter() to a pipeline in an unbreaking way. I believe you could do the same for the T2I XL adapter too. Would you be open to create a PR for this?

Additionally, if you could provide some results with this addition, that would be great for us to judge if having this method integrated is worthwhile.

Cc: @yiyixuxu

@jquintanilla4
Copy link
Contributor Author

jquintanilla4 commented Dec 18, 2023

Sorry for the late reply. I was traveling. Yeah i'll create a PR and add the relevant result examples as you suggested. Thanks for your patience

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants