.load_ip_adapter in StableDiffusionXLAdapterPipeline #6246

jquintanilla4 · 2023-12-20T10:13:39Z

What does this PR do?

This PR is referenced in the following issue #6167.
I have purposely laser focused on the StableDiffusionXLAdapterPipeline. I did not touch the regular Stable Diffusion Adapter Pipeline because I didn't wanna go poking and changing stuff that was not previously discussed in the issue, as tempting as it might be 😄.

In short, I added the .load_ip_adapter to the StableDiffusionXLAdapterPipeline. It works great using the regular SDXL models with 30+ steps. It works okay with SDXL Turbo at 6-7 steps. It works not so well with LCM-Lora at 4 steps.

The following is code and images from a temporary notebook to test the results.

LCM-LoRA

import torch
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, LCMScheduler
from diffusers.utils import load_image

model_id = 'stabilityai/stable-diffusion-xl-base-1.0'
lcm_lora_id = 'latent-consistency/lcm-lora-sdxl'
t2i_adapter_id = 'TencentARC/t2i-adapter-depth-midas-sdxl-1.0'

adapter = T2IAdapter.from_pretrained(t2i_adapter_id, torch_dtype=torch.float16, variant='fp16')

pipe = StableDiffusionXLAdapterPipeline.from_pretrained(model_id, adapter=adapter, torch_dtype=torch.float16, variant='fp16')
pipe.load_ip_adapter('h94/IP-Adapter', subfolder='sdxl_models', weight_name='ip-adapter_sdxl.safetensors')
pipe.load_lora_weights(lcm_lora_id)
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()

init_image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/statue.png")
depth_map = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/depth.png")
prompt = "'best quality, high quality"

image = pipe(prompt,
            image=depth_map,
            ip_adapter_image=init_image,
            num_inference_steps=4,
            guidance_scale=1.0,
            adapter_conditioning_scale=0.6,
            adapter_conditioning_factor=1.0,
            ).images[0]
image.resize((512, 512))

SDXL Turbo

import torch
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter
from diffusers.utils import load_image

model_id = 'Lykon/dreamshaper-xl-turbo'
t2i_adapter_id = 'TencentARC/t2i-adapter-depth-midas-sdxl-1.0'

adapter = T2IAdapter.from_pretrained(t2i_adapter_id, torch_dtype=torch.float16, variant='fp16')

pipe = StableDiffusionXLAdapterPipeline.from_pretrained(model_id, adapter=adapter, torch_dtype=torch.float16, variant='fp16')
pipe.load_ip_adapter('h94/IP-Adapter', subfolder='sdxl_models', weight_name='ip-adapter_sdxl.safetensors')
pipe.enable_model_cpu_offload()

init_image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/statue.png")
depth_map = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/depth.png")
prompt = "'best quality, high quality"

for i in range(3):
    image = pipe(prompt,
                 image=depth_map,
                 ip_adapter_image=init_image,
                 num_inference_steps=7,
                 guidance_scale=1.0,
                 adapter_conditioning_scale=0.6,
                 adapter_conditioning_factor=1.0,
                 ).images[0]
    display(image.resize((512, 512)))
    if i == 2:
        del image

Standard SDXL

import torch
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter
from diffusers.utils import load_image

model_id = 'stabilityai/stable-diffusion-xl-base-1.0'
t2i_adapter_id = 'TencentARC/t2i-adapter-depth-midas-sdxl-1.0'

adapter = T2IAdapter.from_pretrained(t2i_adapter_id, torch_dtype=torch.float16, variant='fp16')

pipe = StableDiffusionXLAdapterPipeline.from_pretrained(model_id, adapter=adapter, torch_dtype=torch.float16, variant='fp16')
pipe.load_ip_adapter('h94/IP-Adapter', subfolder='sdxl_models', weight_name='ip-adapter_sdxl.safetensors')
pipe.enable_model_cpu_offload()

init_image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/statue.png")
depth_map = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/depth.png")
prompt = "'best quality, high quality"

for i in range(3):
    image = pipe(prompt,
                 image=depth_map,
                 ip_adapter_image=init_image,
                 num_inference_steps=50,
                 guidance_scale=6.5,
                 adapter_conditioning_scale=0.7,
                 adapter_conditioning_factor=1.0,
                 ).images[0]
    display(image)
    if i == 2:
        del image

Notes

The inference speed of the pipeline on all three types of checkpoints(and LoRAs) were quite quick, since t2i-adapters are less resource heavy.
I ran pytest tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py and got the following erros:

================================================================================================================================================= short test summary info ==================================================================================================================================================FAILED tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py::StableDiffusionXLAdapterPipelineFastTests::test_components_function - AssertionError: False is not true
FAILED tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py::StableDiffusionXLAdapterPipelineFastTests::test_inference_batch_single_identical - assert 0.000998795 < 0.0001
FAILED tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py::StableDiffusionXLAdapterPipelineFastTests::test_save_load_optional_components - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA__index_select)
FAILED tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py::StableDiffusionXLMultiAdapterPipelineFastTests::test_attention_slicing_forward_pass - AssertionError: 0.0011808276 not less than 0.001 : Attention slicing should not affect the inference results
FAILED tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py::StableDiffusionXLMultiAdapterPipelineFastTests::test_components_function - AssertionError: False is not true
FAILED tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py::StableDiffusionXLMultiAdapterPipelineFastTests::test_save_load_optional_components - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA__index_select)
================================================================================================================================== 6 failed, 50 passed, 1 skipped, 39 warnings in 49.32s ===================================================================================================================

Although the above errors were present, the updated pipeline ran smoothly inside a notebook.
I made 1-2 simple changes to the annotation numbering in the pipeline when i noticed it was out of order, so it would be easier to understand and update in the future.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@sayakpaul @yiyixuxu

…Pipeline Merging upstream to fork

HuggingFaceDocBuilderDev · 2023-12-20T10:20:05Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

patrickvonplaten · 2023-12-26T20:26:40Z

@yiyixuxu can you take a look here?

yiyixuxu

thank you! can we fix the tests?

you need to run 1make quality for the code quality test
the other fast test failure should go away once you add the missing comma in _optional_components = []

src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py

jquintanilla4 · 2023-12-31T07:54:30Z

Ahhh sorry just saw your message. Holidays 😅 super busy with family and friends. Apologies. I'll try to get on it asap.

Happy holidays.

…xl_adapter.py Co-authored-by: YiYi Xu <[email protected]>

jquintanilla4 · 2024-01-03T04:56:17Z

@yiyixuxu made the changes you suggested. Ran the make quality a couple of times and fixed the issues until no errors were shown. I ran pytest tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py and I'm getting these failures:

================================================================================================================================================= short test summary info ==================================================================================================================================================
FAILED tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py::StableDiffusionXLAdapterPipelineFastTests::test_components_function - AssertionError: False is not true
FAILED tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py::StableDiffusionXLAdapterPipelineFastTests::test_inference_batch_single_identical - assert 0.000998795 < 0.0001
FAILED tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py::StableDiffusionXLMultiAdapterPipelineFastTests::test_attention_slicing_forward_pass - AssertionError: 0.0011808276 not less than 0.001 : Attention slicing should not affect the inference results
FAILED tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py::StableDiffusionXLMultiAdapterPipelineFastTests::test_components_function - AssertionError: False is not true
================================================================================================================================== 4 failed, 52 passed, 1 skipped, 41 warnings in 48.72s ===================================================================================================================================

As a result, i keep failing the "Fast tests for PRs / Fast PyTorch Pipeline CPU tests (pull_request)"

jquintanilla4 · 2024-01-03T05:22:47Z

After doing some research. I think the reason why i keep getting failures for the components is because of the following block of code:

        self.register_modules(
            vae=vae,
            text_encoder=text_encoder,
            text_encoder_2=text_encoder_2,
            tokenizer=tokenizer,
            tokenizer_2=tokenizer_2,
            unet=unet,
            adapter=adapter,
            scheduler=scheduler,
            feature_extractor=feature_extractor,
            image_encoder=image_encoder,

i added the following two lines(265-266):

            feature_extractor=feature_extractor,
            image_encoder=image_encoder,

Which are not present in the components dictionary of the test_stable_diffusion_xl_adapter.py script, hence the mismatch. Maybe I'm wrong. But thought I mention it.

yiyixuxu · 2024-01-09T02:56:01Z

@jquintanilla4 you can update the tests! similar to this PR here https://github.com/huggingface/diffusers/pull/6293/files

…4/diffusers into loadipa_sdxlAdapterPipeline

jquintanilla4 · 2024-01-11T05:20:56Z

@yiyixuxu I added the changes to the test and now all checks have passed 😄 thanks for your feedback

yiyixuxu

looking great:)

sayakpaul

So good!

* Added testing notebook and .load_ip_adapter to XLAdapterPipeline * Added annotations * deleted testing notebook * Update src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py Co-authored-by: YiYi Xu <[email protected]> * code clean up * Add feature_extractor and image_encoder to components --------- Co-authored-by: YiYi Xu <[email protected]>

jquintanilla4 added 4 commits December 20, 2023 02:10

Added testing notebook and .load_ip_adapter to XLAdapterPipeline

9130a86

Added annotations

87703ff

Merge remote-tracking branch 'upstream/main' into loadipa_sdxlAdapter…

066d074

…Pipeline Merging upstream to fork

deleted testing notebook

673bdbe

jquintanilla4 changed the title ~~Loadipa sdxl adapter pipeline~~ .load_ip_adapter in StableDiffusionXLAdapterPipeline Dec 20, 2023

patrickvonplaten requested a review from yiyixuxu December 26, 2023 20:26

yiyixuxu reviewed Dec 28, 2023

View reviewed changes

src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py Outdated Show resolved Hide resolved

jquintanilla4 and others added 3 commits January 3, 2024 11:25

Update src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_…

1d16c38

…xl_adapter.py Co-authored-by: YiYi Xu <[email protected]>

code clean up

07a67b1

Merge branch 'main' into loadipa_sdxlAdapterPipeline

1732bd8

Merge branch 'main' into loadipa_sdxlAdapterPipeline

9025834

jquintanilla4 and others added 3 commits January 11, 2024 12:49

Add feature_extractor and image_encoder to components

2853937

Merge branch 'loadipa_sdxlAdapterPipeline' of github.com:jquintanilla…

6f35dae

…4/diffusers into loadipa_sdxlAdapterPipeline

Merge branch 'main' into loadipa_sdxlAdapterPipeline

a3f144d

yiyixuxu approved these changes Jan 11, 2024

View reviewed changes

yiyixuxu requested a review from sayakpaul January 11, 2024 07:04

sayakpaul approved these changes Jan 11, 2024

View reviewed changes

sayakpaul merged commit da843b3 into huggingface:main Jan 11, 2024
14 checks passed

jquintanilla4 deleted the loadipa_sdxlAdapterPipeline branch January 11, 2024 08:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

.load_ip_adapter in StableDiffusionXLAdapterPipeline #6246

.load_ip_adapter in StableDiffusionXLAdapterPipeline #6246

jquintanilla4 commented Dec 20, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented Dec 20, 2023

patrickvonplaten commented Dec 26, 2023

yiyixuxu left a comment

jquintanilla4 commented Dec 31, 2023

jquintanilla4 commented Jan 3, 2024 •

edited

Loading

jquintanilla4 commented Jan 3, 2024 •

edited

Loading

yiyixuxu commented Jan 9, 2024

jquintanilla4 commented Jan 11, 2024

yiyixuxu left a comment

sayakpaul left a comment

.load_ip_adapter in StableDiffusionXLAdapterPipeline #6246

.load_ip_adapter in StableDiffusionXLAdapterPipeline #6246

Conversation

jquintanilla4 commented Dec 20, 2023 • edited Loading

What does this PR do?

Before submitting

Who can review?

HuggingFaceDocBuilderDev commented Dec 20, 2023

patrickvonplaten commented Dec 26, 2023

yiyixuxu left a comment

Choose a reason for hiding this comment

jquintanilla4 commented Dec 31, 2023

jquintanilla4 commented Jan 3, 2024 • edited Loading

jquintanilla4 commented Jan 3, 2024 • edited Loading

yiyixuxu commented Jan 9, 2024

jquintanilla4 commented Jan 11, 2024

yiyixuxu left a comment

Choose a reason for hiding this comment

sayakpaul left a comment

Choose a reason for hiding this comment

jquintanilla4 commented Dec 20, 2023 •

edited

Loading

jquintanilla4 commented Jan 3, 2024 •

edited

Loading

jquintanilla4 commented Jan 3, 2024 •

edited

Loading