use 🧨diffusers model

[keturn]

→ Moved to #1583

[Can't change the working branch of an existing PR.]

---

I think the plan is that we keep the public APIs in ldm.invoke.generator stable while swapping out the implementations to be diffusers-based.

That looks like it'll be primarily in the make_image methods of those Generators.

It might be possible to split things up by the different tasks (txt2img, inpainting, etc) to separate PRs? Which I'll be in favor of if that makes smaller PRs, but I don't know yet whether that will help that much.

Usage

Add a section to your models.yaml like this:

diffusers-1.5:
  description: Diffusers version of Stable Diffusion version 1.5
  format: diffusers
  repo_name: runwayml/stable-diffusion-v1-5

Note the format: diffusers.
The repo_name is as it appears on huggingface.co.

To Do: txt2img

• don't load redundant models! (i.e. both the ckpt and the diffusers formats)
• allow scheduler selection
• support extra scheduler parameters (e.g. DDIM's eta).
• honor float16 setting for loading model
• honor free_gpu_mem with features from 🤗 Accelerate
• update callback users with the new signature.
  • at least done for invoke_ai_web_server. Not sure if the other instances are still in use?
• fix prompt fragment weighting. Refer to WeightedFrozenCLIPEmbedder.
• honor threshold
• honor safety-checker setting
• update models.yaml.example
• update configure_invokeai (formerly preload_models)
  • see #non-interactive default model selection

To Do: txt2img

• make sure we use the correct seeded noise

waiting on upstream diffusers

• fix seamless. (it looks like it's trying to work, but the padding is off somehow.) see Image output tiling for seamless textures with Stable Diffusion huggingface/diffusers#556
• models.diffusion.cross_attention_control might be an obstacle, as that's not in stock diffusers yet and it meddles with some internals. The prompt-to-prompt authors do have a reference implementation that uses diffusers: https://nbviewer.org/github/google/prompt-to-prompt/blob/main/prompt-to-prompt_stable.ipynb

To Do: inpainting

• remove this kludge that has something to do with inpainting? https://github.com/keturn/InvokeAI/blob/f49317c4f5e4a29c747aaaa433e617ff8ae98f13/ldm/generate.py#L1004-L1006

discussion thread: https://discord.com/channels/1020123559063990373/1031668022294884392

[keturn]

and hey, I already hit the first obstacle to using a stock diffusers pipeline: the stock pipelines take in the prompt as text, but Invoke does its own handling of the text and wants to pass in the data for the CLIP text embeddings instead.

This is fine, diffusers pretty much expects most applications doing anything interesting will hit the point of needing to customize their pipeline anyway. It just means a bit more code is required in order to get even the basic proof-of-concept up.

[patrickvonplaten]

Very cool to see that diffusers can be useful to serve as a backend for this library. If you need any help with the migration or require additional features, we're very open to help 🤗

[keturn]

Patrick, why do you say that almost like it's a surprise? 😄 Was serving as an application backend not the plan for diffusers all along? Don't make me second-guess myself here. It'll make me look bad in front of the Invoke devs! 🙈

As for what diffusers could do to help, a fine place to start would be the refactoring the StableDiffusionPipeline to aid reusability and extensibility: huggingface/diffusers#551 (comment)

[keturn]

I've pushed a proof of concept for txt2img. It is super rough, but it does succeed in producing an image for a prompt.

I've updated this PR's main description with a checklist of things we need to do to support it for real.

[patrickvonplaten]

ed a proof of concept for txt2img. It is super rough, but it does succeed in producing an image for a prompt.

I've updated this PR's main description with a checklist of things we need to do to support it for real.

Haha that sounds good - we've starting factoring out methods as done in this PR: huggingface/diffusers#1224 - the __call__ method should get cleaner bit by bit ;-)

[keturn]

Update: made model loading much better. made output much worse.

Like no-longer-recognizable worse. But I committed anyway because it does run, and it's so much easier to fiddle with now that it's not taking extra gigabytes of RAM.

I suspect this implementation of get_learned_conditionings:

InvokeAI/ldm/invoke/generator/diffusers_pipeline.py

Lines 327 to 333 in b39d04d

	text_fragments = c[0]
	text_input = self._tokenize(text_fragments)
	with torch.inference_mode():
	token_ids = text_input.input_ids.to(self.text_encoder.device)
	text_embeddings = self.text_encoder(token_ids)[0]
	return text_embeddings, text_input.input_ids

but maybe it's something else, like configure_model_padding.

[keturn]

fixed! I didn't notice it was making 256px images instead of 512.

[keturn]

Added initial support for switching schedulers. Some of them look like they need further configuration.

[keturn]

Found the missing bit. k_lms and k_euler schedulers fixed.

[keturn]

The current test failure seems to be the same as the failure in development rather than anything specific to this PR.

[keturn]

Pushed support for img2img. Seems to be working, at least with DDIM. LMS and Euler don't do so well.

Might be a few things to follow up on to get proper reproducible-with-seed results.

[keturn]

→ Moved to #1583

[Can't change the working branch of an existing PR.]