use 🧨diffusers model #1583

keturn · 2022-11-27T17:43:51Z

The goal is to reduce the amount of model code InvokeAI has to maintain by integrating https://github.com/huggingface/diffusers , using that to replace the existing ldm (descended from the original CompVis implementation).

I think the plan is that we keep the public APIs in ldm.invoke.generator stable while swapping out the implementations to be diffusers-based.

Discord discussion thread: https://discord.com/channels/1020123559063990373/1031668022294884392

[This is a continuation of #1384. The branch is now hosted in the InvokeAI repo instead of a fork for easier collaboration.]

Usage

Add a section to your models.yaml like this:

diffusers-1.5:
  description: Diffusers version of Stable Diffusion version 1.5
  format: diffusers
  repo_id: runwayml/stable-diffusion-v1-5

Note the format: diffusers.
The repo_id is as it appears on huggingface.co.

Sub-Tasks

i.e. things keturn would love to delegate.

[diffusers]: import CompVis-flavored ckpt models #1690
[diffusers]: model management (configure and pre-load) #1997
[diffusers]: Model Cache #1777
[diffusers]: EmbeddingManager #1778
[diffusers]: Model Manager Web UI #2160
[diffusers]: main.py #1779
[installer enhancement]: xformers #1876
[diffusers]: threshold #2042
- implemented, waiting on acceptance testing

To Do: txt2img

waiting on upstream diffusers

fix seamless. (it looks like it's trying to work, but the padding is off somehow.) see Image output tiling for seamless textures with Stable Diffusion huggingface/diffusers#556
models.diffusion.cross_attention_control might be an obstacle, as that's not in stock diffusers yet and it meddles with some internals. The prompt-to-prompt authors do have a reference implementation that uses diffusers: https://nbviewer.org/github/google/prompt-to-prompt/blob/main/prompt-to-prompt_stable.ipynb

To Do: img2img

bug: crashes when strength is too low (zero timestamps. upstream bug in img2img pipeline?)
make sure we use the correct seeded noise
make work with inpainting model
decide if we need to keep --inpaint_replace now that --strength works. (Or should it apply to the results of the infill method?)
- decision: drop inpaint_replace

To Do: txt2img2img (high-res optimization)

rewrite to our diffusers-based pipeline

To Do: inpainting

work with the inpainting-specific model
make sure masks are being used correctly
remove this kludge that has something to do with inpainting? https://github.com/keturn/InvokeAI/blob/f49317c4f5e4a29c747aaaa433e617ff8ae98f13/ldm/generate.py#L1004-L1006
- see use 🧨diffusers model #1583 (comment)

To Do: embiggen

embiggen!
- (I took a quick look: it mostly goes through invoke's generate API instead of directly accessing the model or schedulers, so hopefully doesn't need to change much.)

Stable Diffusion 2.x support

(I think we can merge the PR without this, but we'll want it before release.)

black images from stable-diffusion-2.1
- seems to only apply to the v-prediction model. stable-diffusion-2.1-base works fine.
- also works fine when xformers is enabled.

damian0815 · 2022-11-27T17:50:00Z

~~currently getting AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings' doing txt2img~~

fixed, fix was "read the instructions in the top of the PR about putting a format: diffusers model in your models.yaml"

damian0815 · 2022-11-27T18:17:34Z

looks like this needs torch==1.13 on macOS, otherwise it crashes in diffusers lib with an error about views having to be contiguous

damian0815 · 2022-11-27T18:23:56Z

something weird is going on because this is not a cat playing with a ball in the forest -s 10 -S 1699080397 -W 512 -H 512 -C 7.5 -A k_lms

k_euler produces the same composition

keturn · 2022-11-27T18:25:00Z

Seems to have worked on mps here with torch 1.12: https://github.com/invoke-ai/InvokeAI/actions/runs/3559296062/jobs/5978550236

click for banana sushi output

Does torch 1.13 on mac perform any better with this diffusers implementation? Or is it still much much slower than torch 1.12 with the old implementation?

damian0815 · 2022-11-27T19:16:27Z

Seems to have worked on mps here with torch 1.12: https://github.com/invoke-ai/InvokeAI/actions/runs/3559296062/jobs/5978550236

that's actually x64, not m1/mps (check the install requirements step, calls x64 python) - github does not offer M1 hosts afaik.

Does torch 1.13 on mac perform any better with this diffusers implementation? Or is it still much much slower than torch 1.12 with the old implementation?

it seems slow but i haven't paid too much attention

damian0815 · 2022-11-27T19:22:21Z

something is definitely broke on mps because this is supposedly banana sushi -s 10 -S 42 -W 512 -H 512 -C 7.5 -A k_lms

damian0815 · 2022-11-27T19:26:54Z

apricot sushi produces exactly the same image, as does empty string. ok, so that means that on MPS the prompt embeddings tensor is being zero'd (or inf'd?) somehow. i'll look into it later.

keturn · 2022-11-27T20:23:52Z

that's actually x64, not m1/mps (check the install requirements step, calls x64 python) - github does not offer M1 hosts afaik.

yeah, hmm, I think you're right. Well that makes it very misleading to have a check named mac-mps-cpu

damian0815 · 2022-11-27T20:28:00Z

there is shared memory shenanigans going on in self.clip_embedder.encode (diffusers_pipeline.py line 309) on MPS that means that the second call overwrites the first-returned tensor. .clone() should fix it, testing now.

damian0815 · 2022-11-27T22:42:36Z

ok there's some deep in the weeds bug in pytorch, because:

conditioned_next_x - unconditioned_next_x

result = all zeros

conditioned_next_x.clone() - unconditioned_next_x

result = looks reasonable

i don't know why this is happening and i don't know what to do about it

keturn · 2022-11-29T03:09:46Z

Some change I just pulled in from development introduced a crash on model load, so I threw this in there as a stopgap: invoke-ai/InvokeAI@185aa24 (#1583)

If I'm reading things correctly, embedding_manager is currently a submodel defined by

InvokeAI/configs/stable-diffusion/v1-inference.yaml

Lines 29 to 30 in 8423be5

    
           personalization_config: 
        
             target: ldm.modules.embedding_manager.EmbeddingManager

The model configs like https://huggingface.co/runwayml/stable-diffusion-v1-5/blob/main/model_index.json have no such personalization_config.

Does it feel necessary to have that be data-driven, or is that class reference something we can hardcode?

keturn · 2022-11-29T03:28:00Z

conditioned_next_x - unconditioned_next_x

result = all zeros

That's weird. Both on the same device and same dtype?

keturn · 2022-11-29T22:39:34Z

development brought in a few more references to model.embedding_manager on the default code path, so I've patched over them in a few more places.

It was a kludge in one place before, but now it's spreading. Should probably the the next thing we tackle in this branch.

keturn · 2022-11-30T05:22:39Z

I had it create an EmbeddingManager. I am not sure if it's working yet but at least it's back to not-crashing.

Also pushed a couple of fixes for deprecated diffusers things, cleaning up some of the warning messages it was spewing.

damian0815 · 2022-11-30T19:14:06Z

realising i never answered your question about the personalization_config keturn - yes, i think it can be hardcoded

ldm/models/diffusion/shared_invokeai_diffusion.py

keturn · 2022-11-30T22:38:01Z

oof. re-resolving all the conflicts after the entirety of 2.2 was rebased was a doozy, but I think I did it okay.

keturn · 2022-11-30T22:43:32Z

🚧 PLZ HOLD. DO NOT PUSH TO THIS BRANCH FOR A BIT, I WILL NEED TO FORCE-PUSH IT. 🚧

oh fiddlesticks.

this branch was based off of develop, so it contains that history.

but that history all got squashed away when it merged in to main.

that means I'm going to have to rebase this branch on main so it's not dragging in the duplicate history... okay.

and update associated things in Generate & Generator to not instantly fail when that happens

Remove IPNDM scheduler; it is not behaving.

- put try: blocks around places where the system tries to load an embedding which is incompatible with the currently loaded model

- Preferences are stored in a file named text-inversion-training/preferences.conf - Currently the resume-from-checkpoint option is not working correctly. Possible bug in textual_inversion_training.py?

- Front end doesn't do anything yet!!!! - Made change to model name parsing in CLI to support ability to have merged models with the "+" character in their names.

- recommend ckpt version of inpainting-1.5 to user - fix get_noise() bug in ckpt version of omnibus.py

- update scripts will now fetch new INITIAL_MODELS.yaml so that configure_invokeai.py will know about the diffusers versions.

- added configure_invokeai.py to menu - menu defaults to browser-based invoke

- Add information on how formats have changed and the upgrade process. - Add short bug list.

lstein

I'm ready to merge this in.

mauwii · 2023-01-15T14:24:53Z

Not working if one switched to diffusers and back, hope this hasn't been done by too many users 😅

│ /Users/mauwii/git/invoke-ai/InvokeAI/ldm/invoke/model_manager.py:765 in migrate_models           │
│                                                                                                  │
│   762 │   │   │   source = models_dir /model                                                     │
│   763 │   │   │   if source.exists():                                                            │
│   764 │   │   │   │   print(f'DEBUG: Moving {models_dir / model} into hub')                      │
│ ❱ 765 │   │   │   │   move(models_dir / model, hub)                                              │
│   766 │   │                                                                                      │
│   767 │   │   # anything else gets moved into the diffusers directory                            │
│   768 │   │   diffusers = models_dir / 'diffusers'                                               │
│                                                                                                  │
│ /Users/mauwii/.pyenv/versions/3.10.9/lib/python3.10/shutil.py:814 in move                        │
│                                                                                                  │
│    811 │   │   real_dst = os.path.join(dst, _basename(src))                                      │
│    812 │   │                                                                                     │
│    813 │   │   if os.path.exists(real_dst):                                                      │
│ ❱  814 │   │   │   raise Error("Destination path '%s' already exists" % real_dst)                │
│    815 │   try:                                                                                  │
│    816 │   │   os.rename(src, real_dst)                                                          │
│    817 │   except OSError:                                                                       │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
Error: Destination path '/Users/mauwii/git/invoke-ai/InvokeAI/models/hub/models--CompVis--stable-diffusion-safety-checker' already exists

same for models/hub/models--bert-base-uncased and /models/hub/models--openai--clip-vit-large-patch14

Having 2.1 as the default model is also kind of inconvenient, since it is not really working on my mac (while 1.5 is working better):

2.1 with k_heun	1.5 with k_heun

Since I created a fresh venv I assume that I am not the only User with this issue 🤔

Just to have it in place: MacBook Air M1 - 16GB - 8 GPU - MacOS 13.1

hipsterusername · 2023-01-15T14:45:15Z

Re: models--CompVis--stable-diffusion-safety-checker' already exists - Deleting the \hub\ folder if you're trying to switch back resolves the issue.

As far as 2.1 - I believe that might just be model quality, but we should be able to confirm by testing w/ a different tool, if anyone can offer to do that.

…odel (#2367) This PR attempts to fix `--free_gpu_mem` option that was not working in CKPT-based diffuser model after #1583. I noticed that the memory usage after #1583 did not decrease after generating an image when `--free_gpu_mem` option was enabled. It turns out that the option was not propagated into `Generator` instance, hence the generation will always run without the memory saving procedure. This PR also related to #2326. Initially, I was trying to make `--free_gpu_mem` works on 🤗 diffuser model as well. In the process, I noticed that InvokeAI will raise an exception when `--free_gpu_mem` is enabled. I tried to quickly fix it by simply ignoring the exception and produce a warning message to user's console.

keturn mentioned this pull request Nov 27, 2022

use 🧨diffusers model #1384

Closed

15 tasks

damian0815 reviewed Nov 30, 2022

View reviewed changes

ldm/models/diffusion/shared_invokeai_diffusion.py Show resolved Hide resolved

keturn changed the base branch from development to main November 30, 2022 21:55

keturn added 10 commits November 30, 2022 14:54

initial commit of DiffusionPipeline class

1d43512

spike: proof of concept using diffusers for txt2img

58ea3bf

doc: type hints for Generator

dcfdb83

refactor(model_cache): factor out load_ckpt

9b274bd

model_cache: add ability to load a diffusers model pipeline

4c3858e

and update associated things in Generate & Generator to not instantly fail when that happens

model_cache: fix model default image dimensions

ae9b482

txt2img: support switching diffusers schedulers

d55e229

diffusers: let the scheduler do its scaling of the initial latents

1e98f4b

Remove IPNDM scheduler; it is not behaving.

web server: update image_progress callback for diffusers data

05a1d68

diffusers: restore prompt weighting feature

e99faeb

lstein added 3 commits January 12, 2023 19:07

don't crash out on incompatible embeddings

b9cd54d

- put try: blocks around places where the system tries to load an embedding which is incompatible with the currently loaded model

add support for checkpoint resuming

0518c37

textual inversion preferences are saved and restored between sessions

de5130a

- Preferences are stored in a file named text-inversion-training/preferences.conf - Currently the resume-from-checkpoint option is not working correctly. Possible bug in textual_inversion_training.py?

patrickvonplaten mentioned this pull request Jan 13, 2023

Weighted Prompts for Diffusers stable diffusion pipeline huggingface/diffusers#1506

Closed

lstein and others added 10 commits January 13, 2023 13:04

copy learned_embeddings.bin into right location

1e2f871

add front end for diffusers model merging

e717c02

- Front end doesn't do anything yet!!!! - Made change to model name parsing in CLI to support ability to have merged models with the "+" character in their names.

improve inpainting experience

2d3e5cf

- recommend ckpt version of inpainting-1.5 to user - fix get_noise() bug in ckpt version of omnibus.py

update environment*yml

b6e83d6

tweak instructions to install HuggingFace token

1a045c9

bump version number

75426d0

enhance update scripts

fb28078

- update scripts will now fetch new INITIAL_MODELS.yaml so that configure_invokeai.py will know about the diffusers versions.

enhance invoke.sh/invoke.bat launchers

2a3fcfb

- added configure_invokeai.py to menu - menu defaults to browser-based invoke

remove conda workflow (#2321)

84274e2

fix token_ids has shape torch.Size([79]) - expected [77]

dfc24dc

lstein marked this pull request as ready for review January 15, 2023 13:14

lstein requested review from mauwii and tildebyte as code owners January 15, 2023 13:14

lstein added 2 commits January 15, 2023 08:17

Merge branch 'main' into dev/diffusers

a492bf3

update CHANGELOG.md with 2.3.* info

8c450d4

- Add information on how formats have changed and the upgrade process. - Add short bug list.

lstein approved these changes Jan 15, 2023

View reviewed changes

lstein merged commit 6fdbc19 into main Jan 15, 2023

lstein deleted the dev/diffusers branch January 15, 2023 14:22

keturn added this to the 2.3 🧨 milestone Jan 15, 2023

mauwii restored the dev/diffusers branch January 16, 2023 20:25

addianto mentioned this pull request Jan 19, 2023

Make sure --free_gpu_mem still works when using CKPT-based diffuser model #2367

Merged

keturn deleted the dev/diffusers branch January 19, 2023 20:19

keturn mentioned this pull request Jan 20, 2023

[bug]: swap doesn't work when xformers is enabled #2328

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use 🧨diffusers model #1583

use 🧨diffusers model #1583

keturn commented Nov 27, 2022 •

edited

Loading

damian0815 commented Nov 27, 2022 •

edited

Loading

damian0815 commented Nov 27, 2022 •

edited

Loading

damian0815 commented Nov 27, 2022 •

edited

Loading

keturn commented Nov 27, 2022 •

edited

Loading

damian0815 commented Nov 27, 2022

damian0815 commented Nov 27, 2022

damian0815 commented Nov 27, 2022

keturn commented Nov 27, 2022

damian0815 commented Nov 27, 2022

damian0815 commented Nov 27, 2022 •

edited

Loading

keturn commented Nov 29, 2022

keturn commented Nov 29, 2022

keturn commented Nov 29, 2022

keturn commented Nov 30, 2022

damian0815 commented Nov 30, 2022

keturn commented Nov 30, 2022

keturn commented Nov 30, 2022 •

edited

Loading

lstein left a comment

mauwii commented Jan 15, 2023 •

edited

Loading

hipsterusername commented Jan 15, 2023

use 🧨diffusers model #1583

use 🧨diffusers model #1583

Conversation

keturn commented Nov 27, 2022 • edited Loading

Usage

Sub-Tasks

To Do: txt2img

waiting on upstream diffusers

To Do: img2img

To Do: txt2img2img (high-res optimization)

To Do: inpainting

To Do: embiggen

Stable Diffusion 2.x support

damian0815 commented Nov 27, 2022 • edited Loading

damian0815 commented Nov 27, 2022 • edited Loading

damian0815 commented Nov 27, 2022 • edited Loading

keturn commented Nov 27, 2022 • edited Loading

damian0815 commented Nov 27, 2022

damian0815 commented Nov 27, 2022

damian0815 commented Nov 27, 2022

keturn commented Nov 27, 2022

damian0815 commented Nov 27, 2022

damian0815 commented Nov 27, 2022 • edited Loading

keturn commented Nov 29, 2022

keturn commented Nov 29, 2022

keturn commented Nov 29, 2022

keturn commented Nov 30, 2022

damian0815 commented Nov 30, 2022

keturn commented Nov 30, 2022

keturn commented Nov 30, 2022 • edited Loading

lstein left a comment

Choose a reason for hiding this comment

mauwii commented Jan 15, 2023 • edited Loading

hipsterusername commented Jan 15, 2023

keturn commented Nov 27, 2022 •

edited

Loading

damian0815 commented Nov 27, 2022 •

edited

Loading

damian0815 commented Nov 27, 2022 •

edited

Loading

damian0815 commented Nov 27, 2022 •

edited

Loading

keturn commented Nov 27, 2022 •

edited

Loading

damian0815 commented Nov 27, 2022 •

edited

Loading

keturn commented Nov 30, 2022 •

edited

Loading

mauwii commented Jan 15, 2023 •

edited

Loading