Support for stable-diffusion-3-medium model

[rupeshs]

Is your feature request related to a problem? Please describe.
No.

Describe the solution you'd like.
Support SD3 medium model (https://huggingface.co/stabilityai/stable-diffusion-3-medium).

Describe alternatives you've considered.
It seems like this PR(#7232) is stale

[sayakpaul]

Coming up in some hours ;)

[tin2tin]

https://huggingface.co/stabilityai/stable-diffusion-3-medium/commit/fc46bd4cf48394d33a813fe81f4241568cc8b85b

So, do we need to wait for an update of Diffusers?

[painebenjamin]

https://huggingface.co/stabilityai/stable-diffusion-3-medium/commit/fc46bd4cf48394d33a813fe81f4241568cc8b85b

So, do we need to wait for an update of Diffusers?

At some point in the near future the SD3 branch will be merged into diffusers which will add support for it. See this pull request for details.

At the time of writing, diffusers-formatted weights (and control files like model_index.json) are not available at the huggingface repository, so even if you pull that branch, it will not work yet. Give it a few hours and that will likely change.

[sayakpaul]

We will let everyone know once y'all are good to go. Till then, have patience :)

[Yakonrus]

It’s strange that Stability AI weren’t give you accessin advance

[tin2tin]

The files for diffusers are up, but gated files will only be accessible from the Huggingface platform, as there is no way, afaik, to get access when running diffusers locally, and only downloading it with diffusers from HF.
https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers

[painebenjamin]

We can use the code from the HF space to run it without needing to use the repository that is gated behind SAI's approval. You will still need to be logged into Huggingface and request access to the first repository, but you shouldn't have to wait for approval.

pip install git+https://github.com/huggingface/diffusers.git@sd3

Then...

import torch
from diffusers import StableDiffusion3Pipeline

pipe = StableDiffusion3Pipeline.from_pretrained(
    "stabilityai/stable-diffusion-3-medium",
    revision="refs/pr/26",
    torch_dtype=torch.float16,
)
pipe.to("cuda")

image = pipe(
    prompt="a photo of a cat holding a sign that says hello world",
    negative_prompt="",
    num_inference_steps=28,
    height=1024,
    width=1024,
    guidance_scale=7.0,
).images[0]

image.save("sd3_hello_world.png")

Happy generating!

[haofanwang]

You guys are awesome!

[rolux]

Happy generating!

Thanks a lot!

[OlegRuban-ai]

Hello! Is it possible to do something with optimization? On Comfy it runs with 12 GB GPU, and with diffusers - 21.

[sayakpaul]

Yeah. The documentation includes all optimization related bits :)

[tin2tin]

Hmm... I can't get access to the gated files, even though I accepted everything, and I am logged in, but trying to access them from my local computer through Diffusers just doesn't remove the gate blocking. Neither does accessing a mirrored single file, I can download it, but because of the gate, it doesn't work. Gated files does not play well with Diffusers at all.

[s9anus98a]

We can use the code from the HF space to run it without needing to use the repository that is gated behind SAI's approval. You will still need to be logged into Huggingface and request access to the first repository, but you shouldn't have to wait for approval.

pip install git+https://github.com/huggingface/diffusers.git@sd3

Then...

import torch
from diffusers import StableDiffusion3Pipeline

pipe = StableDiffusion3Pipeline.from_pretrained(
    "stabilityai/stable-diffusion-3-medium",
    revision="refs/pr/26",
    torch_dtype=torch.float16,
)
pipe.to("cuda")

image = pipe(
    prompt="a photo of a cat holding a sign that says hello world",
    negative_prompt="",
    num_inference_steps=28,
    height=1024,
    width=1024,
    guidance_scale=7.0,
).images[0]

image.save("sd3_hello_world.png")

Happy generating!

can anyone run sd3 medium on colab free? i tried and it take out all the space. 78GB. WTF??

[7iFinalBoSS]

We can use the code from the HF space to run it without needing to use the repository that is gated behind SAI's approval. You will still need to be logged into Huggingface and request access to the first repository, but you shouldn't have to wait for approval.

pip install git+https://github.com/huggingface/diffusers.git@sd3

Then...

import torch
from diffusers import StableDiffusion3Pipeline

pipe = StableDiffusion3Pipeline.from_pretrained(
    "stabilityai/stable-diffusion-3-medium",
    revision="refs/pr/26",
    torch_dtype=torch.float16,
)
pipe.to("cuda")

image = pipe(
    prompt="a photo of a cat holding a sign that says hello world",
    negative_prompt="",
    num_inference_steps=28,
    height=1024,
    width=1024,
    guidance_scale=7.0,
).images[0]

image.save("sd3_hello_world.png")

Happy generating!

can anyone run sd3 medium on colab free? i tried and it take out all the space. 78GB. WTF??

share your colab link...!

[rohitgandikota]

We can use the code from the HF space to run it without needing to use the repository that is gated behind SAI's approval. You will still need to be logged into Huggingface and request access to the first repository, but you shouldn't have to wait for approval.

pip install git+https://github.com/huggingface/diffusers.git@sd3

Then...

import torch
from diffusers import StableDiffusion3Pipeline

pipe = StableDiffusion3Pipeline.from_pretrained(
    "stabilityai/stable-diffusion-3-medium",
    revision="refs/pr/26",
    torch_dtype=torch.float16,
)
pipe.to("cuda")

image = pipe(
    prompt="a photo of a cat holding a sign that says hello world",
    negative_prompt="",
    num_inference_steps=28,
    height=1024,
    width=1024,
    guidance_scale=7.0,
).images[0]

image.save("sd3_hello_world.png")

Happy generating!

can anyone run sd3 medium on colab free? i tried and it take out all the space. 78GB. WTF??

I am trying locally and it takes around 20GB with float16 precision - you can go even low with bfloat16!

[rolux]

Yeah. The documentation includes all optimization related bits :)

Currently, loading the T5 text encoder separately fails with

OSError: stabilityai/stable-diffusion-3-medium-diffusers does not appear to have a file named text_encoder_3/model-00004-of-00004.safetensors. Checkout 'https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers/tree/main' for available files.

... but you're probably aware of that ;)

[rolux]

Yeah. The documentation includes all optimization related bits :)

Currently, loading the T5 text encoder separately fails with

OSError: stabilityai/stable-diffusion-3-medium-diffusers does not appear to have a file named text_encoder_3/model-00004-of-00004.safetensors. Checkout 'https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers/tree/main' for available files.

... but you're probably aware of that ;)

Seems to be fixed in the latest revision.

[s9anus98a]

what happen to sd3 model?

prompt: photo of a young woman with long, wavy brown hair lying down in grass, top down shot, summer, warm, laughing, joy, fun,

[rupeshs]

@OlegRuban-ai Try SD3 for low VRAM systems: https://github.com/rupeshs/sd3-low-vram

[DN6]

Closing as completed.