[FEAT] DDUF format #10037

SunMarc · 2024-11-27T17:07:22Z

What does this PR do?

This PR adds support to DDUF format. More docs about this format here.

Requires transformers PR : https://github.com/huggingface/transformers/pull/35217/files (merged)
Requires huggingface_hub PR: huggingface/huggingface_hub#2692 (released)

Example

Load from the hub

pipe = StableDiffusion3Pipeline.from_pretrained("DDUF/stable-diffusion-3-medium-diffusers-DDUF",
                                                dduf_file="stable-diffusion-3-medium-diffusers.dduf")
pipe = pipe.to("cuda")

image = pipe(
    "A cat holding a sign that says hello world",
    negative_prompt="",
    num_inference_steps=28,
    guidance_scale=7.0,
).images[0]
image.save("cat.png")

Save to DDUF format

from huggingface_hub import export_folder_as_dduf

pipe = StableDiffusion3Pipeline.from_pretrained("stabilityai/stable-diffusion-3-medium-diffusers", 
                                                torch_dtype=torch.float16)

pipe.save_pretrained(save_directory="stable-diffusion-3-medium-diffusers")

export_folder_as_dduf("stable-diffusion-3-medium-diffusers.dduf", folder_path="stable-diffusion-3-medium-diffusers")

TODO list:

stream directly from the .dduf
replace loader with huggingface_hub parser DDUF parser v0.1 huggingface_hub#2692 (comment)
SD pipeline
Flux pipeline

HuggingFaceDocBuilderDev · 2024-11-27T17:14:06Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

sayakpaul

Thanks for starting this! Apart from the initial comments I left, I have a few higher level questions:

We're assuming that the archive will contain all the pipeline-related files in the diffusers format. I am fine with that. This means we'll be shipping DDUF only at the pipeline-level for now, correct?

sayakpaul · 2024-11-28T08:56:54Z

src/diffusers/pipelines/pipeline_utils.py

+        dduf_format: bool = False,
+        dduf_filename: Optional[Union[str, os.PathLike]] = None,


As mentioned over Slack, why do we need two arguments here? How about just dduf_filename?

I will just use that then !

I think that most, if not all, the components of the file name should be inferred. One of the reasons to use DDUF would be to provide confidence about the contents based on filename inspection, so we should try to make it work with something like an optional prefix, but build the rest automatically. So something like dduf_name="my-lora" resulting in the library saving to "my-lora-stable-diffusion-3-medium-diffusers-fp16.dduf", or something like that.

ok to use some kind of convention, but i don't think it should be an absolute requirement, ie. people can rename their files if they want to (but the metadata should be inside the file anyways)

Yeah that is the plan.

src/diffusers/pipelines/pipeline_utils.py

sayakpaul · 2024-11-28T10:16:51Z

I just tried out bitsanbdytes and it seems to work as well. Checkpoint: https://huggingface.co/DDUF/Flux.1-Dev-Tnf4-TE2NF4 (private for now)

Serialization

from diffusers import FluxTransformer2DModel, DiffusionPipeline 
from diffusers import BitsAndBytesConfig
from transformers import BitsAndBytesConfig as BnbConfig
from transformers import T5EncoderModel 
from huggingface_hub import create_repo, upload_folder
import torch 

repo_id = "DDUF/Flux.1-Dev-Tnf4-TE2NF4"
repo_id = create_repo(repo_id=repo_id, exist_ok=True).repo_id
dtype = torch.bfloat16

nf4_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=dtype,
)
transformer = FluxTransformer2DModel.from_pretrained(
    "black-forest-labs/FLUX.1-dev", 
    subfolder="transformer", 
    quantization_config=nf4_config, 
    torch_dtype=dtype
)
nf4_config = BnbConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=dtype,
)
text_encoder_2 = T5EncoderModel.from_pretrained(
    "black-forest-labs/FLUX.1-dev", 
    subfolder="text_encoder_2", 
    quantization_config=nf4_config,
    torch_dtype=dtype
)

pipeline = DiffusionPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev", 
    transformer=transformer, 
    text_encoder_2=text_encoder_2,
    torch_dtype=dtype
)
pipeline.enable_model_cpu_offload()

image = pipeline(
    "a cat standing by the sea waiting for its companion to come and console it.",
    guidance_scale=4.5,
    num_inference_steps=50,
    max_sequence_length=512,
    generator=torch.manual_seed(0)
).images[0]
image.save("dduf_out.png")

folder = "flux.1-dduf"
pipeline.save_pretrained(folder, dduf_filename="Flux.1-Dev-Tnf4-TE2NF4.dduf")
upload_folder(repo_id=repo_id, folder_path=folder, create_pr=True)

Loading

from diffusers import DiffusionPipeline 
import torch

repo_id = "DDUF/Flux.1-Dev-Tnf4-TE2NF4"
pipeline = DiffusionPipeline.from_pretrained(
    repo_id, dduf="Flux.1-Dev-Tnf4-TE2NF4.dduf", torch_dtype=torch.bfloat16
)
pipeline.enable_model_cpu_offload()

image = pipeline(
    "a cat standing by the sea waiting for its companion to come and console it.",
    guidance_scale=4.5,
    num_inference_steps=50,
    max_sequence_length=512,
    generator=torch.manual_seed(0)
).images[0]
image.save("dduf_out.png")

pcuenca · 2024-11-28T14:04:24Z

src/diffusers/pipelines/pipeline_utils.py

+                    # remove tar archive to free memory
+                    os.remove(tar_file_path)
+                    # rename folder to match the name of the dduf archive
+                    os.rename(extract_to, tar_file_path)


I think that expanding the archive and removing it could be problematic.

If I understand correctly, we would be removing the symlink to the blob, but not the blob itself. We won't be saving disk space.

If we did remove the blob, I think it'd trigger another download next time we call .from_pretrained.

Removing the reference could maybe have consequences when checking for newer versions in the Hub (not sure).

When we change to using zips, I wonder if we could extract files progressively in memory during the load process, and keeping the big .dduf file as is. Having a disk snapshot is convenient, but I can see no easy solution to avoid duplicating space.

I agree. I will check if it is possible to stream directly from the zip or at least extract progressively in memory as you said

maybe this? https://github.com/leenr/gzip-stream/blob/master/gzip_stream.py

Thanks for the link ! I'll have a look !

BTW on the Hub side we want to constraint the types of Zip a lot:

no compression only (to make sure it's deduplicated well by XetHub)

single disk (internal simplifications of Zip)

no encryption etc

To ensure restrictions are met, we'll provide tooling in Python to save/load DDUF files in a consistent way: huggingface/huggingface_hub#2692. This will be part of huggingface_hub to that it can be easily reused by any library without having necessarily diffusers as a dependency.

src/diffusers/pipelines/pipeline_utils.py

julien-c

BTW a bit orthogonal but do you have examples of quantized diffusers repos right now?

sayakpaul · 2024-11-29T02:22:58Z

BTW a bit orthogonal but do you have examples of quantized diffusers repos right now?

This is one example: https://huggingface.co/sayakpaul/FLUX.1-Fill-dev-nf4. Note that it doesn't include all the models needed to run Flux as it's not typical to quantize the VAE and the other CLIP-based text encoder.

src/diffusers/pipelines/pipeline_utils.py

sayakpaul · 2024-11-29T03:16:51Z

@SunMarc I don't think loading is working:

from diffusers import DiffusionPipeline

repo_id = "DDUF/stable-diffusion-3-medium-diffusers-DDUF"
filename = "stable-diffusion-3-medium-diffusers.dduf"
pipeline = DiffusionPipeline.from_pretrained(repo_id, dduf=filename)

Leads to:

Error

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/fsx/sayak/diffusers/load_dduf.py", line 23, in <module>
    pipeline = DiffusionPipeline.from_pretrained(repo_id, dduf=filename)
  File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/fsx/sayak/diffusers/src/diffusers/pipelines/pipeline_utils.py", line 776, in from_pretrained
    cached_folder = cls.download(
  File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/fsx/sayak/diffusers/src/diffusers/pipelines/pipeline_utils.py", line 1379, in download
    config_file = hf_hub_download(
  File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 862, in hf_hub_download
    return _hf_hub_download_to_cache_dir(
  File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 925, in _hf_hub_download_to_cache_dir
    (url_to_download, etag, commit_hash, expected_size, head_call_error) = _get_metadata_or_catch_error(
  File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1376, in _get_metadata_or_catch_error
    metadata = get_hf_file_metadata(
  File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1296, in get_hf_file_metadata
    r = _request_wrapper(
  File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 277, in _request_wrapper
    response = _request_wrapper(
  File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 301, in _request_wrapper
    hf_raise_for_status(response)
  File "/fsx/sayak/miniconda3/envs/diffusers/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 417, in hf_raise_for_status
    raise _format(EntryNotFoundError, message, response) from e
huggingface_hub.errors.EntryNotFoundError: 404 Client Error. (Request ID: Root=1-674931ef-1fdc6f274e6f862c31db623e;2056ffe1-9bd0-4667-806f-273827e95101)

Entry Not Found for url: https://huggingface.co/DDUF/stable-diffusion-3-medium-diffusers-DDUF/resolve/main/model_index.json.

which is expected.

Update: I have pushed my updates in sayak-dduf which fixes it (it's incomplete but gives a heads-up). Feel free to cherry-pick commits from it. It has progressive extraction with a progressbar as well.

Co-authored-by: Sayak Paul <[email protected]>

SunMarc · 2024-12-04T12:57:58Z

Update: I have pushed my updates in sayak-dduf which fixes it (it's incomplete but gives a heads-up). Feel free to cherry-pick commits from it. It has progressive extraction with a progressbar as well.

Merged !

src/diffusers/configuration_utils.py

src/diffusers/pipelines/pipeline_utils.py

src/diffusers/configuration_utils.py

sayakpaul

Thanks! Left a single comment.

src/diffusers/models/modeling_utils.py

tests/pipelines/test_pipelines.py

sayakpaul

Thanks for handling the final changes.

yiyixuxu · 2025-01-06T18:21:00Z

@DN6 can you do a review here? it is a single file

src/diffusers/models/model_loading_utils.py

DN6 · 2025-01-07T07:17:51Z

tests/pipelines/test_pipelines.py

@@ -981,6 +983,18 @@ def test_download_ignore_files(self):
            assert not any(f in ["vae/diffusion_pytorch_model.bin", "text_encoder/config.json"] for f in files)
            assert len(files) == 14

+    def test_download_dduf_with_custom_pipeline_raises_error(self):
+        with self.assertRaises(NotImplementedError):


Think it might be easier to maintain if we introduce a class attribute into the PipelineTesterMixin supports_dduf=False and set it to True for the pipelines that work with DDUF.

We can make the these tests conditional on that attribute. Similar to how we skip tests for models with custom attention processors

diffusers/tests/models/transformers/test_models_transformer_mochi.py

Line 32 in f1e0c7c

uses_custom_attn_processor = True

diffusers/tests/models/test_modeling_common.py

Line 548 in f1e0c7c

if self.uses_custom_attn_processor:

WDTY @sayakpaul ?

I can look into it tomorrow, sounds good.

Maybe it is easier to set to True by default then set to to False for pipeline that doesn't work ? This helps finding which pipeline doesn't work with dduf and it makes sure that when adding a model, we try to make it compatible with dduf

LMK what you think about the last commit

src/diffusers/configuration_utils.py

src/diffusers/utils/hub_utils.py

Co-authored-by: YiYi Xu <[email protected]>

yiyixuxu

thanks for working on this! @SunMarc

tests/pipelines/test_pipelines.py

sayakpaul

Thanks, Marc!

DN6

LGTM 👍🏽 Good to merge after resolving conflicts.

sayakpaul · 2025-01-12T15:11:39Z

@SunMarc I proceeded with resolving the conflicts. LMK.

sayakpaul · 2025-01-12T15:35:30Z

@SunMarc not sure why the test is failing. Could you give it a look?

SunMarc added 2 commits November 27, 2024 18:01

load and save dduf archive

1fb86e3

style

0389333

SunMarc changed the title ~~# [FEAT] DDUF format~~ [FEAT] DDUF format Nov 27, 2024

sayakpaul changed the title ~~[FEAT] DDUF format~~ [FEAT] [WIP] DDUF format Nov 28, 2024

sayakpaul marked this pull request as draft November 28, 2024 08:56

sayakpaul reviewed Nov 28, 2024

View reviewed changes

pcuenca reviewed Nov 28, 2024

View reviewed changes

switch to zip uncompressed

2eeda25

julien-c reviewed Nov 28, 2024

View reviewed changes

src/diffusers/pipelines/pipeline_utils.py Outdated Show resolved Hide resolved

julien-c reviewed Nov 28, 2024

View reviewed changes

sayakpaul reviewed Nov 29, 2024

View reviewed changes

sayakpaul added 2 commits November 30, 2024 08:56

Merge branch 'main' into dduf

cbee7cb

updates

d840867

yiyixuxu added the roadmap Add to current release roadmap label Dec 3, 2024

Wauplin mentioned this pull request Dec 4, 2024

DDUF parser v0.1 huggingface/huggingface_hub#2692

Merged

5 tasks

sayakpaul and others added 3 commits December 4, 2024 17:34

Merge branch 'main' into dduf

78135f1

Update src/diffusers/pipelines/pipeline_utils.py

7d2c7d5

Co-authored-by: Sayak Paul <[email protected]>

Update src/diffusers/pipelines/pipeline_utils.py

e66c4d0

Co-authored-by: Sayak Paul <[email protected]>

SunMarc added 2 commits December 4, 2024 13:03

first draft

b14bffe

remove print

1cd5155

SunMarc mentioned this pull request Dec 4, 2024

[FEAT] Compatibility with dduf format from diffusers huggingface/transformers#35093

Draft

sayakpaul reviewed Dec 5, 2024

View reviewed changes

src/diffusers/configuration_utils.py Outdated Show resolved Hide resolved

sayakpaul reviewed Dec 5, 2024

View reviewed changes

src/diffusers/pipelines/pipeline_utils.py Outdated Show resolved Hide resolved

sayakpaul reviewed Dec 5, 2024

View reviewed changes

src/diffusers/pipelines/pipeline_utils.py Outdated Show resolved Hide resolved

sayakpaul reviewed Dec 5, 2024

View reviewed changes

src/diffusers/configuration_utils.py Outdated Show resolved Hide resolved

SunMarc requested review from sayakpaul and yiyixuxu January 3, 2025 11:39

Merge branch 'main' into dduf

faa0cac

sayakpaul reviewed Jan 3, 2025

View reviewed changes

src/diffusers/models/modeling_utils.py Outdated Show resolved Hide resolved

SunMarc added 2 commits January 3, 2025 14:52

add test for sharded checkpoint

0205cc8

Merge branch 'dduf' of github.com:huggingface/diffusers into dduf

9cda4c1

SunMarc requested a review from sayakpaul January 3, 2025 14:12

Merge branch 'main' into dduf

cd0734e

sayakpaul reviewed Jan 4, 2025

View reviewed changes

tests/pipelines/test_pipelines.py Outdated Show resolved Hide resolved

sayakpaul approved these changes Jan 4, 2025

View reviewed changes

SunMarc added 2 commits January 6, 2025 14:55

add suggestions

5037d39

Merge branch 'dduf' of github.com:huggingface/diffusers into dduf

c9e08da

yiyixuxu reviewed Jan 6, 2025

View reviewed changes

src/diffusers/models/model_loading_utils.py Outdated Show resolved Hide resolved

DN6 reviewed Jan 7, 2025

View reviewed changes

SunMarc and others added 4 commits January 7, 2025 16:19

Update src/diffusers/models/model_loading_utils.py

02a368b

Co-authored-by: YiYi Xu <[email protected]>

from suggestions

7bc9347

add class attributes to flag dduf tests

fff5954

last one

da402da

SunMarc requested review from DN6 and yiyixuxu January 7, 2025 17:00

yiyixuxu approved these changes Jan 7, 2025

View reviewed changes

Merge branch 'main' into dduf

9ebbf84

sayakpaul reviewed Jan 8, 2025

View reviewed changes

tests/pipelines/test_pipelines.py Show resolved Hide resolved

sayakpaul approved these changes Jan 8, 2025

View reviewed changes

Merge branch 'main' into dduf

aaaa947

DN6 approved these changes Jan 11, 2025

View reviewed changes

resolve conflicts.

290b88d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEAT] DDUF format #10037

[FEAT] DDUF format #10037

SunMarc commented Nov 27, 2024 •

edited by sayakpaul

Loading

HuggingFaceDocBuilderDev commented Nov 27, 2024

sayakpaul left a comment

sayakpaul Nov 28, 2024

SunMarc Nov 28, 2024

pcuenca Nov 28, 2024 •

edited

Loading

julien-c Dec 4, 2024

sayakpaul Dec 10, 2024

sayakpaul commented Nov 28, 2024 •

edited

Loading

pcuenca Nov 28, 2024 •

edited

Loading

SunMarc Nov 28, 2024

vladmandic Nov 28, 2024

SunMarc Nov 28, 2024

julien-c Nov 28, 2024

Wauplin Dec 4, 2024

julien-c left a comment

sayakpaul commented Nov 29, 2024

sayakpaul commented Nov 29, 2024 •

edited

Loading

SunMarc commented Dec 4, 2024

sayakpaul left a comment

sayakpaul left a comment

yiyixuxu commented Jan 6, 2025

DN6 Jan 7, 2025

SunMarc Jan 7, 2025

sayakpaul Jan 7, 2025

SunMarc Jan 7, 2025

SunMarc Jan 7, 2025

yiyixuxu left a comment

sayakpaul left a comment

DN6 left a comment

sayakpaul commented Jan 12, 2025

sayakpaul commented Jan 12, 2025

		dduf_format: bool = False,
		dduf_filename: Optional[Union[str, os.PathLike]] = None,

[FEAT] DDUF format #10037

Are you sure you want to change the base?

[FEAT] DDUF format #10037

Conversation

SunMarc commented Nov 27, 2024 • edited by sayakpaul Loading

What does this PR do?

Example

Load from the hub

Save to DDUF format

HuggingFaceDocBuilderDev commented Nov 27, 2024

sayakpaul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pcuenca Nov 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sayakpaul commented Nov 28, 2024 • edited Loading

pcuenca Nov 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

julien-c left a comment

Choose a reason for hiding this comment

sayakpaul commented Nov 29, 2024

sayakpaul commented Nov 29, 2024 • edited Loading

SunMarc commented Dec 4, 2024

sayakpaul left a comment

Choose a reason for hiding this comment

sayakpaul left a comment

Choose a reason for hiding this comment

yiyixuxu commented Jan 6, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yiyixuxu left a comment

Choose a reason for hiding this comment

sayakpaul left a comment

Choose a reason for hiding this comment

DN6 left a comment

Choose a reason for hiding this comment

sayakpaul commented Jan 12, 2025

sayakpaul commented Jan 12, 2025

SunMarc commented Nov 27, 2024 •

edited by sayakpaul

Loading

pcuenca Nov 28, 2024 •

edited

Loading

sayakpaul commented Nov 28, 2024 •

edited

Loading

pcuenca Nov 28, 2024 •

edited

Loading

sayakpaul commented Nov 29, 2024 •

edited

Loading