Image output tiling for seamless textures with Stable Diffusion

[torrinworx]

Is your feature request related to a problem? Please describe.
Currently there is no way to create seamless textures with Stable Diffusion, a crucial feature that is missing.

Describe the solution you'd like
Something similar to this pull on the sd-webui repo: Sygil-Dev/sygil-webui#911

A simple argument in the StableDiffusionPipeline that would enable seamless texture generation for 3D applications.

[anton-l]

Hi @torrinworx! As the Sygil-Dev/sygil-webui#911 PR suggests, you can make the Stable Diffusion models tile-able by patching the torch.nn.Conv2d before loading the pipeline:

# add global options to models
def patch_conv(**patch):
    cls = torch.nn.Conv2d
    init = cls.__init__
    def __init__(self, *args, **kwargs):
        return init(self, *args, **kwargs, **patch)
    cls.__init__ = __init__

patch_conv(padding_mode='circular')
print("patched for tiling")

Native support for tiling in diffusers is unlikely to come, as it would either be hacky, or complicate the model the model design with additional arguments :)

[shirayu]

FYI:
In my app, I toggle to be tile-able or not after pipe loading by change padding_mode in each layer.

1. First, save conv layers and their original padding modes
2. Change padding mode on each generation

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patrickvonplaten]

Could we make this a community pipeline maybe? :-)

[camenduru]

is it possible for flax
maybe like this: https://flax.readthedocs.io/en/latest/api_reference/_autosummary/flax.linen.Conv.html?highlight=circular

update: I test it working 🎉

[uyo9ko]

Hi @torrinworx! As the sd-webui/stable-diffusion-webui#911 PR suggests, you can make the Stable Diffusion models tile-able by patching the torch.nn.Conv2d before loading the pipeline:

# add global options to models
def patch_conv(**patch):
    cls = torch.nn.Conv2d
    init = cls.__init__
    def __init__(self, *args, **kwargs):
        return init(self, *args, **kwargs, **patch)
    cls.__init__ = __init__

patch_conv(padding_mode='circular')
print("patched for tiling")

Native support for tiling in diffusers is unlikely to come, as it would either be hacky, or complicate the model the model design with additional arguments :)

hello, i follow your advice , add this snippet before loading the pipeline, but there is an error when loading pipeline:

pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
    model_path,
    revision="fp16", 
    torch_dtype=torch.float16,
    use_auth_token=True
)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-29-796abcd4f547>](https://localhost:8080/#) in <module>
      6     revision="fp16",
      7     torch_dtype=torch.float16,
----> 8     use_auth_token=True
      9 )
     10 pipe = pipe.to(device)

8 frames
[<ipython-input-19-17c9e93293f0>](https://localhost:8080/#) in __init__(self, *args, **kwargs)
      4     init = cls.__init__
      5     def __init__(self, *args, **kwargs):
----> 6         return init(self, *args, **kwargs, **patch)
      7     cls.__init__ = __init__
      8 

TypeError: __init__() got multiple values for keyword argument 'padding_mode'

can you tell me why?

[patrickvonplaten]

@anton-l, I think we can allow to adapt the conv mode with a nice diffusers API here no?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patrickvonplaten]

Linking some first PRs here:

• split VAE decode for high-res upsampling #1521
• 8k Stable Diffusion with tiled VAE #1441

[patrickvonplaten]

cc @patil-suraj @anton-l does any of you have time to look into tiling?

[keturn]

Note this must be applied to both the diffusion model and the decoder (VAE).

[patil-suraj]

That's the plan @keturn, still experimenting with it.

[keturn]

begone, stalebot!

[asomoza]

@yiyixuxu

I didn't know that PR existed, very nice!

Even so, If you don't want to add this code the the SD pipelines, I think it would be better to just write it as a technique in the new section since is just like 20 lines of code and it wouldn't need to be maintained or updated which is a known issue with community pipelines.

[cmdr2]

My 2 cents (maintainer of Easy Diffusion) - I'd also request that tiling be included in the main code, because tiling is a feature present in pretty much every Stable Diffusion software (including Easy Diffusion). InvokeAI has it, auto1111 has it, and it's a very common use-case.

I genuinely understand that it's a tricky balance between keeping the project lean vs empowering users. But in this particular case, given the context, I believe it makes sense to tilt towards including it. But that's obviously your call.

It's a regular task for me to fix our tiling code every few releases of diffusers, because something changed that broke tiling. I don't see that changing with making it a community pipeline. People keep patching the code in this ticket, and people will keep patching the code in a community pipeline. But the bottom line is that tiling will keep breaking every few releases (because it isn't a core feature with automated tests).

So to me it doesn't really matter whether the code is here in this ticket, or in a community pipeline. If it isn't in the core, it'll very likely continue breaking every few releases. And given the number of people maintaining the code in this ticket for so long (to keep tiling updated with diffusers), I'd say it signals the utility of this feature.

I'm not sure tiling is a different pipeline in the first place. It's an option for an image, using the same pipeline. Regardless of whether we're doing SDXL or Img2Img or Inpaint etc. But anyway, that's semantics. :)

[yiyixuxu]

@asomoza @cmdr2
thanks for your feedback! I hear y'all

I just realized that we merged this PR #6031
I think with this, we don't have to patch it up this way anymore?

I think we can add it to StableDiffusionMixin (

diffusers/src/diffusers/pipelines/pipeline_utils.py

Line 1654 in 4974b84

class StableDiffusionMixin:

) and make it automatically available to all pipelines in SD family - this way we do not need to add any code to the pipelines

[yiyixuxu]

hey sorry guys

I looked into this and really can't come up with a clean way to support this feature from diffusers

Also I now agree it does not make too much sense to make community pipelines for it since it should be a feature every single SD pipeline can use; so I added it to the "community script" section in the community folder for now #7358. This will also most likely be added to our docs later

[cmdr2]

Thanks @yiyixuxu ! That sounds fair enough, thanks for trying!

Just curious, what's the complication with adding a set_seamless_tiling() function to StableDiffusionMixin, which accepts the tiling type as the argument? Not challenging, just curious to understand the complication. :) Thanks!

[yiyixuxu]

@cmdr2

I think we would be able to support it if it's just to modify the padding_mode attribute - but patching the forward method is just way too hacky and it's better to be done outside of diffusers

[cmdr2]

@yiyixuxu Yeah, that makes sense. I don't have a good suggestion either, still thinking about it. Maybe a wrapper class (like the LoRA Conv class), that is applied when seamless-tiling is enabled, and removed when disabled? Not sure yet whether this will play well with the LoRA Conv class, and whether it'll work well with torch.compile. @sayakpaul - I was wondering if you had a suggestion regarding torch.compile? thanks

I agree that hacks should be avoided, so brainstorming about ways to make this happen without bad code.

Because this feature is really useful, especially when it comes to 3D models and game-dev. For e.g. here's a tweet by Adobe Substance 3D (from yesterday), which shows its ability to generate seamless textures using prompts - https://twitter.com/Substance3D/status/1769778370227683514

[sayakpaul]

Not sure yet whether this will play well with the LoRA Conv class, and whether it'll work well with torch.compile. @sayakpaul - I was wondering if you had a suggestion regarding torch.compile? thanks

Should work as expected. If not, please let us know. Would be more than happy to investigate :)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[samiede]

I've recently had a lot of issues with sdxl and tiling, I've noticed that images come out wonky or display very strong artifacts. I've used the above snippets to generate some examples:
First images I've generated with 1.5 with the above technique:


and with the same seed, also sdxl:

Here's another example

is this expected behavior?

[yiyixuxu]

cc @asomoza here

[asomoza]

Probably depends on the model and the prompt, SDXL understands better a good prompt, so if you get bad results you'll need to make a better prompt (see the ceramic floor):

	generated	tiled
brick wall
plywood
ceramic floor
ceramic floor seamless uniform pattern

[samiede]

I can get good looking examples from sdxl with a lot of experimentation, but one big issue that we are seeing is that lower resolutions (e.g. here 512x512) result in these very much destroyed and artefacted images. They are always looking similar, and you can see the patterns also emerge in the last example I posted. Here's the artefacted examples:

[asomoza]

SDXL wasn't trained in 512x512 images or even in 768x768, the results will almost always be bad. You'll need to use 1024x1024 or some of the resolutions in which SDXL was trained.

[samiede]

Yes, I am aware of that, the thing I'm wondering about is why the distribution seems to be getting thrown so much stronger when tiling is enabled, i.e. this is a 768x768 generation from sdxl 0.9 without tiling vs with tiling.

Admittedly, 512x512 doesn't produce any useful results with or without tiling

[asomoza]

yeah, I don't know the "true" answer to that, but probably you'll get the same answer I gave you anywhere you ask, not many people might spend time and resources in understanding why a model doesn't work well when you use it in some specific way outside of its domain.

But in the meantime, we can safely assume that this is not a diffusers issue, maybe you can try asking in the Stability AI repo since they trained the model and maybe they did some experiments with it about this.

Edit: I forgot about this, if you need to work with lower resolutions, maybe this will work: https://github.com/bytedance/res-adapter

[samiede]

Yes, I agree, it's definitively not a diffusers issue, sorry for highjacking the thread, I found it on my quest to understand :)

Thanks for the link, I'll check the adapter out!

[chirag4798]

I also had to replace padding_mode for TransposeConv2d layers as well, here's what worked for me

import torch
from diffusers import StableDiffusionPipeline, StableDiffusionXLPipeline


def flatten(model: torch.nn.Module):
    """
    Recursively flattens the model to retrieve all layers.
    """
    children = list(model.children())
    flattened = []

    if children == []:
        return model

    for child in children:
        try:
            flattened.extend(flatten(child))
        except TypeError:
            flattened.append(flatten(child))
    return flattened


def seamless_tiling(pipeline):
    """
    Enables seamless tiling for specific layers in the pipeline.
    """
    targets = [pipeline.vae, pipeline.text_encoder, pipeline.unet]

    if hasattr(pipeline, "text_encoder_2"):
        targets.append(pipeline.text_encoder_2)
    if pipeline.image_encoder is not None:
        targets.append(pipeline.image_encoder)

    layers = [
        layer
        for target in targets
        for layer in flatten(target)
        if isinstance(layer, (torch.nn.Conv2d, torch.nn.ConvTranspose2d))
    ]

    for layer in layers:
        layer.padding_mode = "circular"


def main(model_id="SG161222/RealVisXL_V4.0"):
    pipeline = StableDiffusionXLPipeline.from_pretrained(
        model_id, torch_dtype=torch.float16, use_safetensors=True
    )

    prompt = "A seamless pattern showcasing a nature inspired graphic print with flowers and vines"
    pipeline.enable_model_cpu_offload()

    seamless_tiling(pipeline=pipeline)
    image = pipeline(
        prompt=prompt,
        width=1024,
        height=1024,
        num_inference_steps=40,
        guidance_scale=3,
        num_images_per_prompt=1,
    ).images[0]

    torch.cuda.empty_cache()
    image.save(f"image.png")


if __name__ == "__main__":
    main()

Output:

Tiled Output:

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sayakpaul]

Cc: @asomoza WDYT about this?

[asomoza]

we can close this now, we added it as a community script with this PR