Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support SageAttention #1820

Open
sdbds opened this issue Dec 4, 2024 · 4 comments
Open

Support SageAttention #1820

sdbds opened this issue Dec 4, 2024 · 4 comments

Comments

@sdbds
Copy link
Contributor

sdbds commented Dec 4, 2024

image

Looks like a faster method similar to xformers and SDP
https://github.com/thu-ml/SageAttention

image

@kohya-ss
Copy link
Owner

kohya-ss commented Dec 4, 2024

This is very interesting. It appears to be a quantized version, so the accuracy may be inferior, but this shouldn't be an issue if we use --fp8_base.

I'm curious if Windows is supported.

@67372a
Copy link

67372a commented Dec 4, 2024

Triton is listed as a requirement, there are working wheels for Triton for Windows at https://github.com/woct0rdho/triton-windows

@kohya-ss
Copy link
Owner

kohya-ss commented Dec 9, 2024

I confirmed that sageattention works on Windows by following the steps here: https://www.reddit.com/r/StableDiffusion/comments/1h7hunp/how_to_run_hunyuanvideo_on_a_single_24gb_vram_card/

Documenting these steps would be difficult, so I will not do so.
However, if the user has installed SageAttention themselves, it may be possible to make the scripts compatible with SageAttention.

The problem is that some scripts have SDPA hardcoded, so I think introducing a global option would be a solution.

@67372a
Copy link

67372a commented Dec 9, 2024

I did do a quick test with SDXL and Flux loras, the approach to have it replace SDPA globally (as below) doesn't work due to incompatible head_dims.

from sageattention import sageattn
import torch.nn.functional as F

F.scaled_dot_product_attention = sageattn

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants