Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add sdpa and flash_attention2 support to speech2text #33716

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
+367 −17
Open

add sdpa and flash_attention2 support to speech2text #33716

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
5481df5
start sdpa for speech to text
Sep 25, 2024
2f02207
add flash attention as well.
Sep 25, 2024
1ba581c
add speech2text to docs
Sep 25, 2024
bd53345
move default tests to eager
Sep 25, 2024
41d5b3a
update encoder attention maks
Sep 25, 2024
0f7673a
bug fix for when to expand embeddings in speech2text
Sep 26, 2024
cb821cc
also fix the same bug in SpeechT5 beacuse of #Copied comment
Sep 26, 2024
855ed90
fixup
Sep 26, 2024
5900f1e
[run-slow] speech2text
Sep 26, 2024
9922454
add "test_flash_attn_2_generate_reuse_cache" inspired by whisper
Sep 26, 2024
b594f7b
Merge branch 'main' into sdpa_speech2text
Sep 26, 2024
54d24c4
[run-slow] speech2text
Sep 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/source/en/perf_infer_gpu_one.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ FlashAttention-2 is currently supported for the following architectures:
* [OPT](https://huggingface.co/docs/transformers/model_doc/opt#transformers.OPTModel)
* [Phi](https://huggingface.co/docs/transformers/model_doc/phi#transformers.PhiModel)
* [Phi3](https://huggingface.co/docs/transformers/model_doc/phi3#transformers.Phi3Model)
* [Speech2Text](https://huggingface.co/docs/transformers/en/model_doc/speech_to_text)
* [StableLm](https://huggingface.co/docs/transformers/model_doc/stablelm#transformers.StableLmModel)
* [Starcoder2](https://huggingface.co/docs/transformers/model_doc/starcoder2#transformers.Starcoder2Model)
* [Qwen2](https://huggingface.co/docs/transformers/model_doc/qwen2#transformers.Qwen2Model)
Expand Down Expand Up @@ -251,6 +252,7 @@ For now, Transformers supports SDPA inference and training for the following arc
* [Whisper](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperModel)
* [mBart](https://huggingface.co/docs/transformers/model_doc/mbart#transformers.MBartModel)
* [Mistral](https://huggingface.co/docs/transformers/model_doc/mistral#transformers.MistralModel)
* [Speech2Text](https://huggingface.co/docs/transformers/en/model_doc/speech_to_text)
* [Mixtral](https://huggingface.co/docs/transformers/model_doc/mixtral#transformers.MixtralModel)
* [StableLm](https://huggingface.co/docs/transformers/model_doc/stablelm#transformers.StableLmModel)
* [Starcoder2](https://huggingface.co/docs/transformers/model_doc/starcoder2#transformers.Starcoder2Model)
Expand Down
Loading