Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Universal Speculative Decoding CandidateGenerator #35029

Open
wants to merge 72 commits into
base: main
Choose a base branch
from
+581 −16
Open

Universal Speculative Decoding CandidateGenerator #35029

Show file tree
Hide file tree
Changes from 60 commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
aa7e01a
move `TestAssistedCandidateGeneratorDifferentTokenizers` into a new t…
keyboardAnt Nov 28, 2024
f6b7f20
refactor
keyboardAnt Nov 28, 2024
0ded37c
NOTHING. add space to rerun github actions tests
keyboardAnt Nov 28, 2024
d48b69b
remove it...
keyboardAnt Nov 28, 2024
b47e33a
`UniversalSpeculativeDecodingGenerator`
keyboardAnt Nov 16, 2024
8a99129
Use `UniversalSpeculativeDecodingGenerator` when `generation_config.d…
keyboardAnt Nov 16, 2024
4649bd2
assistant tokenizes only the target's new suffix
keyboardAnt Nov 16, 2024
f199c94
formatting
keyboardAnt Nov 16, 2024
19c0057
fix code
jmamou Nov 21, 2024
acf5a4b
fix code
jmamou Nov 24, 2024
3712117
formatting
keyboardAnt Nov 24, 2024
63f2f46
add `TestGenerateWithDifferentModels`
keyboardAnt Nov 24, 2024
6ac33f1
`TestGenerateWithDifferentModels` parameterize on `do_sample`
keyboardAnt Nov 24, 2024
6938311
`AssistantVocabMapping` & `AssistantVocabMappingCache`
keyboardAnt Nov 24, 2024
5a0db3b
formatting
keyboardAnt Nov 24, 2024
92f8ad3
`AssistantToTargetTranslator`: `get_target_input_ids` & `get_target_l…
keyboardAnt Nov 24, 2024
7c8708e
improve `_get_assistant_to_target_input_ids` & formatting
keyboardAnt Nov 24, 2024
880d0ae
renaming
keyboardAnt Nov 24, 2024
d9b5e74
WIP: debugging `min_new_tokens`
keyboardAnt Nov 25, 2024
25974d5
fix get_target_ids
jmamou Nov 25, 2024
b8636ab
`UniversalSpeculativeDecodingGenerator`
keyboardAnt Nov 16, 2024
1ef46b7
assistant tokenizes only the target's new suffix
keyboardAnt Nov 16, 2024
f8e94eb
formatting
keyboardAnt Nov 16, 2024
439db84
fix code
jmamou Nov 21, 2024
643901d
fix code
jmamou Nov 24, 2024
77097ff
formatting
keyboardAnt Nov 24, 2024
d08b4f0
`TestGenerateWithDifferentModels` parameterize on `do_sample`
keyboardAnt Nov 24, 2024
f242dc1
`AssistantVocabMapping` & `AssistantVocabMappingCache`
keyboardAnt Nov 24, 2024
ede1176
formatting
keyboardAnt Nov 24, 2024
511ee96
`AssistantToTargetTranslator`: `get_target_input_ids` & `get_target_l…
keyboardAnt Nov 24, 2024
5e47945
improve `_get_assistant_to_target_input_ids` & formatting
keyboardAnt Nov 24, 2024
25a4349
renaming
keyboardAnt Nov 24, 2024
95fe744
WIP: debugging `min_new_tokens`
keyboardAnt Nov 25, 2024
0ad88b2
fix get_target_ids
jmamou Nov 25, 2024
bc5fa61
fix device issue
jmamou Nov 25, 2024
41a5670
fix get_assistant_input_ids
jmamou Nov 25, 2024
44f7ba7
add `TestAssistedCandidateGeneratorDifferentTokenizers`
keyboardAnt Nov 26, 2024
57aafcc
formatting
keyboardAnt Nov 26, 2024
6f95c33
`AssistantVocabTranslatorCache` refactor & tests
keyboardAnt Nov 26, 2024
078f763
revert changes in `src/transformers/generation/logits_process.py`
keyboardAnt Nov 26, 2024
faac2fc
refactor `AssistedCandidateGenerator`
keyboardAnt Nov 26, 2024
76a2dd3
refactor `AssistedCandidateGeneratorDifferentTokenizers`
keyboardAnt Nov 26, 2024
43e96e7
formatting
keyboardAnt Nov 26, 2024
e63cb9d
refactor `UniversalSpeculativeDecodingGenerator`
keyboardAnt Nov 26, 2024
8aa6020
fix negative value for max_new_tokens
jmamou Nov 26, 2024
2169973
fix generation length target + attention_mask vs. assistant + attent
jmamou Nov 26, 2024
c6da827
fix device
jmamou Nov 26, 2024
2cf9e8e
fix negative max_new_tokens bug
jmamou Nov 27, 2024
a1c0d05
fix UAG
jmamou Nov 28, 2024
d830091
minor
jmamou Nov 28, 2024
19d0cce
formatting
keyboardAnt Nov 28, 2024
5b8217d
`AssistedCandidateGeneratorDifferentTokenizers` `lookbehind`s init
keyboardAnt Nov 28, 2024
9b0126a
resolve conflict & formatting
keyboardAnt Nov 30, 2024
578d0b3
rerun CI tests
keyboardAnt Nov 30, 2024
7db2695
remove space...
keyboardAnt Nov 30, 2024
fb69900
remove old code
keyboardAnt Dec 3, 2024
e40c775
fix candidate_input_ids device
jmamou Dec 4, 2024
b5ce873
minor
jmamou Dec 4, 2024
bfccdea
Merge pull request #4 from keyboardAnt/fix_device
keyboardAnt Dec 5, 2024
d34d7ea
formatting
keyboardAnt Dec 5, 2024
9d4d9f9
Fix prepare + apply (#7)
jmamou Dec 17, 2024
4e92e9c
Add unittests for Universal Assisted generation
gauravj14 Dec 12, 2024
3fe2d31
Merge branch 'main' into usd
jmamou Dec 18, 2024
a350b1c
fix style
jmamou Dec 18, 2024
e047adf
update tests
jmamou Dec 18, 2024
011f595
Remove unused import and fix `test_speculation_depth` test
gauravjain14 Dec 17, 2024
2652490
exclude special and reserved tokens from tokenizer for UAG
gauravjain14 Dec 18, 2024
701edbb
mv `test_universal_assisted_generation.py` to `generation/test_candid…
gauravjain14 Dec 19, 2024
7088978
Merge pull request #8 from keyboardAnt/unit_tests_usd
gauravjain14 Dec 19, 2024
3b89341
Remove unused imports and fix style using `make style` (#9)
gauravjain14 Dec 20, 2024
e43dba8
formatting
keyboardAnt Dec 21, 2024
a529795
Swap gated `meta-llama/llama-3.2` with `allenai/llama` (#10)
gauravjain14 Dec 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading