Remove unnecessary array allocations in generation process and enable caching #308
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
brandonwillard
force-pushed
the
fix-sequence-scaling
branch
from
8f7836d
to
1a5082f
Compare
brandonwillard
commented
Oct 2, 2023
brandonwillard
added
text
brandonwillard
force-pushed
the
fix-sequence-scaling
branch
6 times, most recently
from
dc551ff
to
d55bbc1
Compare
brandonwillard
changed the title
brandonwillard
force-pushed
the
fix-sequence-scaling
branch
from
d55bbc1
to
50352a7
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR removes some unnecessary array allocations during the generation process that affect scaling in max tokens and adds KV caching.
Perhaps the biggest non-cache-based change is that the method
Sequence.update_token_idshas been removed; otherwise, the dimensions for arrays returned bySequence.stepare fixed (i.e. no squeezing). This makes the dimensions inSequence.__call__clearer and allows us to simplify the loop (e.g. no need to duplicate the steps in the auto-regression loop before starting the loop).