Adding more token encoding types #1254
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
codestar12
reviewed
Jun 5, 2024
codestar12
suggested changes
Jun 5, 2024
XiaohanZhangCMU
approved these changes
Jun 5, 2024
dakinggg
reviewed
Jun 5, 2024
dakinggg
reviewed
Jun 6, 2024
…undry into saaketh/tok_encodings merging main
|
Ran the following scripts successfully: Using the shakespeare text file here, models trained with/without this branch have deterministic loss curves. One set of runs was with global batch size 32, the other with global batch size 256. See wandb project here. Foundry regression tests are partially borked right now because of a small bug that's getting addressed in the release branch, but the tests that did run all succeeded. See here. |
dakinggg
approved these changes
Jun 6, 2024
There was a problem hiding this comment.
Thanks @snarayan21! And thanks for leaving the documentation better than you found it :)
KuuCi
pushed a commit
that referenced
this pull request
Jun 7, 2024
* add more token encoing types * add more token encoing types * add tests * add tests * ft support, tests * linting is shortening my lifespan * linting is shortening my lifespan * long tensor * long tensor * long tensor * feedbacc * import * import --------- Co-authored-by: Daniel King <[email protected]> (cherry picked from commit 42c2d9a)
KuuCi
pushed a commit
that referenced
this pull request
Jun 7, 2024
* add more token encoing types * add more token encoing types * add tests * add tests * ft support, tests * linting is shortening my lifespan * linting is shortening my lifespan * long tensor * long tensor * long tensor * feedbacc * import * import --------- Co-authored-by: Daniel King <[email protected]>
KuuCi
pushed a commit
that referenced
this pull request
Jun 7, 2024
* add more token encoing types * add more token encoing types * add tests * add tests * ft support, tests * linting is shortening my lifespan * linting is shortening my lifespan * long tensor * long tensor * long tensor * feedbacc * import * import --------- Co-authored-by: Daniel King <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Depending on the vocab size, users can encode their token IDs using various int formats. Previously, we only allowed for int64, which covers an absurdly high vocab size. Enabling decoding tokens in uint32 or uint16, for example, would let people save space on their datasets since the max vocab sizes supported would be ~4 million with uint32, or ~65k with uint16. This has been added to both the text and finetuning datasets.
This PR also lets users specify their MDS dataset columns using
ndarraytypes to enable automatically encoding/decoding samples. This was already present for finetuning dataset, so the functionality has been added for the generic text dataset. Accordingly, I've changed the default value in our MDS conversion scripts to usendarray:uint32instead ofbytesand made relevant changes to get this working.Added unit tests checking that this works for text and finetuning datasets and that an error is thrown for uncompatible encoding types.
Moved a util function that was applicable to both the text and finetuning dataloaders to a common location to import. This had been written twice to perform the same functionality.
Ran the following scripts successfully:
convert_dataset_hf.py,convert_dataset_json.py,convert_finetuning_dataset.py,convert_text_to_mds.py. Updated data prep readme to have instructions forconvert_text_to_mds.py.Using the shakespeare text file here, models trained with/without this branch have deterministic loss curves. One set of runs was with global batch size 32, the other with global batch size 256. See wandb project here.
Foundry regression tests are partially borked right now because of a small bug that's getting addressed in the release branch, but the tests that did run all succeeded. See here.