Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Preprocessing at scale #291

Merged
merged 17 commits into from
Jan 22, 2025
Merged

Preprocessing at scale #291

merged 17 commits into from
Jan 22, 2025

Conversation

le1nux
Copy link
Member

@le1nux le1nux commented Jan 15, 2025 •
edited
Loading

What does this PR do?

This PR adds functionality (including endpoint) to create chunks from a set of tokenized file.
Each tokenized file is split into the targeted number of chunks. We combine the splits (same split_id) from all files into the respective chunk. This means each chunk contains one split from each tokenized file.

General Changes

  • Dataset chunking (including random shuffling of individual chunks)
  • Added TokenizedFileWriter for writing out tokenized data (list of numpy arrays) in the pbin format.
  • Added slicing functionality to the __get_item function of PackedMemMapDatasetBase

Breaking Changes

  • ..

Checklist before submitting final PR

  • My PR is minimal and addresses one issue in isolation
  • I have merged the latest version of the target branch into this feature branch
  • I have reviewed my own code w.r.t. correct implementation, missing type hints, proper documentation, etc.
  • I have run a sample config for model training
  • I have checked that all tests run through (python tests/tests.py)
  • I have updated the internal changelog (CHANGELOG_DEV.md)

le1nux added 14 commits January 15, 2025 21:31
@le1nux
feat: implemented chunking calculation
bc01afe
@le1nux
chore: Merge branch 'main' into preprocessing_at_scale
0ff30e1
@le1nux
feat: added slicing to PackedMemMapDatasetBase
e938ce4
@le1nux
feat: implemented TokenizedFileWriter
9446a0f
@le1nux
feat: added chunk shuffling
ba33e25
@le1nux
feat: addded create_shuffled_dataset_chunk api endpoint
36347ff
@le1nux
feat: added test for Chunking.get_file_chunk
e43739a
@le1nux
feat: added test for shuffle_file_chunks_in_place
ca2b298
@le1nux
refactor: fixed faulty index in lorem_ipsum_long.pbin
c75a2c5
@le1nux
feat: added test for TokenizedFileWriter.write_tokenized_dataset
376fd4f
@le1nux
feat: added more testing to TokenizedFileWriter.write_tokenized_dataset
42a6cae
@le1nux
feat: added end to end test test_create_shuffled_dataset_chunk
3d17633
@le1nux
feat: added slicing tests for PackedMemMapDatasetBase
781a008
@le1nux
fix: fixed failing test test_skipped_and_distributed_dataloader_from_…
65477cd 
…config
@le1nux le1nux requested review from mali-git and fromm-m January 20, 2025 10:25
@le1nux le1nux added the enhancement New feature or request label Jan 20, 2025
@le1nux le1nux marked this pull request as ready for review January 20, 2025 10:25
fromm-m
fromm-m approved these changes Jan 20, 2025
View reviewed changes
Copy link
Member

@fromm-m fromm-m left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

mali-git
mali-git requested changes Jan 21, 2025
View reviewed changes
src/modalities/__main__.py Outdated Show resolved Hide resolved
src/modalities/__main__.py Outdated Show resolved Hide resolved
src/modalities/api.py Outdated Show resolved Hide resolved
src/modalities/api.py Show resolved Hide resolved
src/modalities/api.py Outdated Show resolved Hide resolved
src/modalities/dataloader/dataset.py Outdated Show resolved Hide resolved
src/modalities/dataloader/preprocessing/chunking/create_chunks.py Show resolved Hide resolved
src/modalities/dataloader/preprocessing/chunking/create_chunks.py Outdated Show resolved Hide resolved
src/modalities/dataloader/preprocessing/tokenization/tokenized_file_writer.py Outdated Show resolved Hide resolved
src/modalities/dataloader/preprocessing/tokenization/tokenized_file_writer.py Outdated Show resolved Hide resolved
@le1nux le1nux requested a review from mali-git January 22, 2025 10:33
mali-git
mali-git approved these changes Jan 22, 2025
View reviewed changes
Copy link
Member

@mali-git mali-git left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

@le1nux le1nux merged commit fe4a3be into main Jan 22, 2025
3 checks passed
@le1nux le1nux deleted the preprocessing_at_scale branch January 22, 2025 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fromm-m fromm-m fromm-m approved these changes

@mali-git mali-git mali-git approved these changes

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@le1nux @mali-git @fromm-m