Code

Preprocessing at scale#291

Code

Merged

le1nux merged 17 commits intomainModalities/modalities:mainfrom preprocessing_at_scaleModalities/modalities:preprocessing_at_scaleCopy head branch name to clipboardJan 22, 2025

+725-30

Commits

Commits on Jan 15, 2025

• feat: implemented chunking calculation

  le1nux

  committed
  bc01afe

  Copy full SHA for bc01afe

Commits on Jan 16, 2025

• chore: Merge branch 'main' into preprocessing_at_scale

  le1nux

  committed
  0ff30e1

  Copy full SHA for 0ff30e1

• feat: added slicing to PackedMemMapDatasetBase

  le1nux

  committed
  e938ce4

  Copy full SHA for e938ce4

Commits on Jan 17, 2025

• feat: implemented TokenizedFileWriter

  le1nux

  committed
  9446a0f

  Copy full SHA for 9446a0f

• feat: added chunk shuffling

  le1nux

  committed
  ba33e25

  Copy full SHA for ba33e25

Commits on Jan 18, 2025

• feat: addded create_shuffled_dataset_chunk api endpoint

  le1nux

  committed
  36347ff

  Copy full SHA for 36347ff

• feat: added test for Chunking.get_file_chunk

  le1nux

  committed
  e43739a

  Copy full SHA for e43739a

• feat: added test for shuffle_file_chunks_in_place

  le1nux

  committed
  ca2b298

  Copy full SHA for ca2b298

• refactor: fixed faulty index in lorem_ipsum_long.pbin

  le1nux

  committed
  c75a2c5

  Copy full SHA for c75a2c5

• feat: added test for TokenizedFileWriter.write_tokenized_dataset

  le1nux

  committed
  376fd4f

  Copy full SHA for 376fd4f

• feat: added more testing to TokenizedFileWriter.write_tokenized_dataset

  le1nux

  committed
  42a6cae

  Copy full SHA for 42a6cae

Commits on Jan 19, 2025

• feat: added end to end test test_create_shuffled_dataset_chunk

  le1nux

  committed
  3d17633

  Copy full SHA for 3d17633

• feat: added slicing tests for PackedMemMapDatasetBase

  le1nux

  committed
  781a008

  Copy full SHA for 781a008

• fix: fixed failing test test_skipped_and_distributed_dataloader_from_config

  le1nux

  committed
  65477cd

  Copy full SHA for 65477cd

Commits on Jan 21, 2025

• refactor: simplified slicing in PackedMemMapDatasetBase

  le1nux

  committed
  8722b16

  Copy full SHA for 8722b16

• refactor: improved code based on review comments

  le1nux

  committed
  d3a1795

  Copy full SHA for d3a1795

Commits on Jan 22, 2025

• chore: added missing type annotations

  le1nux

  committed
  4bc671b

  Copy full SHA for 4bc671b