Code

Add support for auto packing ratio#683

Code

Merged

irenedea merged 26 commits intomosaicml:mainmosaicml/llm-foundry:mainfrom irenedea:packing-collatorirenedea/llm-foundry:packing-collatorCopy head branch name to clipboardNov 5, 2023

+587-154

Commits

Commits on Sep 22, 2023

• Add support for auto packing ratio

  irenedea

  committed
  6d53fca

Commits on Oct 17, 2023

• Add test

  irenedea

  committed
  1c0f157

Commits on Oct 19, 2023

• Refactor and change to generator

  irenedea

  committed
  b32bac0

• Add simple tests

  irenedea

  committed
  d9dcdbc

• Add auto packing tests

  irenedea

  committed
  93d7926

• Add auto packing to test_dataloader

  irenedea

  committed
  ec71fec

• Merge branch 'main' into packing-collator

  irenedea

  authored
  239fce4

• use correct max leftovers to keep

  irenedea

  committed
  0db972a

Commits on Oct 20, 2023

• Handle dataspec change

  irenedea

  committed
  6c321d3

• Merge branch 'main' into packing-collator

  irenedea

  authored
  d6793bb

• Add dataloader test

  irenedea

  committed
  a852c23

Commits on Oct 21, 2023

• Add distributed autopacking

  irenedea

  committed
  d48fb97

• Update comments for profile_packing script refactor

  irenedea

  committed
  8c08405

• add torch cuda check

  irenedea

  committed
  6aab1ad

• Use 0 workers for profiling because one batch is loaded per worker and we only load a single batch.

  irenedea

  committed
  aeffb4b

• Merge branch 'main' into packing-collator

  irenedea

  authored
  044bb00

Commits on Oct 24, 2023

• Fix code quality

  irenedea

  committed
  96b4829

• Merge branch 'main' into packing-collator

  irenedea

  authored
  2ad0c31

Commits on Oct 25, 2023

• Merge branch 'main' into packing-collator

  irenedea

  authored
  83e8d3a

Commits on Oct 28, 2023

• Merge branch 'main' into packing-collator

  dakinggg

  authored
  f8ba32f

Commits on Nov 2, 2023

• Merge branch 'main' into packing-collator

  irenedea

  authored
  1ee68b8

• Address PR comments

  irenedea

  committed
  d88cdcc

• Merge branch 'main' into packing-collator

  irenedea

  authored
  913c47f

Commits on Nov 3, 2023

• Set random seed for auto packing to make it deterministic

  irenedea

  committed
  57cb170

• Fix typo

  Show description for de6b45d
  

  irenedea

  and

  dakinggg

  authored
  de6b45d

Commits on Nov 4, 2023

• Update max_leftover_bins_to_keep to keep all and remove unused variables

  irenedea

  committed
  2ff88c2