Code

update runners#3472

Code

Closed

KevDevSha wants to merge 70 commits intodevmosaicml/composer:devfrom kevin/emu-allowlisted-runnersmosaicml/composer:kevin/emu-allowlisted-runnersCopy head branch name to clipboard

+76-58

Commits

Commits on Jul 21, 2024

• bump (#3383)

  Show description for e72bbaf
  

  2 peopleauthored and

  mvpatel2000

  committed
  e72bbaf

  Copy full SHA for e72bbaf

• Fix backward compatibility caused by missing eval metrics class (#3385)

  Show description for 9c4b0ba
  

  3 peoplecommitted
  9c4b0ba

  Copy full SHA for 9c4b0ba

• Bump version v0.23.2 (#3386)

  Show description for e85e738
  

  bigning

  authored and

  mvpatel2000

  committed
  e85e738

  Copy full SHA for e85e738

• Restore dev version (#3388)

  Show description for afa2e39
  

  bigning

  authored and

  mvpatel2000

  committed
  afa2e39

  Copy full SHA for afa2e39

• Only requires databricks-sdk when inside the Databricks platform (#3389)

  

  antoinebrl

  authored and

  mvpatel2000

  committed
  4cbb4a2

  Copy full SHA for 4cbb4a2

• Update packaging requirement from <24.1,>=21.3.0 to >=21.3.0,<24.2 (#3392)

  Show description for 735aa6f
  

  dependabot[bot]

  authored and

  mvpatel2000

  committed
  735aa6f

  Copy full SHA for 735aa6f

• Bump cryptography from 42.0.6 to 42.0.8 (#3391)

  Show description for db1325a
  

  dependabot[bot]

  authored and

  mvpatel2000

  committed
  db1325a

  Copy full SHA for db1325a

• Skip extra dataset state load (#3393)

  Show description for 7778fcf
  mvpatel2000

  committed
  7778fcf

  Copy full SHA for 7778fcf

• Remove FSDP restriction from PyTorch 1.13 (#3395)

  Show description for 919fe91
  mvpatel2000

  committed
  919fe91

  Copy full SHA for 919fe91

• Check for 'CUDA error: out of memory' with auto-microbatching (#3400)

  

  JAEarly

  authored and

  mvpatel2000

  committed
  b07b82e

  Copy full SHA for b07b82e

• Add tokens to iterations (#3374)

  

  b-chu

  authored and

  mvpatel2000

  committed
  6298d76

  Copy full SHA for 6298d76

• Busy wait utils in dist (#3396)

  

  dakinggg

  authored and

  mvpatel2000

  committed
  9500fd1

  Copy full SHA for 9500fd1

• Add buffering time to mlflow logger (#3401)

  Show description for a1c581d
  

  chenmoneygithub

  authored and

  mvpatel2000

  committed
  a1c581d

  Copy full SHA for a1c581d

• Update _patch_pytorch.py (#3402)

  mvpatel2000

  committed
  fffa335

  Copy full SHA for fffa335

• Add pynvml to mlflow dep group (#3404)

  

  dakinggg

  authored and

  mvpatel2000

  committed
  3e1396e

  Copy full SHA for 3e1396e

• min/max flagging added to system_metrics_monitor with only non-redundant, necessary gpu metrics logged (#3373)

  Show description for 0eb1eee
  

  3 peoplecommitted
  0eb1eee

  Copy full SHA for 0eb1eee

• simplify launcher (#3398)

  mvpatel2000

  committed
  0ee83f7

  Copy full SHA for 0ee83f7

• Optionally use flash-attn's CE loss for metrics (#3394)

  Show description for 04ba0b6
  

  3 peoplecommitted
  04ba0b6

  Copy full SHA for 04ba0b6

• log image fix (#3286)

  Show description for 1dfd3bc
  

  4 peoplecommitted
  1dfd3bc

  Copy full SHA for 1dfd3bc

• [ckpt-rewr] Save state dict API (#3372)

  

  eracah

  authored and

  mvpatel2000

  committed
  f7e17de

  Copy full SHA for f7e17de

• Revert "Optionally use flash-attn's CE loss for metrics (#3394)" (#3408)

  Show description for 0a1a6a4
  

  snarayan21

  authored and

  mvpatel2000

  committed
  0a1a6a4

  Copy full SHA for 0a1a6a4

• CPU tests image fix (#3409)

  Show description for 0d6ef26
  

  snarayan21

  authored and

  mvpatel2000

  committed
  0d6ef26

  Copy full SHA for 0d6ef26

• Add setter for epoch in iteration (#3407)

  

  b-chu

  authored and

  mvpatel2000

  committed
  dac1995

  Copy full SHA for dac1995

• Move pillow dep as required (#3412)

  Show description for 567c6e5
  mvpatel2000

  committed
  567c6e5

  Copy full SHA for 567c6e5

• fixing mlflow logging to Databricks workspace file paths with /Shared/ prefix (#3410)

  Show description for f26a1d3
  

  JackZ-db

  and

  mvpatel2000

  committed
  f26a1d3

  Copy full SHA for f26a1d3

• Bump version v0.23.3 (#3414)

  Show description for 894a192
  

  karan6181

  authored and

  mvpatel2000

  committed
  894a192

  Copy full SHA for 894a192

• Update numpy requirement from <1.27.0,>=1.21.5 to >=1.21.5,<2.1.0 (#3406)

  Show description for 459a019
  

  3 peoplecommitted
  459a019

  Copy full SHA for 459a019

• Restore dev version (#3417)

  

  karan6181

  authored and

  mvpatel2000

  committed
  7a4644a

  Copy full SHA for 7a4644a

• Save checkpoint to disk for API with new save layout (#3399)

  

  eracah

  authored and

  mvpatel2000

  committed
  94f1ec1

  Copy full SHA for 94f1ec1

• fix typing (#3419)

  mvpatel2000

  committed
  d420765

  Copy full SHA for d420765

• Fixes some typing issues (#3418)

  

  dakinggg

  authored and

  mvpatel2000

  committed
  ba17897

  Copy full SHA for ba17897

• Fix small things (#3420)

  

  b-chu

  authored and

  mvpatel2000

  committed
  4e8ed2e

  Copy full SHA for 4e8ed2e

• Bump coverage[toml] from 7.5.3 to 7.5.4 (#3422)

  Show description for 5ba56ac
  

  dependabot[bot]

  authored and

  mvpatel2000

  committed
  5ba56ac

  Copy full SHA for 5ba56ac

• Update psutil requirement from <6,>=5.8.0 to >=5.8.0,<7 (#3424)

  Show description for abfd78c
  

  dependabot[bot]

  and

  mvpatel2000

  committed
  abfd78c

  Copy full SHA for abfd78c

• Add support for variable length dataloaders in DDP (#3416)

  Show description for d3e95a9
  

  JAEarly

  and

  mvpatel2000

  committed
  d3e95a9

  Copy full SHA for d3e95a9

• Hsdp + MoE CI tests (#3378)

  Show description for 84c4723
  

  4 peoplecommitted
  84c4723

  Copy full SHA for 84c4723

• bumping mlflow to 2.14.1 (#3425)

  

  JackZ-db

  authored and

  mvpatel2000

  committed
  4501305

  Copy full SHA for 4501305

• Skip HSDP + TP pytests that require torch 2.3 or above (#3426)

  Show description for a7218d1
  

  2 peopleauthored and

  mvpatel2000

  committed
  a7218d1

  Copy full SHA for a7218d1

• remove codeql (#3429)

  mvpatel2000

  committed
  8361862

  Copy full SHA for 8361862

• Remove save overwrite (#3431)

  Show description for 0b74933
  mvpatel2000

  committed
  0b74933

  Copy full SHA for 0b74933

• LeDocs (#3430)

  

  snarayan21

  authored and

  mvpatel2000

  committed
  dd3e7f9

  Copy full SHA for dd3e7f9

• Lower the system metrics logging frequency to reduce MLflow server's load (#3436)

  Show description for ac4bd59
  

  chenmoneygithub

  authored and

  mvpatel2000

  committed
  ac4bd59

  Copy full SHA for ac4bd59

• Update paramiko requirement from <3,>=2.11.0 to >=3.4.0,<4 (#3439)

  Show description for 38e5e51
  

  dependabot[bot]

  authored and

  mvpatel2000

  committed
  38e5e51

  Copy full SHA for 38e5e51

• bump versions (#3433)

  mvpatel2000

  committed
  6b461d0

  Copy full SHA for 6b461d0

• fix eval after all (#3445)

  mvpatel2000

  committed
  6bac335

  Copy full SHA for 6bac335

• skip log (#3446)

  mvpatel2000

  committed
  3cd6e6d

  Copy full SHA for 3cd6e6d

• Remove MosaicMLLambdaEvalClient (#3432)

  

  aspfohl

  authored and

  mvpatel2000

  committed
  cf76c96

  Copy full SHA for cf76c96

• Relax hf hub pin (#3435)

  

  dakinggg

  authored and

  mvpatel2000

  committed
  8fbca38

  Copy full SHA for 8fbca38

• Pytest skip 2 (#3448)

  Show description for 54d58c9
  

  3 peoplecommitted
  54d58c9

  Copy full SHA for 54d58c9

• bump version (#3450)

  

  XiaohanZhangCMU

  authored and

  mvpatel2000

  committed
  5a129d1

  Copy full SHA for 5a129d1

• Bump ipykernel from 6.29.2 to 6.29.5 (#3459)

  Show description for a0806f6
  

  dependabot[bot]

  authored and

  mvpatel2000

  committed
  a0806f6

  Copy full SHA for a0806f6

• Update torchmetrics requirement (#3460)

  Show description for 4b71141
  

  dependabot[bot]

  authored and

  mvpatel2000

  committed
  4b71141

  Copy full SHA for 4b71141

• Bump databricks-sdk from 0.28.0 to 0.29.0 (#3456)

  Show description for 89db4e2
  

  dependabot[bot]

  authored and

  mvpatel2000

  committed
  89db4e2

  Copy full SHA for 89db4e2

• [Checkpoint] Fix symlink issue where symlink file uploaded before checkpoint files upload (#3376)

  Show description for 6df01ba
  

  2 peopleauthored and

  mvpatel2000

  committed
  6df01ba

  Copy full SHA for 6df01ba

• Correctly process parallelism_config['tp'] when it's a dict (#3434)

  Show description for e951f0a
  

  snarayan21

  and

  mvpatel2000

  committed
  e951f0a

  Copy full SHA for e951f0a

• [checkpoint v2] Download api (#3447)

  Show description for 6dec835
  

  bigning

  authored and

  mvpatel2000

  committed
  6dec835

  Copy full SHA for 6dec835

• removed exception from logger (#3464)

  

  jjanezhang

  authored and

  mvpatel2000

  committed
  18795f1

  Copy full SHA for 18795f1

• fixed docs for mfu (#3469)

  

  JackZ-db

  authored and

  mvpatel2000

  committed
  11bad57

  Copy full SHA for 11bad57

• add comment (#3470)

  mvpatel2000

  committed
  74c7d3b

  Copy full SHA for 74c7d3b

• Change pytorch eval for FP8 to default to fall back to BF16 (#3454)

  

  j316chuck

  authored and

  mvpatel2000

  committed
  14bc187

  Copy full SHA for 14bc187

• Fix checkpoint events (#3468)

  

  b-chu

  authored and

  mvpatel2000

  committed
  a5dc155

  Copy full SHA for a5dc155

• Add mosaicmllogger attr for fit start (#3467)

  

  ethanma-db

  authored and

  mvpatel2000

  committed
  69b8b23

  Copy full SHA for 69b8b23

• Bump coverage[toml] from 7.5.4 to 7.6.0 (#3471)

  Show description for 15c329e
  

  dependabot[bot]

  authored and

  mvpatel2000

  committed
  15c329e

  Copy full SHA for 15c329e

• Bump flash attention to 2.6.1 (#3476)

  

  dakinggg

  authored and

  mvpatel2000

  committed
  8a09a3b

  Copy full SHA for 8a09a3b

Commits on Jul 22, 2024

• cpu

  KevDevSha

  committed
  779ff3e

  Copy full SHA for 779ff3e

• gpu

  KevDevSha

  committed
  2c0eac2

  Copy full SHA for 2c0eac2

• Merge branch 'dev' into kevin/emu-allowlisted-runners

  KevDevSha

  authored
  bf1843a

  Copy full SHA for bf1843a

• coverage fix

  KevDevSha

  committed
  2b3ff8c

  Copy full SHA for 2b3ff8c

• lint

  KevDevSha

  committed
  bd6515c

  Copy full SHA for bd6515c

• Update pr-cpu.yaml

  KevDevSha

  authored
  4488555

  Copy full SHA for 4488555