Code

Phi2 rewrite#1058

Code

Merged

winglian merged 21 commits intomainaxolotl-ai-cloud/axolotl:mainfrom phi2-rewriteaxolotl-ai-cloud/axolotl:phi2-rewriteCopy head branch name to clipboardJan 8, 2024

+228-97

Commits

Commits on Jan 7, 2024

• restore to current phi modeling code from phi-2

  winglian

  committed
  f2fb132

• enable gradient checkpointing

  winglian

  committed
  1d4e2ac

• don't cast everything to float32 all the time

  winglian

  committed
  421c319

• gradient checkpointing for phi2 ParallelBlock module too

  winglian

  committed
  ba3214b

• fix enabling flash attn for phi2

  winglian

  committed
  6b14db8

• add comment about import

  winglian

  committed
  158b369

• fix phi2 example

  winglian

  committed
  1e4ff39

• fix model type check for tokenizer

  winglian

  committed
  e24c7b2

Commits on Jan 8, 2024

• revert float32 -> bf16 casting changes

  winglian

  committed
  18f1731

• support fused dense flash attn

  winglian

  committed
  066ced8

• fix the repo for flash-attn

  winglian

  committed
  c30396f

• add package name for subdir pkg

  winglian

  committed
  83c8060

• fix the data collator when not using sample packing

  winglian

  committed
  b4c6158

• install packaging for pytests in ci

  winglian

  committed
  95abeee

• also fix setup to not install flash attn fused dense subdir if not extras

  winglian

  committed
  3ac199e

• split out the fused-dense-lib in extra requires

  winglian

  committed
  a174ca3

• don't train w group_by_length for phi

  winglian

  committed
  a4cd052

• update integration test to use phi2

  winglian

  committed
  b8f6d1e

• set max steps and save steps for phi e2e tests

  winglian

  committed
  649b45e

• try to workaround ssave issue in ci

  winglian

  committed
  632413d

• skip phi2 e2e test for now

  winglian

  committed
  c923282