Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upstream Main: Fused Ops and Kernels, FSDP and Memory Fixes #35

Merged
merged 8 commits into from
Jun 7, 2024
Merged

Upstream Main: Fused Ops and Kernels, FSDP and Memory Fixes #35

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
c3c1cdd
Linting and Formatting for FMS-Acceleration-Peft package (#24)
achew010 May 28, 2024
25171a0
Workaround Low-Mem-Mode Patch for GPTQ-LoRA (#26)
achew010 May 29, 2024
70cdf71
Group memory field names with prefix and minor fixes (#27)
achew010 May 30, 2024
06a1af9
Initial Addition of FusedOps and Kernels Plugin With Model Patcher (#25)
fabianlim May 30, 2024
b2b8fe6
fix fsdp casting issue for autogptq and fused ops (#28)
fabianlim May 30, 2024
8103238
Add MLP & QLoRA Fused Ops and Kernels, Mixtral (#29)
fabianlim Jun 2, 2024
00febdc
Address Incorrect Ignoring of Base Layer Modules for FSDP with Kernel…
fabianlim Jun 7, 2024
bfde526
Shift GPU Memory Computation to End of Benchmarking Script (#30)
achew010 Jun 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/format.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@ jobs:
matrix:
plugin_name:
- "framework"
# - "accelerated-peft" # enable later
- "accelerated-peft"
- "fused-ops-and-kernels"

steps:
- uses: actions/checkout@v4
Expand Down
4 changes: 1 addition & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ For example:
- GPTQ-LoRA: 22-44 % token throughput increase on 1 GPU as compared to using Hugging Face BNB QLoRA
- GPTQ-LoRA: Straightforward integration with multiple GPU as compared to using Hugging Face BNB QLoRA

*Huggingface BNB QLoRA numbers taken with legacy approaches, but we are aware of [this issue](https://github.com/foundation-model-stack/fms-acceleration/issues/10) and will update our benches*.
*The above includes numbers using fusedOps-and-kernels and actual impl coming soon, see below*.

**This package is in BETA and is under development. Expect breaking changes!**
Expand All @@ -32,7 +31,7 @@ Plugin | Description | Depends | License | Status
--|--|--|--|--
[framework](./plugins/framework/README.md) | This acceleration framework for integration with huggingface trainers | | | Beta
[accelerated-peft](./plugins/accelerated-peft/README.md) | For PEFT-training, e.g., 4bit QLoRA. | Huggingface<br>AutoGPTQ | Apache 2.0<br>MIT | Beta
fusedOps-and-kernels | Fused LoRA and triton kernels (e.g., fast cross-entropy, rms, rope) | -- | Apache 2.0 with exclusions. | Coming Soon
[fused-op-and-kernels](./plugins/fused-ops-and-kernels/README.md) | Fused LoRA and triton kernels (e.g., fast cross-entropy, rms, rope) | -- | Apache 2.0 [(contains extracted code)](./plugins/fused-ops-and-kernels/README.md#code-extracted-from-unsloth)| Beta
MOE-training-acceleration | [MegaBlocks](https://github.com/databricks/megablocks) inspired triton Kernels and acclerations for Mixture-of-Expert models | | Apache 2.0 | Coming Soon

## Usage with FMS HF Tuning
Expand Down Expand Up @@ -175,7 +174,6 @@ The benchmarks can be reproduced [with the provided scripts](./scripts/benchmark

See below CSV files for various results:
- [A100-80GB](./scripts/benchmarks/refs/a100_80gb.csv).
- [L40-40GB](./scripts/benchmarks/refs/l40_40gb.csv).

### Code Architecture

Expand Down
Loading
Loading