Skip to content

Commit

Permalink
put back bf16 for baseline bnb
Browse files Browse the repository at this point in the history
Signed-off-by: Yu Chin Fabian Lim <[email protected]>
  • Loading branch information
fabianlim committed Oct 13, 2024
1 parent 50e7e20 commit a50ff63
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 1 deletion.
2 changes: 1 addition & 1 deletion plugins/accelerated-peft/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Plugin | Description | Depends | Loading | Augmentation | Callbacks


### Key Points
- fix upcasting (resulting in slowdown) issue for `bnb` plugin, originally discovered by inventors of [Unsloth](https://unsloth.ai/blog/mistral-benchmark).
- fix upcasting (resulting in slowdown) issue for `bnb` plugin, originally discovered by inventors of [Unsloth](https://unsloth.ai/blog/mistral-benchmark). **NOTE**: we recommend using *mixed precision* when using 4bit quant for better performance, as per our benchmarks.
- `bnb` properly configured to work with FSDP following [this guide](https://huggingface.co/docs/bitsandbytes/main/en/fsdp_qlora).
- `triton_v2` kernels are not yet properly integrated into huggingface optimum.
- `triton_v2` kernels are [the only 4bit kernels that work for training](https://github.com/AutoGPTQ/AutoGPTQ/issues/633).
Expand Down
1 change: 1 addition & 0 deletions scripts/benchmarks/scenarios.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ scenarios:
framework_config:
- baseline-peft-bnb
arguments:
bf16: True
learning_rate: 2e-4
torch_dtype: bfloat16
peft_method: lora
Expand Down

0 comments on commit a50ff63

Please sign in to comment.