Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Dropout in Fused LoRA Operations #102

Merged
merged 6 commits into from
Nov 8, 2024
Merged

Fix Dropout in Fused LoRA Operations #102

merged 6 commits into from
Nov 8, 2024

Conversation

fabianlim
Copy link
Contributor

@fabianlim fabianlim commented Nov 6, 2024

This PR fixes #97.

  • update bnb and auto_gptq fused-lora to correctly account for dropout in the backward.
  • fix tests to correct patch model, and compute and test input gradients for test_adapter_gradients_match_with_attention_layer
  • unfortunately we cannot test input_grads for GPTQ in test_adapter_gradients_match_with_attention_layer, this is a limitation of unable to properly load a small GPTQ model.

In addition we also disable the MLP fused op for certain models, and removed the old fast_quantized_peft plugin. We defer this to another PR

Regerssions

Looks quite good. The node I got is a little slower, but everything seems to be inline.

image image
image image
benchmarks.csv

Signed-off-by: Yu Chin Fabian Lim <[email protected]>
Signed-off-by: Yu Chin Fabian Lim <[email protected]>
Signed-off-by: Yu Chin Fabian Lim <[email protected]>
Signed-off-by: Yu Chin Fabian Lim <[email protected]>
Signed-off-by: Yu Chin Fabian Lim <[email protected]>
Signed-off-by: Yu Chin Fabian Lim <[email protected]>
@fabianlim fabianlim changed the title Fix Dropout in Fix Dropout in Fused LoRA Operations Nov 6, 2024
@fabianlim fabianlim requested a review from anhuong November 6, 2024 17:04
@fabianlim fabianlim changed the title Fix Dropout in Fused LoRA Operations Fix Dropout in Fused LoRA Operations, Dsiable MLP Fused Op for non-SwiGLU, Remove Fast Quantized Peft Nov 8, 2024
@fabianlim fabianlim force-pushed the fix/lora-drop branch 2 times, most recently from 263c75e to b6a8d21 Compare November 8, 2024 05:34
@fabianlim fabianlim changed the title Fix Dropout in Fused LoRA Operations, Dsiable MLP Fused Op for non-SwiGLU, Remove Fast Quantized Peft Fix Dropout in Fused LoRA Operations Nov 8, 2024
@fabianlim fabianlim merged commit d767e33 into main Nov 8, 2024
11 checks passed
@fabianlim fabianlim deleted the fix/lora-drop branch November 8, 2024 11:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Dropout for LoRA Fused Ops Incorrectly Implemented. Also should be more careful about activation
1 participant