-
Notifications
You must be signed in to change notification settings - Fork 429
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Integrate PEFT LoRA with HuggingFaceModel #2829
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! thanks for adding the tiny mistral fixtures and better typing throughout!
Co-authored-by: Irene Dea <[email protected]>
Co-authored-by: Irene Dea <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does the huggingface class actually need to be in Composer? I suppose it's a useful general utility...
1 minor nit. I think this makes sense otherwise
This PR integrates PEFT LoRA with the
HuggingFaceModel
base class.Manual test (requires a Foundry PR also), comparing the current lora implementation on foundry main (no fsdp), this implementation (no fsdp), and this implementation (with fsdp)
TODO:
mpt-7b-lora-after-fsdp-14-vcRBaL
no-lora-after-3-ObUYGR
mpt-7b-lora-after-fsdp-14-vcRBaL