Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate PEFT LoRA with HuggingFaceModel #2829

Merged
merged 76 commits into from
Jan 29, 2024
Merged

Conversation

dakinggg
Copy link
Contributor

@dakinggg dakinggg commented Jan 9, 2024

This PR integrates PEFT LoRA with the HuggingFaceModel base class.

Manual test (requires a Foundry PR also), comparing the current lora implementation on foundry main (no fsdp), this implementation (no fsdp), and this implementation (with fsdp)
Screenshot 2024-01-18 at 12 16 23 AM

TODO:

  • rerun lora test (after refactor) mpt-7b-lora-after-fsdp-14-vcRBaL
  • rerun non lora test (without peft installed, after refactor) no-lora-after-3-ObUYGR
  • test that a small checkpoint shows up in object store mpt-7b-lora-after-fsdp-14-vcRBaL

@dakinggg dakinggg requested a review from irenedea January 26, 2024 18:53
Copy link
Contributor

@irenedea irenedea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! thanks for adding the tiny mistral fixtures and better typing throughout!

tests/models/test_hf_model.py Outdated Show resolved Hide resolved
tests/models/test_hf_model.py Outdated Show resolved Hide resolved
Copy link
Contributor

@mvpatel2000 mvpatel2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the huggingface class actually need to be in Composer? I suppose it's a useful general utility...

1 minor nit. I think this makes sense otherwise

composer/models/huggingface.py Outdated Show resolved Hide resolved
@dakinggg dakinggg enabled auto-merge (squash) January 29, 2024 18:17
@dakinggg dakinggg merged commit 74c90bd into mosaicml:dev Jan 29, 2024
17 checks passed
ShashankMosaicML pushed a commit to ShashankMosaicML/composer that referenced this pull request Feb 3, 2024
ShashankMosaicML pushed a commit to ShashankMosaicML/composer that referenced this pull request Feb 3, 2024
@dakinggg dakinggg deleted the composer_lora branch February 10, 2024 07:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants