Support PEFT models in pipelines

[Rocketknight1]

Our pipelines check for supported classes but aren't PEFT-aware, so they get confused and throw errors for PEFT models. This PR changes the supported class check to use the actual underlying model of a PEFT model.

Fixes #29395

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Rocketknight1]

Hi @B-Step62, this branch should fix the issue now! Try pip install --upgrade git+https://github.com/huggingface/transformers.git@fix_peft_model_in_pipelines and let me know if you have any issues. The example you sent me seems to be working now.

If you don't find any further problems, I'll ping for core maintainer review and merge this soon.

[B-Step62]

Thanks for the quick fix, @Rocketknight1! I have tested my examples and confirmed the error message no longer is printed:)

[Rocketknight1]

cc @amyeroberts for core maintainer review in that case

[amyeroberts]

Thanks for adding this support!

Change looks good - just a request to abstract out the peft model check

[amyeroberts]

Hmmm, checking if Peft is in the name seems like a flaky way to check this.

What I'd suggest is adding a more general is_peft_model utility under modeling_utils.py which is tested (and we check with @younesbelkada is correct) which can then be used everywhere. It should also check is_peft_available.

[younesbelkada]

yes you can do if the model is an isntance of Peftxxx class, however we do support already peft model inference: https://huggingface.co/docs/transformers/peft , users that pass a Peftxxx class can be considered as bad intent, as they only need to pass a valid path to an adapter model or first load a peft model using AutoModelForxxx and pass that to the pipeline. See for example:

transformers/tests/peft_integration/test_peft_integration.py

Line 441 in b45c0f5

def test_peft_pipeline(self):

[younesbelkada]

Thanks for the PR !
IMO we shouldn't support that and rather educate the users properly on how to use PEFT + pipeline --> first load the peft adapter with AutoModelxxx interface then pass that model to the pipeline or pass a valid path to a Peft model!
See my comment here: #29517 (comment)

[younesbelkada]

Also note creating a brand new PEFT model with get_peft_model will lead to having a model with random adapter weights attached on it, therefore you'll mostl likely get gibberish output, I think that's also a bad intent as well :/
The typical workflow should be
1- train your adapters with PEFT
2- push them / save them somewhere after training
3- load them back with AutoModelxxx interface
4- do whatever
What do you think?

[Rocketknight1]

My bad! Your solution sounds good - I guess we can close this PR and just post something in the original issue at #29395

[younesbelkada]

ok ok sounds good! sorry for being responsive late 😭 ok yes the proposed plan sounds good!

[B-Step62]

Thank you for working on this issue @younesbelkada @Rocketknight1 !

2- push them / save them somewhere after training
3- load them back with AutoModelxxx interface

Unfortunately, this solution doesn't work as it is in our case. We want to save/load the adapter (+ the base model and tokenizer) locally as pipeline, without needing to interaction with the hub, to avoid the flakiness of downloading heavy files over network. Let me ask a few follow-up questions with this regards:

• Is there a way to casting the PEFT model into AutoModelxxx without pushing to the hub?
• Also vice-versa, once loading the model as AutoModelxxx, can we distinguish it is a PEFT model? It seems is_peft_model uses isinstance so my guess is it doesn't work.

[younesbelkada]

Thanks @B-Step62 for the clarification !
1- Yes you can also save it locally
2- When you load a peft model as AutoModelxxx you can distinguish if it's a peft model by looking into the attribute _hf_peft_config_loaded: getattr(model, "_hf_peft_config_loaded", False)