Support inputs_embeds #687
Conversation
in MPTForCausalLM. It is checked in the base model, and it is actually a common practice to pass both during autoregressive generation. Embeds are used first, then once the kvcache is nonempty, iids are used instead
|
@samhavens here is what HF expects wrt these two args: https://github.com/huggingface/transformers/blob/9333bf0769561c048700377c2e0813221ab9d2c9/src/transformers/models/llama/modeling_llama.py#L955-L963 |
@dakinggg That validation happens in the case model, but it the CausalLM model they can both be there https://github.com/huggingface/transformers/blob/9333bf0769561c048700377c2e0813221ab9d2c9/src/transformers/models/llama/modeling_llama.py#L1140-L1150 I think this is because in |
@dakinggg do you mean new tests or add these edge cases to the 2 tests that have input embeds |
|
@samhavens add these edge cases to existing tests is fine (assuming that fits the tests). Anything that tests the right thing happens for those combinations is sufficient. |
* support inputs_embeds * update tests to test inputs_embeds * make iids optional inputs to fwd * remove check for both iids and inputs_embeds in MPTForCausalLM. It is checked in the base model, and it is actually a common practice to pass both during autoregressive generation. Embeds are used first, then once the kvcache is nonempty, iids are used instead * reorder kwargs * add more tests * fix device merge artifact in test_model.oy * fix generate test * yapf
* Add eval loader to eval script * small input tests * updates * fix typing and formatting * fixes, add tests * remove circular dependency * tests pass * nits + small fixes * add metrics at the end, refactor to put icl/gauntlet as helpers * NOT * metrics instead of models, add unit tests * Move tests into directories * add copyright to inits * fix relative paths * fixes * revert gauntlet test change * Support inputs_embeds (#687) * support inputs_embeds * update tests to test inputs_embeds * make iids optional inputs to fwd * remove check for both iids and inputs_embeds in MPTForCausalLM. It is checked in the base model, and it is actually a common practice to pass both during autoregressive generation. Embeds are used first, then once the kvcache is nonempty, iids are used instead * reorder kwargs * add more tests * fix device merge artifact in test_model.oy * fix generate test * yapf * Better error message when test does not complete (#769) * run script tests first * comment out * ascripts -> scripts * bad dirs * try this * hacks * add a note about a_scripts --------- Co-authored-by: Sam Havens <[email protected]>
* Add eval loader to eval script * small input tests * updates * fix typing and formatting * fixes, add tests * remove circular dependency * tests pass * nits + small fixes * add metrics at the end, refactor to put icl/gauntlet as helpers * NOT * metrics instead of models, add unit tests * Move tests into directories * add copyright to inits * fix relative paths * fixes * revert gauntlet test change * Support inputs_embeds (mosaicml#687) * support inputs_embeds * update tests to test inputs_embeds * make iids optional inputs to fwd * remove check for both iids and inputs_embeds in MPTForCausalLM. It is checked in the base model, and it is actually a common practice to pass both during autoregressive generation. Embeds are used first, then once the kvcache is nonempty, iids are used instead * reorder kwargs * add more tests * fix device merge artifact in test_model.oy * fix generate test * yapf * Better error message when test does not complete (mosaicml#769) * run script tests first * comment out * ascripts -> scripts * bad dirs * try this * hacks * add a note about a_scripts --------- Co-authored-by: Sam Havens <[email protected]>
This allows users to pass in embeddings directly instead of looking them up based on
input_ids. This is useful for PEFT (prompt/prefix-tuning) and multimodal models.