Support num_key_value_heads

[jiqing-feng]

Hi @echarlaix .

This PR enables models with num_key_value_heads like mistral to generate the correct shape of past_key_values. Could you please help review it after 1440 merged? Thx!

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[echarlaix]

Thanks for the addition @jiqing-feng, can you take a look at the failing tests ?

[jiqing-feng]

Thanks for the addition @jiqing-feng, can you take a look at the failing tests ?

Yes. I have checked the failed tests. The ipex testing indicates that I didn't install intel_extension_for_pytorch, and the INC testing indicates that "the maximum number of trials specified has been reached and no quantized model meeting the specified accuracy tolerance has been found.".

I think these are not related to my changes

[echarlaix]

Yes. I have checked the failed tests. The ipex testing indicates that I didn't install intel_extension_for_pytorch, and the INC testing indicates that "the maximum number of trials specified has been reached and no quantized model meeting the specified accuracy tolerance has been found.".

I think these are not related to my changes

Ok I'll take care of fixing it myself then