KeyError: 'mixtral' from transformers.models.auto.configuration_auto

[wasertech]

Hello community!

So it's the first time I try to serve a Mixtral model specifically TheBloke/Nous-Hermes-2-Mixtral-8x7B-DPO-AWQ with vLLM.

Only issue, transformers raises the following error:

+ python app.py --host 0.0.0.0 --port 5085 --model TheBloke/Nous-Hermes-2-Mixtral-8x7B-DPO-AWQ --tokenizer TheBloke/Nous-Hermes-2-Mixtral-8x7B-DPO-AWQ --dtype half --max-model-len 8192 --tensor-parallel-size 1 --gpu-memory-utilization 0.75 --quantization awq
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): huggingface.co:443
DEBUG:urllib3.connectionpool:https://huggingface.co:443 "HEAD /TheBloke/Nous-Hermes-2-Mixtral-8x7B-DPO-AWQ/resolve/main/config.json HTTP/1.1" 200 0
Traceback (most recent call last):
  File "app.py", line 143, in <module>
    engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/usr/local/lib/python3.8/dist-packages/vllm/engine/async_llm_engine.py", line 489, in from_engine_args
    engine_configs = engine_args.create_engine_configs()
  File "/usr/local/lib/python3.8/dist-packages/vllm/engine/arg_utils.py", line 198, in create_engine_configs
    model_config = ModelConfig(self.model, self.tokenizer,
  File "/usr/local/lib/python3.8/dist-packages/vllm/config.py", line 91, in __init__
    self.hf_config = get_config(self.model, trust_remote_code, revision)
  File "/usr/local/lib/python3.8/dist-packages/vllm/transformers_utils/config.py", line 23, in get_config
    config = AutoConfig.from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/configuration_auto.py", line 1064, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/configuration_auto.py", line 761, in __getitem__
    raise KeyError(key)
KeyError: 'mixtral'

I've not found much on the net about a similar issue. I found https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/discussions/30 and https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/discussions/9 which both suggest to update to transformers>=4.36.1 but I'm using:

>>> import transformers; print(transformers.__version__)
4.37.2

Also I'm using vLLM v0.3.0. ( I saw errors with mixtral and vLLM before v0.2.1 but still same error...)

Has someone already experienced such an issue with Mixtral and vLLM? How did you solve it?

[wasertech]

Never mind I just failed to send the new updated image to my cloud provider. Plus some other cloud specific issues. Fixed now. If you run into this error just update and you shouldn't anymore.