Deepseek

[ehartford]

Requesting support for Deepseek-V3 and DeepSeek-V2.5-1210

[ehartford]

With Deepseek-v3 I get this:

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = '/home/ubuntu/datasets/models/DeepSeek-V3-bf16'
quant_path = '/home/ubuntu/datasets/models/DeepSeek-V3-awq'
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }

# Load model
model = AutoAWQForCausalLM.from_pretrained(model_path, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

# Quantize
model.quantize(tokenizer, quant_config=quant_config)

# Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

print(f'Model is quantized and saved at "{quant_path}"')

python quant.py 
Traceback (most recent call last):
  File "/home/ubuntu/quant.py", line 9, in <module>
    model = AutoAWQForCausalLM.from_pretrained(model_path, trust_remote_code=True)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/miniconda3/envs/awq/lib/python3.11/site-packages/awq/models/auto.py", line 73, in from_pretrained
    model_type = check_and_get_model_type(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/miniconda3/envs/awq/lib/python3.11/site-packages/awq/models/auto.py", line 49, in check_and_get_model_type
    raise TypeError(f"{config.model_type} isn't supported yet.")
TypeError: deepseek_v3 isn't supported yet.

[ehartford]

https://huggingface.co/deepseek-ai/DeepSeek-V2.5-1210

https://huggingface.co/deepseek-ai/DeepSeek-V3