Quantization Not Working as Expected

[sdt03]

When attempting to load the model with quantization options in AirLLM, the system is not applying the specified compression and downloads the entire model instead. I used the following code snippet to apply compressions:

model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct", compression='8bit')

The model should be downloaded with the specified quantization, applying the compression as indicated by the compression argument, however, full model is downloaded, ignoring the compression setting, which prevents the expected quantization from being applied.

Is there a way to ensure that the quantized version of the model is loaded instead?