from_gguf only supports CPU?

[sparqyeti]

creating a quantized var builder is done with the code below:
in file: candle-transformers/src/quantized_var_builder.rs

impl VarBuilder {
    pub fn from_gguf<P: AsRef<std::path::Path>>(p: P) -> Result<Self> {
        let mut file = std::fs::File::open(p)?;
        let content = candle::quantized::gguf_file::Content::read(&mut file)?;
        let mut data = std::collections::HashMap::new();
        for tensor_name in content.tensor_infos.keys() {
            let tensor = content.tensor(&mut file, tensor_name)?;
            data.insert(tensor_name.to_string(), Arc::new(tensor));
        }
        Ok(Self {
            data: Arc::new(data),
            path: Vec::new(),
            device: Device::Cpu,
        })
    }

this implies GGUF only supports CPU and not GPU? Am I missing something is this actually a constraint.

[LaurentMazare]

Yes, quantized tensors are only supported on cpu at the moment.

[danielclough]

Related open issues

Support for quantisation: #359
CUDA support for QMatMul: #655
Error: no cuda implementation for qmatmul: #696
Quantized models on Cuda: #1250
You are here: #1486

[hubertshelley]

metal fixed : #1627

[LaurentMazare]

Closing this now in favor of #1250 , #1754 should also help with cuda support.