Multi-backend refactor: Alpha release ( INTEL ONLY )

[Titus-von-Koeller]

This space is intended to receive feedback from users that are willing to help us by alpha testing the current implementation of the Intel backends (Intel CPU and XPU).

Issues to discuss could be

• bugs
• installation
• needed docs
• performance
• ease of use
• ... you name it ...

Thanks everyone for you kind support and please remember to have a constructive tone 🤗

[BBC-Esq]

I'd be interested in trying it out. How can I do that?

[Njbx]

I am also interested thank you

[epage480]

I'm interested as well!

[notsyncing]

Hello, I'm wondering is the intel backend ready to use? My attempts are all failed with

Unable to get JIT kernel for brgemm. 

for either load_in_4bit or load_in_8bit

or

NotImplementedError: Could not run 'torch_ipex::woq_linear.xpu' with arguments from the 'CPU' backend.

despite the model and inputs are all on the cpu.

on torch 2.5.1+cxx11.abi and intel_extension_for_pytorch 2.5.10+xpu and bitsandbytes 0.45.0.dev0 from https://github.com/bitsandbytes-foundation/bitsandbytes/releases/download/continuous-release_multi-backend-refactor/bitsandbytes-0.45.0.dev0-py3-none-manylinux_2_24_x86_64.whl

[jiqing-feng]

You can follow the transformers guide. Can you try llama2-7b or gpt2 model? I suppose these 2 models work well.

[jiqing-feng]

Oh, I think you installed the wrong ipex, please uninstall your current ipex and just run pip install intel_extension_for_pytorch

[notsyncing]

I created a new venv, installed from https://pytorch-extension.intel.com/installation?platform=cpu&version=v2.5.0%2Bcpu&os=linux%2Fwsl2&package=pip, and created a fresh test.py:

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, pipeline
import torch

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

model_4bit = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-1.5B",
    quantization_config=quantization_config,
)

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-1.5B")

generator = pipeline("text-generation", model=model_4bit, tokenizer=tokenizer)
print(generator("def helloWorld() {"))

still got:

Unable to get JIT kernel for brgemm. Params: M=5, N=5, K=128, str_a=1, str_b=1, brgemm_type=1, beta=0, a_trans=0, unroll_hint=1, lda=1536, ldb=5, ldc=5, config=0, b_vnni=0

I also tried gpt2 model, still almost the same error:

Unable to get JIT kernel for brgemm. Params: M=5, N=5, K=64, str_a=1, str_b=1, brgemm_type=1, beta=0, a_trans=0, unroll_hint=1, lda=2304, ldb=5, ldc=5, config=0, b_vnni=0

[notsyncing]

I found a similar error when googling: https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-AWQ/discussions/2, which is also trying to run a model on cpu.
Looks like it's not a bitsandbytes issue now.

[jiqing-feng]

Sure, please let me know if you found it's a BNB issue, thanks