It seems that the current version of bitsandbytes is not compatible with my CUDA 12.4 library.

[Yukiiceeee]

System Info

CUDA12.4

Reproduction

When using bitsandbytes, I noticed that it requires a libcusparse.so.11 file. However, my CUDA version is 12.4, which only includes libcusparse.so.12 but not libcusparse.so.11, resulting in an error. Isn’t bitsandbytes supposed to support CUDA 12.4? After searching for solutions, I received feedback suggesting that I should downgrade my CUDA version. Why is this issue happening?

The errors are as follows:

warnings.warn(
>> world size:1; torch rank:0; OS local rank:0; device:cuda:0;
Could not find the bitsandbytes CUDA binary at PosixPath('/d1/.conda/newcmoe/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda118.so')
Could not load bitsandbytes native library: /d1/.conda/newcmoe/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "/d1/.conda/newcmoe/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 104, in <module>
    lib = get_native_library()
  File "/d1/.conda/newcmoe/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 91, in get_native_library
    dll = ct.cdll.LoadLibrary(str(binary_path))
  File "/d1/.conda/newcmoe/lib/python3.10/ctypes/__init__.py", line 452, in LoadLibrary
    return self._dlltype(name)
  File "/d1/.conda/newcmoe/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /d1/.conda/newcmoe/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: cannot open shared object file: No such file or directory

CUDA Setup failed despite CUDA being available. Please run the following command to get more information:

python -m bitsandbytes

Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

Expected behavior

It should be able for CUDA12.4.

[matthewdouglas]

It would appear that you have a PyTorch installation built with CUDA 11.8. It should work if you upgrade to a CUDA 12.4 build of PyTorch.