Could not load bitsandbytes native library: libcusparse.so.11: cannot open shared object file: No such file or directory

[joshgura]

System Info

arch, kohya_ss, python 3.10

LD_LIBRARY_PATH ==

echo $LD_LIBRARY_PATH
:/opt/cuda/lib64/:/opt/cuda/nvvm/lib64/:/opt/cuda/extras/CUPTI/lib64/:/opt/cuda/targets/x86_64-linux/lib/

outpout of $PATH

echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/opt/cuda/bin:/opt/cuda/nsight_compute:/opt/cuda/nsight_systems/bin:/var/lib/flatpak/exports/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/username/.local/share/yabridge

Reproduction

execute the following commands:

pip install bitsandbytes
python -m bitsandbytes

get this error:

Could not load bitsandbytes native library: libcusparse.so.11: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "/home/username/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 85, in <module>
    lib = get_native_library()
  File "/home/username/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 72, in get_native_library
    dll = ct.cdll.LoadLibrary(str(binary_path))
  File "/usr/lib/python3.10/ctypes/__init__.py", line 452, in LoadLibrary
    return self._dlltype(name)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcusparse.so.11: cannot open shared object file: No such file or directory

CUDA Setup failed despite CUDA being available. Please run the following command to get more information:

python -m bitsandbytes

Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
and open an issue at: https://github.com/bitsandbytes-foundation/bitsandbytes/issues

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ BUG REPORT INFORMATION ++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
CUDA specs: CUDASpecs(highest_compute_capability=(8, 6), cuda_version_string='118', cuda_version_tuple=(11, 8))
PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: (8, 6).
To manually override the PyTorch CUDA version please see: https://github.com/TimDettmers/bitsandbytes/blob/main/docs/source/nonpytorchcuda.mdx
The directory listed in your path is found to be non-existent: local/thing
The directory listed in your path is found to be non-existent: @/tmp/.ICE-unix/1346,unix/thing
The directory listed in your path is found to be non-existent: /org/freedesktop/DisplayManager/Session0
The directory listed in your path is found to be non-existent: /org/freedesktop/DisplayManager/Seat0
The directory listed in your path is found to be non-existent: //debuginfod.archlinux.org 
Found duplicate CUDA runtime files (see below).

We select the PyTorch default CUDA runtime, which is 11.8,
but this might mismatch with the CUDA version that is needed for bitsandbytes.
To override this behavior set the `BNB_CUDA_VERSION=<version string, e.g. 122>` environmental variable.

For example, if you want to use the CUDA version 122,
    BNB_CUDA_VERSION=122 python ...

OR set the environmental variable in your .bashrc:
    export BNB_CUDA_VERSION=122

In the case of a manual override, make sure you set LD_LIBRARY_PATH, e.g.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.2,
* Found CUDA runtime at: /opt/cuda/lib64/libcudart.so.12.6.77
* Found CUDA runtime at: /opt/cuda/lib64/libcudart.so
* Found CUDA runtime at: /opt/cuda/lib64/libcudart.so.12
* Found CUDA runtime at: /opt/cuda/targets/x86_64-linux/lib/libcudart.so.12.6.77
* Found CUDA runtime at: /opt/cuda/targets/x86_64-linux/lib/libcudart.so
* Found CUDA runtime at: /opt/cuda/targets/x86_64-linux/lib/libcudart.so.12
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Checking that the library is importable and CUDA is callable...
Couldn't load the bitsandbytes library, likely due to missing binaries.
Please ensure bitsandbytes is properly installed.

For source installations, compile the binaries with `cmake -DCOMPUTE_BACKEND=cuda -S .`.
See the documentation for more details if needed.

Trying a simple check anyway, but this will likely fail...
Traceback (most recent call last):
  File "/home/username/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/diagnostics/main.py", line 66, in main
    sanity_check()
  File "/home/username/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/diagnostics/main.py", line 40, in sanity_check
    adam.step()
  File "/home/username/kohya_ss/venv/lib/python3.10/site-packages/torch/optim/optimizer.py", line 373, in wrapper
    out = func(*args, **kwargs)
  File "/home/username/kohya_ss/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/username/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/optim/optimizer.py", line 291, in step
    self.update_step(group, p, gindex, pindex)
  File "/home/username/kohya_ss/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/username/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/optim/optimizer.py", line 521, in update_step
    F.optimizer_update_32bit(
  File "/home/username/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/functional.py", line 1571, in optimizer_update_32bit
    optim_func = str2optimizer32bit[optimizer_name][0]
NameError: name 'str2optimizer32bit' is not defined
Above we output some debug information.
Please provide this info when creating an issue via https://github.com/TimDettmers/bitsandbytes/issues/new/choose
WARNING: Please be sure to sanitize sensitive info from the output before posting it.

Expected behavior

expected behavior: libcusparse.so.11 should be installed by bitsandbytes if it is a dependency.

[matthewdouglas]

Hi @joshgura,

Since you've installed PyTorch with a CUDA 11.8 build, the libcusparse.so.11 should be available as it is included as a dependency. Unfortunately it's not sure where to find the library that was distributed with PyTorch. However, since your machine also has a CUDA 12 toolkit installed, it's trying to look there.

If you could, I would recommend switching to a CUDA 12.x build of PyTorch.

Alternatively, you can add ~/kohya_ss/venv/lib/python3.10/site-packages/nvidia/cusparse/lib/ to LD_LIBRARY_PATH as one option. This is where libcusparse.so.11 should be. You may additionally need to do the same for libcublas.so.11.

[joshgura]

Thanks MatthewDouglas!

I think Kohya set up a version of pytorch that it needed, but I am game to try tinkering with the requirements.txt. (Unless it is using system wide installs of things, which is ambiguous in the sense that it doesn't seem to be documented afaik)

I managed to get Kohya working (again) this way: by setting up a Conda venv (carefully as possible, because it is capable of causing headaches for days) because it has easier or better access to packages than pip, and installed another appropriate cuda driver with that, so when kohya and bitsandbytes went a-searchin, it didn't have to get lost in my system which has the duplicate library problem.. But you know what, I'm going to try what you said as well, because it sounds simpler and easier and I am 100% for easier and simpler.