Cuda error bitsandbytes - Arch linux

[NulliferBones]

Hello, i'm getting this error when trying to start training
I can't figure out what to do. I tried visiting the links and they all seem to 404.
Anyone able to help me fix this error?

CUDA SETUP: Loading binary /home/nullifer/Documents/AI/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so...
libcusparse.so.11: cannot open shared object file: No such file or directory
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://raw.githubusercontent.com/TimDettmers/bitsandbytes/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local

[b-fission]

The install script for solution 2 is here: https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/install_cuda.py

Try installing CUDA Toolkit 11.8 or higher.

[NulliferBones]

I do have cuda installed

[b-fission]

You can do
export LD_LIBRARY_PATH=/path/to/your/cuda-12.5/lib64
and then run gui.sh on the same terminal session. I'm not 100% sure if the toolkit's lib64 is present at its install path, or a subfolder. You'll need to double-check in that.

[b-fission]

it says cuda toolkit found when i boot it up, which is odd

That's only because pytorch includes a subset of the cuda libs, which get detected.
bitsandbytes requires some additional cuda libs, which is why the toolkit is needed to successfully run.

[NulliferBones]

still got this error

Traceback (most recent call last):
  File "/home/nullifer/Documents/AI/KohyaSS/kohya_ss/sd-scripts/train_network.py", line 1242, in <module>
    trainer.train(args)
  File "/home/nullifer/Documents/AI/KohyaSS/kohya_ss/sd-scripts/train_network.py", line 1012, in train
    optimizer.step()
  File "/home/nullifer/Documents/AI/KohyaSS/kohya_ss/venv/lib/python3.10/site-packages/accelerate/optimizer.py", line 132, in step
    self.scaler.step(self.optimizer, closure)
  File "/home/nullifer/Documents/AI/KohyaSS/kohya_ss/venv/lib/python3.10/site-packages/torch/cuda/amp/grad_scaler.py", line 416, in step
    retval = self._maybe_opt_step(optimizer, optimizer_state, *args, **kwargs)
  File "/home/nullifer/Documents/AI/KohyaSS/kohya_ss/venv/lib/python3.10/site-packages/torch/cuda/amp/grad_scaler.py", line 315, in _maybe_opt_step
    retval = optimizer.step(*args, **kwargs)
  File "/home/nullifer/Documents/AI/KohyaSS/kohya_ss/venv/lib/python3.10/site-packages/accelerate/optimizer.py", line 185, in patched_step
    return method(*args, **kwargs)
  File "/home/nullifer/Documents/AI/KohyaSS/kohya_ss/venv/lib/python3.10/site-packages/torch/optim/lr_scheduler.py", line 68, in wrapper
    return wrapped(*args, **kwargs)
  File "/home/nullifer/Documents/AI/KohyaSS/kohya_ss/venv/lib/python3.10/site-packages/torch/optim/optimizer.py", line 373, in wrapper
    out = func(*args, **kwargs)
  File "/home/nullifer/Documents/AI/KohyaSS/kohya_ss/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/nullifer/Documents/AI/KohyaSS/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/optim/optimizer.py", line 287, in step
    self.update_step(group, p, gindex, pindex)
  File "/home/nullifer/Documents/AI/KohyaSS/kohya_ss/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/nullifer/Documents/AI/KohyaSS/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/optim/optimizer.py", line 546, in update_step
    F.optimizer_update_8bit_blockwise(
  File "/home/nullifer/Documents/AI/KohyaSS/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/functional.py", line 1777, in optimizer_update_8bit_blockwise
    optim_func = str2optimizer8bit_blockwise[optimizer_name][0]
NameError: name 'str2optimizer8bit_blockwise' is not defined
steps:   0%|                                                                                        | 0/1600 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "/home/nullifer/Documents/AI/KohyaSS/kohya_ss/venv/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/nullifer/Documents/AI/KohyaSS/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/home/nullifer/Documents/AI/KohyaSS/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1017, in launch_command
    simple_launcher(args)
  File "/home/nullifer/Documents/AI/KohyaSS/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 637, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/nullifer/Documents/AI/KohyaSS/kohya_ss/venv/bin/python', '/home/nullifer/Documents/AI/KohyaSS/kohya_ss/sd-scripts/train_network.py', '--config_file', '/home/nullifer/Documents/AI/2/model/config_lora-20240801-010754.toml']' returned non-zero exit status 1.

[b-fission]

Maybe LD_LIBRARY_PATH isn't correct. Do you know the path to the cuda toolkit's libcudart.so.12 file?

[NulliferBones]

yeah its in opt/cuda/lib64
that's what i exported

[b-fission]

Ok, so it wants libcusparse.so.11 which isn't part of the toolkit since you've installed version 12.

For your system, try this...
export LD_LIBRARY_PATH=/home/nullifer/Documents/AI/KohyaSS/kohya_ss/venv/lib/python3.10/site-packages/nvidia/cusparse/lib

If it works, then you won't need the cuda-12 toolkit.
kohya_ss currently expects toolkit 11.8, so you shouldn't try to mix and match v11 and v12.

[NulliferBones]

not related to this, but recently for automatic1111 i started receiving this error. do not understand what caused it, no one is responding there maybe i can get some help here

Check TCMalloc: libtcmalloc_minimal.so.4
/home/nullifer/Documents/AI/WEBUI/webuirc/webui.sh: line 258: bc: command not found
/home/nullifer/Documents/AI/WEBUI/webuirc/webui.sh: line 258: [: -eq: unary operator expected
libtcmalloc_minimal.so.4 is linked with libc.so,execute LD_PRELOAD=/usr/lib/libtcmalloc_minimal.so.4

i tried that execute message but didn't seem to help

[b-fission]

The only error I see there is the bc tool isn't installed.
The parts about libtcmalloc_minimal.so.4 is normal log output, nothing wrong there.

[NulliferBones]

alright i installed BC and that error is gone.
only message remaing apart from the future warning resume download
is

Check TCMalloc: libtcmalloc_minimal.so.4
libtcmalloc_minimal.so.4 is linked with libc.so,execute LD_PRELOAD=/usr/lib/libtcmalloc_minimal.so.4

[b-fission]

Yeah, it looks fine.

[NulliferBones]

thanks again