CUda error in UBUNTU 24.04

[kainatquaderee]

I get this after start train. dont know what is the problem. I can run automatic1111 and comfy UI fine
I tried LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/lib/x86_64-linux-gnu/" still didn't work.
nvidia-driver-550 and cuda version is 12.4

/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:183: UserWarning: Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes


  warn(msg)
================================================================================
The following directories listed in your path were found to be non-existent: {PosixPath('/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/venv/lib/python3.10/site-packages/cv2/../../lib64')}
/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:183: UserWarning: /media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/venv/lib/python3.10/site-packages/cv2/../../lib64: did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0', 'libcudart.so.12.1', 'libcudart.so.12.2'] as expected! Searching further paths...
  warn(msg)
The following directories listed in your path were found to be non-existent: {PosixPath('@/tmp/.ICE-unix/1680,unix/kainat-quadereee-B550M-DS3H'), PosixPath('local/kainat-quadereee-B550M-DS3H')}
The following directories listed in your path were found to be non-existent: {PosixPath('/etc/xdg/xdg-plasma')}
The following directories listed in your path were found to be non-existent: {PosixPath('/org/freedesktop/DisplayManager/Session0')}
The following directories listed in your path were found to be non-existent: {PosixPath('/home/kainatq/.gtkrc'), PosixPath('/etc/gtk/gtkrc')}
The following directories listed in your path were found to be non-existent: {PosixPath('0'), PosixPath('1')}
The following directories listed in your path were found to be non-existent: {PosixPath('/etc/gtk-2.0/gtkrc')}
The following directories listed in your path were found to be non-existent: {PosixPath('/Sessions/3')}
The following directories listed in your path were found to be non-existent: {PosixPath('/org/freedesktop/DisplayManager/Seat0')}
The following directories listed in your path were found to be non-existent: {PosixPath('/usr/share/hunspell-bdic')}
The following directories listed in your path were found to be non-existent: {PosixPath('/Windows/1')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary /media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda118.so...
libcusparse.so.11: cannot open shared object file: No such file or directory
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://raw.githubusercontent.com/TimDettmers/bitsandbytes/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
Traceback (most recent call last):
  File "/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/sd-scripts/train_network.py", line 1242, in <module>
    trainer.train(args)
  File "/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/sd-scripts/train_network.py", line 367, in train
    optimizer_name, optimizer_args, optimizer = train_util.get_optimizer(args, trainable_params)
  File "/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/sd-scripts/library/train_util.py", line 4322, in get_optimizer
    import bitsandbytes as bnb
  File "/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/__init__.py", line 6, in <module>
    from . import cuda_setup, research, utils
  File "/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/research/__init__.py", line 2, in <module>
    from .autograd._functions import (
  File "/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/research/autograd/_functions.py", line 8, in <module>
    from bitsandbytes.autograd._functions import GlobalOutlierPooler, MatmulLtState
  File "/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/autograd/__init__.py", line 1, in <module>
    from ._functions import get_inverse_transform_indices, undo_layout
  File "/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py", line 10, in <module>
    import bitsandbytes.functional as F
  File "/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/functional.py", line 17, in <module>
    from .cextension import COMPILED_WITH_CUDA, lib
  File "/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 17, in <module>
    raise RuntimeError('''
RuntimeError: 
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
Traceback (most recent call last):
  File "/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/venv/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1017, in launch_command
    simple_launcher(args)
  File "/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 637, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/venv/bin/python', '/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/sd-scripts/train_network.py', '--config_file', '/media/kainatq/Games_And_Others/stable/stable-diffusion-webui/repositories/kohya_ss/outputs/ban/model/config_lora-20241222-163355.toml']' returned non-zero exit status 1.