Improvement suggestions for the multi-backend-refactor installation instructions #1219
Replies: 7 comments 21 replies
-
|
I just tested out the multi-backend-refactor for ROCm (Ubuntu 22.04 LTS HWE, ROCm 6.0 (using the standard AMD ROCm repo)) on RDNA3 navi3x gfx1000 (W7900 and 7900XTX). The basic installation instructions worked fine for me. I tested loading a llama3 model w/ Quick perf notes on the W7900 and a Llama 3 8B model, using torch.nograd() - 2.3.0+rocm6.0:
I tested a couple times just to make sure it wasn't a fluke, but looks like for gfx1100 at least, load_in_8bit is very slow for some reason. |
Beta Was this translation helpful? Give feedback.
-
|
Really cool, thanks so much @lhl for your valuable feedback, really appreciated! cc @pnunna93 (from AMD) |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jianan-gu @mingfeima @ashokei @Kanya-Mo, As mentioned in the Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Hey @sanchitintel, thanks for getting in touch. I think for general reporting and discussion topics, this discussion is better suited. I would like to keep these threads here mostly about the installation process + instructions. Be sure to always tag me, so that it doesn't slip my radar. Currently, we're really short on resources and otherwise it go unnoticed for too long. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
how do I compile it from source? is the installation for both HIP and CUDA the same? |
Beta Was this translation helpful? Give feedback.
-
|
also, just curious but do you think it's possible to achieve a sort-of CPU + GPU training with some of the layers offloaded to the CPU with bitsandbytes? Or maybe even an NPU in the future? |
Beta Was this translation helpful? Give feedback.
-
The torch version is for rocm6.0. Please reinstall 6.1 torch and build bitsandbytes again: |
Beta Was this translation helpful? Give feedback.
-
|
Alright, this worked for me too. Didn't know that ROCm support was this strict. Thanks for this. What kind of metrics and data would be useful for you guys for me to test? |
Beta Was this translation helpful? Give feedback.
-
|
This was helpful for me, as I have been having issues building but this thread made me realize that I had torch 2.3.1+rocm5.7 installed. Uninstalling it and reinstalling it with Also for anyone else who might be facing a similar situation as me, I was getting this error while trying to build: This fixed it for me: |
Beta Was this translation helpful? Give feedback.
-
@PatchouliPatch In principle these things could be possible going forward if they're possible in vanilla PyTorch. Currently, the multi-backend approach is still implemented with some custom logic, but in the upcoming weeks I'm going to refactor things so that the BNB operations currently in the Either way, I think that's not the main use-case we're aiming for, but with the upcoming refactor things like that might be possible in the framework of what PyTorch allows. We'd be happy to hear and support to a certain degree, if folks choose to experiment with this in the future. Regarding NPU, that of course also depends if someone from the community is willing to contribute the implementations for that. At BNB we'll continue to focus on CUDA implementations, but are happy to assist with and merge community implementations, as is the case now with AMD and Intel. The idea is that Apple Silicon follows next. In principle every other platform can be supported if there's enough interest from the community and they take the lead on implementation and continued maintenance thereof. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I just tested out building bitsandbytes from ROCm installation instructions Environment details:
After navigating to the "multi-backend-refactor" branch, and following the installation instructions, I get these errors when trying to perform the step
I was able to fix these errors with:
Then, when at the step to I was able to fix these errors with:
Furthermore, when trying to run the example script
I realize these issues may be due to the fact that I'm using a Docker container as my environment, but figured I'd share my process regardless. (Extra) Here are some Dockerfile snippets I used to help build bitsandbytes on the multi-backend-refactor branch: Update:To get the example script Relevant pip modules for torch that I used:
If I recall correctly, |
Beta Was this translation helpful? Give feedback.
-
|
@mohamedyassin1 , please use rocm/pytorch:latest docker, it has all hip packages and pytorch installed. If you'd like to use a rocm base docker and install pytorch manually you can use rocm/dev-ubuntu-22.04:6.1-complete, it comes with all hip packages. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you! that helps a lot |
Beta Was this translation helpful? Give feedback.
-
|
@mohamedyassin1 Thank you so much for your well written and detailed post, as well as giving things fully reproducible through providing the Dockerfile, really cool and helpful! @pnunna93 Is there anything we can improve in the AMD related docs based on this feedback? On another note, I think while the Docker approach is really helpful, we shouldn't depend on only that for our main docs. A lot of people can't or don't want to use Docker or only have access to a machine without admin, for example in an academic lab. Looking at it from that perspective, it would be great if one could install the missing system dependencies with Conda-forge, because that doesn't require sudo.. Not sure if that's possible? Would you be so kind to submit a PR to update the docs for the example? That would be super helpful! The example worked fine for you otherwise, @mohamedyassin1 ? |
Beta Was this translation helpful? Give feedback.
-
|
@Titus-von-Koeller Can do! Will double check the example and update documentation accordingly through a PR. |
Beta Was this translation helpful? Give feedback.
-
|
@Titus-von-Koeller, I have linked ROCm official install instructions to the documentation in packaging PR. It provides alternatives to install ROCm directly. |
Beta Was this translation helpful? Give feedback.
-
|
Heya, I had the unfortunate event of my SSD corrupting so I had to reinstall Ubuntu. For some reason, Bitsandbytes compiles fine but fails when I run (torch) user@chingu:~/bitsandbytes$ python3 -m bitsandbytes
Could not find the bitsandbytes CUDA binary at PosixPath('/home/bitsandbytes/bitsandbytes/libbitsandbytes_hip.so')
Could not load bitsandbytes native library: /home/bitsandbytes/bitsandbytes/libbitsandbytes_cpu.so: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "/home/bitsandbytes/bitsandbytes/cextension.py", line 124, in <module>
lib = get_native_library()
^^^^^^^^^^^^^^^^^^^^
File "/home/bitsandbytes/bitsandbytes/cextension.py", line 104, in get_native_library
dll = ct.cdll.LoadLibrary(str(binary_path))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/anaconda3/envs/torch/lib/python3.11/ctypes/__init__.py", line 454, in LoadLibrary
return self._dlltype(name)
^^^^^^^^^^^^^^^^^^^
File "/home/anaconda3/envs/torch/lib/python3.11/ctypes/__init__.py", line 376, in __init__
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /home/bitsandbytes/bitsandbytes/libbitsandbytes_cpu.so: cannot open shared object file: No such file or directory
CUDA Setup failed despite CUDA being available. Please run the following command to get more information:
python -m bitsandbytes
Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ BUG REPORT INFORMATION ++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
CUDA specs: CUDASpecs(highest_compute_capability=(11, 0), cuda_version_string='61', cuda_version_tuple=(6, 1))
PyTorch settings found: CUDA_VERSION=61, Highest Compute Capability: (11, 0).
Library not found: /home/gabriel/bitsandbytes/bitsandbytes/libbitsandbytes_hip.so. Maybe you need to compile it from source?
If you compiled from source, try again with `make CUDA_VERSION=DETECTED_CUDA_VERSION`,
for example, `make CUDA_VERSION=113`.
The CUDA version for the compile might depend on your conda install, if using conda.
Inspect CUDA version via `conda list | grep cuda`.
WARNING: CUDA versions lower than 11 are currently not supported for LLM.int8().
You will be only to use 8-bit optimizers and quantization routines!
To manually override the PyTorch CUDA version please see: https://github.com/TimDettmers/bitsandbytes/blob/main/docs/source/nonpytorchcuda.mdx
The directory listed in your path is found to be non-existent: local/chingu
The directory listed in your path is found to be non-existent: @/tmp/.ICE-unix/1630,unix/chingu
The directory listed in your path is found to be non-existent: /etc/xdg/xdg-ubuntu
The directory listed in your path is found to be non-existent: /org/gnome/Terminal/screen/135c10b6_a8a1_4974_8ac8_294d12aa05a3
CUDA SETUP: WARNING! CUDA runtime files not found in any environmental path.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Checking that the library is importable and CUDA is callable...
Couldn't load the bitsandbytes library, likely due to missing binaries.
Please ensure bitsandbytes is properly installed.
For source installations, compile the binaries with `cmake -DCOMPUTE_BACKEND=cuda -S .`.
See the documentation for more details if needed.
Trying a simple check anyway, but this will likely fail...
Traceback (most recent call last):
File "/home/bitsandbytes/bitsandbytes/diagnostics/main.py", line 66, in main
sanity_check()
File "/home/bitsandbytes/bitsandbytes/diagnostics/main.py", line 40, in sanity_check
adam.step()
File "/home/anaconda3/envs/torch/lib/python3.11/site-packages/torch/optim/optimizer.py", line 487, in wrapper
out = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/anaconda3/envs/torch/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/bitsandbytes/bitsandbytes/optim/optimizer.py", line 287, in step
self.update_step(group, p, gindex, pindex)
File "/home/anaconda3/envs/torch/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/bitsandbytes/bitsandbytes/optim/optimizer.py", line 496, in update_step
F.optimizer_update_32bit(
File "/home/bitsandbytes/bitsandbytes/functional.py", line 1160, in optimizer_update_32bit
return backends[g.device.type].optimizer_update_32bit(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bitsandbytes/bitsandbytes/backends/cuda.py", line 870, in optimizer_update_32bit
optim_func = str2optimizer32bit[optimizer_name][0]
^^^^^^^^^^^^^^^^^^
NameError: name 'str2optimizer32bit' is not defined
Above we output some debug information.
Please provide this info when creating an issue via https://github.com/TimDettmers/bitsandbytes/issues/new/choose
WARNING: Please be sure to sanitize sensitive info from the output before posting it.am I doing anything wrong? does this only work with certain python versions? |
Beta Was this translation helpful? Give feedback.
-
|
Nevermind, solved with: apparently, the instructions above were inadequate |
Beta Was this translation helpful? Give feedback.
-
|
@PatchouliPatch thanks a lot for the feedback! @pnunna93 I think the part that's outputting would profit from an update that detects if rocm is the available and add's a fitting installation command there. We haven't tested that yet, but should we consider people having both cuda and rocm? Either way, I think that string should also contain a |
Beta Was this translation helpful? Give feedback.
-
|
Hello! I have two questions/issues:
OS = Ubuntu 24.04 but I have also tried it on WSL2 on windows with Ubuntu 24.04 with the same results. The pytorch and intel-extension-for-pytorch versions are as follows: I have verified that the intel-extension-for-pytorch works on my setup by running a few examples from their documentation. The transformers and bitsandbytes versions are: I am compiling bitsandbytes from source (as instructed by the installation guide) using: Output of What could I be doing wrong?
|
Beta Was this translation helpful? Give feedback.
-
|
I had to ad this to my .bashrc file for cmake to work. Fedora doesn't have rocm installed on /opt/. `export HIP_PATH=/usr export ROCM_PATH=/usr cmake -DCOMPUTE_BACKEND=hip -S . -DBNB_ROCM_ARCH="gfx1100" worked as intended afterwards. |
Beta Was this translation helpful? Give feedback.
-
|
Just to confirm: the mulit-backend-refactor works fine here on rocm 6.2 with a mi100 (gfx908) gpu for 8bit and 4bit quantization, was well as the 8bit optimizer |
Beta Was this translation helpful? Give feedback.
-
Please help us out with snippets and recommendations to make the experience as pain-free as possible.
Beta Was this translation helpful? Give feedback.
All reactions