[Bug] KoboldCpp - Version 1.47.1/1.47.2.yr0-ROCm programme crash and close

[Richard7656]

My Computer:
Operating System: Windows 11 Professional 22H2 (Chinese Traditional)
CPU: AMD Ryzen 5 5600 6-Core Processor 3.50 GHz
GPU: GIGABYTE AMD Radeon RX 6650 XT (Driver Version 23.10.2)
RAM: 64GB

KoboldCpp - Version 1.47.1/1.47.2.yr0-ROCm

After press “Launch” to run the programme, the error message

“rocBLAS error: Cannot read C:\Users\richa\AppData\Local\Temp_MEI60562/library/TensileLibrary.dat: No such file or dire ctory “

at the bottom and the programme crash and close.

In KoboldCpp - Version 1.46.1.yr0-ROCm, the programme can still be launched except the problem of reply with garbage characters in certain condition.

[jamesalster]

I'm having the same issue, system:

Operating System: Windows 11 Professional 22H2
CPU: AMD Ryzen 5 5800X
GPU: AMD Radeon RX 6700 XT (Driver Version 23.10.2)
RAM: 32GB

KoboldCpp - Version 1.47.2.yr0-ROCm

[Kazaflow]

@Richard7656 @jamesalster
I will assume both of you tried to use the pre-compiled .exe file, right? That one for now only supports: gfx906, gfx1030, gfx1100, gfx1101, gfx1102 GPUs.
For other GPUs, in your cases gfx1032 (rx 6650 xt) and gfx1031 (rx 6700 xt) you need to compile the exe from source, IF they were supported by the HIP SDK.
There is a way to get unsupported GPUs to work, but that requires certain workarounds.

[brknsoul]

certain workarounds

Please tell us about these workarounds!

I have a 6600 xt (gfx1032) and I get the same error.

[Kazaflow]

@Richard7656 @jamesalster @brknsoul
Well, first you would have to follow the steps described in this comment ggerganov#1087 (comment), but beware when you're supposed to run the command with rmake.py you have to use x64 Native Tools Command Prompt for VS as admin or else it will error out when compiling at this step.

After having done these steps you can compile the Koboldcpp from source by following these steps https://github.com/YellowRoseCx/koboldcpp-rocm/releases/tag/Windows-v1.43-ROCm, even though the instructions are under the 1.43 release, they also apply to higher versions of the ROCm branch.
The DAMDGPU_TARGETS in the first CMake command have to be adjusted for the respective arch so (gfx1031,gfx1032, etc).

Then you can run make_pyinstaller_exe_rocm_only.bat instead of the suggested make_pyinst_rocm_hybrid_henk_yellow.bat, since including OPENBLAS and CLBLAST is kind of unnecessary when you're going to use HIPBlas anyway.

This is at least what I did on my RX 5700 XT (gfx1010), if you don't want to do these steps, you could wait until YellowRoseCx themself manages to compile the EXE for your LLVM arches, or I could try helping you if you have issues (though I am pretty clueless, so I don't know everything).

[ccbadd]

Why remove, stop compile gfx908?

[Kazaflow]

Why remove, stop compile gfx908?

@ccbadd What do you mean?
Edit: Because compiling Tensile Libraries works, not sure if they are usable but I can't check that.

[brknsoul]

@Kazaflow, running python rmake.py (in native tools cmd prompt as admin) results in ;

CMake Error at C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.25/Modules/CMakeTestCXXCompiler.cmake:63 (message):
  The C++ compiler

    "C:/Program Files/AMD/ROCm/5.5/bin/clang++.exe"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: E:/llm/rocBLAS/build/release/CMakeFiles/CMakeScratch/TryCompile-1zh7mt

Full error;
cmakefail.txt

[Kazaflow]

@brknsoul
Hmm, I think maybe disabling Windows defender or just excluding the folder could work (it might do something weird).

But it seems it doesn't find something it tries to link. Your cmakefail file also says,

See also "E:/llm/rocBLAS/build/release/CMakeFiles/CMakeOutput.log".
See also "E:/llm/rocBLAS/build/release/CMakeFiles/CMakeError.log".

I think it would tell us what exactly it failed at.
I'm not well acquainted with C/C++ but I suspect that it fails at this step, so I guess some of the stuff there is the issue:

C:\PROGRA~1\AMD\ROCm\5.5\bin\CLANG_~1.EXE -fuse-ld=lld-link -nostartfiles -nostdlib -DWIN32 -DWIN32_LEAN_AND_MEAN -DNOMINMAX -D_CRT_SECURE_NO_WARNINGS -D_SILENCE_EXPERIMENTAL_FILESYSTEM_DEPRECATION_WARNING -Wno-ignored-attributes -DHIP_CLANG_HCC_COMPAT_MODE=1 -fms-extensions -fms-compatibility -D__HIP_ROCclr__=1 -D__HIP_PLATFORM_AMD__=1  -DWIN32 -DWIN32_LEAN_AND_MEAN -DNOMINMAX -D_CRT_SECURE_NO_WARNINGS -D_SILENCE_EXPERIMENTAL_FILESYSTEM_DEPRECATION_WARNING -Wno-ignored-attributes -DHIP_CLANG_HCC_COMPAT_MODE=1 -fms-extensions -fms-compatibility -D__HIP_ROCclr__=1 -D__HIP_PLATFORM_AMD__=1   -O0 -g -Xclang -gcodeview -D_DEBUG -D_DLL -D_MT -Xclang --dependent-lib=msvcrtd -Xlinker /subsystem:console CMakeFiles/cmTC_8f356.dir/testCXXCompiler.cxx.obj -o cmTC_8f356.exe -Xlinker /MANIFEST:EMBED -Xlinker /implib:cmTC_8f356.lib -Xlinker /pdb:cmTC_8f356.pdb -Xlinker /version:0.0   -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -loldnames```

[brknsoul]

I guess I'll wait until YellowRose makes an exe for gfx1032

[Kazaflow]

Well, that could take some time, since gfx1032 for the HIP SDK has to come first (and its not like AMD seems to be picking up pace as far as I know). Kinda sucks it doesn't work like 1010. Only way for any gfx1032 seems to be Linux, with that HSA overwrite for the time being.
Not sure about gfx1031 though I managed to compile the tensile files for it, so at least those still may work.

[ccbadd]

Why remove, stop compile gfx908?

@ccbadd What do you mean? Edit: Because compiling Tensile Libraries works, not sure if they are usable but I can't check that.

Sorry for the delayed response, but my question is that gfx908 is one of the included fully supported libraries (MI100). When I read your earlier response it sounds like it was just dropped from the default compile list. For whatever reason, I am having problems compiling myself as anything multi gpu outputs gibberish. I have come to believe that the problem is actually with recent changes to llama.cpp and the memory pool changes they are trying to make.

[Kazaflow]

@ccbadd
It's fine, yeah multigpu is as far as I know broken on llama, on koboldcpp they made "Hotfix 1.48.1 - Fixed issues with Multi-GPU setups." which seems to fix it for Nvidia, but I don't know about AMD, though a single gfx908 should work?
It's fully supported on Linux, but on Windows the Tensile Libraries are missing, since HIP SDK is only shipped with gfx906, 1030, 1100/1/2, so you would have to try compiling them, though apparently some LLVMs are not working regardless, like gfx1032 for now.
Also, the compiling "tutorial" I linked there was from 1.43 ROCm Kcpp, now there is a proper section where gfx908 is in the list. https://github.com/YellowRoseCx/koboldcpp-rocm#compiling-for-amd-on-windows

[ccbadd]

@Kazaflow I'm not running Windows so that explains the difference. MI100s are only supported in Linux. AS for llama.cpp, the last release this weekend rolled a bunch of stuff back and I was able to compile and get multi gpu working again. Hopefully this works it's way back into Koboldcpp soon. One more thing, it does not say it anywhere but when you launch koboldcpp from the command line you need to add --usecublas for it to work as there is no usehipblas option. I didn't see that documented anywhere.

[henk717]

As others pointed out, people running in to this issue can blame AMD for not caring about their GPU and not shipping it as part of their SDK. This is not a bug, this is AMD not shipping what we need.
Of course people like YellowRose and other AMD community members keep looking for a way to get more GPU's enabled trough unofficial support, but Koboldcpp's ROCm fork is working as intended.

Those effected can use the regular Koboldcpp edition with CLBlast which does support your GPU.