Llama3/GGUF models do not load after v1.67 on Windows.

[dmjohn0x]

This has been an ongoing problem since 1.68.0, but I had hoped somebody would mention it, as I was fighting not to add 2FA to my account, thus was not allowed to comment or leave feedback. But there has been several releases and not a soul has mentioned it, so I finally gave in to report this problem.

On Windows, no version after 1.67.0 actually loads Llama3/gguf models. It always quickly returns an access violation error on the screen and immediately closes upon loading. Im currently running windows 11 23H2, on a Ryzen 7700X w/ 32GB Ram and a 7900 XT w/ 20GB Vram. All of the 1.67.0 releases run perfectly for me. But every version after that crashes when I go to run a modern model.

[klop08]

Same for me, the last Version that worked was koboldcpp_rocm4allV3.exe. Any other version just throws Access violation errors loading any model if I use Hipblas. Probably rocm 5.7 related issues. Maybe "built by deleting the entire stock AMD ROCm 5.7.1 GPU code folder and replacing it with only ROCm-4-All-5.7.1 Tensile Library files before compiling." would help.

[klop08]

Update: Probably not a rocm issue. Didn't realize that 6.1 already dropped. Same issue here. But I think the problem could be Windows 11 itself after all. I noticed that after I download it, the Windows Defender Smart Screen issues a warning for the app not being trusted. It works after I allow it once. If I close it and start again, I don't get any warnings anymore and I only get Access violation.
Eventviewer doesn't say anything, but I would guess something from windows is blocking Koboldcpp from accessing the GPU.

[dmjohn0x]

Yeah, I dont get any defender prompts. (Probably because I've deactivated Windows Defender) So I don't actually know what the issue is. Just that launching a model in anything after 1.67.0 results in some access violation error then it closes almost instantaneously. I tried many models, and they all run fine on 1.67.y0 rocm4allv3.exe, but nothing works on 1.68,1.69, or 1.69.1

[cloak1505]

Every .exe release is always "untrusted". It costs developers money to get certificates to become a "known publisher". The SmartScreen popup just warns you about running an unrecognized app the first time you run it. I don't think "allow .exe from unknown publisher to run but not use GPU" is a thing. Works for me.

[klop08]

I'm aware of the "untrusted" warning. As I mentioned, it works after I allow it, but only once. If I start the .exe file again, I don't get any warning anymore, but I only encounter Access Violation issues after that. If I download it again, I get the warning again, and it works again only once. It's very weird, and maybe there is a correlation. But its only a guess.

What OS and patch are you using? Are you also on Windows 11 23H2?

[klop08]

1.70 also doesn't work properly. Even running it via Python in Windows gives me the same access violation at 0x0000000 loading any model. The .exe at least works once after the download. Still don't know what the issue is.

[dmjohn0x]

Can confirm. 1.70 yields the same violation errors loading models in Win11 23H2 as all previous models going back to v1.68.

1.67.0 Rocm4Allv3 was the last version to work on Windows.

[RachaelsDen]

The same here. I noticed some support for ROCM under WSL now, so I tried to run it that way as well. While I didn't get the access violation message, it would just freeze, forcing me to kill the process.

I'm using a 7900 XTX if that helps any. I have tried using DDU to remove and reinstall my GPU drivers, but that didn't seem to help. I was wondering if trying the pro drivers might make any difference.

Little added info, this was with the most recent AMD drivers as of today (7/21/2024) , 24.7.1 on Windows 11 Pro 23H2

[RachaelsDen]

Will add that compiling from sourcing using ROCm 5.7 and 6.1 both result in the same error. I'll keep digging.

[Krxon]

[klop08]

Well you are using Vulkan. The Issues mentiond only apply on Rocm. You probably also can test it and see it yourself if you chose the Rocm option. Vulkan works fine, but is way slowler in my experience.

[Tatalebuj]

Oh, I didn't realize that. I'll go try the 1.67 version with ROCM to see what the difference is. Thanks.

[Tatalebuj]

Well you are using Vulkan. The Issues mentiond only apply on Rocm. You probably also can test it and see it yourself if you chose the Rocm option. Vulkan works fine, but is way slowler in my experience.

I was able to run the same GGUF model using 1.70 and ROCM.

Does that help?

[dmjohn0x]

1.71 Still doesnt work on Windows. 1.67.yr0 is the last version working on windows, Tatalebuj is likely using Linux and didnt read the title or failed to mention it. YellowRoseCX has yet to acknowledge that versions after 1.67 no longer run on windows. Likely the same issue we were having prior to 1.67 when several releases wouldnt run on Windows.

[Tatalebuj]

1.71 Still doesnt work on Windows. 1.67.yr0 is the last version working on windows,

I attempted to load WSL once, but I am pretty sure I've done an upgrade since and swapped hard drives, but maybe? Could those files I installed help run the windows version of Koboldcpp_ROCM.exe? Happy to look where ever to help figure that out, I'm just not familiar with how to do so.

Tatalebuj is likely using Linux and didnt read the title or failed to mention it.

I'm using Windows Pro 11. And I'm running the koboldcpp_rocm.exe file out of my KoboldCPP_ROCM folder. I download the latest .exe from YellowRose, and the one I'm currently using says it is 1.70 and here is my latest run using a new .GGUF model. I'll attach what the txt says on my screen (I tried to paste, but the text got very ugly and I couldn't figure out how to properly format it.)
ScrnShot.txt

YellowRoseCX has yet to acknowledge that versions after 1.67 no longer run on windows. Likely the same issue we were having prior to 1.67 when several releases wouldnt run on Windows.

I don't know what that is, but I'm happy to learn that I'm somehow using Linux, because I have no capability with Linux (I don't know hardly anything about it and do not remember installing it specifically.) I do have several other Local LM applications loaded that I used for other purposes from time to time - though I've not found a way to consistently get the systems working, so I'm really not very good at this - so if running OLLAMA or Docker Dekstop could cause me to run KoboldCPP_ROCM in Linux, then maybe you are right.

[nnyanpasu]

strange, but my llama3 launches normally on win11 with the latest updates and drivers. In any case, I'm here to ask how you are with launching llama3.1? I haven't experimented with the settings yet, but 3.1 always crashes.
.\KoboldCPP-v1.70.yr0-ROCm.exe --usecublas --gpulayers 55 --contextsize 16384 .\L3-Umbral-Mind-RP-v3.0-8B.Q8_0.gguf

[klop08]

Can confirm koboldcpp_v1.72.yr0-rocm_6.1.2.exe finally works again. But the most recent "normal" Version koboldcpp_v1.72.yr0-rocm, doesn't work.
So as @dmjohn0x said hopefully new future 6.1.2 builds will work as well.

[YellowRoseCx]

does this version work for you guys? https://github.com/YellowRoseCx/koboldcpp-rocm/releases/tag/v1.73.1.yr1-ROCm

[klop08]

ggml_cuda_init: found 1 ROCm devices: Device 0: AMD Radeon RX 7900 XT, compute capability 11.0, VMM: no Traceback (most recent call last): File "koboldcpp.py", line 4881, in <module> File "koboldcpp.py", line 4526, in main File "koboldcpp.py", line 894, in load_model OSError: exception: access violation reading 0x0000000000000000 [16960] Failed to execute script 'koboldcpp' due to unhandled exception!

No, it doesn't. At least for me. (Running it with ROCm 6.1.2, Windows 11 RX 7900XT). Back to using "koboldcpp_v1.72.yr0-rocm_6.1.2".

[dmjohn0x]

does this version work for you guys? https://github.com/YellowRoseCx/koboldcpp-rocm/releases/tag/v1.73.1.yr1-ROCm

No. 1.72.yr0-rocm_6.1.2 Was working. The version that said to ignore... But this new version doesn't run on windows with the 7900XT:

A strange observation I've made is that almost everyone who is having issues since 1.68 in this thread seems to be specifically running a 7900XT.

[nnyanpasu]

does this version work for you guys? https://github.com/YellowRoseCx/koboldcpp-rocm/releases/tag/v1.73.1.yr1-ROCm

for me - yes, win11 23h2 7900xtx

[klop08]

Just an Info. Newest Version "KoboldCPP-v1.75.2.yr1-ROCm" seem to work!

[dmjohn0x]

I still cannot run any version higher than "koboldcpp_v1.72.yr0-rocm_6.1.2.exe" on my 7900XT... I dont understand why only the version you've tagged as "IGNORE" is the only version that has actually worked since 1.67.yr0_rocm4allV3.exe...

If you have any reasoning for this or know what I can do to use newer versions, I'd be incredibly appreciative.

[YellowRoseCx]

have you tried other models? bc at this point, there's nothing wrong with the program or the rocblas files. It seems like only some people with 7900xt might get this error and as you mentioned a few days ago, at one point, it worked for 2 days and then stopped working the 3rd day, so I think it's more something with your computer setup. (I don't mean that to sound rude, but the program wouldn't have changed just sitting on your pc)

Since you have a 7900XT, have you tried using WSL?

[Krxon]

Since you have a 7900XT, have you tried using WSL?
Nope. But as I said before, I will not spend any more time on this.
I did testing because you asked :)
Version * Koboldcpp_v1.72.yr0-rocm_6.1.2 * still works totally fine - most of my gguf models and sd 1.5 models are working with it.
Few other ppl reported this as well, where koboldcpp_v1.72.yr0-rocm_6.1.2.exe is the last one working.
Maybe some newer versions will work again, until then I am saving for 4090
Thanks a lot for the efforts, though :)

[eo2day]

Can attest that this is still happening with the latest ROCm build release.
CPU: Ryzen 7 7800X3D 8 Core.
GPU: AMD 7900xt
Latest Windows 11 - 23H2.

I run Comfyui and ForgeUI (SD web ui) just fine.

PS D:\LLMs\Kobold> .\koboldcpp_rocm.exe
***
Welcome to KoboldCpp - Version 1.76.yr1-ROCm
For command line arguments, please refer to --help
***
Auto Selected HIP Backend...

Auto Recommended GPU Layers: 43
Attempting to use hipBLAS library for faster prompt ingestion. A compatible AMD GPU will be required.
Initializing dynamic library: koboldcpp_hipblas.dll
==========
Namespace(model='', model_param='D:/LLMs/Kobold/Models/mythalion-kimiko-v2.Q4_K_M.gguf', port=5001, port_param=5001, host='', launch=True, config=None, threads=7, usecublas=['normal', '0'], usevulkan=None, useclblast=None, usecpu=False, contextsize=4096, gpulayers=43, tensor_split=None, checkforupdates=False, ropeconfig=[0.0, 10000.0], blasbatchsize=2048, blasthreads=7, lora=None, noshift=False, nommap=False, usemlock=False, noavx2=False, debugmode=0, onready='', benchmark=None, prompt='', promptlimit=100, multiuser=1, remotetunnel=False, highpriority=False, foreground=False, preloadstory=None, quiet=False, ssl=None, nocertify=False, mmproj=None, password=None, ignoremissing=False, chatcompletionsadapter=None, flashattention=True, quantkv=0, forceversion=0, smartcontext=False, unpack='', nomodel=False, showgui=False, skiplauncher=False, hordemodelname='', hordeworkername='', hordekey='', hordemaxctx=0, hordegenlen=0, sdmodel='', sdthreads=7, sdclamped=0, sdvae='', sdvaeauto=False, sdquant=False, sdlora='', sdloramult=1.0, whispermodel='', hordeconfig=None, sdconfig=None, noblas=False)
==========
Loading model: D:\LLMs\Kobold\Models\mythalion-kimiko-v2.Q4_K_M.gguf

The reported GGUF Arch is: llama
Arch Category: 0

---
Identified as GGUF model: (ver 6)
Attempting to Load...
---
Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead!
It means that the RoPE values written above will be replaced by the RoPE values indicated after loading.
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 0 |
Traceback (most recent call last):
  File "koboldcpp.py", line 4937, in <module>
  File "koboldcpp.py", line 4580, in main
  File "koboldcpp.py", line 890, in load_model
OSError: exception: access violation reading 0x0000000000000000
[12428] Failed to execute script 'koboldcpp' due to unhandled exception!

[dmjohn0x]

if any of you get time to try this out, I've made a new test build: https://github.com/YellowRoseCx/koboldcpp-rocm/releases/tag/v1.76.yr0-ROCm-7900xt

That doesn't work, nor does 1.78 or the latest 1.79.yr0. I dont know what you did differently for koboldcpp_v1.72.yr0-rocm_6.1.2.exe, but thats the last version I've got that works. And the non 6.1.2.exe version doesnt work. So I'd focus on whatever you did different there.

[PhialsBasement]

if any of you get time to try this out, I've made a new test build: https://github.com/YellowRoseCx/koboldcpp-rocm/releases/tag/v1.76.yr0-ROCm-7900xt

That doesn't work, nor does 1.78 or the latest 1.79.yr0. I dont know what you did differently for koboldcpp_v1.72.yr0-rocm_6.1.2.exe, but thats the last version I've got that works. And the non 6.1.2.exe version doesnt work. So I'd focus on whatever you did different there.

Whats your rocm version?

[zac-market]

For what it's worth I'm using v1.77.yr1-ROCm, and it has been working absolutely beautifully on both my 6800 XT and 7900 XTX.

I'm on Windows 11 Pro 24H2.