Build failure: llama-cpp

[suhr]

llama-cpp fails to build with cudaSupport = true

  nixpkgs.overlays = [
    inputs.nur.overlay
    (final: prev: {
      llama-cpp = prev.llama-cpp.override { cudaSupport = true; };
    })
  ];

Build log

/build/source/ggml.c: In function 'ggml_init':
/build/source/ggml.c:2230:9: error: implicit declaration of function 'ggml_init_cublas' [-Werror=implicit-function-declaration]
 2230 |         ggml_init_cublas();
      |         ^~~~~~~~~~~~~~~~
/build/source/ggml.c: In function 'ggml_compute_forward':
/build/source/ggml.c:13707:21: error: implicit declaration of function 'ggml_cuda_compute_forward'; did you mean 'ggml_compute_forward'? [-Werror=implicit-function-declaration]
13707 |     bool skip_cpu = ggml_cuda_compute_forward(params, tensor);
      |                     ^~~~~~~~~~~~~~~~~~~~~~~~~
      |                     ggml_compute_forward
/build/source/ggml.c: In function 'ggml_get_n_tasks':
/build/source/ggml.c:15557:21: error: implicit declaration of function 'ggml_cuda_can_mul_mat'; did you mean 'ggml_can_mul_mat'? [-Werror=implicit-function-declaration]
15557 |                 if (ggml_cuda_can_mul_mat(node->src[0], node->src[1], node)) {
      |                     ^~~~~~~~~~~~~~~~~~~~~
      |                     ggml_can_mul_mat
cc1: some warnings being treated as errors

Notify maintainers

@dit7ya @elohmeier

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.4, NixOS, 24.05 (Uakari), 24.05.20231204.2c7f3c0`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.19.2`
 - channels(root): `"nixos"`
 - channels(suhr): `""`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

---

Add a 👍 reaction to issues you find important.

[blurgyy]

It seems it needs openblasSupport = false for llama-cpp to build with cuda/rocm support. Related: #257760 (comment)

[teto]

the openblasSupport = false worked for me until I updated to the latest nixos-unstable:

       >   "-L/nix/store/sxr2bhshikyqylhbdk3zjwlxkw5pxns7-cudatoolkit-11.8.0-merged/bin/..//lib64/stubs"
       >   "-L/nix/store/sxr2bhshikyqylhbdk3zjwlxkw5pxns7-cudatoolkit-11.8.0-merged/bin/..//lib64"
       >
       >
       >   #$ CUDAFE_FLAGS=
       >
       >   #$ PTXAS_FLAGS=
       >
       >   #$ rm tmp/a_dlink.reg.c
       >
       >   #$ gcc -D__CUDA_ARCH__=520 -D__CUDA_ARCH_LIST__=520 -E -x c++
       >   -DCUDA_DOUBLE_MATH_FUNCTIONS -D__CUDACC__ -D__NVCC__
       >   "-I/nix/store/sxr2bhshikyqylhbdk3zjwlxkw5pxns7-cudatoolkit-11.8.0-merged/bin/..//include"
       >   -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=8
       >   -D__CUDACC_VER_BUILD__=89 -D__CUDA_API_VER_MAJOR__=11
       >   -D__CUDA_API_VER_MINOR__=8 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include
       >   "cuda_runtime.h" -m64 "CMakeCUDACompilerId.cu" -o
       >   "tmp/CMakeCUDACompilerId.cpp1.ii"
       >
       >   In file included from
       >   /nix/store/i1xwyzjcyyz2msv89blck1afz4am979n-cudatoolkit-11.8.0/include/cuda_runtime.h:83,
       >
       >
       >                    from <command-line>:
       >
       >
       >   /nix/store/i1xwyzjcyyz2msv89blck1afz4am979n-cudatoolkit-11.8.0/include/crt/host_config.h:132:2:
       >   error: #error -- unsupported GNU version! gcc versions later than 11 are
       >   not supported! The nvcc flag '-allow-unsupported-compiler' can be used to
       >   override this version check; however, using an unsupported host compiler
       >   may cause compilation failure or incorrect run time execution.  Use at your
       >   own risk.
       >
       >     132 | #error -- unsupported GNU version! gcc versions later than 11 are not supported! The nvcc flag '-allow-unsupported-compiler' can be used to override this version check; however, using an unsupported host compiler may cause compilation failure or incorrect run time execution. Use at your own risk.
       >         |  ^~~~~
       >
       >   # --error 0x1 --
       >
       >
       >
       >
       >
       > Call Stack (most recent call first):
       >   /nix/store/vnhl4zdy7igx9gd3q1d548vwzz15a9ma-cmake-3.27.7/share/cmake-3.27/Modules/CMakeDetermineCompilerId.cmake:8 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
       >   /nix/store/vnhl4zdy7igx9gd3q1d548vwzz15a9ma-cmake-3.27.7/share/cmake-3.27/Modules/CMakeDetermineCompilerId.cmake:53 (__determine_compiler_id_test)
       >   /nix/store/vnhl4zdy7igx9gd3q1d548vwzz15a9ma-cmake-3.27.7/share/cmake-3.27/Modules/CMakeDetermineCUDACompiler.cmake:307 (CMAKE_DETERMINE_COMPILER_ID)
       >   CMakeLists.txt:266 (enable_language)
       >
       > 
       > -- Configuring incomplete, errors occurred!
       For full logs, run 'nix log /nix/store/1yx1q8s57ld4ccsn9d3sk6ndcq5yka99-llama-cpp-1573.drv'.
error: 1 dependencies of derivation '/nix/store/b9pr1axpr2sf748xsf8sm4hkrjfvk57b-ollama-0.1.11.drv' failed to build

I fixerd it with

diff --git a/pkgs/by-name/ll/llama-cpp/package.nix b/pkgs/by-name/ll/llama-cpp/package.nix
index 77a452a26cb0..adb610ddfb7a 100644
--- a/pkgs/by-name/ll/llama-cpp/package.nix
+++ b/pkgs/by-name/ll/llama-cpp/package.nix
@@ -4,7 +4,7 @@
 , fetchFromGitHub
 , fetchpatch
 , nix-update-script
-, stdenv
+, gcc11Stdenv
 , symlinkJoin
 
 , config
@@ -23,6 +23,9 @@
 }:
 
 let
+
+  stdenv = gcc11Stdenv;
+
   cudatoolkit_joined = symlinkJoin {
     name = "${cudaPackages.cudatoolkit.name}-merged";
     paths = [
     ```

[zopieux]

Or alternatively to patching nixpkgs itself, which is annoying, here as a dependency of ollama:

with pkgs; (ollama.override { llama-cpp = (llama-cpp.override { stdenv = gcc11Stdenv; }); }

[phillipzink]

Or alternatively to patching nixpkgs itself, which is annoying, here as a dependency of ollama:

with pkgs; (ollama.override { llama-cpp = (llama-cpp.override { stdenv = gcc11Stdenv; }); }

Where should this line be added? Somewhere in configuration.nix?

[zopieux]

Where should this line be added?

Where you would have otherwise used just ollama, for example in environment.systemPackages. Or alternatively, as an nixos overlay (see docs).

[SomeoneSerge]

This:

nixpkgs/pkgs/by-name/ll/llama-cpp/package.nix

Line 39 in bfbfb34

stdenv.mkDerivation (finalAttrs: {

needs to be rewritten like so:

nixpkgs/pkgs/development/libraries/opencv/4.x.nix

Lines 90 to 93 in bfbfb34

	# It's necessary to consistently use backendStdenv when building with CUDA
	# support, otherwise we get libstdc++ errors downstream
	stdenv = throw "Use effectiveStdenv instead";
	effectiveStdenv = if enableCuda then cudaPackages.backendStdenv else inputs.stdenv;

nixpkgs/pkgs/development/libraries/opencv/4.x.nix

Line 244 in bfbfb34

effectiveStdenv.mkDerivation {

At least until #226165 had been addressed

If anybody gets around to do this, please try also to remove any references to the deprecated cudaPackages.cudatoolkit:

nixpkgs/pkgs/by-name/ll/llama-cpp/package.nix

Lines 26 to 36 in bfbfb34

	cudatoolkit_joined = symlinkJoin {
	name = "${cudaPackages.cudatoolkit.name}-merged";
	paths = [
	cudaPackages.cudatoolkit.lib
	cudaPackages.cudatoolkit.out
	] ++ lib.optionals (lib.versionOlder cudaPackages.cudatoolkit.version "11") [
	# for some reason some of the required libs are in the targets/x86_64-linux
	# directory; not sure why but this works around it
	"${cudaPackages.cudatoolkit}/targets/${stdenv.system}"
	];
	};

Instead, add cuda_nvcc to nativeBuildInputs, and normal libraries (cuda_cudart, libcublas, etc) to buildInputs.

Thanks!

[zopieux]

Tangent: thanks @SomeoneSerge for your contributions on improving the CUDA landscape in the nix ecosystem. You seem to be distilling a lot of good practices and guidelines around CUDA packaging in various GitHub issue comments that sadly won't get any visibility once closed. Please consider adding this knowledge to the manual, which currently is very dry in terms of examples. Cheers! 👍

[suhr]

Have to open this back, since #275807 not only does not fix the issue, but also breaks the workaround.

[suhr]

I tried to adapt https://github.com/ggerganov/llama.cpp/blob/master/.devops/nix/package.nix, but I get the following build error:

[61/101] Linking C executable bin/test-c
FAILED: bin/test-c
: && /nix/store/36cm1ws5sikci8wl4swyfgbk1vr2ydyg-gcc-wrapper-11.4.0/bin/gcc -O3 -DNDEBUG  tests/CMakeFiles/test-c.dir/test-c.c>
/nix/store/fzlkaj1ax7gl655blfcr6zzvml1vx3bj-binutils-2.40/bin/ld: warning: libcuda.so.1, needed by libllama.so, not found (try>
/nix/store/fzlkaj1ax7gl655blfcr6zzvml1vx3bj-binutils-2.40/bin/ld: libllama.so: undefined reference to `cuMemCreate'
/nix/store/fzlkaj1ax7gl655blfcr6zzvml1vx3bj-binutils-2.40/bin/ld: libllama.so: undefined reference to `cuMemAddressReserve'
/nix/store/fzlkaj1ax7gl655blfcr6zzvml1vx3bj-binutils-2.40/bin/ld: libllama.so: undefined reference to `cuMemSetAccess'
/nix/store/fzlkaj1ax7gl655blfcr6zzvml1vx3bj-binutils-2.40/bin/ld: libllama.so: undefined reference to `cuDeviceGet'
/nix/store/fzlkaj1ax7gl655blfcr6zzvml1vx3bj-binutils-2.40/bin/ld: libllama.so: undefined reference to `cuGetErrorString'
/nix/store/fzlkaj1ax7gl655blfcr6zzvml1vx3bj-binutils-2.40/bin/ld: libllama.so: undefined reference to `cuDeviceGetAttribute'
/nix/store/fzlkaj1ax7gl655blfcr6zzvml1vx3bj-binutils-2.40/bin/ld: libllama.so: undefined reference to `cuMemMap'
/nix/store/fzlkaj1ax7gl655blfcr6zzvml1vx3bj-binutils-2.40/bin/ld: libllama.so: undefined reference to `cuMemRelease'
/nix/store/fzlkaj1ax7gl655blfcr6zzvml1vx3bj-binutils-2.40/bin/ld: libllama.so: undefined reference to `cuMemGetAllocationGranu>
collect2: error: ld returned 1 exit status

[happysalada]

hey, to address the cudaSupport failure, I went ahead and created #277709
this might not solve the whole problem, but hopefully at least part of it.

[SomeoneSerge]

/nix/store/fzlkaj1ax7gl655blfcr6zzvml1vx3bj-binutils-2.40/bin/ld: libllama.so: undefined reference to `cuMemGetAllocationGranu>
collect2: error: ld returned 1 exit status

This is https://gitlab.kitware.com/cmake/cmake/-/issues/25536 (which is most likely not a cmake issue, but an error on nvidia's part). We've added a hostfix for the redist packages (e.g. cuda_cudart but not cudatoolkit): #276780