Generalize VGPU drivers installation #93

cmd-ntrf · 2021-02-10T15:49:38Z

Need MC branch keystone

ocaisa · 2021-02-12T13:18:41Z

site/profile/manifests/gpu.pp

+  ensure_packages(['kernel-devel'], {ensure => 'installed'})
+  ensure_packages(['dkms'], {
+    'require' => Yumrepo['epel']
+  })


There's something wrong with this because it is not catching this requirement:

Feb 12 13:08:53 gpu-node3 puppet-agent[1084]: (/Stage[main]/Profile::Gpu::Install::Deps/File[/etc/nvidia]/ensure) created Feb 12 13:09:07 gpu-node3 yum[2001]: Installed: kernel-devel-3.10.0-1160.15.2.el7.x86_64 Feb 12 13:09:07 gpu-node3 puppet-agent[1084]: (/Stage[main]/Profile::Gpu::Install::Deps/Package[kernel-devel]/ensure) created Feb 12 13:09:31 gpu-node3 puppet-agent[1084]: (/Stage[main]/Profile::Gpu::Install::Vgpu::Bin/Exec[vgpu-driver-install-bin]/returns) % Total % Received % Xferd Average Feb 12 13:09:31 gpu-node3 puppet-agent[1084]: (/Stage[main]/Profile::Gpu::Install::Vgpu::Bin/Exec[vgpu-driver-install-bin]/returns) Dload U Feb 12 13:09:31 gpu-node3 puppet-agent[1084]: [244B blob data] Feb 12 13:09:31 gpu-node3 puppet-agent[1084]: (/Stage[main]/Profile::Gpu::Install::Vgpu::Bin/Exec[vgpu-driver-install-bin]/returns) Verifying archive integrity... OK Feb 12 13:09:31 gpu-node3 puppet-agent[1084]: (/Stage[main]/Profile::Gpu::Install::Vgpu::Bin/Exec[vgpu-driver-install-bin]/returns) Uncompressing NVIDIA Accelerated Graphics Feb 12 13:09:31 gpu-node3 puppet-agent[1084]: (/Stage[main]/Profile::Gpu::Install::Vgpu::Bin/Exec[vgpu-driver-install-bin]/returns) ERROR: Unable to find the development too Feb 12 13:09:31 gpu-node3 puppet-agent[1084]: (/Stage[main]/Profile::Gpu::Install::Vgpu::Bin/Exec[vgpu-driver-install-bin]/returns) ERROR: Installation has failed. Please s Feb 12 13:09:31 gpu-node3 puppet-agent[1084]: (/Stage[main]/Profile::Gpu::Install::Vgpu::Bin/Exec[vgpu-driver-install-bin]/returns) Welcome to the NVIDIA Software Installer Feb 12 13:09:31 gpu-node3 puppet-agent[1084]: (/Stage[main]/Profile::Gpu::Install::Vgpu::Bin/Exec[vgpu-driver-install-bin]/returns) Detected 2 CPUs online; setting concurren Feb 12 13:09:31 gpu-node3 puppet-agent[1084]: (/Stage[main]/Profile::Gpu::Install::Vgpu::Bin/Exec[vgpu-driver-install-bin]/returns) Tagging shared libraries with chcon -t te Feb 12 13:09:31 gpu-node3 puppet-agent[1084]: (/Stage[main]/Profile::Gpu::Install::Vgpu::Bin/Exec[vgpu-driver-install-bin]/returns) Installing NVIDIA driver version 450.89. Feb 12 13:09:31 gpu-node3 puppet-agent[1084]: (/Stage[main]/Profile::Gpu::Install::Vgpu::Bin/Exec[vgpu-driver-install-bin]/returns) For some distributions, Nouveau can be di Feb 12 13:09:31 gpu-node3 puppet-agent[1084]: (/Stage[main]/Profile::Gpu::Install::Vgpu::Bin/Exec[vgpu-driver-install-bin]/returns) One or more modprobe configuration files Feb 12 13:09:31 gpu-node3 puppet-agent[1084]: 'curl -L https://hpsrepo.fz-juelich.de/jusuf/nvidia/NVIDIA-Driver.latest -o /tmp/NVIDIA-driver.run && sh /tmp/NVIDIA-driver.run Feb 12 13:09:31 gpu-node3 puppet-agent[1084]: (/Stage[main]/Profile::Gpu::Install::Vgpu::Bin/Exec[vgpu-driver-install-bin]/returns) change from 'notrun' to ['0'] failed: 'cu

It is installing kernel-devel but not recognising that the dkms requirement is not met.

Later I see dkms being installed (which brings in gcc), so the second time around things will succeed

Strategy shift: I have added an explicit requirement on dkms and kernel-devel package on the Exec[vgpu-driver-install-bin] resource and removed the deps class.

Confirmed, that works, the drivers get installed at the right time.

ocaisa · 2021-02-12T15:08:35Z

There is one last remaining issue (for me). I see the symlinks being created for the latest nvidia driver (not the one I installed). This happens long before the driver is installed. It looks like this is expected, as after the driver install, the /usr/lib64/nvidia looks like:

[centos@gpu-node1 ~]$ ls /usr/lib64/nvidia/
gridd  libGLX_indirect.so.0  libnvidia-fatbinaryloader.so.460.32.03

so most of those links are removed, but not created for the installed driver (450.89).

ocaisa · 2021-02-12T15:14:28Z

I'll see what it is like in 30 minutes when puppet runs again

ocaisa · 2021-02-12T15:22:16Z

Yes, these got replaced on the next run, with the caveat that there are now two broken links in that directory:

lrwxrwxrwx. 1 root root  46 Feb 12 15:19 libnvidia-fatbinaryloader.so.450.89 -> /usr/lib64/libnvidia-fatbinaryloader.so.450.89
lrwxrwxrwx. 1 root root  49 Feb 12 14:51 libnvidia-fatbinaryloader.so.460.32.03 -> /usr/lib64/libnvidia-fatbinaryloader.so.460.32.03

cmd-ntrf · 2021-02-12T15:35:46Z

The symlink issue is a tough one.

The driver version is determined by a Puppet fact. Facts are resolved before puppet run. On the first puppet run, nvidia_driver_vers.sh is executed first. Since the drivers are not installed, the driver version cannot be determined from dkms or nvidia-smi because they are not installed yet, so the script defaults on fetching the drivers version from NVIDIA website.

The following Puppet run, nvidia_driver_vers.sh can now use etiher nvidia-smi or dkms and returns a different version number, although the symlinks with the other driver version have already been created, and removing the previously created symlinks is complicated.

Ideally, we would be able to know on the first Puppet run, what is the VGPUs driver version, or we create the symlink with an exec script instead of a Puppet file resource.

ocaisa · 2021-02-12T15:39:58Z

Something did remove the previous symlinks, I suspect that may have been the nvidia driver installation, which is perhaps why that single broken link libnvidia-fatbinaryloader.so.460.32.03 -> /usr/lib64/libnvidia-fatbinaryloader.so.460.32.03 was left over (since it doesn't actually exist).

ocaisa · 2021-02-16T15:17:29Z

Just to record this somewhere, vGPUs have some limitations when it comes to CUDA (correct as of 11.2, see NVIDIA docs):

NVIDIA vGPU does not support the following NVIDIA CUDA Toolkit features:

- Unified Memory
- Development tools such as IDEs, debuggers, profilers, and utilities as listed under CUDA Toolkit Major Components in NVIDIA CUDA Toolkit Release Notes for CUDA 11.2
- Tracing and profiling through the CUDA Profiling Tools Interface (CUPTI)

There features are only available in pass-through mode.

Avoid creation of broken symlink when dealing with VGPU drivers.

site/profile/manifests/gpu.pp

cmd-ntrf added the enhancement New feature or request label Feb 10, 2021

cmd-ntrf self-assigned this Feb 10, 2021

cmd-ntrf force-pushed the dev/vgpu2 branch from 4efcb1f to 58c13d3 Compare February 11, 2021 14:38

ocaisa reviewed Feb 12, 2021

View reviewed changes

cmd-ntrf force-pushed the dev/vgpu2 branch from 0eb1d68 to e650304 Compare February 15, 2021 17:31

cmd-ntrf force-pushed the dev/vgpu2 branch from e650304 to 774a14a Compare February 16, 2021 15:54

cmd-ntrf and others added 17 commits February 16, 2021 16:22

Add cloud provide and region to static fact template

7cd727f

Replace List by Array

15fee35

Fix arbutus hieradata

50f490c

Add missing quote

410ebce

Fix source for vgpu repo in arbutus

7650a48

Decompose vgpu install

2ee6563

Move dkms dependency up

696a14e

Add class to install nvidia vgpu driver from bin installer

c403153

Add hieradata for openstack jusuf-cloud

473b463

Fix vgpu package name

ec21ef3

Fix split command

7d53a36

Add missing include for vgpu and bin

9fca4a3

Fix jusuf-cloud gridd source key in hieradata

6042478

Move gpu driver dependencies higher up

4ae44db

Make sure installer succeeds

d763b07

Add explicit requirement on dkms and kernel-devel

bc36283

Do not execute dkms autoinstall with vgpu binary

8664e64

cmd-ntrf force-pushed the dev/vgpu2 branch from a8818f2 to 8664e64 Compare February 16, 2021 21:23

Refactor nvidia symlink creation

54ea3ec

Avoid creation of broken symlink when dealing with VGPU drivers.

ocaisa reviewed Feb 23, 2021

View reviewed changes

site/profile/manifests/gpu.pp Show resolved Hide resolved

Make gpu symlink location configurable

9e8bbc9

cmd-ntrf force-pushed the dev/vgpu2 branch from 639c1e2 to 9e8bbc9 Compare February 23, 2021 20:02

cmd-ntrf merged commit fb3c903 into master Feb 24, 2021

cmd-ntrf deleted the dev/vgpu2 branch February 24, 2021 16:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generalize VGPU drivers installation #93

Generalize VGPU drivers installation #93

cmd-ntrf commented Feb 10, 2021

ocaisa Feb 12, 2021

ocaisa Feb 12, 2021

cmd-ntrf Feb 12, 2021

ocaisa Feb 12, 2021

ocaisa commented Feb 12, 2021 •

edited

Loading

ocaisa commented Feb 12, 2021

ocaisa commented Feb 12, 2021

cmd-ntrf commented Feb 12, 2021

ocaisa commented Feb 12, 2021

ocaisa commented Feb 16, 2021 •

edited

Loading

Generalize VGPU drivers installation #93

Generalize VGPU drivers installation #93

Conversation

cmd-ntrf commented Feb 10, 2021

ocaisa Feb 12, 2021

Choose a reason for hiding this comment

ocaisa Feb 12, 2021

Choose a reason for hiding this comment

cmd-ntrf Feb 12, 2021

Choose a reason for hiding this comment

ocaisa Feb 12, 2021

Choose a reason for hiding this comment

ocaisa commented Feb 12, 2021 • edited Loading

ocaisa commented Feb 12, 2021

ocaisa commented Feb 12, 2021

cmd-ntrf commented Feb 12, 2021

ocaisa commented Feb 12, 2021

ocaisa commented Feb 16, 2021 • edited Loading

ocaisa commented Feb 12, 2021 •

edited

Loading

ocaisa commented Feb 16, 2021 •

edited

Loading