Use NVIDIA GPU Operator for GPU nodes

[davidspek]

NVIDIA provides a GPU operator that handles deploying the DaemonSet, installing the driver, setting up the container runtime, and provides a metrics endpoint for Prometheus to scrape. It would be nice if this was deployed on GPU nodes automatically so people can use them without further configuration.
Docs for installing the operator: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/getting-started.html#install-kubernetes

Requirements are that the NVIDIA driver is not installed on the host system and that nouveau drivers are disabled. For Ubuntu based systems you can disable nouveau at boot with the following kernel parameters, avoiding the need to reboot nodes:

nomodeset modprobe.blacklist=nouveau nouveau.blacklist=1