From c09b87e9741da69620dba003266eddf97d81c7ce Mon Sep 17 00:00:00 2001 From: Mitali Paygude Date: Wed, 24 Jan 2024 15:12:55 -0800 Subject: [PATCH] Add troubleshooting section for labeling nodes with reserved labels (#6758) * Add troubleshooting section for labeling nodes with reserved labels * Update docs/content/en/docs/troubleshooting/troubleshooting.md Co-authored-by: Vivek Koppuru * Update docs/content/en/docs/troubleshooting/troubleshooting.md Co-authored-by: Vivek Koppuru * Update docs/content/en/docs/troubleshooting/troubleshooting.md Co-authored-by: Chris Negus * Update docs/content/en/docs/troubleshooting/troubleshooting.md Co-authored-by: Drew Flower * Update docs/content/en/docs/troubleshooting/troubleshooting.md Co-authored-by: Drew Flower * Update docs/content/en/docs/troubleshooting/troubleshooting.md Co-authored-by: Drew Flower --------- Co-authored-by: Vivek Koppuru Co-authored-by: Chris Negus Co-authored-by: Drew Flower --- .../docs/troubleshooting/troubleshooting.md | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/docs/content/en/docs/troubleshooting/troubleshooting.md b/docs/content/en/docs/troubleshooting/troubleshooting.md index 56ad85e7e56e..54f07ce0ad73 100755 --- a/docs/content/en/docs/troubleshooting/troubleshooting.md +++ b/docs/content/en/docs/troubleshooting/troubleshooting.md @@ -500,6 +500,49 @@ To fix it, make sure to validate the network/firewall settings from the workload * [Prerequisite Checklist for EKS Anywhere on Snow]({{< relref "../getting-started/snow/snow-getstarted/#prerequisite-checklist" >}}) * [Requirements for EKS Anywhere on Nutanix Cloud Infrastructure]({{< relref "../getting-started/nutanix/nutanix-prereq" >}}) +### Labeling nodes with reserved labels such as `node-role.kubernetes.io` fails with kubeadm error during bootstrap + +If cluster creation or upgrade fails to complete successfully and kubelet throws an error similar to the one below, please refer to this section. The cluster spec for EKS Anywhere create or upgrade should look like: + +``` +. +. + controlPlaneConfiguration: + count: 2 + endpoint: + host: "192.168.x.x" + labels: + "node-role.kubernetes.io/control-plane": "cp" + workerNodeGroupConfigurations: + - count: 2 + labels: + "node-role.kubernetes.io/worker": "worker" +. +. +``` + +If your cluster spec looks like the above one for either the control plane configuration and/or worker node configuration, you might run into the below kubelet error: +``` +unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels: [node-role.kubernetes.io/worker]. +--node-labels in the 'kubernetes.io' namespace must begin with an allowed prefix (kubelet.kubernetes.io, node.kubernetes.io) or be in the specifically allowed set (beta.kubernetes.io/arch, beta.kubernetes.io/instance-type, beta.kubernetes.io/os, failure-domain.beta.kubernetes.io/region, failure-domain.beta.kubernetes.io/zone, kubernetes.io/arch, kubernetes.io/hostname, kubernetes.io/os, node.kubernetes.io/instance-type, topology.kubernetes.io/region, topology.kubernetes.io/zone) +``` +Self-assigning node labels such as `node-role.kubernetes.io` using the kubelet `--node-labels` flag is not possible due to a security measure imposed by the NodeRestriction admission controller that kubeadm enables by default. + +Assigning such labels to nodes can be done after the bootstrap process has completed: + +``` +kubectl label nodes node-role.kubernetes.io/worker="" +``` +For convenience, here are example one-liners to do this post-installation: + +``` +# For Kubernetes 1.19 (kubeadm 1.19 sets only the node-role.kubernetes.io/master label) +kubectl get nodes --no-headers -l '!node-role.kubernetes.io/master' -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}' | xargs -I{} kubectl label node {} node-role.kubernetes.io/worker='' + +# For Kubernetes >= 1.20 (kubeadm >= 1.20 sets the node-role.kubernetes.io/control-plane label) +kubectl get nodes --no-headers -l '!node-role.kubernetes.io/control-plane' -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}' | xargs -I{} kubectl label node {} node-role.kubernetes.io/worker='' +``` + ## Bare Metal troubleshooting ### Creating new workload cluster hangs or fails