teleport-kube-agent-app should be evictable by cluster-autoscaler in order to scale down nodes

[AndiDog]

cluster-autoscaler 2023-12-21T12:53:50.957115967+01:00 I1221 11:53:50.957076       1 cluster.go:164] node ip-10-0-66-124.eu-west-2.compute.internal cannot be removed: non-daemonset, non-mirrored, non-pdb-assigned kube-system pod present: teleport-kube-agent-0

Tasks

Preview Give feedback

1. Enable high availability mode and podDisruptionBudget teleport-kube-agent-app#30
   +0
   
2. Test and release v0.7.0
3. Report back to Phoenix and apply the fix

[tuladhar]

Problem

teleport-kube-agent runs in kube-system namespace. And by default, kube-system pods prevent cluster-autoscaler from removing nodes unless:

• A. Pod has annotation cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
• B. Pod has PDB

Solution

• In case of teleport-kube-agent we enable PDB with maxUnavailable: 1 and turning on high-availability mode with 2 replicas.