Use super-admin Kubeconfig for kube-vip from Kubernetes v1.29 onwards

[abhay-krishna]

Before Kubernetes v1.29, the default Kubeconfig file admin.conf generated by kubeadm was bound to system:masters Group. In Kubernetes v1.29, a change was made to kubeadm to provision a separate Kubeconfig file called super-admin.conf which is a break-glass Group that can bypass RBAC, and in addition, scoping down the admin.conf User's access to the permissions provided by the cluster-admin ClusterRole. It is to be noted that in a multi-cluster setup, the super-admin.conf is only created on the primary control plane node, which runs kubeadm init. This causes a bootstrap issue with EKS-A's kube-vip deployment because the static pod manifest is hardcoded to use admin.conf but kube-vip requires elevated permissions to access the API server during cluster initialization, which are available only when using the super-admin.conf.

This issue was first reported by CAPV maintainers in Kubernetes Slack kube-vip channel (Initial issue report, Workarounds discussion) when they were trying to add Kubernetes v1.29 support. Since then, the CAPV maintainers have found a workaround that involves using the super-admin.conf only for kubeadm init and then migrating to using the admin.conf after the cluster has been initialized. This PR basically ports the workaround to our cluster templates for EKS-A's kube-vip deployment to work with Kubernetes v1.29.

This is similar to the logic we added to the Bottlerocket kubeadm-bootstrap host container to fix the same issue for clusters using Bottlerocket nodes.

$ kubectl --kubeconfig vsphere-1-29-test-ubuntu/vsphere-1-29-test-ubuntu-eks-a-cluster.kubeconfig get nodes -o wide       
NAME                                        STATUS   ROLES           AGE   VERSION               INTERNAL-IP     EXTERNAL-IP     OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
vsphere-1-29-test-ubuntu-4nwtt              Ready    control-plane   19h   v1.29.0-eks-a5ec690   196.18.91.211   196.18.91.211   Ubuntu 20.04.6 LTS   5.4.0-170-generic   containerd://1.7.12-0-g71909c181
vsphere-1-29-test-ubuntu-fptnj              Ready    control-plane   19h   v1.29.0-eks-a5ec690   196.18.96.242   196.18.96.242   Ubuntu 20.04.6 LTS   5.4.0-170-generic   containerd://1.7.12-0-g71909c181
vsphere-1-29-test-ubuntu-md-0-vwxjf-rg6b7   Ready    <none>          19h   v1.29.0-eks-a5ec690   196.18.34.251   196.18.34.251   Ubuntu 20.04.6 LTS   5.4.0-170-generic   containerd://1.7.12-0-g71909c181

References:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[codecov]

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (6dc52b5) 72.03% compared to head (b3f522d) 73.47%.
Report is 3 commits behind head on main.

Files	Patch %	Lines
pkg/providers/snow/apibuilder.go	60.00%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7368      +/-   ##
==========================================
+ Coverage   72.03%   73.47%   +1.44%     
==========================================
  Files         579      579              
  Lines       44704    36344    -8360     
==========================================
- Hits        32202    26705    -5497     
+ Misses      10740     7875    -2865     
- Partials     1762     1764       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[abhinavmpandey08]

/approve
/lgtm
/approve

[eks-distro-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavmpandey08

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [abhinavmpandey08]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment