SPIKE: EKS discovery #2609

T-Kukawka · 2023-06-21T09:23:03Z

First step to achieve EKS deployments integrated with GS is the discovery phase. In the short SPIKE we want to discover how easily it is to deploy the EKS clusters, what is missing from our setup and what we have to build on top to integrate fully with our offering.

Tasks

Give feedback

create EKS cluster
document EKS setup
gather notes for further development
consider testsing pipelines and talk with tinkerers
implement kubectl gs basic templating for cluster creation
Options

calvix · 2023-06-28T07:38:34Z

issues that I encountered when trying to create a proper EKS cluster with CAPI:

chart -operator cannot be scheduled on the EKS cluster because there is no master/control-plane node and the chart-operator pod has node selector that pins it to a master node
chart operator is trying to run in bootstrap bot which means hostNetwork and connecting to api via localhost - that obviously doesnt work as EKS has no master nodes and api is not reachable
due to that issues no apps can be installed,
if we use helmRelease in order to install CNI we could be able to run chart operator in normal mode like in vintage, but that means cilium needs to be able to connect to API and that won't work as currently dns-operator-aws only reconciles AWSCluster so there is no DNS pointing to the API EKS ELB

T-Kukawka · 2023-06-28T09:41:33Z

as discussed with Product we will move on with the dns-operator-aws so we unblock rest of deployments

calvix · 2023-06-28T13:16:31Z

so the next steps:

make cluster-eks app available for deployment - add-eks-templates cluster-eks#2
~~adjust dns-operator-aws to reconcile EKS cluster and create apropriate DNS record (most important is API dns) - currently blocked 🟥 by Merge aws-resolver-rules-operator & dns-aws-operator #2112~~
add CNI, CPI, and other basic components as HealmRelease to the cluster-eks app add-cni-csi-coredns cluster-eks#4
make chart-operator run in the basic non-bootstrap mode for the EKS cluster (similar to how it runs in vintage) and make it run on non-master nodes eks-run-chart-operator-in-normal-mode cluster-apps-operator#363
check all apps and fix broken apps
figure out how to deal with corednsas its installed by AWS and adoption do not work add-corends-adopter cluster-eks#6

calvix · 2023-06-30T12:03:08Z

With further progress I found out that the solution with dns-operator-aws is not really compatible with our custom DNS api.clusterID.basedomain DNS wont work as the EKS does not allow adding extra CN to the API SSL certs meaning we have to use the EKS endpoint URL everywhere including cilium

So I got inspired by the solution they have in cluster-cloud-director which uses a hook that patches the HelmRelease resource afterwards - https://github.com/giantswarm/cluster-cloud-director/blob/main/helm/cluster-cloud-director/templates/update-values-hook-job.yaml

Instead, I did a job that wait until the Cluster.Spec.ControlPlaneEndpoint value is populated and then create a CM with the EKS endpoint value which is consumed by the cilium HelmRelease and is passing the right value to the helm chart.

This solved Cilium app and the cluster got networking up

The next step was to make chart-operator work - in cluster-apps-operator I add a single if that decides if the cluster is EKS and in that case it will set values for chart operator to run in normal mode - giantswarm/cluster-apps-operator#363

So the result is the EKS cluster created with cluster-eks repo which has CNI working and chart-operator can run and install additional apps. There is still an issue with adopting the corends app which does not work for some reason. Some other non-important default apps might be failing as well.

I updated the TODO list, the PRs need reviews

calvix · 2023-07-07T14:44:46Z

app etcd-kubernetes-resources-count-exporter in default-apps-aws do not make sense for EKS as the ETCD is not exposed so there it should not be there, we might need to introduce a mechanism to disable certain apps from the bundle

calvix · 2023-07-07T14:48:32Z

current state:

 #: kk get no
NAME                                        STATUS   ROLES    AGE   VERSION
ip-10-0-147-56.eu-west-2.compute.internal   Ready    worker   19m   v1.24.13-eks-0a21954
ip-10-0-4-202.eu-west-2.compute.internal    Ready    worker   19m   v1.24.13-eks-0a21954
ip-10-0-43-122.eu-west-2.compute.internal   Ready    worker   19m   v1.24.13-eks-0a21954
[local] _GS/cluster-eks/helm/cluster-eks | git: add-corends-adopter | k8s: vac0eks-capi-admin@vac0eks
 #: kk get po
NAME                                                            READY   STATUS              RESTARTS   AGE
aws-pod-identity-webhook-app-57df4547fc-2s2ns                   0/1     ContainerCreating   0          7m15s
aws-pod-identity-webhook-app-57df4547fc-nqnk6                   0/1     ContainerCreating   0          7m15s
aws-pod-identity-webhook-app-57df4547fc-wcggs                   0/1     ContainerCreating   0          7m15s
aws-pod-identity-webhook-restarter-28145685-fwhlh               0/1     Completed           0          3m10s
capi-node-labeler-c6pwj                                         1/1     Running             0          9m38s
capi-node-labeler-lr7h9                                         1/1     Running             0          9m38s
capi-node-labeler-nw65w                                         1/1     Running             0          9m38s
cert-exporter-daemonset-2p9st                                   1/1     Running             0          9m38s
cert-exporter-daemonset-pblkw                                   1/1     Running             0          9m38s
cert-exporter-daemonset-tjlt5                                   1/1     Running             0          9m38s
cert-exporter-deployment-5d9dc8cd46-fqv2b                       1/1     Running             0          9m38s
cert-manager-cainjector-9cc6b5c4d-f66fl                         1/1     Running             0          6m11s
cert-manager-controller-7c688c6db-44zdt                         1/1     Running             0          6m11s
cert-manager-webhook-6c5ddc7678-j6lmz                           1/1     Running             0          6m11s
cert-manager-webhook-6c5ddc7678-mw4c5                           1/1     Running             0          6m11s
cilium-cfhtx                                                    1/1     Running             0          11m
cilium-hl56d                                                    1/1     Running             0          11m
cilium-n25st                                                    1/1     Running             0          11m
cilium-operator-f6ffd6cc4-42j4d                                 1/1     Running             0          11m
cilium-operator-f6ffd6cc4-wck9j                                 1/1     Running             0          11m
coredns-adopter-g2ddb                                           0/1     Completed           0          20m
coredns-controlplane-56565947c4-z48pf                           1/1     Running             0          8m45s
coredns-workers-6858cbbdbb-27jq2                                1/1     Running             0          8m44s
coredns-workers-6858cbbdbb-nf5mt                                1/1     Running             0          8m45s
etcd-kubernetes-resources-count-exporter-674c49c5b9-z8wmk       0/1     Pending             0          10m
external-dns-5d6545fd79-mp792                                   1/1     Running             0          41s
hubble-relay-88c84b559-rpshj                                    1/1     Running             0          11m
hubble-ui-5c7475d499-2wl89                                      2/2     Running             0          11m
metrics-server-75f9465bdc-db7jd                                 1/1     Running             0          10m
metrics-server-75f9465bdc-pm44g                                 1/1     Running             0          10m
net-exporter-9jpfn                                              1/1     Running             0          7m16s
net-exporter-f4c2j                                              1/1     Running             0          7m16s
net-exporter-wv4rc                                              1/1     Running             0          7m16s
node-exporter-node-exporter-6kqc5                               1/1     Running             0          7m17s
node-exporter-node-exporter-drwjc                               1/1     Running             0          7m17s
node-exporter-node-exporter-z2lcg                               1/1     Running             0          7m17s
prometheus-operator-app-kube-state-metrics-d7f4ff68d-x6lc8      1/1     Running             0          7m5s
prometheus-operator-app-operator-5474bb6778-gwxjq               1/1     Running             0          42s
vertical-pod-autoscaler-admission-controller-754fd4b4b9-jjqg6   1/1     Running             0          6m49s
vertical-pod-autoscaler-admission-controller-754fd4b4b9-pltxs   1/1     Running             0          6m49s
vertical-pod-autoscaler-recommender-5cf6659f65-sjn5s            1/1     Running             0          6m49s
vertical-pod-autoscaler-updater-5b445bbb66-rbjhc                1/1     Running             0          6m49s

cluster is created and available to use, all default apps (except etcd exporter) are running

calvix · 2023-07-07T14:56:26Z

The current state is that the cluster can be created and is usable for basic usage, but there is no ingress.

Ingress - for that, we need proper DNS

TODO for DNS:

reconcile AWSManagedControlPlane in dns-oprator-aws and create a hosted zone + ingress DNS record
reconcile AWSManagedCluster in irsa-operator
reconcile AWSManagedControlPlane in capa-iam-operator in order to create a IAM role for external-dns app which is necessary for the creation of the ingress DNS record

other stories that would cover other teams would be:

EKS kubeconfig and automation of opsctl login - the proper kubeconfig needs AWS credentials in order to work ( there is very short-lived kubeconfig in the CAPI MC that can be used for short debug but its not feasible for normal work)
check if monitoring works
CI tests for cluster-eks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SPIKE: EKS discovery #2609

SPIKE: EKS discovery #2609

T-Kukawka commented Jun 21, 2023 •

edited

Loading

Tasks

calvix commented Jun 28, 2023

T-Kukawka commented Jun 28, 2023

calvix commented Jun 28, 2023 •

edited

Loading

calvix commented Jun 30, 2023 •

edited

Loading

calvix commented Jul 7, 2023

calvix commented Jul 7, 2023 •

edited

Loading

calvix commented Jul 7, 2023 •

edited

Loading

calvix commented Aug 4, 2023 •

edited by T-Kukawka

Loading

calvix commented Aug 22, 2023

SPIKE: EKS discovery #2609

SPIKE: EKS discovery #2609

Comments

T-Kukawka commented Jun 21, 2023 • edited Loading

Tasks

calvix commented Jun 28, 2023

T-Kukawka commented Jun 28, 2023

calvix commented Jun 28, 2023 • edited Loading

calvix commented Jun 30, 2023 • edited Loading

calvix commented Jul 7, 2023

calvix commented Jul 7, 2023 • edited Loading

calvix commented Jul 7, 2023 • edited Loading

calvix commented Aug 4, 2023 • edited by T-Kukawka Loading

calvix commented Aug 22, 2023

T-Kukawka commented Jun 21, 2023 •

edited

Loading

calvix commented Jun 28, 2023 •

edited

Loading

calvix commented Jun 30, 2023 •

edited

Loading

calvix commented Jul 7, 2023 •

edited

Loading

calvix commented Jul 7, 2023 •

edited

Loading

calvix commented Aug 4, 2023 •

edited by T-Kukawka

Loading