v0.1.0 - Fully automated permanent and workload cluster(s) deploy

• Complete error-free installation of permanent management cluster and all the workload clusters currently present in the repo
• Ability to add new workload clusters, fully automatic including all components health validation.
• Standardise on cluster name and cluster namespace.
• Cleanup script fixed to match the naming standard and auto-discover all CAPI clusters, plus other fixes.

Installation

deploy permanent management cluster on AWS (using temp kind cluster and then pivot)

./scripts/deploy.sh

flux on management cluster will apply CAPI manifests that are currently present in the repo.

When script is complete run script to finalize workload clusters (install cilium which currently is not vi CRS - due to dynamic KAS address) and flux secret (WIP to eliminate this step).
This script without arguments will discover all workload clusters and perform all necessary adjustments:

./scripts/workload-cluster.sh

Adding a new cluster

Hands free with just one script!

To add a new cluster create config env for it by copying existing file (./config/cluster-<num>.env) and modifying values. This is intended to be manual as script can't or shouldn't guess this values (or too difficult in bash like calc next CIDR)

./scripts/workload-cluster.sh -n cluster-02

This will generate all necessary files and add the cluster to mgmt kustomization list too. Then it will be pushed to the repo (example commit from the script: 92ee7e0), where flux will apply it and capi will provision it. The ./scripts/workload-cluster.sh is still waiting for the cluster to come up and finalize the installation.

on mgmt cluster:

% k get cluster -A
NAMESPACE      NAME           PHASE          AGE   VERSION
cluster-01     cluster-01     Provisioned    12m
cluster-02     cluster-02     Provisioning   60s
cluster-mgmt   cluster-mgmt   Provisioned    13m

Hubble

Current setup enables hubble relay and UI.

cilium hubble ui

https://docs.cilium.io/en/v1.11/gettingstarted/hubble/#hubble-ui
or in CLI with instructions from here: https://docs.cilium.io/en/v1.11/gettingstarted/hubble_setup/

more details: #10

Integrate Kubefed, Kong; install calico with CRS; improve deploy and cleanup scripts

• Install Kong ingress controller on workload cluster.
• Install KubeFed on management cluster.
• Use ClusterResourceSet to install calico CNI on mgmt and workload cluster.
• Fix bug when performing CAPI management cluster pivot.
• Improve cleanup by moving all clusters back to kind cluster and deleting them in parallel.
• Add AWS cleanup script for cases when CAPI cleanup is not working. (the script doesn't filter any resources by cluster name, just removes everything that attracts charges)

There is still a bug in deletion of workload cluster due to Kong SecurityGroup. At the final stage of the workload cluster VPC needs to be remove manually from the AWS console.

Current setup is management cluster and 1 workload cluster.

%
% ######      AWS Management Cluster
%
% k config get-contexts
CURRENT   NAME              CLUSTER     AUTHINFO     NAMESPACE
          dev-admin@dev     dev         dev-admin
          kind-kind         kind-kind   kind-kind
*         mgmt-admin@mgmt   mgmt        mgmt-admin
% k get pods -A | grep -vE "kube-system|cert-manager"
NAMESPACE                           NAME                                                                      READY   STATUS    RESTARTS   AGE
capa-system                         capa-controller-manager-68655647dc-krlxm                                  1/1     Running   0          49m
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-58db4b5555-hgrbc                1/1     Running   0          49m
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-86c4dcbc4c-2cdkw            1/1     Running   0          49m
capi-system                         capi-controller-manager-7985b86b97-kjph8                                  1/1     Running   0          49m
flux-system                         helm-controller-68686dc594-529fs                                          1/1     Running   0          12m
flux-system                         kustomize-controller-6b565c6c5d-hkbhs                                     1/1     Running   0          12m
flux-system                         notification-controller-6c656746-6hrvf                                    1/1     Running   0          12m
flux-system                         source-controller-c65ddffbb-pjbd4                                         1/1     Running   0          12m
kube-federation-system              kubefed-admission-webhook-75b96bbcdb-f4rnm                                1/1     Running   0          11m
kube-federation-system              kubefed-controller-manager-58978c7cdb-bkbxw                               1/1     Running   0          10m
kube-federation-system              kubefed-controller-manager-58978c7cdb-pndzd                               1/1     Running   0          10m
%
% ######      Workload Cluster
%
% k config use-context dev-admin@dev
Switched to context "dev-admin@dev".
% flux get all
NAME                            REVISION                                SUSPENDED       READY   MESSAGE
gitrepository/flux-system       feature/kubefed-and-kong/c532aa4        False           True    stored artifact for revision 'feature/kubefed-and-kong/c532aa463f4c50088751fa0f6bb2a2b9ab32b297'

NAME                    REVISION                                                                SUSPENDED       READY   MESSAGE
helmrepository/kong     88d3b67840c3f22102541243a73a583b47d21ecbb49ec126ed1798d6eb393282        False           True    stored artifact for revision '88d3b67840c3f22102541243a73a583b47d21ecbb49ec126ed1798d6eb393282'

NAME                    REVISION        SUSPENDED       READY   MESSAGE
helmchart/kong-kong     2.8.2           False           True    pulled 'kong' chart with version '2.8.2'

NAME                            REVISION                                SUSPENDED       READY   MESSAGE
kustomization/flux-system       feature/kubefed-and-kong/c532aa4        False           True    Applied revision: feature/kubefed-and-kong/c532aa4
kustomization/infrastructure    feature/kubefed-and-kong/c532aa4        False           True    Applied revision: feature/kubefed-and-kong/c532aa4
kustomization/tenants           feature/kubefed-and-kong/c532aa4        False           True    Applied revision: feature/kubefed-and-kong/c532aa4

% k get pod -A | grep -v kube-system
NAMESPACE     NAME                                                                      READY   STATUS    RESTARTS   AGE
flux-system   helm-controller-68686dc594-fwn6p                                          1/1     Running   0          18m
flux-system   kustomize-controller-6b565c6c5d-nb2nq                                     1/1     Running   0          18m
flux-system   notification-controller-6c656746-pfqb2                                    1/1     Running   0          18m
flux-system   source-controller-c65ddffbb-98llw                                         1/1     Running   0          18m
kong          kong-kong-6c746dfcfb-hjl9b                                                2/2     Running   2          16m

Permanent CAPI management cluster and workload cluster applied by FluxCD

First iteration of CAPI + FluxCD.
./scripts/deploy-bootstrap-cluster.sh - implements bootstrap and pivot approach as described here: https://cluster-api.sigs.k8s.io/clusterctl/commands/move.html#bootstrap--pivot except that temporary management cluster is not deleted. This is because without it currently it is not clear how to manage the permanent management cluster.
Following that, running scripts/install-flux.sh installs FluxCD on permanent management cluster and applies config which includes workload cluster definition.

% k config use-context mgmt-admin@mgmt 
Switched to context "mgmt-admin@mgmt".
% k get nodes 
NAME                                              STATUS   ROLES                  AGE     VERSION
ip-10-0-222-144.ap-southeast-2.compute.internal   Ready    <none>                 3m50s   v1.22.8
ip-10-0-230-230.ap-southeast-2.compute.internal   Ready    control-plane,master   4m51s   v1.22.8

connect to dev cluster:

% clusterctl get kubeconfig dev -n cluster-dev > dev.kubeconfig 
% k get nodes --kubeconfig=./dev.kubeconfig --context dev-admin@dev
NAME                                             STATUS     ROLES                  AGE     VERSION
ip-10-0-196-17.ap-southeast-2.compute.internal   NotReady   <none>                 108s    v1.21.11
ip-10-0-231-67.ap-southeast-2.compute.internal   NotReady   control-plane,master   2m43s   v1.21.11

(still needs CNI to properly finish the install)