From 272d44b5df333a9dbbd93f562f81420dd00b6648 Mon Sep 17 00:00:00 2001 From: John Garbutt Date: Mon, 9 Oct 2023 14:02:33 +0100 Subject: [PATCH 1/9] Separate out the README --- README.rst | 89 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 87 insertions(+), 2 deletions(-) diff --git a/README.rst b/README.rst index 9d22d56..a9e6821 100644 --- a/README.rst +++ b/README.rst @@ -4,10 +4,92 @@ magnum-capi-helm OpenStack Magnum driver using helm to create k8s clusters with Cluster API. -The long stated aim of this driver is to merge -in upstream Magnum: + +The driver uses capi-helm-charts to create the +k8s resources needed to create a k8s cluster +using Cluster API, including various useful +add ons like a CNI and a monitoring stack. +https://github.com/stackhpc/capi-helm-charts + +Note, the above helm charts are intended to be +a way to share a reference method to create K8s +on OpenStack. The charts are not expected or +indented to be specific to Magnum. The hope is +they can also be used by ArgoCD, Flux or Azimuth +to create k8s clusters on OpenStack. + +Work on this driver started upstream aroun October 2021. +After failing to get merged during Bobcat, +we created this downstream repo as a stop gap to help +those wanting to use this driver now. https://specs.openstack.org/openstack/magnum-specs/specs/bobcat/clusterapi-driver.html +Installation and Dependencies +============================= + +If you install this python package within your Magnum virtual env, +it should be picked up by Magnum::: + + git clone https://github.com/stackhpc/magnum-capi-helm.git + cd magnum-capi-helm + pip install -e . + +The driver requires access to a Cluster API management cluster. +For more information, please see: +https://cluster-api.sigs.k8s.io/user/quick-start + +To access the above Cluster API management cluster, +you need to configure where the kubeconfig file +lives::: + + [capi_helm] + kubeconfig_file = /etc/magnum/kubeconfig + +To create a cluster, first you will need an image that +has been built to include kubernetes. +There are community maintained packer build pipelines here: +https://image-builder.sigs.k8s.io/capi/capi.html + +The above image needs to have the correct os-distro +property set when uploaded to Glance. For example::: + + curl -fo ubuntu-focal-kube-v1.28.1-230831-1150.qcow2 'https://object.arcus.openstack.hpc.cam.ac.uk/azimuth-images/ubuntu-focal-kube-v1.28.1-230831-1150.qcow2?AWSAccessKeyId=c5bd0fa15bae4e08b305a52aac97c3a6&Expires=1725019898&Signature=%2FXW2ywkA%2FQ8bCUiJkiLCWBAf81M%3D' + openstack image create ubuntu-focal-kube-v1.28.1 \ + --file ubuntu-focal-kube-v1.28.1-230831-1150.qcow2 \ + --disk-format qcow2 \ + --container-format bare \ + --public + openstack image set ubuntu-focal-kube-v1.28.1 --os-distro capi-kubeadm-cloudinit --os-version 20.04 + openstack image set ubuntu-focal-kube-v1.28.1 \ + --os-distro capi-kubeadm-cloudinit + +DevStack Setup +============== + +Did you want to try this driver in DevStack? +Please try our setup script in this repo: +`devstack/contrib/new-devstack.sh` + +The above devstack script includes creating k3s based +Cluster API management cluster. + +Features +======== + +The driver currently supports, create, delete, upgrade and +updates to node groups and their sizes. + +The CAPI helm charts are currently being tested +with K8s 1.26, 1.27 and 1.28: +https://github.com/stackhpc/capi-helm-charts/blob/main/.github/workflows/ensure-capi-images.yaml#L9 + +The currently supported labels include: + +* TODO + +History +======= + The helm charts used by this driver started out in August 2021 to build a template for creating K8s on OpenStack using Cluster API. @@ -51,3 +133,6 @@ The hope is that helm provides a better interface for per operator additions to clusters, and should allow for helm to be updated to support new Kubernetes versions, independently from the core Magnum code. + + + From d10bca5fa2d7c2291b37e679810ee2797d62d975 Mon Sep 17 00:00:00 2001 From: John Garbutt Date: Mon, 9 Oct 2023 14:31:47 +0100 Subject: [PATCH 2/9] Fill out feature description --- README.rst | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/README.rst b/README.rst index a9e6821..a72878e 100644 --- a/README.rst +++ b/README.rst @@ -83,9 +83,35 @@ The CAPI helm charts are currently being tested with K8s 1.26, 1.27 and 1.28: https://github.com/stackhpc/capi-helm-charts/blob/main/.github/workflows/ensure-capi-images.yaml#L9 -The currently supported labels include: - -* TODO +The driver respects the following cluster and template properties: + +* image_id +* keypair +* fixed_network, fixed_subnet (if missing, new one is created) +* external_network_id +* dns_nameserver + +The driver supports the following labels: + +* monitoring_enabled: default is off, change to "true" to enable +* kube_dashboard_enabled: defalt is on, change to "false" to disable +* octavia_provider: default is "amphora" +* fixed_subnet_cidr: default is "10.0.0.0/24" +* extra_network_name: default is "", change to name of additional network, + which can be useful if using Manila with the CephFS Native driver. + +We have found upgrade with ClusterAPI doesn't work well without +using a loadbalancer, even with a single node control plane, +so we currently ignore the "master-lb-enabled" flag. + +NOTE: +We are working in Cluster API provider OpenStack to add the ability +to store the etcd state on a cinder volume, separate from the root +disk. This is a big feature gap for clouds where most of your +root disks are on spinning disk Ceph, which is not fast enough +for etcd to operate correctly, but equally you don't have enough +ssd based Ceph to put all controller root disks on that Ceph: +https://github.com/kubernetes-sigs/cluster-api-provider-openstack/pull/1668 History ======= From 567430df5866a1841bf6c8debb56b622a3151aee Mon Sep 17 00:00:00 2001 From: John Garbutt Date: Mon, 9 Oct 2023 14:39:29 +0100 Subject: [PATCH 3/9] Add note on CNI choice. --- README.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.rst b/README.rst index a72878e..666c98a 100644 --- a/README.rst +++ b/README.rst @@ -100,6 +100,9 @@ The driver supports the following labels: * extra_network_name: default is "", change to name of additional network, which can be useful if using Manila with the CephFS Native driver. +Currently all clusters use the Calico CNI. While Cilium is also supported +in the helm charts, it is not currently reguarlly tested. + We have found upgrade with ClusterAPI doesn't work well without using a loadbalancer, even with a single node control plane, so we currently ignore the "master-lb-enabled" flag. From b66d39cae59e55cc51428c274850fa6fa0906030 Mon Sep 17 00:00:00 2001 From: John Garbutt Date: Mon, 9 Oct 2023 14:40:57 +0100 Subject: [PATCH 4/9] Pin oslo_db to unbreak tests --- requirements.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index fd3d4dc..e5c50e3 100644 --- a/requirements.txt +++ b/requirements.txt @@ -3,6 +3,7 @@ # process, which may cause wedges in the gate later. pbr>=2.0 # Apache-2.0 -magnum oslo_log oslo_utils +oslo_db<=14.1.0 # work around for broken magnum +magnum From ca1fcdfcdd380422cd747d4b223c7804c3ac0243 Mon Sep 17 00:00:00 2001 From: John Garbutt Date: Mon, 9 Oct 2023 14:52:46 +0100 Subject: [PATCH 5/9] Fix up tox by pinning to Antelope --- magnum_capi_helm/driver.py | 5 +---- requirements.txt | 3 +-- tox.ini | 2 +- 3 files changed, 3 insertions(+), 7 deletions(-) diff --git a/magnum_capi_helm/driver.py b/magnum_capi_helm/driver.py index 751c1cd..2ab9198 100644 --- a/magnum_capi_helm/driver.py +++ b/magnum_capi_helm/driver.py @@ -447,10 +447,7 @@ def _ensure_certificate_secrets(self, context, cluster): # Cluster API looks for specific named secrets for each of the CAs, # and generates them if they don't exist, so we create them here # with the correct certificates in - for ( - name, - data, - ) in ca_certificates.get_certificate_string_data( + for (name, data,) in ca_certificates.get_certificate_string_data( context, cluster ).items(): self._k8s_client.apply_secret( diff --git a/requirements.txt b/requirements.txt index e5c50e3..42d90d4 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5,5 +5,4 @@ pbr>=2.0 # Apache-2.0 oslo_log oslo_utils -oslo_db<=14.1.0 # work around for broken magnum -magnum +magnum<17.0.0 # test against 2023.1 for now diff --git a/tox.ini b/tox.ini index e288980..4b8fc7e 100644 --- a/tox.ini +++ b/tox.ini @@ -10,7 +10,7 @@ setenv = OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 -deps = -c{env:UPPER_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master} +deps = -c{env:UPPER_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/2023.1} -r{toxinidir}/requirements.txt -r{toxinidir}/test-requirements.txt commands = stestr run {posargs} From d569e1cfc45991ead7e07b44e611f16b10bd2b4a Mon Sep 17 00:00:00 2001 From: John Garbutt Date: Mon, 9 Oct 2023 14:54:04 +0100 Subject: [PATCH 6/9] Add readme note on the version --- README.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.rst b/README.rst index 666c98a..a5b3f80 100644 --- a/README.rst +++ b/README.rst @@ -34,6 +34,8 @@ it should be picked up by Magnum::: cd magnum-capi-helm pip install -e . +We currently run the unit tests against the 2023.1 version of Magnum. + The driver requires access to a Cluster API management cluster. For more information, please see: https://cluster-api.sigs.k8s.io/user/quick-start From 47c9858182d81bc394086e0e6645dc48e066d48e Mon Sep 17 00:00:00 2001 From: scrungus Date: Mon, 9 Oct 2023 16:24:18 +0100 Subject: [PATCH 7/9] more info --- README.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.rst b/README.rst index a5b3f80..b1840ac 100644 --- a/README.rst +++ b/README.rst @@ -27,6 +27,8 @@ https://specs.openstack.org/openstack/magnum-specs/specs/bobcat/clusterapi-drive Installation and Dependencies ============================= +For a kolla-ansible deployment, you can follow `this `__ guide. + If you install this python package within your Magnum virtual env, it should be picked up by Magnum::: @@ -52,6 +54,9 @@ has been built to include kubernetes. There are community maintained packer build pipelines here: https://image-builder.sigs.k8s.io/capi/capi.html +Or you can grab prebuilt images from our `azimuth image releases `__. +Images are available in the `manifest.json` file, and are named in the format `ubuntu---`. + The above image needs to have the correct os-distro property set when uploaded to Glance. For example::: From 80efda6d886ab9c905e8c31443ac19d857d269e6 Mon Sep 17 00:00:00 2001 From: John Garbutt Date: Mon, 9 Oct 2023 16:27:21 +0100 Subject: [PATCH 8/9] Add note about endpoint_type config --- README.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/README.rst b/README.rst index b1840ac..ec06304 100644 --- a/README.rst +++ b/README.rst @@ -49,6 +49,15 @@ lives::: [capi_helm] kubeconfig_file = /etc/magnum/kubeconfig +Note that the driver makes use of the nova_client config that will +already be setup in your Magnum configuration to generate application +credentials that reference the correct endpoint. You need to make +sure this uses an endpoint that can be reached from inside the +magnum clusters you create, by having something like this::: + + [nova_client] + endpoint_type = publicURL + To create a cluster, first you will need an image that has been built to include kubernetes. There are community maintained packer build pipelines here: From e807e814147b92520d8f7b56bc78609d8e61ab84 Mon Sep 17 00:00:00 2001 From: John Garbutt Date: Mon, 9 Oct 2023 16:40:12 +0100 Subject: [PATCH 9/9] Add note about creating a template --- README.rst | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/README.rst b/README.rst index ec06304..7f5a37b 100644 --- a/README.rst +++ b/README.rst @@ -79,6 +79,33 @@ property set when uploaded to Glance. For example::: openstack image set ubuntu-focal-kube-v1.28.1 \ --os-distro capi-kubeadm-cloudinit +Finally, this means you can now create a template, and then a cluster, +get the kubeconfig to access it, then run sonaboy to test it, +doing something like this::: + + openstack coe cluster template create new_driver \ + --coe kubernetes \ + --label octavia_provider=ovn \ + --image $(openstack image show ubuntu-focal-kube-v1.28.1 -c id -f value) \ + --external-network public \ + --master-flavor ds2G20 \ + --flavor ds2G20 \ + --public \ + --master-lb-enabled + + openstack coe cluster create devstacktest \ + --cluster-template new_driver \ + --master-count 1 \ + --node-count 2 + openstack coe cluster list + + mkdir -p ~/clusters/devstacktest + cd ~/clusters/devstacktest + openstack coe cluster config devstacktest + export KUBECONFIG=~/clusters/kubernetes-cluster/config + kubectl get nodes + sonobuoy run --mode quick --wait + DevStack Setup ============== @@ -111,7 +138,7 @@ The driver supports the following labels: * monitoring_enabled: default is off, change to "true" to enable * kube_dashboard_enabled: defalt is on, change to "false" to disable -* octavia_provider: default is "amphora" +* octavia_provider: default is "amphora", ovn is also an option * fixed_subnet_cidr: default is "10.0.0.0/24" * extra_network_name: default is "", change to name of additional network, which can be useful if using Manila with the CephFS Native driver.