diff --git a/README.md b/README.md index ff9ad5d14d9..c598033db41 100644 --- a/README.md +++ b/README.md @@ -19,12 +19,19 @@ Kubernetes-native declarative infrastructure for Azure. -## What is the Cluster API Provider Azure +## What is the Cluster API Provider Azure (CAPZ) The [Cluster API][cluster_api] brings declarative, Kubernetes-style APIs to cluster creation, configuration and management. -The API itself is shared across multiple cloud providers allowing for true Azure -hybrid deployments of Kubernetes. +The API itself is shared across multiple cloud providers allowing for true Azure hybrid deployments of Kubernetes. + +CAPZ enables efficient management at scale of self-managed or managed (AKS) clusters on Azure. Furthermore, the CAPZ management cluster can be utilized with the automatically installed Azure Service Operator (ASO) installation dependency to manage any Azure infrastructure. For more information see the [roadmap high level vision](https://capz.sigs.k8s.io/roadmap#high-level-vision). + +## Documentation + +Please see our [Book](https://capz.sigs.k8s.io) for in-depth user documentation. + +Additional docs can be found in the `/docs` directory, and the [index is here](https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/docs/README.md). ## Quick Start @@ -58,12 +65,6 @@ For more information on Kubernetes version support, see the [Cluster API book](h ------ -## Documentation - -Please see our [Book](https://capz.sigs.k8s.io) for in-depth user documentation. - -Additional docs can be found in the `/docs` directory, and the [index is here](https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/docs/README.md). - ## Getting involved and contributing Are you interested in contributing to cluster-api-provider-azure? We, the diff --git a/docs/README.md b/docs/README.md index bb24b76eb9e..372af9dd6c8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -23,3 +23,9 @@ ## Troubleshooting - [Troubleshooting guide](https://capz.sigs.k8s.io/topics/troubleshooting.html) + +## Docs contributors + +To run the link check linter, execute the following command from the root of the repository: + +`find . -name '*.md' -not -path './node_modules/*' -exec markdown-link-check '{}' --config .markdownlinkcheck.json -q ';'` diff --git a/docs/book/book.toml b/docs/book/book.toml index ed59d3c2dc5..b44632294ad 100644 --- a/docs/book/book.toml +++ b/docs/book/book.toml @@ -10,7 +10,7 @@ build-dir = "bookout" create-missing = false [output.html] -curly-quotes = true +smart-punctionation = true git-repository-url = "https://sigs.k8s.io/cluster-api-provider-azure" [preprocessor.tabulate] diff --git a/docs/book/install-and-build.sh b/docs/book/install-and-build.sh index 2a012f93c00..5e2e4692611 100755 --- a/docs/book/install-and-build.sh +++ b/docs/book/install-and-build.sh @@ -23,8 +23,8 @@ cd "${KUBE_ROOT}" || exit 1 os=$(go env GOOS) arch=$(go env GOARCH) -mdBookVersion="v0.4.35" -genCRDAPIReferenceDocsVersion="11fe95cbdcb91e9c25446fc99e6f2cdd8cbeb91a" +mdBookVersion="v0.4.40" +genCRDAPIReferenceDocsVersion="642c3aa7441d324f54f5a9a6a1841ffffacf5aeb" # translate arch to rust's conventions (if we can) if [[ ${arch} == "amd64" ]]; then diff --git a/docs/book/src/SUMMARY.md b/docs/book/src/SUMMARY.md index 9857c0f30ac..cea635b41df 100644 --- a/docs/book/src/SUMMARY.md +++ b/docs/book/src/SUMMARY.md @@ -1,45 +1,51 @@ # Summary [Introduction](./introduction.md) +[Getting Started](./getting-started.md) [Roadmap](./roadmap.md) -- [Topics](./topics/topics.md) - - [Getting Started](./topics/getting-started.md) - - [Troubleshooting](./topics/troubleshooting.md) - - [AAD Integration](./topics/aad-integration.md) - - [Addons](./topics/addons.md) - - [API Server Endpoint](./topics/api-server-endpoint.md) +- [General Topics](./topics/topics.md) - [Azure Service Operator](./topics/aso.md) - - [Cloud Provider Config](./topics/cloud-provider-config.md) - [ClusterClass](./topics/clusterclass.md) - - [Control Plane Outbound Load Balancer](./topics/control-plane-outbound-lb.md) - - [Custom Images](./topics/custom-images.md) - - [Custom Private DNS Zone Name](./topics/custom-dns.md) - - [Custom VM Extensions](./topics/custom-vm-extensions.md) - - [Disks](./topics/disks.md) - - [Data Disks](./topics/data-disks.md) - - [OS Disk](./topics/os-disk.md) - - [Disk Encryption](./topics/disk-encryption.md) - - [Dual-Stack](./topics/dual-stack.md) - - [Externally managed Azure infrastructure](./topics/externally-managed-azure-infrastructure.md) - - [Failure Domains](./topics/failure-domains.md) - - [Flatcar](./topics/flatcar.md) - - [GPU-enabled Clusters](./topics/gpu.md) - - [Confidential VMs](./topics/confidential-vms.md) - - [Trusted Launch for VMs](./topics/trusted-launch-for-vms.md) - [Identities](./topics/identities.md) + - [AAD Integration](./topics/aad-integration.md) - [Identity use cases](./topics/identities-use-cases.md) - [Multi-tenancy](./topics/multitenancy.md) - [Workload Identity](./topics/workload-identity.md) - - [IPv6](./topics/ipv6.md) - - [Machine Pools (VMSS)](./topics/machinepools.md) - - [Managed Clusters (AKS)](./topics/managedcluster.md) - - [Node Outbound Connection](./topics/node-outbound-connection.md) - - [Spot Virtual Machines](./topics/spot-vms.md) - - [SSH Access to nodes](./topics/ssh-access.md) - - [Virtual Networks](./topics/custom-vnet.md) - - [VM Identity](./topics/vm-identity.md) - - [Windows](./topics/windows.md) - - [WebAssembly / WASI Pods](./topics/wasi.md) +- [Managed Clusters (AKS)](./managed/managed.md) + - [Adopting Clusters](./managed/adopting-clusters.md) + - [ASO Managed Clusters (AKS)](./managed/asomanagedcluster.md) + - [Managed Clusters (AKS)](./managed/managedcluster.md) + - [Managed Clusters - BYO VMSS Nodes](./managed/managedcluster-join-vmss.md) + - [Troubleshooting](./managed/troubleshooting.md) +- [Self-managed Clusters](./self-managed/self-managed.md) + - [Addons](./self-managed/addons.md) + - [API Server Endpoint](./self-managed/api-server-endpoint.md) + - [Cloud Provider Config](./self-managed/cloud-provider-config.md) + - [Confidential VMs](./self-managed/confidential-vms.md) + - [Control Plane Outbound Load Balancer](./self-managed/control-plane-outbound-lb.md) + - [Custom Images](./self-managed/custom-images.md) + - [Custom Private DNS Zone Name](./self-managed/custom-dns.md) + - [Custom VM Extensions](./self-managed/custom-vm-extensions.md) + - [Disks](./self-managed/disks.md) + - [Data Disks](./self-managed/data-disks.md) + - [Disk Encryption](./self-managed/disk-encryption.md) + - [OS Disk](./self-managed/os-disk.md) + - [Dual-Stack](./self-managed/dual-stack.md) + - [Externally managed Azure infrastructure](./self-managed/externally-managed-azure-infrastructure.md) + - [Failure Domains](./self-managed/failure-domains.md) + - [Flatcar](./self-managed/flatcar.md) + - [GPU-enabled Clusters](./self-managed/gpu.md) + - [IPv6](./self-managed/ipv6.md) + - [Machine Pools (VMSS)](./self-managed/machinepools.md) + - [Node Outbound Connection](./self-managed/node-outbound-connection.md) + - [Spot Virtual Machines](./self-managed/spot-vms.md) + - [SSH Access to nodes](./self-managed/ssh-access.md) + - [Troubleshooting](./self-managed/troubleshooting.md) + - [Trusted Launch for VMs](./self-managed/trusted-launch-for-vms.md) + - [Virtual Networks](./self-managed/custom-vnet.md) + - [VM Identity](./self-managed/vm-identity.md) + - [WebAssembly / WASI Pods](./self-managed/wasi.md) + - [Windows](./self-managed/windows.md) - [Development](./developers/development.md) - [Kubernetes Developers](./developers/kubernetes-developers.md) - [Releasing](./developers/releasing.md) diff --git a/docs/book/src/developers/development.md b/docs/book/src/developers/development.md index ede226333f3..0845fa0ede4 100644 --- a/docs/book/src/developers/development.md +++ b/docs/book/src/developers/development.md @@ -122,7 +122,7 @@ Makefile targets and scripts are offered to work with go modules: ### Setting up the environment -Your must have the Azure credentials as outlined in the [getting started prerequisites](../topics/getting-started.md#Prerequisites) section. +You must have the Azure credentials as outlined in the [getting started prerequisites](../getting-started.md#Prerequisites) section. ### Tilt Requirements @@ -163,7 +163,7 @@ kustomize_substitutions: AZURE_CLIENT_ID: ``` -You should have these values saved from the [getting started prerequisites](../topics/getting-started.md#Prerequisites) section. +You should have these values saved from the [getting started prerequisites](../getting-started.md#Prerequisites) section. To build a kind cluster and start Tilt, just run: @@ -218,7 +218,7 @@ kustomize_substitutions: EOF ``` -Make sure to replace the credentials with the values from the [getting started prerequisites](../topics/getting-started.md#Prerequisites) section. +Make sure to replace the credentials with the values from the [getting started prerequisites](../getting-started.md#Prerequisites) section. > `$REGISTRY` should be in the format `docker.io/` @@ -245,7 +245,7 @@ To delete the cluster: make delete-workload-cluster ``` -> Check out the [troubleshooting guide](../topics/troubleshooting.md) for common errors you might run into. +> Check out the [self-managed](../self-managed/troubleshooting.md) and [managed](../managed/troubleshooting.md) troubleshooting guides for common errors you might run into. #### Viewing Telemetry @@ -414,7 +414,7 @@ Create the cluster: make create-cluster ``` -> Check out the [troubleshooting](../topics/troubleshooting.md) guide for common errors you might run into. +> Check out the [self-managed](../self-managed/troubleshooting.md) and [managed](../managed/troubleshooting.md) troubleshooting guides for common errors you might run into ### Instrumenting Telemetry Telemetry is the key to operational transparency. We strive to provide insight into the internal behavior of the @@ -571,7 +571,7 @@ You can optionally set the following variables: | `KUBERNETES_VERSION` | Desired Kubernetes version to test. You can pass in a definitive released version, e.g., "v1.24.0". If you want to use pre-released CI bits of a particular release you may use the "latest-" prefix, e.g., "latest-1.24"; you may use the very latest built CI bits from the kubernetes/kubernetes master branch by passing in "latest". If you provide a `KUBERNETES_VERSION` environment variable, you may not also use `CI_VERSION` (below). Use only one configuration variable to declare the version of Kubernetes to test. | | `CI_VERSION` | Provide a custom CI version of Kubernetes (e.g., `v1.25.0-alpha.0.597+aa49dffc7f24dc`). If not specified, this will be determined from the KUBERNETES_VERSION above if it is an unreleased version. If you provide a `CI_VERSION` environment variable, you may not also use `KUBERNETES_VERSION` (above). | | `TEST_CCM` | Build a cluster that uses custom versions of the Azure cloud-provider cloud-controller-manager and node-controller-manager images | -| `EXP_MACHINE_POOL` | Use [Machine Pool](../topics/machinepools.md) for worker machines. | +| `EXP_MACHINE_POOL` | Use [Machine Pool](../self-managed/machinepools.md) for worker machines. | | `TEST_WINDOWS` | Build a cluster that has Windows worker nodes. | | `REGISTRY` | Registry to push any custom k8s images or cloud provider images built. | | `CLUSTER_TEMPLATE` | Use a custom cluster template. It can be a path to a template under templates/, a path on the host or a link. If the value is not set, the script will choose the appropriate cluster template based on existing environment variables. | diff --git a/docs/book/src/developers/releasing.md b/docs/book/src/developers/releasing.md index 162c346f76a..8574033506f 100644 --- a/docs/book/src/developers/releasing.md +++ b/docs/book/src/developers/releasing.md @@ -156,7 +156,7 @@ Note: this step requires access to the Netlify site. If you don't have access, p #### Minor/Major Releases -1. Follow the communications process for [pre-releases](#pre-releases) +1. Follow the communications process for [patch-releases](#patch-releases) 2. An announcement email is sent to `kubernetes-sig-azure@googlegroups.com` and `kubernetes-sig-cluster-lifecycle@googlegroups.com` with the subject `[ANNOUNCE] cluster-api-provider-azure has been released` [semver]: https://semver.org/#semantic-versioning-200 diff --git a/docs/book/src/topics/getting-started.md b/docs/book/src/getting-started.md similarity index 95% rename from docs/book/src/topics/getting-started.md rename to docs/book/src/getting-started.md index 48cbdb74388..9578411de5f 100644 --- a/docs/book/src/topics/getting-started.md +++ b/docs/book/src/getting-started.md @@ -45,7 +45,7 @@ An Azure Service Principal is needed for deploying Azure resources. The below in ``` 5. Create an Azure Service Principal by running the following command or skip this step and use a previously created Azure Service Principal. - NOTE: the "owner" role is required to be able to create role assignments for [system-assigned managed identity](vm-identity.md). + NOTE: the "owner" role is required to be able to create role assignments for system-assigned managed identity. ```bash az ad sp create-for-rbac --role contributor --scopes="/subscriptions/${AZURE_SUBSCRIPTION_ID}" @@ -84,7 +84,7 @@ For example, if your password is `foo'blah$`, you should do `export AZURE_CLIENT

Warning

-The capability to set credentials using environment variables is now deprecated and will be removed in future releases, the recommended approach is to use `AzureClusterIdentity` as explained [here](multitenancy.md) +The capability to set credentials using environment variables is now deprecated and will be removed in future releases, the recommended approach is to use `AzureClusterIdentity` as explained [here](./topics/multitenancy.md) @@ -92,7 +92,7 @@ The capability to set credentials using environment variables is now deprecated ### Building your first cluster Check out the [Cluster API Quick Start](https://cluster-api.sigs.k8s.io/user/quick-start.html) to create your first Kubernetes cluster on Azure using Cluster API. Make sure to select the "Azure" tabs. -If you are looking to install additional ASO CRDs, set `ADDITIONAL_ASO_CRDS` to the list of CRDs you want to install. Refer to adding additional CRDs for Azure Service Operator [here](aso.md#Using-aso-for-non-capz-resources). +If you are looking to install additional ASO CRDs, set `ADDITIONAL_ASO_CRDS` to the list of CRDs you want to install. Refer to adding additional CRDs for Azure Service Operator [here](./topics/aso.md#Using-aso-for-non-capz-resources).

Warning

diff --git a/docs/book/src/managed/adopting-clusters.md b/docs/book/src/managed/adopting-clusters.md new file mode 100644 index 00000000000..b5463fa271b --- /dev/null +++ b/docs/book/src/managed/adopting-clusters.md @@ -0,0 +1,85 @@ +## Adopting Existing AKS Clusters + +### Option 1: Using the new AzureASOManaged API + + +The [AzureASOManagedControlPlane and related APIs](./asomanagedcluster.md) support +adoption as a first-class use case. Going forward, this method is likely to be easier, more reliable, include +more features, and better supported for adopting AKS clusters than Option 2 below. + +To adopt an AKS cluster into a full Cluster API Cluster, create an ASO ManagedCluster and associated +ManagedClustersAgentPool resources annotated with `sigs.k8s.io/cluster-api-provider-azure-adopt=true`. The +annotation may also be added to existing ASO resources to trigger adoption. CAPZ will automatically scaffold +the Cluster API resources like the Cluster, AzureASOManagedCluster, AzureASOManagedControlPlane, MachinePools, +and AzureASOManagedMachinePools. The [`asoctl import +azure-resource`](https://azure.github.io/azure-service-operator/tools/asoctl/#import-azure-resource) command +can help generate the required YAML. + +Caveats: +- The `asoctl import azure-resource` command has at least [one known + bug](https://github.com/Azure/azure-service-operator/issues/3805) requiring the YAML it generates to be + edited before it can be applied to a cluster. +- CAPZ currently only records the ASO resources in the CAPZ resources' `spec.resources` that it needs to + function, which include the ManagedCluster, its ResourceGroup, and associated ManagedClustersAgentPools. + Other resources owned by the ManagedCluster like Kubernetes extensions or Fleet memberships are not + currently imported to the CAPZ specs. +- Configuring the automatically generated Cluster API resources is not currently possible. If you need to + change something like the `metadata.name` of a resource from what CAPZ generates, create the Cluster API + resources manually referencing the pre-existing resources. +- Adopting existing clusters created with the GA AzureManagedControlPlane API to the experimental API with + this method is theoretically possible, but untested. Care should be taken to prevent CAPZ from reconciling + two different representations of the same underlying Azure resources. + +### Option 2: Using the current AzureManagedControlPlane API + + + +CAPZ can adopt some AKS clusters created by other means under its management. This works by crafting CAPI and +CAPZ manifests which describe the existing cluster and creating those resources on the CAPI management +cluster. This approach is limited to clusters which can be described by the CAPZ API, which includes the +following constraints: + +- the cluster operates within a single Virtual Network and Subnet +- the cluster's Virtual Network exists outside of the AKS-managed `MC_*` resource group +- the cluster's Virtual Network and Subnet are not shared with any other resources outside the context of this cluster + +To ensure CAPZ does not introduce any unwarranted changes while adopting an existing cluster, carefully review +the [entire AzureManagedControlPlane spec](../reference/v1beta1-api#infrastructure.cluster.x-k8s.io/v1beta1.AzureManagedControlPlaneSpec) +and specify _every_ field in the CAPZ resource. CAPZ's webhooks apply defaults to many fields which may not +match the existing cluster. + +Specific AKS features not represented in the CAPZ API, like those from a newer AKS API version than CAPZ uses, +do not need to be specified in the CAPZ resources to remain configured the way they are. CAPZ will still not +be able to manage that configuration, but it will not modify any settings beyond those for which it has +knowledge. + +By default, CAPZ will not make any changes to or delete any pre-existing Resource Group, Virtual Network, or +Subnet resources. To opt-in to CAPZ management for those clusters, tag those resources with the following +before creating the CAPZ resources: `sigs.k8s.io_cluster-api-provider-azure_cluster_: owned`. +Managed Cluster and Agent Pool resources do not need this tag in order to be adopted. + +After applying the CAPI and CAPZ resources for the cluster, other means of managing the cluster should be +disabled to avoid ongoing conflicts with CAPZ's reconciliation process. + +#### Pitfalls + +The following describes some specific pieces of configuration that deserve particularly careful attention, +adapted from https://gist.github.com/mtougeron/1e5d7a30df396cd4728a26b2555e0ef0#file-capz-md. + +- Make sure `AzureManagedControlPlane.metadata.name` matches the AKS cluster name +- Set the `AzureManagedControlPlane.spec.virtualNetwork` fields to match your existing VNET +- Make sure the `AzureManagedControlPlane.spec.sshPublicKey` matches what was set on the AKS cluster. (including any potential newlines included in the base64 encoding) + - NOTE: This is a required field in CAPZ, if you don't know what public key was used, you can _change_ or _set_ it via the Azure CLI however before attempting to import the cluster. +- Make sure the `Cluster.spec.clusterNetwork` settings match properly to what you are using in AKS +- Make sure the `AzureManagedControlPlane.spec.dnsServiceIP` matches what is set in AKS +- Set the tag `sigs.k8s.io_cluster-api-provider-azure_cluster_` = `owned` on the AKS cluster +- Set the tag `sigs.k8s.io_cluster-api-provider-azure_role` = `common` on the AKS cluster + +NOTE: Several fields, like `networkPlugin`, if not set on the AKS cluster at creation time, will mean that CAPZ will not be able to set that field. AKS doesn't allow such fields to be changed if not set at creation. However, if it was set at creation time, CAPZ will be able to successfully change/manage the field. diff --git a/docs/book/src/managed/asomanagedcluster.md b/docs/book/src/managed/asomanagedcluster.md new file mode 100644 index 00000000000..398d5dd6639 --- /dev/null +++ b/docs/book/src/managed/asomanagedcluster.md @@ -0,0 +1,54 @@ +## ASO Managed Clusters (AKS) + +- **Feature status:** alpha, not experimental, fully supported +- **Feature gate:** MachinePool=true + +New in CAPZ v1.15.0 is a new flavor of APIs that addresses the following limitations of +the existing CAPZ APIs for advanced use cases for provisioning AKS clusters: + +- A limited set of Azure resource types can be represented. +- A limited set of Azure resource topologies can be expressed. e.g. Only a single Virtual Network resource can + be reconciled for each CAPZ-managed AKS cluster. +- For each Azure resource type supported by CAPZ, CAPZ generally only uses a single Azure API version to + define resources of that type. +- For each Azure API version known by CAPZ, only a subset of fields defined in that version by the Azure API + spec are exposed by the CAPZ API. + +This new API defines new AzureASOManagedCluster, AzureASOManagedControlPlane, and +AzureASOManagedMachinePool resources. An AzureASOManagedCluster might look like this: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 +kind: AzureASOManagedCluster +metadata: + name: my-cluster + namespace: default +spec: + resources: + - apiVersion: resources.azure.com/v1api20200601 + kind: ResourceGroup + metadata: + name: my-resource-group + spec: + location: eastus +``` + +See [here](https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/templates/cluster-template-aks-aso.yaml) for a full AKS example using all the new resources. + +The main element of the new API is `spec.resources` in each new resource, which defines arbitrary, literal ASO +resources inline to be managed by CAPZ. These inline ASO resource definitions take the place of almost all +other configuration currently defined by CAPZ. e.g. Instead of a CAPZ-specific `spec.location` field on the +existing AzureManagedControlPlane, the same value would be expected to be set on an ASO ManagedCluster +resource defined in an AzureASOManagedControlPlane's `spec.resources`. This pattern allows users to define, in +full, any ASO-supported version of a resource type in any of these new CAPZ resources. + +The obvious tradeoff with this new style of API is that CAPZ resource definitions can become more verbose for +basic use cases. To address this, CAPZ still offers flavor templates that use this API with all of the +boilerplate predefined to serve as a starting point for customization. + +The overall theme of this API is to leverage ASO as much as possible for representing Azure resources in the +Kubernetes API, thereby making CAPZ the thinnest possible translation layer between ASO and Cluster API. + +This experiment will help inform CAPZ whether this pattern may be a candidate for a potential v2 API. This +functionality is enabled by default and can be disabled with the `ASOAPI` feature flag (set by the `EXP_ASO_API` environment variable). +Please try it out and offer any feedback! diff --git a/docs/book/src/managed/managed.md b/docs/book/src/managed/managed.md new file mode 100644 index 00000000000..f3a24ccf199 --- /dev/null +++ b/docs/book/src/managed/managed.md @@ -0,0 +1,40 @@ +# Managed Clusters + +This section contains information specific to configuring managed (AKS) Kubernetes clusters using Cluster API Provider for Azure (CAPZ). +See [self-managed clusters](../self-managed/self-managed.md) for information specific to provisioning these clusters. + +Documents under the Managed Clusters area: +- [Adopting Clusters](managed.md) +- [ASO Managed Clusters](asomanagedcluster.md) +- [Managed Clusters](managedcluster.md) +- [Managed Clusters - Join VMSS Nodes](adopting-clusters.md) +- [Troubleshooting](troubleshooting.md) + +## CAPZ's Managed Cluster versus CAPZ's ASOManaged Cluster + +There are two APIs which can create AKS clusters and it's important to understand the differences. + +**ManagedCluster** - is the original API to provision AKS clusters introduced with the [0.4.4 release](https://github.com/kubernetes-sigs/cluster-api-provider-azure/releases/tag/v0.4.4). This more closely matches the CAPI style for YAML code for other providers. The original code was based on directly using the Azure Go SDK, but in the [1.11.0 release](https://github.com/kubernetes-sigs/cluster-api-provider-azure/releases/tag/v1.11.0) was switched to utilize Azure Service Operator (ASO) as a dependency for provisioning. This supports the preview API, but does not natively support all AKS features available. The ManagedCluster API will eventually be deprecated, [see the roadmap for more information](../roadmap.md). + +**ASOManagedCluster** - was created in the [1.15 release of CAPZ](https://github.com/kubernetes-sigs/cluster-api-provider-azure/releases/tag/v1.15.0) and creates a CAPI-compliant wrapper around the existing ASO definitions for AKS clusters. It has 100% API coverage for the preview and current AKS APIs via the ASO AKS CRDs. This is the long-term plan to support provisioning AKS clusters using CAPZ. [See the roadmap for more information](../roadmap.md) + +### Why is ASOManagedCluster the future? + +The biggest challenge with ManagedCluster is attempting to keep up with the velocity of feature changes to the frequently changing AKS API. This model requires every single feature to be added into code directly. Even with the simplification of code to utilize ASO as a dependency, there still is quite a bit of time required to keep up with these features. Ultimately, this is an unsustainable path long-term. The [asoManaged*Patches feature](managedcluster.md#warning-warning-this-is-meant-to-be-used-sparingly-to-enable-features-for-development-and-testing-that-are-not-otherwise-represented-in-the-capz-api-misconfiguration-that-conflicts-with-capzs-normal-mode-of-operation-is-possible) enables patching/adding various AKS fields via ASO to help try to fill some of these gaps, but this comes at the cost of unknown full testing. + +ASOManagedCluster enables 100% API coverage natively and is easy to keep up with since it is primarily a dependency update to CAPZ. Furthermore it has the advantage of making it easier to import existing deployed clusters which have no existing CAPZ or ASO YAML code defined. + +For the complete history of this transition, see the [Managing Azure Resources with Azure Service Operator](https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/docs/proposals/20230123-azure-service-operator.md) and [Automate AKS Features available in CAPZ](https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/docs/proposals/20231122-automate-aks-features.md) design proposal documents. + +## ASO's AKS versus CAPZ's ASOManaged AKS + +It is possible to not utilize CAPZ at all and simply utilize ASO to provision an [AKS cluster definition directly](https://azure.github.io/azure-service-operator/reference/containerservice/v1api20231001/#containerservice.azure.com/v1api20231001.ManagedCluster). The advantages that CAPZ brings over this approach are the following: + - Robust Testing - CAPZ is utilized to test Kubernetes and AKS using this code with numerous end-to-end tests. ASO has no AKS-specific testing. + - Simplification of Infrastructure as Code (IaC) definitions - With ASO you have to figure out how to put together every field and there are some small examples. CAPZ provides `kustomize` template samples connected to `clusterctl generate template` as well as a [helm chart](https://github.com/mboersma/cluster-api-charts/). + - Management scale - CAPZ enables use of [ClusterClass](../topics/clusterclass.md) so you can have a smaller chunk of code to manage numerous clusters with the same configuration. + - Heterogeneous Kubernetes management - it is possible with CAPZ to manage self-managed (not possible at all with ASO) and managed clusters with a single management control plane and similar IaC definitions. + - Multi-cloud IaC consistency - even though it's still wrapping ASO, there still is some consistency in the code contract to provision Kubernetes clusters with the [~30 other CAPI infrastructure providers](https://cluster-api.sigs.k8s.io/reference/providers). + +## General AKS Best Practices + +A set of best practices for managing AKS clusters is documented here: [https://learn.microsoft.com/azure/aks/best-practices](https://learn.microsoft.com/azure/aks/best-practices) diff --git a/docs/book/src/managed/managedcluster-join-vmss.md b/docs/book/src/managed/managedcluster-join-vmss.md new file mode 100644 index 00000000000..da55fff34c5 --- /dev/null +++ b/docs/book/src/managed/managedcluster-join-vmss.md @@ -0,0 +1,125 @@ +## Joining self-managed VMSS nodes to an AKS control plane + + + +### Installing Addons + +In order for the nodes to become ready, you'll need to install Cloud Provider Azure and a CNI. + +AKS will install Cloud Provider Azure on the self-managed nodes as long as they have the appropriate labels. You can add the required label on the nodes by running the following command on the AKS cluster: + +```bash +kubectl label node kubernetes.azure.com/cluster= +``` + +Repeat this for each node in the MachinePool. + + + +For the CNI, you can install the CNI of your choice. For example, to install Azure CNI, run the following command on the AKS cluster: + +```bash +kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-azure/main/templates/addons/azure-cni-v1.yaml +``` + +### Notes + +Some notes about how this works under the hood: + +- CAPZ will fetch the kubeconfig for the AKS cluster and store it in a secret named `${CLUSTER_NAME}-kubeconfig` in the management cluster. That secret is then used for discovery by the `KubeadmConfig` resource. +- You can customize the `MachinePool`, `AzureMachinePool`, and `KubeadmConfig` resources to your liking. The example above is just a starting point. Note that the key configurations to keep are in the `KubeadmConfig` resource, namely the `files`, `joinConfiguration`, and `preKubeadmCommands` sections. +- The `KubeadmConfig` resource will be used to generate a `kubeadm join` command that will be executed on each node in the VMSS. It uses the cluster kubeconfig for discovery. The `kubeadm init phase upload-config all` is run as a preKubeadmCommand to ensure that the kubeadm and kubelet configurations are uploaded to a ConfigMap. This step would normally be done by the `kubeadm init` command, but since we're not running `kubeadm init` we need to do it manually. + +### Creating the MachinePool + +You can add a self-managed VMSS node pool to any CAPZ-managed AKS cluster by applying the following resources to the management cluster: + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: MachinePool +metadata: + name: ${CLUSTER_NAME}-vmss + namespace: default +spec: + clusterName: ${CLUSTER_NAME} + replicas: ${WORKER_MACHINE_COUNT} + template: + spec: + bootstrap: + configRef: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfig + name: ${CLUSTER_NAME}-vmss + clusterName: ${CLUSTER_NAME} + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: AzureMachinePool + name: ${CLUSTER_NAME}-vmss + version: ${KUBERNETES_VERSION} +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: AzureMachinePool +metadata: + name: ${CLUSTER_NAME}-vmss + namespace: default +spec: + location: ${AZURE_LOCATION} + strategy: + rollingUpdate: + deletePolicy: Oldest + maxSurge: 25% + maxUnavailable: 1 + type: RollingUpdate + template: + osDisk: + diskSizeGB: 30 + managedDisk: + storageAccountType: Premium_LRS + osType: Linux + sshPublicKey: ${AZURE_SSH_PUBLIC_KEY_B64:=""} + vmSize: ${AZURE_NODE_MACHINE_TYPE} +--- +apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 +kind: KubeadmConfig +metadata: + name: ${CLUSTER_NAME}-vmss + namespace: default +spec: + files: + - contentFrom: + secret: + key: worker-node-azure.json + name: ${CLUSTER_NAME}-vmss-azure-json + owner: root:root + path: /etc/kubernetes/azure.json + permissions: "0644" + - contentFrom: + secret: + key: value + name: ${CLUSTER_NAME}-kubeconfig + owner: root:root + path: /etc/kubernetes/admin.conf + permissions: "0644" + joinConfiguration: + discovery: + file: + kubeConfigPath: /etc/kubernetes/admin.conf + nodeRegistration: + kubeletExtraArgs: + cloud-provider: external + name: '{{ ds.meta_data["local_hostname"] }}' + preKubeadmCommands: + - kubeadm init phase upload-config all + ``` diff --git a/docs/book/src/topics/managedcluster.md b/docs/book/src/managed/managedcluster.md similarity index 53% rename from docs/book/src/topics/managedcluster.md rename to docs/book/src/managed/managedcluster.md index 4ed76a40d7a..a2f769d0e22 100644 --- a/docs/book/src/topics/managedcluster.md +++ b/docs/book/src/managed/managedcluster.md @@ -18,6 +18,18 @@ creating an AzureManagedControlPlane requires at least one AzureManagedMachinePo with `spec.mode` `System`, since AKS expects at least one system pool at creation time. For more documentation on system node pool refer [AKS Docs](https://learn.microsoft.com/azure/aks/use-system-pools) +Sections in this document: +- [Deploy with clusterctl](#deploy-with-clusterctl) +- [Specification walkthrough](#specification) + - [Use an existing Virtual Network to provision an AKS cluster](#use-an-existing-virtual-network-to-provision-an-aks-cluster) + - [Disable Local Accounts in AKS when using Azure Active Directory](#disable-local-accounts-in-aks-when-using-azure-active-directory) + - [AKS Fleet Integration](#aks-fleet-integration) + - [AKS Extensions](#aks-extensions) + - [Security Profile for AKS clusters](#security-profile-for-aks-clusters) + - [Enabling Preview API Features for ManagedClusters](#enabling-preview-api-features-for-managedclusters) + - [OIDC Issuer on AKS](#oidc-issuer-on-aks) + - [Enable AKS features with custom headers](#enable-aks-features-with-custom-headers---aks-custom-headers) + ## Deploy with clusterctl A clusterctl flavor exists to deploy an AKS cluster with CAPZ. This @@ -59,13 +71,7 @@ export CLUSTER_IDENTITY_NAME="cluster-identity" Managed clusters require the Cluster API "MachinePool" feature flag enabled. You can do that via an environment variable thusly: ```bash -export EXP_MACHINE_POOL=true -``` - -Optionally, you can enable the CAPZ "AKSResourceHealth" feature flag as well: - -```bash -export EXP_AKS_RESOURCE_HEALTH=true +export EXP_MACHINE_POOL=true ``` Create a local kind cluster to run the management cluster components: @@ -242,12 +248,7 @@ spec: name: test-subnet ``` -### Enable AKS features with custom headers (--aks-custom-headers) - -CAPZ no longer supports passing custom headers to AKS APIs with `infrastructure.cluster.x-k8s.io/custom-header-` annotations. -Custom headers are deprecated in AKS in favor of new features first landing in preview API versions: -https://github.com/Azure/azure-rest-api-specs/pull/18232 ### Disable Local Accounts in AKS when using Azure Active Directory @@ -337,7 +338,7 @@ az k8s-extension extension-types list-by-cluster --resource-group my-resource-gr For more details, please refer to the [az k8s-extension cli reference](https://learn.microsoft.com/cli/azure/k8s-extension). -### Security Profile for AKS clusters. +### Security Profile for AKS clusters Example for configuring AzureManagedControlPlane with a security profile: @@ -402,7 +403,7 @@ spec: - '{"spec": {"enableCustomCATrust": true}}' ``` -#### OIDC Issuer on AKS +### OIDC Issuer on AKS Setting `AzureManagedControlPlane.Spec.oidcIssuerProfile.enabled` to `true` will enable OIDC issuer profile for the `AzureManagedControlPlane`. Once enabled, you will see a configmap named `-aso-oidc-issuer-profile` in the same namespace as the `AzureManagedControlPlane` resource. This configmap will contain the OIDC issuer profile url under the `oidc-issuer-profile-url` key. @@ -410,307 +411,9 @@ Once OIDC issuer is enabled on the cluster, it's not supported to disable it. To learn more about OIDC and AKS refer [AKS Docs on OIDC issuer](https://learn.microsoft.com/en-us/azure/aks/use-oidc-issuer). +### Enable AKS features with custom headers (--aks-custom-headers) -## Features - -AKS clusters deployed from CAPZ currently only support a limited, -"blessed" configuration. This was primarily to keep the initial -implementation simple. If you'd like to run managed AKS cluster with CAPZ -and need an additional feature, please open a pull request or issue with -details. We're happy to help! - -## Best Practices - -A set of best practices for managing AKS clusters is documented here: https://learn.microsoft.com/azure/aks/best-practices - -## Troubleshooting - -If a user tries to delete the MachinePool which refers to the last system node pool AzureManagedMachinePool webhook will reject deletion, so time stamp never gets set on the AzureManagedMachinePool. However the timestamp would be set on the MachinePool and would be in deletion state. To recover from this state create a new MachinePool manually referencing the AzureManagedMachinePool, edit the required references and finalizers to link the MachinePool to the AzureManagedMachinePool. In the AzureManagedMachinePool remove the owner reference to the old MachinePool, and set it to the new MachinePool. Once the new MachinePool is pointing to the AzureManagedMachinePool you can delete the old MachinePool. To delete the old MachinePool remove the finalizers in that object. - -Here is an Example: - -```yaml -# MachinePool deleted -apiVersion: cluster.x-k8s.io/v1beta1 -kind: MachinePool -metadata: - finalizers: # remove finalizers once new object is pointing to the AzureManagedMachinePool - - machinepool.cluster.x-k8s.io - labels: - cluster.x-k8s.io/cluster-name: capz-managed-aks - name: agentpool0 - namespace: default - ownerReferences: - - apiVersion: cluster.x-k8s.io/v1beta1 - kind: Cluster - name: capz-managed-aks - uid: 152ecf45-0a02-4635-987c-1ebb89055fa2 - uid: ae4a235a-f0fa-4252-928a-0e3b4c61dbea -spec: - clusterName: capz-managed-aks - minReadySeconds: 0 - providerIDList: - - azure:///subscriptions/9107f2fb-e486-a434-a948-52e2929b6f18/resourceGroups/MC_rg_capz-managed-aks_eastus/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool0-10226072-vmss/virtualMachines/0 - replicas: 1 - template: - metadata: {} - spec: - bootstrap: - dataSecretName: "" - clusterName: capz-managed-aks - infrastructureRef: - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 - kind: AzureManagedMachinePool - name: agentpool0 - namespace: default - version: v1.21.2 - ---- -# New Machinepool -apiVersion: cluster.x-k8s.io/v1beta1 -kind: MachinePool -metadata: - finalizers: - - machinepool.cluster.x-k8s.io - generation: 2 - labels: - cluster.x-k8s.io/cluster-name: capz-managed-aks - name: agentpool2 # change the name of the machinepool - namespace: default - ownerReferences: - - apiVersion: cluster.x-k8s.io/v1beta1 - kind: Cluster - name: capz-managed-aks - uid: 152ecf45-0a02-4635-987c-1ebb89055fa2 - # uid: ae4a235a-f0fa-4252-928a-0e3b4c61dbea # remove the uid set for machinepool -spec: - clusterName: capz-managed-aks - minReadySeconds: 0 - providerIDList: - - azure:///subscriptions/9107f2fb-e486-a434-a948-52e2929b6f18/resourceGroups/MC_rg_capz-managed-aks_eastus/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool0-10226072-vmss/virtualMachines/0 - replicas: 1 - template: - metadata: {} - spec: - bootstrap: - dataSecretName: "" - clusterName: capz-managed-aks - infrastructureRef: - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 - kind: AzureManagedMachinePool - name: agentpool0 - namespace: default - version: v1.21.2 -``` - -## Joining self-managed VMSS nodes to an AKS control plane - - - -### Creating the MachinePool - -You can add a self-managed VMSS node pool to any CAPZ-managed AKS cluster by applying the following resources to the management cluster: - -```yaml -apiVersion: cluster.x-k8s.io/v1beta1 -kind: MachinePool -metadata: - name: ${CLUSTER_NAME}-vmss - namespace: default -spec: - clusterName: ${CLUSTER_NAME} - replicas: ${WORKER_MACHINE_COUNT} - template: - spec: - bootstrap: - configRef: - apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 - kind: KubeadmConfig - name: ${CLUSTER_NAME}-vmss - clusterName: ${CLUSTER_NAME} - infrastructureRef: - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 - kind: AzureMachinePool - name: ${CLUSTER_NAME}-vmss - version: ${KUBERNETES_VERSION} ---- -apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 -kind: AzureMachinePool -metadata: - name: ${CLUSTER_NAME}-vmss - namespace: default -spec: - location: ${AZURE_LOCATION} - strategy: - rollingUpdate: - deletePolicy: Oldest - maxSurge: 25% - maxUnavailable: 1 - type: RollingUpdate - template: - osDisk: - diskSizeGB: 30 - managedDisk: - storageAccountType: Premium_LRS - osType: Linux - sshPublicKey: ${AZURE_SSH_PUBLIC_KEY_B64:=""} - vmSize: ${AZURE_NODE_MACHINE_TYPE} ---- -apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 -kind: KubeadmConfig -metadata: - name: ${CLUSTER_NAME}-vmss - namespace: default -spec: - files: - - contentFrom: - secret: - key: worker-node-azure.json - name: ${CLUSTER_NAME}-vmss-azure-json - owner: root:root - path: /etc/kubernetes/azure.json - permissions: "0644" - - contentFrom: - secret: - key: value - name: ${CLUSTER_NAME}-kubeconfig - owner: root:root - path: /etc/kubernetes/admin.conf - permissions: "0644" - joinConfiguration: - discovery: - file: - kubeConfigPath: /etc/kubernetes/admin.conf - nodeRegistration: - kubeletExtraArgs: - cloud-provider: external - name: '{{ ds.meta_data["local_hostname"] }}' - preKubeadmCommands: - - kubeadm init phase upload-config all - ``` - -### Installing Addons - -In order for the nodes to become ready, you'll need to install Cloud Provider Azure and a CNI. - -AKS will install Cloud Provider Azure on the self-managed nodes as long as they have the appropriate labels. You can add the required label on the nodes by running the following command on the AKS cluster: - -```bash -kubectl label node kubernetes.azure.com/cluster= -``` - -Repeat this for each node in the MachinePool. - - - -For the CNI, you can install the CNI of your choice. For example, to install Azure CNI, run the following command on the AKS cluster: - -```bash -kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-azure/main/templates/addons/azure-cni-v1.yaml -``` - -### Notes - -Some notes about how this works under the hood: - -- CAPZ will fetch the kubeconfig for the AKS cluster and store it in a secret named `${CLUSTER_NAME}-kubeconfig` in the management cluster. That secret is then used for discovery by the `KubeadmConfig` resource. -- You can customize the `MachinePool`, `AzureMachinePool`, and `KubeadmConfig` resources to your liking. The example above is just a starting point. Note that the key configurations to keep are in the `KubeadmConfig` resource, namely the `files`, `joinConfiguration`, and `preKubeadmCommands` sections. -- The `KubeadmConfig` resource will be used to generate a `kubeadm join` command that will be executed on each node in the VMSS. It uses the cluster kubeconfig for discovery. The `kubeadm init phase upload-config all` is run as a preKubeadmCommand to ensure that the kubeadm and kubelet configurations are uploaded to a ConfigMap. This step would normally be done by the `kubeadm init` command, but since we're not running `kubeadm init` we need to do it manually. - -## Adopting Existing AKS Clusters - -### Option 1: Using the experimental ASO-based API - - -The [experimental AzureASOManagedControlPlane and related APIs](/topics/aso.html#experimental-aso-api) support -adoption as a first-class use case. Going forward, this method is likely to be easier, more reliable, include -more features, and better supported for adopting AKS clusters than Option 2 below. - -To adopt an AKS cluster into a full Cluster API Cluster, create an ASO ManagedCluster and associated -ManagedClustersAgentPool resources annotated with `sigs.k8s.io/cluster-api-provider-azure-adopt=true`. The -annotation may also be added to existing ASO resources to trigger adoption. CAPZ will automatically scaffold -the Cluster API resources like the Cluster, AzureASOManagedCluster, AzureASOManagedControlPlane, MachinePools, -and AzureASOManagedMachinePools. The [`asoctl import -azure-resource`](https://azure.github.io/azure-service-operator/tools/asoctl/#import-azure-resource) command -can help generate the required YAML. - -Caveats: -- The `asoctl import azure-resource` command has at least [one known - bug](https://github.com/Azure/azure-service-operator/issues/3805) requiring the YAML it generates to be - edited before it can be applied to a cluster. -- CAPZ currently only records the ASO resources in the CAPZ resources' `spec.resources` that it needs to - function, which include the ManagedCluster, its ResourceGroup, and associated ManagedClustersAgentPools. - Other resources owned by the ManagedCluster like Kubernetes extensions or Fleet memberships are not - currently imported to the CAPZ specs. -- Configuring the automatically generated Cluster API resources is not currently possible. If you need to - change something like the `metadata.name` of a resource from what CAPZ generates, create the Cluster API - resources manually referencing the pre-existing resources. -- Adopting existing clusters created with the GA AzureManagedControlPlane API to the experimental API with - this method is theoretically possible, but untested. Care should be taken to prevent CAPZ from reconciling - two different representations of the same underlying Azure resources. - -### Option 2: Using the current AzureManagedControlPlane API - - - -CAPZ can adopt some AKS clusters created by other means under its management. This works by crafting CAPI and -CAPZ manifests which describe the existing cluster and creating those resources on the CAPI management -cluster. This approach is limited to clusters which can be described by the CAPZ API, which includes the -following constraints: - -- the cluster operates within a single Virtual Network and Subnet -- the cluster's Virtual Network exists outside of the AKS-managed `MC_*` resource group -- the cluster's Virtual Network and Subnet are not shared with any other resources outside the context of this cluster - -To ensure CAPZ does not introduce any unwarranted changes while adopting an existing cluster, carefully review -the [entire AzureManagedControlPlane spec](../reference/v1beta1-api#infrastructure.cluster.x-k8s.io/v1beta1.AzureManagedControlPlaneSpec) -and specify _every_ field in the CAPZ resource. CAPZ's webhooks apply defaults to many fields which may not -match the existing cluster. - -Specific AKS features not represented in the CAPZ API, like those from a newer AKS API version than CAPZ uses, -do not need to be specified in the CAPZ resources to remain configured the way they are. CAPZ will still not -be able to manage that configuration, but it will not modify any settings beyond those for which it has -knowledge. - -By default, CAPZ will not make any changes to or delete any pre-existing Resource Group, Virtual Network, or -Subnet resources. To opt-in to CAPZ management for those clusters, tag those resources with the following -before creating the CAPZ resources: `sigs.k8s.io_cluster-api-provider-azure_cluster_: owned`. -Managed Cluster and Agent Pool resources do not need this tag in order to be adopted. - -After applying the CAPI and CAPZ resources for the cluster, other means of managing the cluster should be -disabled to avoid ongoing conflicts with CAPZ's reconciliation process. - -#### Pitfalls - -The following describes some specific pieces of configuration that deserve particularly careful attention, -adapted from https://gist.github.com/mtougeron/1e5d7a30df396cd4728a26b2555e0ef0#file-capz-md. - -- Make sure `AzureManagedControlPlane.metadata.name` matches the AKS cluster name -- Set the `AzureManagedControlPlane.spec.virtualNetwork` fields to match your existing VNET -- Make sure the `AzureManagedControlPlane.spec.sshPublicKey` matches what was set on the AKS cluster. (including any potential newlines included in the base64 encoding) - - NOTE: This is a required field in CAPZ, if you don't know what public key was used, you can _change_ or _set_ it via the Azure CLI however before attempting to import the cluster. -- Make sure the `Cluster.spec.clusterNetwork` settings match properly to what you are using in AKS -- Make sure the `AzureManagedControlPlane.spec.dnsServiceIP` matches what is set in AKS -- Set the tag `sigs.k8s.io_cluster-api-provider-azure_cluster_` = `owned` on the AKS cluster -- Set the tag `sigs.k8s.io_cluster-api-provider-azure_role` = `common` on the AKS cluster +CAPZ no longer supports passing custom headers to AKS APIs with `infrastructure.cluster.x-k8s.io/custom-header-` annotations. +Custom headers are deprecated in AKS in favor of new features first landing in preview API versions: -NOTE: Several fields, like `networkPlugin`, if not set on the AKS cluster at creation time, will mean that CAPZ will not be able to set that field. AKS doesn't allow such fields to be changed if not set at creation. However, if it was set at creation time, CAPZ will be able to successfully change/manage the field. +https://github.com/Azure/azure-rest-api-specs/pull/18232 \ No newline at end of file diff --git a/docs/book/src/managed/troubleshooting.md b/docs/book/src/managed/troubleshooting.md new file mode 100644 index 00000000000..9dd9c7c9203 --- /dev/null +++ b/docs/book/src/managed/troubleshooting.md @@ -0,0 +1,79 @@ +# Troubleshooting Managed Clusters (AKS) + +If a user tries to delete the MachinePool which refers to the last system node pool AzureManagedMachinePool webhook will reject deletion, so time stamp never gets set on the AzureManagedMachinePool. However the timestamp would be set on the MachinePool and would be in deletion state. To recover from this state create a new MachinePool manually referencing the AzureManagedMachinePool, edit the required references and finalizers to link the MachinePool to the AzureManagedMachinePool. In the AzureManagedMachinePool remove the owner reference to the old MachinePool, and set it to the new MachinePool. Once the new MachinePool is pointing to the AzureManagedMachinePool you can delete the old MachinePool. To delete the old MachinePool remove the finalizers in that object. + +Here is an Example: + +```yaml +# MachinePool deleted +apiVersion: cluster.x-k8s.io/v1beta1 +kind: MachinePool +metadata: + finalizers: # remove finalizers once new object is pointing to the AzureManagedMachinePool + - machinepool.cluster.x-k8s.io + labels: + cluster.x-k8s.io/cluster-name: capz-managed-aks + name: agentpool0 + namespace: default + ownerReferences: + - apiVersion: cluster.x-k8s.io/v1beta1 + kind: Cluster + name: capz-managed-aks + uid: 152ecf45-0a02-4635-987c-1ebb89055fa2 + uid: ae4a235a-f0fa-4252-928a-0e3b4c61dbea +spec: + clusterName: capz-managed-aks + minReadySeconds: 0 + providerIDList: + - azure:///subscriptions/9107f2fb-e486-a434-a948-52e2929b6f18/resourceGroups/MC_rg_capz-managed-aks_eastus/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool0-10226072-vmss/virtualMachines/0 + replicas: 1 + template: + metadata: {} + spec: + bootstrap: + dataSecretName: "" + clusterName: capz-managed-aks + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: AzureManagedMachinePool + name: agentpool0 + namespace: default + version: v1.21.2 + +--- +# New Machinepool +apiVersion: cluster.x-k8s.io/v1beta1 +kind: MachinePool +metadata: + finalizers: + - machinepool.cluster.x-k8s.io + generation: 2 + labels: + cluster.x-k8s.io/cluster-name: capz-managed-aks + name: agentpool2 # change the name of the machinepool + namespace: default + ownerReferences: + - apiVersion: cluster.x-k8s.io/v1beta1 + kind: Cluster + name: capz-managed-aks + uid: 152ecf45-0a02-4635-987c-1ebb89055fa2 + # uid: ae4a235a-f0fa-4252-928a-0e3b4c61dbea # remove the uid set for machinepool +spec: + clusterName: capz-managed-aks + minReadySeconds: 0 + providerIDList: + - azure:///subscriptions/9107f2fb-e486-a434-a948-52e2929b6f18/resourceGroups/MC_rg_capz-managed-aks_eastus/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool0-10226072-vmss/virtualMachines/0 + replicas: 1 + template: + metadata: {} + spec: + bootstrap: + dataSecretName: "" + clusterName: capz-managed-aks + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: AzureManagedMachinePool + name: agentpool0 + namespace: default + version: v1.21.2 +``` diff --git a/docs/book/src/topics/addons.md b/docs/book/src/self-managed/addons.md similarity index 100% rename from docs/book/src/topics/addons.md rename to docs/book/src/self-managed/addons.md diff --git a/docs/book/src/topics/api-server-endpoint.md b/docs/book/src/self-managed/api-server-endpoint.md similarity index 100% rename from docs/book/src/topics/api-server-endpoint.md rename to docs/book/src/self-managed/api-server-endpoint.md diff --git a/docs/book/src/topics/cloud-provider-config.md b/docs/book/src/self-managed/cloud-provider-config.md similarity index 100% rename from docs/book/src/topics/cloud-provider-config.md rename to docs/book/src/self-managed/cloud-provider-config.md diff --git a/docs/book/src/topics/confidential-vms.md b/docs/book/src/self-managed/confidential-vms.md similarity index 100% rename from docs/book/src/topics/confidential-vms.md rename to docs/book/src/self-managed/confidential-vms.md diff --git a/docs/book/src/topics/control-plane-outbound-lb.md b/docs/book/src/self-managed/control-plane-outbound-lb.md similarity index 100% rename from docs/book/src/topics/control-plane-outbound-lb.md rename to docs/book/src/self-managed/control-plane-outbound-lb.md diff --git a/docs/book/src/topics/custom-dns.md b/docs/book/src/self-managed/custom-dns.md similarity index 100% rename from docs/book/src/topics/custom-dns.md rename to docs/book/src/self-managed/custom-dns.md diff --git a/docs/book/src/topics/custom-images.md b/docs/book/src/self-managed/custom-images.md similarity index 100% rename from docs/book/src/topics/custom-images.md rename to docs/book/src/self-managed/custom-images.md diff --git a/docs/book/src/topics/custom-vm-extensions.md b/docs/book/src/self-managed/custom-vm-extensions.md similarity index 100% rename from docs/book/src/topics/custom-vm-extensions.md rename to docs/book/src/self-managed/custom-vm-extensions.md diff --git a/docs/book/src/topics/custom-vnet.md b/docs/book/src/self-managed/custom-vnet.md similarity index 100% rename from docs/book/src/topics/custom-vnet.md rename to docs/book/src/self-managed/custom-vnet.md diff --git a/docs/book/src/topics/data-disks.md b/docs/book/src/self-managed/data-disks.md similarity index 100% rename from docs/book/src/topics/data-disks.md rename to docs/book/src/self-managed/data-disks.md diff --git a/docs/book/src/topics/disk-encryption.md b/docs/book/src/self-managed/disk-encryption.md similarity index 100% rename from docs/book/src/topics/disk-encryption.md rename to docs/book/src/self-managed/disk-encryption.md diff --git a/docs/book/src/topics/disks.md b/docs/book/src/self-managed/disks.md similarity index 100% rename from docs/book/src/topics/disks.md rename to docs/book/src/self-managed/disks.md diff --git a/docs/book/src/topics/dual-stack.md b/docs/book/src/self-managed/dual-stack.md similarity index 100% rename from docs/book/src/topics/dual-stack.md rename to docs/book/src/self-managed/dual-stack.md diff --git a/docs/book/src/topics/externally-managed-azure-infrastructure.md b/docs/book/src/self-managed/externally-managed-azure-infrastructure.md similarity index 100% rename from docs/book/src/topics/externally-managed-azure-infrastructure.md rename to docs/book/src/self-managed/externally-managed-azure-infrastructure.md diff --git a/docs/book/src/topics/failure-domains.md b/docs/book/src/self-managed/failure-domains.md similarity index 100% rename from docs/book/src/topics/failure-domains.md rename to docs/book/src/self-managed/failure-domains.md diff --git a/docs/book/src/topics/flatcar.md b/docs/book/src/self-managed/flatcar.md similarity index 98% rename from docs/book/src/topics/flatcar.md rename to docs/book/src/self-managed/flatcar.md index a3fdd1bdd48..8c06f650574 100644 --- a/docs/book/src/topics/flatcar.md +++ b/docs/book/src/self-managed/flatcar.md @@ -14,7 +14,7 @@ The testing reference images are built using [image-builder](https://github.com/ The reference images are not updated with security fixes. They are intended only to facilitate testing and to help users try out Cluster API for Azure. -The reference images should not be used in a production environment. It is highly recommended to [maintain your own custom image](#building-a-custom-image) instead. +The reference images should not be used in a production environment. It is highly recommended to [maintain your own custom image](./custom-images.md#building-a-custom-image) instead. diff --git a/docs/book/src/topics/gpu.md b/docs/book/src/self-managed/gpu.md similarity index 100% rename from docs/book/src/topics/gpu.md rename to docs/book/src/self-managed/gpu.md diff --git a/docs/book/src/topics/ipv6.md b/docs/book/src/self-managed/ipv6.md similarity index 100% rename from docs/book/src/topics/ipv6.md rename to docs/book/src/self-managed/ipv6.md diff --git a/docs/book/src/topics/machinepools.md b/docs/book/src/self-managed/machinepools.md similarity index 100% rename from docs/book/src/topics/machinepools.md rename to docs/book/src/self-managed/machinepools.md diff --git a/docs/book/src/topics/node-outbound-connection.md b/docs/book/src/self-managed/node-outbound-connection.md similarity index 100% rename from docs/book/src/topics/node-outbound-connection.md rename to docs/book/src/self-managed/node-outbound-connection.md diff --git a/docs/book/src/topics/os-disk.md b/docs/book/src/self-managed/os-disk.md similarity index 100% rename from docs/book/src/topics/os-disk.md rename to docs/book/src/self-managed/os-disk.md diff --git a/docs/book/src/topics/publicmec-clusters.md b/docs/book/src/self-managed/publicmec-clusters.md similarity index 100% rename from docs/book/src/topics/publicmec-clusters.md rename to docs/book/src/self-managed/publicmec-clusters.md diff --git a/docs/book/src/self-managed/self-managed.md b/docs/book/src/self-managed/self-managed.md new file mode 100644 index 00000000000..a43fd0f7032 --- /dev/null +++ b/docs/book/src/self-managed/self-managed.md @@ -0,0 +1,4 @@ +# Self-Managed Clusters + +This section contains information specific to configuring self-managed Kubernetes clusters using Cluster API Provider for Azure (CAPZ). +See [managed clusters](../managed/managed.md) (AKS) for information specific to provisioning these clusters. diff --git a/docs/book/src/topics/spot-vms.md b/docs/book/src/self-managed/spot-vms.md similarity index 100% rename from docs/book/src/topics/spot-vms.md rename to docs/book/src/self-managed/spot-vms.md diff --git a/docs/book/src/topics/ssh-access.md b/docs/book/src/self-managed/ssh-access.md similarity index 100% rename from docs/book/src/topics/ssh-access.md rename to docs/book/src/self-managed/ssh-access.md diff --git a/docs/book/src/topics/troubleshooting.md b/docs/book/src/self-managed/troubleshooting.md similarity index 100% rename from docs/book/src/topics/troubleshooting.md rename to docs/book/src/self-managed/troubleshooting.md diff --git a/docs/book/src/topics/trusted-launch-for-vms.md b/docs/book/src/self-managed/trusted-launch-for-vms.md similarity index 100% rename from docs/book/src/topics/trusted-launch-for-vms.md rename to docs/book/src/self-managed/trusted-launch-for-vms.md diff --git a/docs/book/src/topics/vm-diagnostics.md b/docs/book/src/self-managed/vm-diagnostics.md similarity index 100% rename from docs/book/src/topics/vm-diagnostics.md rename to docs/book/src/self-managed/vm-diagnostics.md diff --git a/docs/book/src/topics/vm-identity.md b/docs/book/src/self-managed/vm-identity.md similarity index 100% rename from docs/book/src/topics/vm-identity.md rename to docs/book/src/self-managed/vm-identity.md diff --git a/docs/book/src/topics/wasi.md b/docs/book/src/self-managed/wasi.md similarity index 100% rename from docs/book/src/topics/wasi.md rename to docs/book/src/self-managed/wasi.md diff --git a/docs/book/src/topics/windows.md b/docs/book/src/self-managed/windows.md similarity index 100% rename from docs/book/src/topics/windows.md rename to docs/book/src/self-managed/windows.md diff --git a/docs/book/src/topics/aso.md b/docs/book/src/topics/aso.md index 0607562d56f..e9513abde6a 100644 --- a/docs/book/src/topics/aso.md +++ b/docs/book/src/topics/aso.md @@ -8,6 +8,8 @@ CAPZ interfaces with Azure to create and manage some types of resources using [A More context around the decision for CAPZ to pivot towards using ASO can be found in the [proposal](https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/docs/proposals/20230123-azure-service-operator.md). +[Visit this page](../managed/asomanagedcluster.md) to learn more about the AzureASOManaged cluster API which provisions an AKS cluster. + ## Primary changes For most users, the introduction of ASO is expected to be fully transparent and backwards compatible. Changes @@ -79,55 +81,3 @@ More details about how ASO manages CRDs can be found [here](https://azure.github **Note:** To install the resource for the newly installed CRDs, make sure that the ASO operator has the authentication to install the resources. Refer [authentication in ASO](https://azure.github.io/azure-service-operator/guide/authentication/) for more details. An example configuration file and demo for `Azure Cache for Redis` can be found [here](https://github.com/Azure-Samples/azure-service-operator-samples/tree/master/azure-votes-redis). - -## ASO-based API - -New in CAPZ v1.15.0 is a new flavor of APIs that addresses the following limitations of -the existing CAPZ APIs for advanced use cases: - -- A limited set of Azure resource types can be represented. -- A limited set of Azure resource topologies can be expressed. e.g. Only a single Virtual Network resource can - be reconciled for each CAPZ-managed AKS cluster. -- For each Azure resource type supported by CAPZ, CAPZ generally only uses a single Azure API version to - define resources of that type. -- For each Azure API version known by CAPZ, only a subset of fields defined in that version by the Azure API - spec are exposed by the CAPZ API. - -This new API defines new AzureASOManagedCluster, AzureASOManagedControlPlane, and -AzureASOManagedMachinePool resources. An AzureASOManagedCluster might look like this: - -```yaml -apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 -kind: AzureASOManagedCluster -metadata: - name: my-cluster - namespace: default -spec: - resources: - - apiVersion: resources.azure.com/v1api20200601 - kind: ResourceGroup - metadata: - name: my-resource-group - spec: - location: eastus -``` - -See [here](https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/templates/cluster-template-aks-aso.yaml) for a full AKS example using all the new resources. - -The main element of the new API is `spec.resources` in each new resource, which defines arbitrary, literal ASO -resources inline to be managed by CAPZ. These inline ASO resource definitions take the place of almost all -other configuration currently defined by CAPZ. e.g. Instead of a CAPZ-specific `spec.location` field on the -existing AzureManagedControlPlane, the same value would be expected to be set on an ASO ManagedCluster -resource defined in an AzureASOManagedControlPlane's `spec.resources`. This pattern allows users to define, in -full, any ASO-supported version of a resource type in any of these new CAPZ resources. - -The obvious tradeoff with this new style of API is that CAPZ resource definitions can become more verbose for -basic use cases. To address this, CAPZ still offers flavor templates that use this API with all of the -boilerplate predefined to serve as a starting point for customization. - -The overall theme of this API is to leverage ASO as much as possible for representing Azure resources in the -Kubernetes API, thereby making CAPZ the thinnest possible translation layer between ASO and Cluster API. - -This experiment will help inform CAPZ whether this pattern may be a candidate for a potential v2 API. This -functionality is enabled by default and can be disabled with the `ASOAPI` feature flag (set by the `EXP_ASO_API` environment variable). -Please try it out and offer any feedback! diff --git a/docs/book/src/topics/identities.md b/docs/book/src/topics/identities.md index 122a7d345ab..284776b438b 100644 --- a/docs/book/src/topics/identities.md +++ b/docs/book/src/topics/identities.md @@ -160,9 +160,9 @@ spec: - ``` -### Assigning VM identities for cloud provider authentication +### Assigning VM identities for cloud provider authentication (self-managed) -When using a user-assigned managed identity to create the workload cluster, a VM identity should also be assigned to each control plane machine in the workload cluster for Azure Cloud Provider to use. See [here](../topics/vm-identity.md#managed-identities) for more information. +When using a user-assigned managed identity to create the workload cluster, a VM identity should also be assigned to each control plane machine in the workload cluster for Azure Cloud Provider to use. See [here](../self-managed/vm-identity.md#managed-identities) for more information. ## Azure Host Identity @@ -172,6 +172,6 @@ The identity assigned to the Azure host which in the control plane provides the - User-assigned Managed Identity - System-assigned Managed Identity - Service Principal -- See details about each type in the [VM identity](vm-identity.md) page +- See details about each type in the [VM identity](../self-managed/vm-identity.md) page More details in [Azure built-in roles documentation](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles). diff --git a/docs/book/src/topics/multitenancy.md b/docs/book/src/topics/multitenancy.md index c028b989725..0ee3a9695cd 100644 --- a/docs/book/src/topics/multitenancy.md +++ b/docs/book/src/topics/multitenancy.md @@ -20,7 +20,7 @@ Please note NamespaceList will take precedence over Selector if both are set.