diff --git a/docs/docs-content/integrations/k3s.md b/docs/docs-content/integrations/k3s.md new file mode 100644 index 0000000000..febbdbcc20 --- /dev/null +++ b/docs/docs-content/integrations/k3s.md @@ -0,0 +1,484 @@ +--- +sidebar_label: 'K3s' +title: 'K3s' +description: "Learn about the K3s pack in Palette Edge and how you can use it within your Kubernetes clusters." +hide_table_of_contents: true +type: "integration" +category: ['kubernetes', 'amd64', 'fips'] +sidebar_class_name: "hide-from-sidebar" +logoUrl: 'https://registry.spectrocloud.com/v1/edge-k3s/blobs/sha256:5b7f8a80330d6938056848905238efbbd085e2f0f86fd03d7be10ccdabf791c7?type=image/png' +tags: ["packs", "kubernetes", "k3s", "edge"] +--- + +K3s is a lightweight distribution of Kubernetes that is specifically designed for resource-constrained environments and Internet of Things (IoT) applications. Unlike standard Kubernetes with multiple binaries for its various components, K3s is packaged as a single binary with a lightweight storage backend and minimal external dependencies. For more information about K3s, refer to [K3s documentation](https://docs.k3s.io/). + + +### Support Lifecycle + +We support other Kubernetes distributions such as K3s, Microk8s, and RKE2 until their official EOL. The EOL is set by the respective owner. Once we stop supporting the minor version, we initiate the deprecation process. Refer to the [Kubernetes Support Lifecycle](kubernetes-support.md#palette-extended-kubernetes-support) guide to learn more. + +## Versions Supported + + + + +### Prerequisites + +- An edge device with AMD64 processor architecture or a Palette Virtual Cluster. +- A minimum of 2 CPU cores and 1 GB memory. + +### Parameters + +Since you can deploy both virtual clusters and Edge clusters using K3s, you have different configuration options depending on the cluster type. + + + + + +|**Parameter**|**Description** | +|-------------|----------------| +| `cluster.config.cluster-cidr`| Specifies the CIDR range that can be used by pods in the cluster. | +| `cluster.config.service-cidr`| Specifies the CIDR range that can be used by services in the cluster.| +| `kube-apiserver-arg`| This parameter contains extra arguments for the Kubernetes API server, such as enabling audit logging, enabling certain authorization modes, and setting profiling and secure-port.| +|`kube-controller-manager-arg` | This parameter describes extra arguments for the Kubernetes Controller Manager, such as enabling certain feature gates and setting profiling. | +| `kubelet-arg` | This parameter contains extra arguments for Kubelet during node registration, such as setting feature gates, protecting kernel defaults, and disabling the read-only port. | +| `pack.palette.config.oidc.identityProvider` | Dynamically enabled OpenID Connect (OIDC) Identity Provider (IDP) setting based on your UI selection when you add the K3s pack to your profile. This parameter appears in the YAML file after you make a selection. Refer to [Configure OIDC Identity Provider](#configure-custom-oidc). | + +You can add cloud-init stages, which allow you to customize your instances declaratively. The cloud-init stages are exposed by [Kairos](https://kairos.io/docs/architecture/cloud-init/), an open source project. For more information, check out the [Cloud Init Stages](../clusters/edge/edge-configuration/cloud-init.md) reference. + + + + +Since you are setting up a virtual cluster inside another Kubernetes cluster, you can configure its pods and services differently than the host cluster. The default configuration file you get includes parameters that offer you a higher degree of customization. These configuration parameters are exposed in the cluster group settings page. + +|**Parameter**|**Description** | +|-------------|----------------| +|`enableHA`| Determines whether the control plane is deployed in high availability mode. If you set this parameter to true, make sure to adjust the number of replicas and use an external datastore. | +|`defaultImageRegistry` | Specifies the default registry from which images are pulled. The value of this parameter will be prepended to all deployed virtual cluster images. If an image has already been deployed as part of the virtual cluster, the deployed images within the virtual cluster will not be rewritten.| +|`sync` | Specifies which Kubernetes resources are synced between the virtual and host clusters. | +|`storage` | Specifies storage settings such as persistence and PVC size. By default, storage of the virtual cluster uses the same storage class as the host cluster, but you can also optionally specify a different storage class. | +|`ingress` | Configures the ingress resource that allows you to access the virtual cluster. | + + + + + +### Usage + +K3s is available for Edge host deployments as well as virtual clusters that you can create from cluster groups. Refer to the [Create an Infrastructure Profile](../profiles/cluster-profiles/create-cluster-profiles/create-infrastructure-profile.md) guide and the [Create and Manage Cluster Groups](../clusters/cluster-groups/create-cluster-group.md) guide for more information. + +:::info +In order to use K3s as part of an Edge deployment, you need to go through the EdgeForge process and specify K3s as your intended Kubernetes distribution when you build your OS image. For more information, refer to the [EdgeForge Workflow](../clusters/edge/edgeforge-workflow/) guide. +::: + +#### Configure OIDC Identity Provider for Edge + +You can modify the configuration file to configure your Edge cluster to use an OpenID Connect (OIDC) Identity Provider (IDP) for authentication. +You can use a custom third-party IDP, such as Okta, or use Palette as your IDP. + +When you add the K3s pack to a cluster profile, Palette displays the OIDC IDP options listed below: + +- **None**: This setting does not require OIDC configuration for the cluster. It displays in the YAML file as `noauth`. + +- **Custom**: This is the default setting and does not require OIDC configuration. However, if desired, it allows you to specify a third-party OIDC provider by configuring OIDC statements in the YAML file as described in [Configure Custom OIDC](kubernetes-edge.md#configure-custom-oidc). This setting displays in the YAML file as `none`. + +- **Palette**: This setting makes Palette the IDP. Any user with a Palette account in the tenant and the proper permissions to view and access the project's resources is able to use kubectl CLI to access cluster. This setting displays in the YAML file as `palette`. When you select **Palette**, all you have to do to enable OIDC for your cluster is create role bindings to configure authorization. You do not need to provide extra parameters such as `oidc-issuer-url` as you need to when you configure a custom OIDC provider. + +- **Inherit from Tenant**: This setting allows you to apply RBAC to multiple clusters and requires you to configure OpenID Connect (OIDC) in **Tenant Settings**. In Tenant Admin scope, navigate to **Tenant Settings** > **SSO**, choose **OIDC**, and provide your third-party IDP details. This setting displays in the YAML file as `tenant`. For more information, check out the [SSO Setup](../user-management/saml-sso/enable-saml.md) guide. + +All the options require you to map a set of users or groups to a Kubernetes RBAC role. To learn how to map a Kubernetes role to users and groups, refer to [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings). + +:::warning + +If your IDP uses Security Assertion Markup Language (SAML) authentication, then the **Inherit from Tenant** option will not work, and you will need to use the **Custom** option instead. This is because Kubernetes supports only OIDC authentication and not SAML authentication. + +::: + +To configure a custom OIDC IDP, choose **Custom** when adding the K3s pack to your profile, and then follow these steps: + +1. Add the following OIDC parameters to the `kube-apiserver-arg` section of your configuration file for your Kubernetes layer when creating a cluster profile. + + ```yaml + cluster: + config: + kube-apiserver-arg: + - oidc-issuer-url="provider URL" + - oidc-client-id="client-id" + - oidc-groups-claim="groups" + - oidc-username-claim="email" + ``` +2. Add the following `clientConfig` section that contains OIDC parameters to your Kubernetes YAML file and replace the placeholders with your third-party OIDC IDP details. The `clientConfig` section must be placed at the root level of the YAML file. + ```yaml + clientConfig: + oidc-issuer-url: "OIDC-ISSUER-URL" + oidc-client-id: "OIDC-CLIENT-ID" + oidc-client-secret: "OIDC-CLIENT-SECRET" + oidc-extra-scope: profile,email,openid + ``` + +After you have configured the IDP for authentication, you can proceed to create role bindings to configure authorization in your cluster. Refer to [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings) for more guidance. + +#### Configure OIDC Identity Provider for Palette Virtual Clusters + +If you are using K3s in a virtual cluster inside of a cluster group, you can also configure OIDC for your cluster. Refer to [Configure OIDC for a Virtual Cluster](../clusters/palette-virtual-clusters/configure-oidc-virtual-cluster.md) for more guidance. + +#### Add a Certificate for Reverse Proxy + +You can use a reverse proxy with a K3s Kubernetes cluster. The reverse proxy allows you to connect to the cluster API of a Palette-managed Kubernetes cluster in private networks or clusters configured with private API endpoints. For more information, refer to the [Spectro Proxy](frp.md) pack guide. + + + + + + +### Prerequisites + +- An edge device with AMD64 processor architecture or a Palette Virtual Cluster. +- A minimum of 2 CPU cores and 1 GB memory. + +### Parameters + +Since you can deploy both virtual clusters and Edge clusters using K3s, you have different configuration options depending on the cluster type. + + + + + +|**Parameter**|**Description** | +|-------------|----------------| +| `cluster.config.cluster-cidr`| Specifies the CIDR range that can be used by pods in the cluster. | +| `cluster.config.service-cidr`| Specifies the CIDR range that can be used by services in the cluster.| +| `kube-apiserver-arg`| This parameter contains extra arguments for the Kubernetes API server, such as enabling audit logging, enabling certain authorization modes, and setting profiling and secure-port.| +|`kube-controller-manager-arg` | This parameter describes extra arguments for the Kubernetes Controller Manager, such as enabling certain feature gates and setting profiling. | +| `kubelet-arg` | This parameter contains extra arguments for Kubelet during node registration, such as setting feature gates, protecting kernel defaults, and disabling the read-only port. | +| `pack.palette.config.oidc.identityProvider` | Dynamically enabled OpenID Connect (OIDC) Identity Provider (IDP) setting based on your UI selection when you add the K3s pack to your profile. This parameter appears in the YAML file after you make a selection. Refer to [Configure OIDC Identity Provider](#configure-custom-oidc). | + +You can add cloud-init stages, which allow you to customize your instances declaratively. The cloud-init stages are exposed by [Kairos](https://kairos.io/docs/architecture/cloud-init/), an open source project. For more information, check out the [Cloud Init Stages](../clusters/edge/edge-configuration/cloud-init.md) reference. + + + + +Since you are setting up a virtual cluster inside another Kubernetes cluster, you can configure its pods and services differently than the host cluster. The default configuration file you get includes parameters that offer you a higher degree of customization. These configuration parameters are exposed in the cluster group settings page. + +|**Parameter**|**Description** | +|-------------|----------------| +|`enableHA`| Determines whether the control plane is deployed in high availability mode. If you set this parameter to true, make sure to adjust the number of replicas and use an external datastore. | +|`defaultImageRegistry` | Specifies the default registry from which images are pulled. The value of this parameter will be prepended to all deployed virtual cluster images. If an image has already been deployed as part of the virtual cluster, the deployed images within the virtual cluster will not be rewritten.| +|`sync` | Specifies which Kubernetes resources are synced between the virtual and host clusters. | +|`storage` | Specifies storage settings such as persistence and PVC size. By default, storage of the virtual cluster uses the same storage class as the host cluster, but you can also optionally specify a different storage class. | +|`ingress` | Configures the ingress resource that allows you to access the virtual cluster. | + + + + + +### Usage + +K3s is available for Edge host deployments as well as virtual clusters that you can create from cluster groups. Refer to the [Create an Infrastructure Profile](../profiles/cluster-profiles/create-cluster-profiles/create-infrastructure-profile.md) guide and the [Create and Manage Cluster Groups](../clusters/cluster-groups/create-cluster-group.md) guide for more information. + +:::info +In order to use K3s as part of an Edge deployment, you need to go through the EdgeForge process and specify K3s as your intended Kubernetes distribution when you build your OS image. For more information, refer to the [EdgeForge Workflow](../clusters/edge/edgeforge-workflow/) guide. +::: + +#### Configure OIDC Identity Provider for Edge + +You can modify the configuration file to configure your Edge cluster to use an OpenID Connect (OIDC) Identity Provider (IDP) for authentication. +You can use a custom third-party IDP, such as Okta, or use Palette as your IDP. + +When you add the K3s pack to a cluster profile, Palette displays the OIDC IDP options listed below: + +- **None**: This setting does not require OIDC configuration for the cluster. It displays in the YAML file as `noauth`. + +- **Custom**: This is the default setting and does not require OIDC configuration. However, if desired, it allows you to specify a third-party OIDC provider by configuring OIDC statements in the YAML file as described in [Configure Custom OIDC](kubernetes-edge.md#configure-custom-oidc). This setting displays in the YAML file as `none`. + +- **Palette**: This setting makes Palette the IDP. Any user with a Palette account in the tenant and the proper permissions to view and access the project's resources is able to use kubectl CLI to access cluster. This setting displays in the YAML file as `palette`. When you select **Palette**, all you have to do to enable OIDC for your cluster is create role bindings to configure authorization. You do not need to provide extra parameters such as `oidc-issuer-url` as you need to when you configure a custom OIDC provider. + +- **Inherit from Tenant**: This setting allows you to apply RBAC to multiple clusters and requires you to configure OpenID Connect (OIDC) in **Tenant Settings**. In Tenant Admin scope, navigate to **Tenant Settings** > **SSO**, choose **OIDC**, and provide your third-party IDP details. This setting displays in the YAML file as `tenant`. For more information, check out the [SSO Setup](../user-management/saml-sso/enable-saml.md) guide. + +All the options require you to map a set of users or groups to a Kubernetes RBAC role. To learn how to map a Kubernetes role to users and groups, refer to [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings). + +:::warning + +If your IDP uses Security Assertion Markup Language (SAML) authentication, then the **Inherit from Tenant** option will not work, and you will need to use the **Custom** option instead. This is because Kubernetes supports only OIDC authentication and not SAML authentication. + +::: + +To configure a custom OIDC IDP, choose **Custom** when adding the K3s pack to your profile, and then follow these steps: + +1. Add the following OIDC parameters to the `kube-apiserver-arg` section of your configuration file for your Kubernetes layer when creating a cluster profile. + + ```yaml + cluster: + config: + kube-apiserver-arg: + - oidc-issuer-url="provider URL" + - oidc-client-id="client-id" + - oidc-groups-claim="groups" + - oidc-username-claim="email" + ``` +2. Add the following `clientConfig` section that contains OIDC parameters to your Kubernetes YAML file and replace the placeholders with your third-party OIDC IDP details. The `clientConfig` section must be placed at the root level of the YAML file. + + ```yaml + clientConfig: + oidc-issuer-url: "OIDC-ISSUER-URL" + oidc-client-id: "OIDC-CLIENT-ID" + oidc-client-secret: "OIDC-CLIENT-SECRET" + oidc-extra-scope: profile,email,openid + ``` + +After you have configured the IDP for authentication, you can proceed to create role bindings to configure authorization in your cluster. Refer to [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings) for more guidance. + +#### Configure OIDC Identity Provider for Palette Virtual Clusters + +If you are using K3s in a virtual cluster inside of a cluster group, you can also configure OIDC for your cluster. Refer to [Configure OIDC for a Virtual Cluster](../clusters/palette-virtual-clusters/configure-oidc-virtual-cluster.md) for more guidance. + +#### Add a Certificate for Reverse Proxy + +You can use a reverse proxy with a K3s Kubernetes cluster. The reverse proxy allows you to connect to the cluster API of a Palette-managed Kubernetes cluster in private networks or clusters configured with private API endpoints. For more information, refer to the [Spectro Proxy](frp.md) pack guide. + + + + + +### Prerequisites + +- An edge device with AMD64 processor architecture or a Palette Virtual Cluster. +- A minimum of 2 CPU cores and 1 GB memory. + +### Parameters + +Since you can deploy both virtual clusters and Edge clusters using K3s, you have different configuration options depending on the cluster type. + + + + + +|**Parameter**|**Description** | +|-------------|----------------| +| `cluster.config.cluster-cidr`| Specifies the CIDR range that can be used by pods in the cluster. | +| `cluster.config.service-cidr`| Specifies the CIDR range that can be used by services in the cluster.| +| `kube-apiserver-arg`| This parameter contains extra arguments for the Kubernetes API server, such as enabling audit logging, enabling certain authorization modes, and setting profiling and secure-port.| +|`kube-controller-manager-arg` | This parameter describes extra arguments for the Kubernetes Controller Manager, such as enabling certain feature gates and setting profiling. | +| `kubelet-arg` | This parameter contains extra arguments for Kubelet during node registration, such as setting feature gates, protecting kernel defaults, and disabling the read-only port. | +| `pack.palette.config.oidc.identityProvider` | Dynamically enabled OpenID Connect (OIDC) Identity Provider (IDP) setting based on your UI selection when you add the K3s pack to your profile. This parameter appears in the YAML file after you make a selection. Refer to [Configure OIDC Identity Provider](#configure-custom-oidc). | + +You can add cloud-init stages, which allow you to customize your instances declaratively. The cloud-init stages are exposed by [Kairos](https://kairos.io/docs/architecture/cloud-init/), an open source project. For more information, check out the [Cloud Init Stages](../clusters/edge/edge-configuration/cloud-init.md) reference. + + + + +Since you are setting up a virtual cluster inside another Kubernetes cluster, you can configure its pods and services differently than the host cluster. The default configuration file you get includes parameters that offer you a higher degree of customization. These configuration parameters are exposed in the cluster group settings page. + +|**Parameter**|**Description** | +|-------------|----------------| +|`enableHA`| Determines whether the control plane is deployed in high availability mode. If you set this parameter to true, make sure to adjust the number of replicas and use an external datastore. | +|`defaultImageRegistry` | Specifies the default registry from which images are pulled. The value of this parameter will be prepended to all deployed virtual cluster images. If an image has already been deployed as part of the virtual cluster, the deployed images within the virtual cluster will not be rewritten.| +|`sync` | Specifies which Kubernetes resources are synced between the virtual and host clusters. | +|`storage` | Specifies storage settings such as persistence and PVC size. By default, storage of the virtual cluster uses the same storage class as the host cluster, but you can also optionally specify a different storage class. | +|`ingress` | Configures the ingress resource that allows you to access the virtual cluster. | + + + + +### Usage + +K3s is available for Edge host deployments as well as virtual clusters that you can create from cluster groups. Refer to the [Create an Infrastructure Profile](../profiles/cluster-profiles/create-cluster-profiles/create-infrastructure-profile.md) guide and the [Create and Manage Cluster Groups](../clusters/cluster-groups/create-cluster-group.md) guide for more information. + +:::info +In order to use K3s as part of an Edge deployment, you need to go through the EdgeForge process and specify K3s as your intended Kubernetes distribution when you build your OS image. For more information, refer to the [EdgeForge Workflow](../clusters/edge/edgeforge-workflow/) guide. +::: + +#### Configure OIDC Identity Provider for Edge + +You can modify the configuration file to configure your Edge cluster to use an OpenID Connect (OIDC) Identity Provider (IDP) for authentication. +You can use a custom third-party IDP, such as Okta, or use Palette as your IDP. + +When you add the K3s pack to a cluster profile, Palette displays the OIDC IDP options listed below: + +- **None**: This setting does not require OIDC configuration for the cluster. It displays in the YAML file as `noauth`. + +- **Custom**: This is the default setting and does not require OIDC configuration. However, if desired, it allows you to specify a third-party OIDC provider by configuring OIDC statements in the YAML file as described in [Configure Custom OIDC](kubernetes-edge.md#configure-custom-oidc). This setting displays in the YAML file as `none`. + +- **Palette**: This setting makes Palette the IDP. Any user with a Palette account in the tenant and the proper permissions to view and access the project's resources is able to use kubectl CLI to access cluster. This setting displays in the YAML file as `palette`. Selecting **Palette** in this setting and creating role bindings to configure authorization are all you need to do to enable OIDC for your cluster. You do not need to provide extra parameters such as `oidc-issuer-url` as you need to when you configure a custom OIDC provider. + +- **Inherit from Tenant**: This setting allows you to apply RBAC to multiple clusters and requires you to configure OpenID Connect (OIDC) in **Tenant Settings**. In Tenant Admin scope, navigate to **Tenant Settings** > **SSO**, choose **OIDC**, and provide your third-party IDP details. This setting displays in the YAML file as `tenant`. For more information, check out the [SSO Setup](../user-management/saml-sso/enable-saml.md) guide. + +All the options require you to map a set of users or groups to a Kubernetes RBAC role. To learn how to map a Kubernetes role to users and groups, refer to [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings). + +:::warning + +If your IDP uses Security Assertion Markup Language (SAML) authentication, then the **Inherit from Tenant** option will not work, and you will need to use the **Custom** option instead. This is because Kubernetes supports only OIDC authentication and not SAML authentication. + +::: + +To configure a custom OIDC IDP, choose **Custom** when adding the K3s pack to your profile, and then follow these steps: + +1. Add the following OIDC parameters to the `kube-apiserver-arg` section of your configuration file for your Kubernetes layer when creating a cluster profile. + + ```yaml + cluster: + config: + kube-apiserver-arg: + - oidc-issuer-url="provider URL" + - oidc-client-id="client-id" + - oidc-groups-claim="groups" + - oidc-username-claim="email" + ``` +2. Add the following `clientConfig` section that contains OIDC parameters to your Kubernetes YAML file and replace the placeholders with your third-party OIDC IDP details. The `clientConfig` section must be placed at the root level of the YAML file. + ```yaml + clientConfig: + oidc-issuer-url: "OIDC-ISSUER-URL" + oidc-client-id: "OIDC-CLIENT-ID" + oidc-client-secret: "OIDC-CLIENT-SECRET" + oidc-extra-scope: profile,email,openid + ``` + +After you have configured the IDP for authentication, you can proceed to create role bindings to configure authorization in your cluster. Refer to [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings) for more guidance. + +#### Configure OIDC Identity Provider for Palette Virtual Clusters + +If you are using K3s in a virtual cluster inside of a cluster group, you can also configure OIDC for your cluster. Refer to [Configure OIDC for a Virtual Cluster](../clusters/palette-virtual-clusters/configure-oidc-virtual-cluster.md) for more guidance. + +#### Add a Certificate for Reverse Proxy + +You can use a reverse proxy with a K3s Kubernetes cluster. The reverse proxy allows you to connect to the cluster API of a Palette-managed Kubernetes cluster in private networks or clusters configured with private API endpoints. For more information, refer to the [Spectro Proxy](frp.md) pack guide. + + + + + +### Prerequisites + +- An edge device with AMD64 processor architecture or a virtual cluster. +- A minimum of 2 CPU cores and 1 GB memory. + +### Parameters + +Since you can deploy both virtual clusters and Edge clusters using K3s, you have different configuration options depending on the cluster type. + + + + + +|**Parameter**|**Description** | +|-------------|----------------| +| `cluster.config.cluster-cidr`| Specifies the CIDR range that can be used by pods in the cluster. | +| `cluster.config.service-cidr`| Specifies the CIDR range that can be used by services in the cluster.| +| `kube-apiserver-arg`| This parameter contains extra arguments for the Kubernetes API server, such as enabling audit logging, enabling certain authorization modes, and setting profiling and secure-port.| +|`kube-controller-manager-arg` | This parameter describes extra arguments for the Kubernetes Controller Manager, such as enabling certain feature gates and setting profiling. | +| `kubelet-arg` | This parameter contains extra arguments for Kubelet during node registration, such as setting feature gates, protecting kernel defaults, and disabling the read-only port. | +| `pack.palette.config.oidc.identityProvider` | Dynamically enabled OpenID Connect (OIDC) Identity Provider (IDP) setting based on your UI selection when you add the K3s pack to your profile. This parameter appears in the YAML file after you make a selection. Refer to [Configure OIDC Identity Provider](#configure-custom-oidc). | + +You can add cloud-init stages, which allow you to customize your instances declaratively. The cloud-init stages are exposed by [Kairos](https://kairos.io/docs/architecture/cloud-init/), an open source project. For more information, check out the [Cloud Init Stages](../clusters/edge/edge-configuration/cloud-init.md) reference. + + + + +Since you are setting up a virtual cluster inside another Kubernetes cluster, you can configure its pods and services differently than the host cluster. The default configuration file you get includes parameters that offer you a higher degree of customization. These configuration parameters are exposed in the cluster group settings page. + +|**Parameter**|**Description** | +|-------------|----------------| +|`enableHA`| Determines whether the control plane is deployed in high availability mode. If you set this parameter to true, make sure to adjust the number of replicas and use an external datastore. | +|`defaultImageRegistry` | Specifies the default registry from which images are pulled. The value of this parameter will be prepended to all deployed virtual cluster images. If an image has already been deployed as part of the virtual cluster, the deployed images within the virtual cluster will not be rewritten.| +|`sync` | Specifies which Kubernetes resources are synced between the virtual and host clusters. | +|`storage` | Specifies storage settings such as persistence and PVC size. By default, storage of the virtual cluster uses the same storage class as the host cluster, but you can also optionally specify a different storage class. | +|`ingress` | Configures the ingress resource that allows you to access the virtual cluster. | + + + + +### Usage + +K3s is available for Edge host deployments as well as virtual clusters that you can create from cluster groups. Refer to the [Create an Infrastructure Profile](../profiles/cluster-profiles/create-cluster-profiles/create-infrastructure-profile.md) guide and the [Create and Manage Cluster Groups](../clusters/cluster-groups/create-cluster-group.md) guide for more information. + +:::info +In order to use K3s as part of an Edge deployment, you need to go through the EdgeForge process and specify K3s as your intended Kubernetes distribution when you build your OS image. For more information, refer to the [EdgeForge Workflow](../clusters/edge/edgeforge-workflow/) guide. +::: + +#### Configure OIDC Identity Provider for Edge + +You can modify the configuration file to configure your Edge cluster to use an OpenID Connect (OIDC) Identity Provider (IDP) for authentication. +You can use a custom third-party IDP, such as Okta, or use Palette as your IDP. + +When you add the K3s pack to a cluster profile, Palette displays the OIDC IDP options listed below: + +- **None**: This setting does not require OIDC configuration for the cluster. It displays in the YAML file as `noauth`. + +- **Custom**: This is the default setting and does not require OIDC configuration. However, if desired, it allows you to specify a third-party OIDC provider by configuring OIDC statements in the YAML file as described in [Configure Custom OIDC](kubernetes-edge.md#configure-custom-oidc). This setting displays in the YAML file as `none`. + +- **Palette**: This setting makes Palette the IDP. Any user with a Palette account in the tenant and the proper permissions to view and access the project's resources is able to use kubectl CLI to access cluster. This setting displays in the YAML file as `palette`. Selecting **Palette** in this setting and creating role bindings to configure authorization are all you need to do to enable OIDC for your cluster. You do not need to provide extra parameters such as `oidc-issuer-url` as you need to when you configure a custom OIDC provider. + +- **Inherit from Tenant**: This setting allows you to apply RBAC to multiple clusters and requires you to configure OpenID Connect (OIDC) in **Tenant Settings**. In Tenant Admin scope, navigate to **Tenant Settings** > **SSO**, choose **OIDC**, and provide your third-party IDP details. This setting displays in the YAML file as `tenant`. For more information, check out the [SSO Setup](../user-management/saml-sso/enable-saml.md) guide. + +All the options require you to map a set of users or groups to a Kubernetes RBAC role. To learn how to map a Kubernetes role to users and groups, refer to [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings). + +:::warning + +If your IDP uses Security Assertion Markup Language (SAML) authentication, then the **Inherit from Tenant** option will not work, and you will need to use the **Custom** option instead. This is because Kubernetes supports only OIDC authentication and not SAML authentication. + +::: + +To configure a custom OIDC IDP, choose **Custom** when adding the K3s pack to your profile, and then follow these steps: + +1. Add the following OIDC parameters to the `kube-apiserver-arg` section of your configuration file for your Kubernetes layer when creating a cluster profile. + + ```yaml + cluster: + config: + kube-apiserver-arg: + - oidc-issuer-url="provider URL" + - oidc-client-id="client-id" + - oidc-groups-claim="groups" + - oidc-username-claim="email" + ``` +2. Add the following `clientConfig` section that contains OIDC parameters to your Kubernetes YAML file and replace the placeholders with your third-party OIDC IDP details. The `clientConfig` section must be placed at the root level of the YAML file. + ```yaml + clientConfig: + oidc-issuer-url: "OIDC-ISSUER-URL" + oidc-client-id: "OIDC-CLIENT-ID" + oidc-client-secret: "OIDC-CLIENT-SECRET" + oidc-extra-scope: profile,email,openid + ``` + +After you have configured the IDP for authentication, you can proceed to create role bindings to configure authorization in your cluster. Refer to [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings) for more guidance. + +#### Configure OIDC Identity Provider for Palette Virtual Clusters + +If you are using K3s in a virtual clusters inside of a cluster group, you can also configure OIDC for your cluster. Refer to [Configure OIDC for a Virtual Cluster](../clusters/palette-virtual-clusters/configure-oidc-virtual-cluster.md) for more guidance. + +#### Add a Certificate for Reverse Proxy + +You can use a reverse proxy with a K3s Kubernetes cluster. The reverse proxy allows you to connect to the cluster API of a Palette-managed Kubernetes cluster in private networks or clusters configured with private API endpoints. For more information, refer to the [Spectro Proxy](frp.md) pack guide. + + + + + +:::warning +All versions less than v1.25.x are considered deprecated. Upgrade to a newer version to take advantage of new features. +::: + + + + + +### Terraform + +You can retrieve details about the K3s pack by using the following Terraform code. + +```hcl +data "spectrocloud_registry" "public_registry" { + name = "Public Repo" +} + +data "spectrocloud_pack_simple" "k8s" { + name = "edge-k3s" + version = "1.27.5" + type = "helm" + registry_uid = data.spectrocloud_registry.public_registry.id +} +``` + + + + diff --git a/docs/docs-content/integrations/kubernetes-edge.md b/docs/docs-content/integrations/kubernetes-edge.md new file mode 100644 index 0000000000..82514ca8c2 --- /dev/null +++ b/docs/docs-content/integrations/kubernetes-edge.md @@ -0,0 +1,1580 @@ +--- +sidebar_label: "Palette eXtended Kubernetes - Edge" +title: "Palette eXtended Kubernetes - Edge" +description: "Learn about the Palette eXtended Kubernetes - Edge pack and how you can use it your host clusters in an edge environment." +hide_table_of_contents: true +type: "integration" +category: ["kubernetes", 'amd64', 'fips'] +sidebar_class_name: "hide-from-sidebar" +logoUrl: "https://registry.spectrocloud.com/v1/k8s-dashboard/blobs/sha256:2de5d88b2573af42d4cc269dff75744c4174ce47cbbeed5445e51a2edd8b7429?type=image/png" +tags: ["packs", "kubernetes", "pxke", "edge"] +--- + +The Palette eXtended Kubernetes - Edge (PXK-E) pack supports Kubernetes clusters set up on Edge hosts installed in isolated locations like grocery stores, restaurants, and similar locations, versus a data center or cloud environment. We offer PXK-E as a core pack in Palette. + +:::info +Review our [Maintenance Policy](maintenance-policy.md) to learn about pack update and deprecation schedules. +::: + +## What is PXK-E? + +PXK-E is a customized version of the open-source Cloud Native Computing Foundation (CNCF) distribution of Kubernetes. This Kubernetes distribution is customized and optimized for edge computing environments and can be deployed through Palette. PXK-E is the Kubernetes distribution Palette defaults to when deploying Edge clusters. + +PXK-E differs from the upstream open-source Kubernetes version by optimizing for operations in an edge computing environment. PXK-E also differentiates itself by using the Kairos open-source project as the base operating system (OS). PXK-E’s use of Kairos means the OS is immutable, which significantly improves the security posture and reduces potential attack surfaces. + +Another differentiator of PXK-E is the carefully reviewed and applied hardening of the OS and Kubernetes. The hardening ranges from removing unused OS kernel modules to using an OS configuration that follows industry best practices. Our custom Kubernetes configuration addresses common deployment security pitfalls and implements industry best practices. + +With PXK-E, you can manage automatic OS upgrades while retaining immutability and the flexibility to roll out changes safely. The A/B partition architecture of Kairos allows for new OS and dependency versions to be installed in a separate partition and mounted at runtime. You can fall back to use the previous partition if issues are identified in the new partition. + +PXK-E manages the underlying OS and the Kubernetes layer together, which reduces the challenge of upgrading and maintaining two separate components. + +PXK-E allows you to apply different flavors of container storage interfaces (CSI) and container network interfaces (CNI). Other open-source Kubernetes distributions such as MicroK8s, RKE2, and K3s come with a default CSI and CNI. There is additional complexity and overhead when you want to consume different interface plugins with traditional Kubernetes distributions. Using PXK-E, you select the interface plugin you want to apply without additional overhead and complexity. + +There are no changes to the Kubernetes source code used in PXK-E, and it follows the same versioning schema as the upstream open-source Kubernetes distribution. + + +:::info + +We also offer Palette eXtended Kubernetes (PXK) for cloud and data center deployments. For more information, refer to the [Palette eXtended Kubernetes](kubernetes.md) guide to learn more about PXK. + +::: + +### PXK and Palette VerteX + +The PXK-E used in [Palette VerteX](../vertex/vertex.md) is compiled and linked with our [NIST-certified FIPS crypto module](../compliance.md#fips-140-2). PXK-E is by default enabled with [Ubuntu Pro](https://ubuntu.com/pro) with FIPS mode enabled. Additionally, the Operating System (OS) is hardened based on the NIST-800 standard. Refer to the [Build Edge Artifacts](../clusters/edge/edgeforge-workflow/build-artifacts.md) guide to learn more on how to build the PXK-E image with FIPS mode enabled. + +The combined usage of PXK-E and Palette VerteX provides a secure and FIPS-compliant experience as the Kubernetes distribution, OS, and management platform VerteX is FIPS-compliant. + + +### Support Lifecycle + +We support PXK-E for N-3 Kubernetes minor versions for a duration of 14 months. The duration exceeds the official EOL by four months. Once we stop supporting the minor version, we initiate the deprecation process. Refer to the [Kubernetes Support Lifecycle](kubernetes-support.md#palette-extended-kubernetes-support) guide to learn more. + +# Versions Supported + + + + + +## Prerequisites + +- A minimum of 2 CPU and 4GB Memory. + + +## Parameters + +| Parameter | Description | +|----------------------------------------------------------|----------------------------------------------------| +| `cluster.config.clusterConfiguration.apiServer.extraArgs` | This parameter contains extra arguments for the Kubernetes API server, such as enabling audit logging, enabling certain authorization modes, and setting profiling and secure-port.| +| `cluster.config.clusterConfiguration.apiServer.extraVolumes` | This parameter describes extra volumes for the Kubernetes API server, such as `audit-log` and `audit-policy`. | +| `cluster.config.clusterConfiguration.controllerManager.extraArgs` | This parameter describes extra arguments for the Kubernetes Controller Manager, such as enabling certain feature gates and setting profiling. | +| `cluster.config.clusterConfiguration.etcd.local.dataDir` | This parameter specifies the data directory for etcd, the distributed key-value store that Kubernetes uses to persist cluster state. | +| `cluster.config.clusterConfiguration.networking.podSubnet` | The IP subnet range to assign to pods. Default: 192.168.0.0/16 | +| `cluster.config.clusterConfiguration.networking.serviceSubnet` | The IP subnet range to assign to services. Default: 192.169.0.0/16 | +| `cluster.config.clusterConfiguration.scheduler.extraArgs` | This parameter contains extra arguments for the Kubernetes scheduler, such as disabling profiling. | +| `cluster.config.initConfiguration.nodeRegistration.kubeletExtraArgs` | This parameter contains extra arguments for kubelet during node registration, such as setting feature gates, protecting kernel defaults, and disabling the read-only port. | +| `pack.palette.config.oidc.identityProvider` | Dynamically enabled OpenID Connect (OIDC) Identity Provider (IDP) setting based on your UI selection when you add the PXK-E pack to your profile. This parameter appears in the YAML file after you make a selection. Refer to [Configure OIDC Identity Provider](#configure-custom-oidc). | + +You can add cloud-init stages exposed by [Kairos](https://kairos.io/docs/architecture/cloud-init/), an open-source project. For more information, check out the [Cloud Init Stages](../clusters/edge/edge-configuration/cloud-init.md) reference. + +You can also use pack settings described in the [Palette eXtended Kubernetes](kubernetes.md) guide. + + +## Usage + +The Kubernetes configuration file is where you can do the following: + +- Manually configure a third-party OIDC IDP. For more information, check out [Configure Custom OIDC](kubernetes-edge.md#configure-custom-oidc). + +- Add a certificate for the Spectro Proxy pack if you want to use a reverse proxy with a Kubernetes cluster. For more information, refer to the [Spectro Proxy](frp.md) guide. + + +#### Configuration Changes + +The PXK-E Kubeadm configuration is updated to dynamically enable OIDC based on your IDP selection by adding the ``identityProvider`` parameter. + +```yaml +pack: + palette: + config: + oidc: + identityProvider: +``` + + +#### Example Kubernetes Configuration File + +```yaml +cluster: + config: | + clusterConfiguration: + apiServer: + extraArgs: + advertise-address: "0.0.0.0" + anonymous-auth: "true" + audit-log-maxage: "30" + audit-log-maxbackup: "10" + audit-log-maxsize: "100" + audit-log-path: /var/log/apiserver/audit.log + audit-policy-file: /etc/kubernetes/audit-policy.yaml + authorization-mode: RBAC,Node + default-not-ready-toleration-seconds: "60" + default-unreachable-toleration-seconds: "60" + disable-admission-plugins: AlwaysAdmit + enable-admission-plugins: AlwaysPullImages,NamespaceLifecycle,ServiceAccount,NodeRestriction + profiling: "false" + secure-port: "6443" + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + extraVolumes: + - hostPath: /var/log/apiserver + mountPath: /var/log/apiserver + name: audit-log + pathType: DirectoryOrCreate + - hostPath: /etc/kubernetes/audit-policy.yaml + mountPath: /etc/kubernetes/audit-policy.yaml + name: audit-policy + pathType: File + readOnly: true + timeoutForControlPlane: 10m0s + controllerManager: + extraArgs: + feature-gates: RotateKubeletServerCertificate=true + pod-eviction-timeout: 1m0s + profiling: "false" + terminated-pod-gc-threshold: "25" + use-service-account-credentials: "true" + dns: {} + kubernetesVersion: v1.26.4 + etcd: + local: + dataDir: "/etc/kubernetes/etcd" + extraArgs: + listen-client-urls: "https://0.0.0.0:2379" + networking: + podSubnet: 192.168.0.0/16 + serviceSubnet: 192.169.0.0/16 + scheduler: + extraArgs: + profiling: "false" + initConfiguration: + localAPIEndpoint: {} + nodeRegistration: + kubeletExtraArgs: + event-qps: "0" + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: "true" + read-only-port: "0" + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + joinConfiguration: + discovery: {} + nodeRegistration: + kubeletExtraArgs: + event-qps: "0" + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: "true" + read-only-port: "0" + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +stages: + initramfs: + - sysctl: + vm.overcommit_memory: 1 + kernel.panic: 10 + kernel.panic_on_oops: 1 + commands: + - ln -s /etc/kubernetes/admin.conf /run/kubeconfig + files: + - path: /etc/hosts + permission: "0644" + content: | + 127.0.0.1 localhost + - path: "/etc/kubernetes/audit-policy.yaml" + owner_string: "root" + permission: 0600 + content: | + apiVersion: audit.k8s.io/v1 + kind: Policy + rules: + - level: None + users: ["system:kube-proxy"] + verbs: ["watch"] + resources: + - group: "" # core + resources: ["endpoints", "services", "services/status"] + - level: None + users: ["system:unsecured"] + namespaces: ["kube-system"] + verbs: ["get"] + resources: + - group: "" # core + resources: ["configmaps"] + - level: None + users: ["kubelet"] # legacy kubelet identity + verbs: ["get"] + resources: + - group: "" # core + resources: ["nodes", "nodes/status"] + - level: None + userGroups: ["system:nodes"] + verbs: ["get"] + resources: + - group: "" # core + resources: ["nodes", "nodes/status"] + - level: None + users: + - system:kube-controller-manager + - system:kube-scheduler + - system:serviceaccount:kube-system:endpoint-controller + verbs: ["get", "update"] + namespaces: ["kube-system"] + resources: + - group: "" # core + resources: ["endpoints"] + - level: None + users: ["system:apiserver"] + verbs: ["get"] + resources: + - group: "" # core + resources: ["namespaces", "namespaces/status", "namespaces/finalize"] + - level: None + users: ["cluster-autoscaler"] + verbs: ["get", "update"] + namespaces: ["kube-system"] + resources: + - group: "" # core + resources: ["configmaps", "endpoints"] + # Don't log HPA fetching metrics. + - level: None + users: + - system:kube-controller-manager + verbs: ["get", "list"] + resources: + - group: "metrics.k8s.io" + # Don't log these read-only URLs. + - level: None + nonResourceURLs: + - /healthz* + - /version + - /swagger* + # Don't log events requests. + - level: None + resources: + - group: "" # core + resources: ["events"] + # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes + - level: Request + users: ["kubelet", "system:node-problem-detector", "system:serviceaccount:kube-system:node-problem-detector"] + verbs: ["update","patch"] + resources: + - group: "" # core + resources: ["nodes/status", "pods/status"] + omitStages: + - "RequestReceived" + - level: Request + userGroups: ["system:nodes"] + verbs: ["update","patch"] + resources: + - group: "" # core + resources: ["nodes/status", "pods/status"] + omitStages: + - "RequestReceived" + # deletecollection calls can be large, don't log responses for expected namespace deletions + - level: Request + users: ["system:serviceaccount:kube-system:namespace-controller"] + verbs: ["deletecollection"] + omitStages: + - "RequestReceived" + # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data, + # so only log at the Metadata level. + - level: Metadata + resources: + - group: "" # core + resources: ["secrets", "configmaps"] + - group: authentication.k8s.io + resources: ["tokenreviews"] + omitStages: + - "RequestReceived" + # Get repsonses can be large; skip them. + - level: Request + verbs: ["get", "list", "watch"] + resources: + - group: "" # core + - group: "admissionregistration.k8s.io" + - group: "apiextensions.k8s.io" + - group: "apiregistration.k8s.io" + - group: "apps" + - group: "authentication.k8s.io" + - group: "authorization.k8s.io" + - group: "autoscaling" + - group: "batch" + - group: "certificates.k8s.io" + - group: "extensions" + - group: "metrics.k8s.io" + - group: "networking.k8s.io" + - group: "policy" + - group: "rbac.authorization.k8s.io" + - group: "settings.k8s.io" + - group: "storage.k8s.io" + omitStages: + - "RequestReceived" + # Default level for known APIs + - level: RequestResponse + resources: + - group: "" # core + - group: "admissionregistration.k8s.io" + - group: "apiextensions.k8s.io" + - group: "apiregistration.k8s.io" + - group: "apps" + - group: "authentication.k8s.io" + - group: "authorization.k8s.io" + - group: "autoscaling" + - group: "batch" + - group: "certificates.k8s.io" + - group: "extensions" + - group: "metrics.k8s.io" + - group: "networking.k8s.io" + - group: "policy" + - group: "rbac.authorization.k8s.io" + - group: "settings.k8s.io" + - group: "storage.k8s.io" + omitStages: + - "RequestReceived" + # Default level for all other requests. + - level: Metadata + omitStages: + - "RequestReceived" +pack: + palette: + config: + oidc: + identityProvider: palette +``` + +### Configure OIDC Identity Provider + +The OIDC IDP feature offers the convenience of managing OIDC at the Kubernetes layer. The OIDC IDP feature is particularly useful for environments that do not have their own IDP configured. In this scenario, you can leverage Palette as an IDP without having to configure a third-party IDP. We also support the ability to take advantage of other OIDC providers by making it possible for you to configure OIDC at the tenant level. For additional flexibility, if you wish to use a different IDP than the one configured at the tenant level, you can select a different IDP by adding the OIDC configuration to your cluster profile. + +When you add the PXK-E pack to a cluster profile, Palette displays the OIDC IDP options listed below. + +All the options require you to map a set of users or groups to a Kubernetes RBAC role. To learn how to map a Kubernetes role to users and groups, refer to [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings). + +You can create a role binding that maps individual users or groups assigned within the OIDC provider's configuration to a role. To learn more, review [Use RBAC with OIDC](kubernetes-edge.md#use-rbac-with-oidc). You can also configure OIDC for virtual clusters. For guidance, refer to [Configure OIDC for a Virtual Cluster](../clusters/palette-virtual-clusters/configure-oidc-virtual-cluster.md). + + +- **None**: This setting does not require OIDC configuration for the cluster. It displays in the YAML file as `noauth`. + +- **Custom**: This is the default setting and does not require OIDC configuration. However, if desired, it allows you to specify a third-party OIDC provider by configuring OIDC statements in the YAML file as described in [Configure Custom OIDC](kubernetes-edge.md#configure-custom-oidc). This setting displays in the YAML file as `none`. + +- **Palette**: This setting makes Palette the IDP. Any user with a Palette account in the tenant and the proper permissions to view and access the project's resources is able to log into the Kubernetes dashboard. This setting displays in the YAML file as `palette`. + +- **Inherit from Tenant**: This setting allows you to apply RBAC to multiple clusters and requires you to configure OpenID Connect (OIDC) in **Tenant Settings**. In Tenant Admin scope, navigate to **Tenant Settings** > **SSO**, choose **OIDC**, and provide your third-party IDP details. This setting displays in the YAML file as `tenant`. For more information, check out the [SSO Setup](../user-management/saml-sso/enable-saml.md) guide. + +:::info + +If your IDP uses Security Assertion Markup Language (SAML) authentication, then the **Inherit from Tenant** option will not work, and you will need to use the **Custom** option instead. This is because Kubernetes supports only OIDC authentication and not SAML authentication. + +::: + + +### Configure Custom OIDC + +Follow these steps to configure a third-party OIDC IDP. + +1. Add the following OIDC parameters to the `apiServer.extraArgs` section of your Kubernetes YAML file when creating a cluster profile. + + +```yaml +cluster: + config: | + clusterConfiguration: + apiServer: + extraArgs: + oidc-issuer-url: "provider URL" + oidc-client-id: "client-id" + oidc-groups-claim: "groups" + oidc-username-claim: "email" +``` + +2. Add the following `clientConfig` section that contains OIDC parameters to your Kubernetes configuration file. + + +```yaml +clientConfig: + oidc-issuer-url: "" + oidc-client-id: "" + oidc-client-secret: "" + oidc-extra-scope: profile,email,openid +``` + +3. Provide third-party OIDC IDP details. + +### Use RBAC with OIDC + +You can create a role binding that uses individual users as the subject or specify a group name as the subject to map many users to a role. The group name is the group assigned in the OIDC provider's configuration. Below is an example. To learn more, review [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings). + +Assume you created a group named `dev-east-2` within an OIDC provider. If you configure the host cluster's Kubernetes pack with all the correct OIDC settings, you could then create a role binding for the `dev-east-2` group. + +In this example, Palette is used as the IDP, and all users in the `dev-east-2` would inherit the `cluster-admin` role. + +![A subject of the type group is assigned as the subject in a RoleBinding](/clusters_cluster-management_cluster-rbac_cluster-subject-group.png) + + + + + + + +## Prerequisites + +- A minimum of 2 CPU and 4GB Memory. + + +## Parameters + +| Parameter | Description | +|----------------------------------------------------------|----------------------------------------------------| +| `cluster.config.clusterConfiguration.apiServer.extraArgs` | This parameter contains extra arguments for the Kubernetes API server, such as enabling audit logging, enabling certain authorization modes, and setting profiling and secure-port.| +| `cluster.config.clusterConfiguration.apiServer.extraVolumes` | This parameter describes extra volumes for the Kubernetes API server, such as `audit-log` and `audit-policy`. | +| `cluster.config.clusterConfiguration.controllerManager.extraArgs` | This parameter describes extra arguments for the Kubernetes Controller Manager, such as enabling certain feature gates and setting profiling. | +| `cluster.config.clusterConfiguration.etcd.local.dataDir` | This parameter specifies the data directory for etcd, the distributed key-value store that Kubernetes uses to persist cluster state. | +| `cluster.config.clusterConfiguration.networking.podSubnet` | The IP subnet range to assign to pods. Default: 192.168.0.0/16 | +| `cluster.config.clusterConfiguration.networking.serviceSubnet` | The IP subnet range to assign to services. Default: 192.169.0.0/16 | +| `cluster.config.clusterConfiguration.scheduler.extraArgs` | This parameter contains extra arguments for the Kubernetes scheduler, such as disabling profiling. | +| `cluster.config.initConfiguration.nodeRegistration.kubeletExtraArgs` | This parameter contains extra arguments for kubelet during node registration, such as setting feature gates, protecting kernel defaults, and disabling the read-only port. | +| `pack.palette.config.oidc.identityProvider` | Dynamically enabled OpenID Connect (OIDC) Identity Provider (IDP) setting based on your UI selection when you add the PXK-E pack to your profile. This parameter appears in the YAML file after you make a selection. Refer to [Configure OIDC Identity Provider](#configure-custom-oidc). | + +You can add cloud-init stages exposed by [Kairos](https://kairos.io/docs/architecture/cloud-init/), an open-source project. For more information, check out the [Cloud Init Stages](../clusters/edge/edge-configuration/cloud-init.md) reference. + +You can also use pack settings described in the [Palette eXtended Kubernetes](kubernetes.md) guide. + + +## Usage + +The Kubeadm configuration file is where you can do the following: + +- Manually configure a third-party OIDC IDP. For more information, check out [Configure Custom OIDC](kubernetes-edge.md#configure-custom-oidc). + +- Add a certificate for the Spectro Proxy pack if you want to use a reverse proxy with a Kubernetes cluster. For more information, refer to the [Spectro Proxy](frp.md) guide. + + +#### Configuration Changes + +The PXK-E Kubeadm configuration is updated to dynamically enable OIDC based on your IDP selection by adding the ``identityProvider`` parameter. + +```yaml +pack: + palette: + config: + oidc: + identityProvider: +``` + + +#### Example Kubeadm Configuration File + +```yaml +cluster: + config: | + clusterConfiguration: + apiServer: + extraArgs: + advertise-address: "0.0.0.0" + anonymous-auth: "true" + audit-log-maxage: "30" + audit-log-maxbackup: "10" + audit-log-maxsize: "100" + audit-log-path: /var/log/apiserver/audit.log + audit-policy-file: /etc/kubernetes/audit-policy.yaml + authorization-mode: RBAC,Node + default-not-ready-toleration-seconds: "60" + default-unreachable-toleration-seconds: "60" + disable-admission-plugins: AlwaysAdmit + enable-admission-plugins: AlwaysPullImages,NamespaceLifecycle,ServiceAccount,NodeRestriction + profiling: "false" + secure-port: "6443" + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + extraVolumes: + - hostPath: /var/log/apiserver + mountPath: /var/log/apiserver + name: audit-log + pathType: DirectoryOrCreate + - hostPath: /etc/kubernetes/audit-policy.yaml + mountPath: /etc/kubernetes/audit-policy.yaml + name: audit-policy + pathType: File + readOnly: true + timeoutForControlPlane: 10m0s + controllerManager: + extraArgs: + feature-gates: RotateKubeletServerCertificate=true + pod-eviction-timeout: 1m0s + profiling: "false" + terminated-pod-gc-threshold: "25" + use-service-account-credentials: "true" + dns: {} + kubernetesVersion: v1.26.4 + etcd: + local: + dataDir: "/etc/kubernetes/etcd" + extraArgs: + listen-client-urls: "https://0.0.0.0:2379" + networking: + podSubnet: 192.168.0.0/16 + serviceSubnet: 192.169.0.0/16 + scheduler: + extraArgs: + profiling: "false" + initConfiguration: + localAPIEndpoint: {} + nodeRegistration: + kubeletExtraArgs: + event-qps: "0" + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: "true" + read-only-port: "0" + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + joinConfiguration: + discovery: {} + nodeRegistration: + kubeletExtraArgs: + event-qps: "0" + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: "true" + read-only-port: "0" + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +stages: + initramfs: + - sysctl: + vm.overcommit_memory: 1 + kernel.panic: 10 + kernel.panic_on_oops: 1 + commands: + - ln -s /etc/kubernetes/admin.conf /run/kubeconfig + files: + - path: /etc/hosts + permission: "0644" + content: | + 127.0.0.1 localhost + - path: "/etc/kubernetes/audit-policy.yaml" + owner_string: "root" + permission: 0600 + content: | + apiVersion: audit.k8s.io/v1 + kind: Policy + rules: + - level: None + users: ["system:kube-proxy"] + verbs: ["watch"] + resources: + - group: "" # core + resources: ["endpoints", "services", "services/status"] + - level: None + users: ["system:unsecured"] + namespaces: ["kube-system"] + verbs: ["get"] + resources: + - group: "" # core + resources: ["configmaps"] + - level: None + users: ["kubelet"] # legacy kubelet identity + verbs: ["get"] + resources: + - group: "" # core + resources: ["nodes", "nodes/status"] + - level: None + userGroups: ["system:nodes"] + verbs: ["get"] + resources: + - group: "" # core + resources: ["nodes", "nodes/status"] + - level: None + users: + - system:kube-controller-manager + - system:kube-scheduler + - system:serviceaccount:kube-system:endpoint-controller + verbs: ["get", "update"] + namespaces: ["kube-system"] + resources: + - group: "" # core + resources: ["endpoints"] + - level: None + users: ["system:apiserver"] + verbs: ["get"] + resources: + - group: "" # core + resources: ["namespaces", "namespaces/status", "namespaces/finalize"] + - level: None + users: ["cluster-autoscaler"] + verbs: ["get", "update"] + namespaces: ["kube-system"] + resources: + - group: "" # core + resources: ["configmaps", "endpoints"] + # Don't log HPA fetching metrics. + - level: None + users: + - system:kube-controller-manager + verbs: ["get", "list"] + resources: + - group: "metrics.k8s.io" + # Don't log these read-only URLs. + - level: None + nonResourceURLs: + - /healthz* + - /version + - /swagger* + # Don't log events requests. + - level: None + resources: + - group: "" # core + resources: ["events"] + # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes + - level: Request + users: ["kubelet", "system:node-problem-detector", "system:serviceaccount:kube-system:node-problem-detector"] + verbs: ["update","patch"] + resources: + - group: "" # core + resources: ["nodes/status", "pods/status"] + omitStages: + - "RequestReceived" + - level: Request + userGroups: ["system:nodes"] + verbs: ["update","patch"] + resources: + - group: "" # core + resources: ["nodes/status", "pods/status"] + omitStages: + - "RequestReceived" + # deletecollection calls can be large, don't log responses for expected namespace deletions + - level: Request + users: ["system:serviceaccount:kube-system:namespace-controller"] + verbs: ["deletecollection"] + omitStages: + - "RequestReceived" + # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data, + # so only log at the Metadata level. + - level: Metadata + resources: + - group: "" # core + resources: ["secrets", "configmaps"] + - group: authentication.k8s.io + resources: ["tokenreviews"] + omitStages: + - "RequestReceived" + # Get repsonses can be large; skip them. + - level: Request + verbs: ["get", "list", "watch"] + resources: + - group: "" # core + - group: "admissionregistration.k8s.io" + - group: "apiextensions.k8s.io" + - group: "apiregistration.k8s.io" + - group: "apps" + - group: "authentication.k8s.io" + - group: "authorization.k8s.io" + - group: "autoscaling" + - group: "batch" + - group: "certificates.k8s.io" + - group: "extensions" + - group: "metrics.k8s.io" + - group: "networking.k8s.io" + - group: "policy" + - group: "rbac.authorization.k8s.io" + - group: "settings.k8s.io" + - group: "storage.k8s.io" + omitStages: + - "RequestReceived" + # Default level for known APIs + - level: RequestResponse + resources: + - group: "" # core + - group: "admissionregistration.k8s.io" + - group: "apiextensions.k8s.io" + - group: "apiregistration.k8s.io" + - group: "apps" + - group: "authentication.k8s.io" + - group: "authorization.k8s.io" + - group: "autoscaling" + - group: "batch" + - group: "certificates.k8s.io" + - group: "extensions" + - group: "metrics.k8s.io" + - group: "networking.k8s.io" + - group: "policy" + - group: "rbac.authorization.k8s.io" + - group: "settings.k8s.io" + - group: "storage.k8s.io" + omitStages: + - "RequestReceived" + # Default level for all other requests. + - level: Metadata + omitStages: + - "RequestReceived" +pack: + palette: + config: + oidc: + identityProvider: palette +``` + +### Configure OIDC Identity Provider + +The OIDC IDP feature offers the convenience of managing OIDC at the Kubernetes layer. The OIDC IDP feature is particularly useful for environments that do not have their own IDP configured. In this scenario, you can leverage Palette as an IDP without having to configure a third-party IDP. We also support the ability to take advantage of other OIDC providers by making it possible for you to configure OIDC at the tenant level. For additional flexibility, if you wish to use a different IDP than the one configured at the tenant level, you can select a different IDP by adding the OIDC configuration to your cluster profile. + +When you add the PXK-E pack to a cluster profile, Palette displays the OIDC IDP options listed below. + +All the options require you to map a set of users or groups to a Kubernetes RBAC role. To learn how to map a Kubernetes role to users and groups, refer to [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings). + +You can create a role binding that maps individual users or groups assigned within the OIDC provider's configuration to a role. To learn more, review [Use RBAC with OIDC](kubernetes-edge.md#use-rbac-with-oidc). You can also configure OIDC for virtual clusters. For guidance, refer to [Configure OIDC for a Virtual Cluster](../clusters/palette-virtual-clusters/configure-oidc-virtual-cluster.md). + + +- **None**: This setting does not require OIDC configuration for the cluster. It displays in the YAML file as `noauth`. + +- **Custom**: This is the default setting and does not require OIDC configuration. However, if desired, it allows you to specify a third-party OIDC provider by configuring OIDC statements in the YAML file as described in [Configure Custom OIDC](kubernetes-edge.md#configure-custom-oidc). This setting displays in the YAML file as `none`. + +- **Palette**: This setting makes Palette the IDP. Any user with a Palette account in the tenant and the proper permissions to view and access the project's resources is able to log into the Kubernetes dashboard. This setting displays in the YAML file as `palette`. + +- **Inherit from Tenant**: This setting allows you to apply RBAC to multiple clusters and requires you to configure OpenID Connect (OIDC) in **Tenant Settings**. In Tenant Admin scope, navigate to **Tenant Settings** > **SSO**, choose **OIDC**, and provide your third-party IDP details. This setting displays in the YAML file as `tenant`. For more information, check out the [SSO Setup](../user-management/saml-sso/enable-saml.md) guide. + +:::info + +If your IDP uses Security Assertion Markup Language (SAML) authentication, then the **Inherit from Tenant** option will not work, and you will need to use the **Custom** option instead. This is because Kubernetes supports only OIDC authentication and not SAML authentication. + +::: + + +### Configure Custom OIDC + +Follow these steps to configure a third-party OIDC IDP. + +1. Add the following OIDC parameters to the `apiServer.extraArgs` section of your Kubernetes YAML file when creating a cluster profile. + + +```yaml +cluster: + config: + clusterConfiguration: + apiServer: + extraArgs: + oidc-issuer-url: "provider URL" + oidc-client-id: "client-id" + oidc-groups-claim: "groups" + oidc-username-claim: "email" +``` + +2. Add the following `kubeadmconfig.clientConfig` section that contains OIDC parameters to your Kubernetes YAML file. + + +```yaml +kubeadmconfig: + clientConfig: + oidc-issuer-url: "" + oidc-client-id: "" + oidc-client-secret: "" + oidc-extra-scope: profile,email,openid +``` + +3. Provide third-party OIDC IDP details. + +### Use RBAC with OIDC + +You can create a role binding that uses individual users as the subject or specify a group name as the subject to map many users to a role. The group name is the group assigned in the OIDC provider's configuration. Below is an example. To learn more, review [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings). + +Assume you created a group named `dev-east-2` within an OIDC provider. If you configure the host cluster's Kubernetes pack with all the correct OIDC settings, you could then create a role binding for the `dev-east-2` group. + +In this example, Palette is used as the IDP, and all users in the `dev-east-2` would inherit the `cluster-admin` role. + +![A subject of the type group is assigned as the subject in a RoleBinding](/clusters_cluster-management_cluster-rbac_cluster-subject-group.png) + + + + + + + +## Prerequisites + +- A minimum of 2 CPU and 4GB Memory. + + +## Parameters + +| Parameter | Description | +|----------------------------------------------------------|----------------------------------------------------| +| `cluster.config.clusterConfiguration.apiServer.extraArgs` | This parameter contains extra arguments for the Kubernetes API server, such as enabling audit logging, enabling certain authorization modes, and setting profiling and secure-port.| +| `cluster.config.clusterConfiguration.apiServer.extraVolumes` | This parameter describes extra volumes for the Kubernetes API server, such as `audit-log` and `audit-policy`. | +| `cluster.config.clusterConfiguration.controllerManager.extraArgs` | This parameter describes extra arguments for the Kubernetes Controller Manager, such as enabling certain feature gates and setting profiling. | +| `cluster.config.clusterConfiguration.etcd.local.dataDir` | This parameter specifies the data directory for etcd, the distributed key-value store that Kubernetes uses to persist cluster state. | +| `cluster.config.clusterConfiguration.networking.podSubnet` | The IP subnet range to assign to pods. Default: 192.168.0.0/16 | +| `cluster.config.clusterConfiguration.networking.serviceSubnet` | The IP subnet range to assign to services. Default: 192.169.0.0/16 | +| `cluster.config.clusterConfiguration.scheduler.extraArgs` | This parameter contains extra arguments for the Kubernetes scheduler, such as disabling profiling. | +| `cluster.config.initConfiguration.nodeRegistration.kubeletExtraArgs` | This parameter contains extra arguments for kubelet during node registration, such as setting feature gates, protecting kernel defaults, and disabling the read-only port. | +| `pack.palette.config.oidc.identityProvider` | Dynamically enabled OpenID Connect (OIDC) Identity Provider (IDP) setting based on your UI selection when you add the PXK-E pack to your profile. This parameter appears in the YAML file after you make a selection. Refer to [Configure OIDC Identity Provider](#configure-custom-oidc). | + +You can add cloud-init stages exposed by [Kairos](https://kairos.io/docs/architecture/cloud-init/), an open-source project. For more information, check out the [Cloud Init Stages](../clusters/edge/edge-configuration/cloud-init.md) reference. + +You can also use pack settings described in the [Palette eXtended Kubernetes](kubernetes.md) guide. + + +## Usage + +The Kubeadm configuration file is where you can do the following: + +- Manually configure a third-party OIDC IDP. For more information, check out [Configure Custom OIDC](kubernetes-edge.md#configure-custom-oidc). + +- Add a certificate for the Spectro Proxy pack if you want to use a reverse proxy with a Kubernetes cluster. For more information, refer to the [Spectro Proxy](frp.md) guide. + + +#### Configuration Changes + +The PXK-E Kubeadm configuration is updated to dynamically enable OIDC based on your IDP selection by adding the ``identityProvider`` parameter. + + +```yaml +pack: + palette: + config: + oidc: + identityProvider: +``` + + +#### Example Kubeadm Configuration File + +```yaml +cluster: + config: | + clusterConfiguration: + apiServer: + extraArgs: + advertise-address: "0.0.0.0" + anonymous-auth: "true" + audit-log-maxage: "30" + audit-log-maxbackup: "10" + audit-log-maxsize: "100" + audit-log-path: /var/log/apiserver/audit.log + audit-policy-file: /etc/kubernetes/audit-policy.yaml + authorization-mode: RBAC,Node + default-not-ready-toleration-seconds: "60" + default-unreachable-toleration-seconds: "60" + disable-admission-plugins: AlwaysAdmit + enable-admission-plugins: AlwaysPullImages,NamespaceLifecycle,ServiceAccount,NodeRestriction + profiling: "false" + secure-port: "6443" + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + extraVolumes: + - hostPath: /var/log/apiserver + mountPath: /var/log/apiserver + name: audit-log + pathType: DirectoryOrCreate + - hostPath: /etc/kubernetes/audit-policy.yaml + mountPath: /etc/kubernetes/audit-policy.yaml + name: audit-policy + pathType: File + readOnly: true + timeoutForControlPlane: 10m0s + controllerManager: + extraArgs: + feature-gates: RotateKubeletServerCertificate=true + pod-eviction-timeout: 1m0s + profiling: "false" + terminated-pod-gc-threshold: "25" + use-service-account-credentials: "true" + dns: {} + kubernetesVersion: v1.26.4 + etcd: + local: + dataDir: "/etc/kubernetes/etcd" + extraArgs: + listen-client-urls: "https://0.0.0.0:2379" + networking: + podSubnet: 192.168.0.0/16 + serviceSubnet: 192.169.0.0/16 + scheduler: + extraArgs: + profiling: "false" + initConfiguration: + localAPIEndpoint: {} + nodeRegistration: + kubeletExtraArgs: + event-qps: "0" + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: "true" + read-only-port: "0" + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + joinConfiguration: + discovery: {} + nodeRegistration: + kubeletExtraArgs: + event-qps: "0" + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: "true" + read-only-port: "0" + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +stages: + initramfs: + - sysctl: + vm.overcommit_memory: 1 + kernel.panic: 10 + kernel.panic_on_oops: 1 + commands: + - ln -s /etc/kubernetes/admin.conf /run/kubeconfig + files: + - path: /etc/hosts + permission: "0644" + content: | + 127.0.0.1 localhost + - path: "/etc/kubernetes/audit-policy.yaml" + owner_string: "root" + permission: 0600 + content: | + apiVersion: audit.k8s.io/v1 + kind: Policy + rules: + - level: None + users: ["system:kube-proxy"] + verbs: ["watch"] + resources: + - group: "" # core + resources: ["endpoints", "services", "services/status"] + - level: None + users: ["system:unsecured"] + namespaces: ["kube-system"] + verbs: ["get"] + resources: + - group: "" # core + resources: ["configmaps"] + - level: None + users: ["kubelet"] # legacy kubelet identity + verbs: ["get"] + resources: + - group: "" # core + resources: ["nodes", "nodes/status"] + - level: None + userGroups: ["system:nodes"] + verbs: ["get"] + resources: + - group: "" # core + resources: ["nodes", "nodes/status"] + - level: None + users: + - system:kube-controller-manager + - system:kube-scheduler + - system:serviceaccount:kube-system:endpoint-controller + verbs: ["get", "update"] + namespaces: ["kube-system"] + resources: + - group: "" # core + resources: ["endpoints"] + - level: None + users: ["system:apiserver"] + verbs: ["get"] + resources: + - group: "" # core + resources: ["namespaces", "namespaces/status", "namespaces/finalize"] + - level: None + users: ["cluster-autoscaler"] + verbs: ["get", "update"] + namespaces: ["kube-system"] + resources: + - group: "" # core + resources: ["configmaps", "endpoints"] + # Don't log HPA fetching metrics. + - level: None + users: + - system:kube-controller-manager + verbs: ["get", "list"] + resources: + - group: "metrics.k8s.io" + # Don't log these read-only URLs. + - level: None + nonResourceURLs: + - /healthz* + - /version + - /swagger* + # Don't log events requests. + - level: None + resources: + - group: "" # core + resources: ["events"] + # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes + - level: Request + users: ["kubelet", "system:node-problem-detector", "system:serviceaccount:kube-system:node-problem-detector"] + verbs: ["update","patch"] + resources: + - group: "" # core + resources: ["nodes/status", "pods/status"] + omitStages: + - "RequestReceived" + - level: Request + userGroups: ["system:nodes"] + verbs: ["update","patch"] + resources: + - group: "" # core + resources: ["nodes/status", "pods/status"] + omitStages: + - "RequestReceived" + # deletecollection calls can be large, don't log responses for expected namespace deletions + - level: Request + users: ["system:serviceaccount:kube-system:namespace-controller"] + verbs: ["deletecollection"] + omitStages: + - "RequestReceived" + # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data, + # so only log at the Metadata level. + - level: Metadata + resources: + - group: "" # core + resources: ["secrets", "configmaps"] + - group: authentication.k8s.io + resources: ["tokenreviews"] + omitStages: + - "RequestReceived" + # Get repsonses can be large; skip them. + - level: Request + verbs: ["get", "list", "watch"] + resources: + - group: "" # core + - group: "admissionregistration.k8s.io" + - group: "apiextensions.k8s.io" + - group: "apiregistration.k8s.io" + - group: "apps" + - group: "authentication.k8s.io" + - group: "authorization.k8s.io" + - group: "autoscaling" + - group: "batch" + - group: "certificates.k8s.io" + - group: "extensions" + - group: "metrics.k8s.io" + - group: "networking.k8s.io" + - group: "policy" + - group: "rbac.authorization.k8s.io" + - group: "settings.k8s.io" + - group: "storage.k8s.io" + omitStages: + - "RequestReceived" + # Default level for known APIs + - level: RequestResponse + resources: + - group: "" # core + - group: "admissionregistration.k8s.io" + - group: "apiextensions.k8s.io" + - group: "apiregistration.k8s.io" + - group: "apps" + - group: "authentication.k8s.io" + - group: "authorization.k8s.io" + - group: "autoscaling" + - group: "batch" + - group: "certificates.k8s.io" + - group: "extensions" + - group: "metrics.k8s.io" + - group: "networking.k8s.io" + - group: "policy" + - group: "rbac.authorization.k8s.io" + - group: "settings.k8s.io" + - group: "storage.k8s.io" + omitStages: + - "RequestReceived" + # Default level for all other requests. + - level: Metadata + omitStages: + - "RequestReceived" +pack: + palette: + config: + oidc: + identityProvider: palette +``` + + +### Configure OIDC Identity Provider + +The OIDC IDP feature offers the convenience of managing OIDC at the Kubernetes layer. The OIDC IDP feature is particularly useful for environments that do not have their own IDP configured. In this scenario, you can leverage Palette as an IDP without having to configure a third-party IDP. We also support the ability to take advantage of other OIDC providers by making it possible for you to configure OIDC at the tenant level. For additional flexibility, if you wish to use a different IDP than the one configured at the tenant level, you can select a different IDP by adding the OIDC configuration to your cluster profile. + +When you add the PXK-E pack to a cluster profile, Palette displays the OIDC IDP options listed below. + +All the options require you to map a set of users or groups to a Kubernetes RBAC role. To learn how to map a Kubernetes role to users and groups, refer to [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings). + +You can create a role binding that maps individual users or groups assigned within the OIDC provider's configuration to a role. To learn more, review [Use RBAC with OIDC](kubernetes-edge.md#use-rbac-with-oidc). You can also configure OIDC for virtual clusters. For guidance, refer to [Configure OIDC for a Virtual Cluster](../clusters/palette-virtual-clusters/configure-oidc-virtual-cluster.md). + + +- **None**: This setting does not require OIDC configuration for the cluster. It displays in the YAML file as `noauth`. + + +- **Custom**: This is the default setting and does not require OIDC configuration. However, if desired, it allows you to specify a third-party OIDC provider by configuring OIDC statements in the YAML file as described in [Configure Custom OIDC](kubernetes-edge.md#configure-custom-oidc). This setting displays in the YAML file as `none`. + + +- **Palette**: This setting makes Palette the IDP. Any user with a Palette account in the tenant and the proper permissions to view and access the project's resources is able to log into the Kubernetes dashboard. This setting displays in the YAML file as `palette`. + + +- **Inherit from Tenant**: This setting allows you to apply RBAC to multiple clusters and requires you to configure OpenID Connect (OIDC) in **Tenant Settings**. In Tenant Admin scope, navigate to **Tenant Settings** > **SSO**, choose **OIDC**, and provide your third-party IDP details. This setting displays in the YAML file as `tenant`. For more information, check out the [SSO Setup](../user-management/saml-sso/enable-saml.md) guide. + +:::info + +If your IDP uses Security Assertion Markup Language (SAML) authentication, then the **Inherit from Tenant** option will not work, and you will need to use the **Custom** option instead. This is because Kubernetes supports only OIDC authentication and not SAML authentication. + +::: + + +### Configure Custom OIDC + +Follow these steps to configure a third-party OIDC IDP. + + +1. Add the following OIDC parameters to the `apiServer.extraArgs` section of your Kubernetes YAML file when creating a cluster profile. + + +```yaml +cluster: + config: + clusterConfiguration: + apiServer: + extraArgs: + oidc-issuer-url: "provider URL" + oidc-client-id: "client-id" + oidc-groups-claim: "groups" + oidc-username-claim: "email" +``` + +2. Add the following `kubeadmconfig.clientConfig` section that contains OIDC parameters to your Kubernetes YAML file. + + +```yaml +kubeadmconfig: + clientConfig: + oidc-issuer-url: "" + oidc-client-id: "" + oidc-client-secret: "" + oidc-extra-scope: profile,email,openid +``` + +3. Provide third-party OIDC IDP details. + +### Use RBAC with OIDC + +You can create a role binding that uses individual users as the subject or specify a group name as the subject to map many users to a role. The group name is the group assigned in the OIDC provider's configuration. Below is an example. To learn more, review [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings). + +Assume you created a group named `dev-east-2` within an OIDC provider. If you configure the host cluster's Kubernetes pack with all the correct OIDC settings, you could then create a role binding for the `dev-east-2` group. + +In this example, Palette is used as the IDP, and all users in the `dev-east-2` would inherit the `cluster-admin` role. + +![A subject of the type group is assigned as the subject in a RoleBinding](/clusters_cluster-management_cluster-rbac_cluster-subject-group.png) + + + + + + + + + +## Prerequisites + +- A minimum of 2 CPU and 4GB Memory. + + +## Parameters + +| Parameter | Description | +|----------------------------------------------------------|----------------------------------------------------| +| `cluster.config.clusterConfiguration.apiServer.extraArgs` | This parameter contains extra arguments for the Kubernetes API server, such as enabling audit logging, enabling certain authorization modes, and setting profiling and secure-port.| +| `cluster.config.clusterConfiguration.apiServer.extraVolumes` | This parameter describes extra volumes for the Kubernetes API server, such as `audit-log` and `audit-policy`. | +| `cluster.config.clusterConfiguration.controllerManager.extraArgs` | This parameter describes extra arguments for the Kubernetes Controller Manager, such as enabling certain feature gates and setting profiling. | +| `cluster.config.clusterConfiguration.etcd.local.dataDir` | This parameter specifies the data directory for etcd, the distributed key-value store that Kubernetes uses to persist cluster state. | +| `cluster.config.clusterConfiguration.networking.podSubnet` | The IP subnet range to assign to pods. Default: 192.168.0.0/16 | +| `cluster.config.clusterConfiguration.networking.serviceSubnet` | The IP subnet range to assign to services. Default: 192.169.0.0/16 | +| `cluster.config.clusterConfiguration.scheduler.extraArgs` | This parameter contains extra arguments for the Kubernetes scheduler, such as disabling profiling. | +| `cluster.config.initConfiguration.nodeRegistration.kubeletExtraArgs` | This parameter contains extra arguments for kubelet during node registration, such as setting feature gates, protecting kernel defaults, and disabling the read-only port. | +| ``pack.palette.config.oidc.identityProvider`` | Dynamically enabled OpenID Connect (OIDC) Identity Provider (IDP) setting based on your UI selection when you add the PXK-E pack to your profile. This parameter appears in the YAML file after you make a selection. Refer to [Configure OIDC Identity Provider](#configure-custom-oidc). | + +You can add cloud-init stages exposed by [Kairos](https://kairos.io/docs/architecture/cloud-init/), an open-source project. For more information, check out the [Cloud Init Stages](../clusters/edge/edge-configuration/cloud-init.md) reference. + +You can also use pack settings described in the [Palette eXtended Kubernetes](kubernetes.md) guide. + + +## Usage + +The Kubeadm configuration file is where you can do the following: + +- Manually configure a third-party OIDC IDP. For more information, check out [Configure Custom OIDC](kubernetes-edge.md#configure-custom-oidc). + +- Add a certificate for the Spectro Proxy pack if you want to use a reverse proxy with a Kubernetes cluster. For more information, refer to the [Spectro Proxy](frp.md) guide. + + +#### Configuration Changes + +The PXK-E Kubeadm configuration is updated to dynamically enable OIDC based on your IDP selection by adding the ``identityProvider`` parameter. + +```yaml +palette: + config: + oidc: + identityProvider: +``` + + +#### Example Kubeadm Configuration File + +```yaml +cluster: + config: | + clusterConfiguration: + apiServer: + extraArgs: + advertise-address: "0.0.0.0" + anonymous-auth: "true" + audit-log-maxage: "30" + audit-log-maxbackup: "10" + audit-log-maxsize: "100" + audit-log-path: /var/log/apiserver/audit.log + audit-policy-file: /etc/kubernetes/audit-policy.yaml + authorization-mode: RBAC,Node + default-not-ready-toleration-seconds: "60" + default-unreachable-toleration-seconds: "60" + disable-admission-plugins: AlwaysAdmit + enable-admission-plugins: AlwaysPullImages,NamespaceLifecycle,ServiceAccount,NodeRestriction + profiling: "false" + secure-port: "6443" + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + extraVolumes: + - hostPath: /var/log/apiserver + mountPath: /var/log/apiserver + name: audit-log + pathType: DirectoryOrCreate + - hostPath: /etc/kubernetes/audit-policy.yaml + mountPath: /etc/kubernetes/audit-policy.yaml + name: audit-policy + pathType: File + readOnly: true + timeoutForControlPlane: 10m0s + controllerManager: + extraArgs: + feature-gates: RotateKubeletServerCertificate=true + pod-eviction-timeout: 1m0s + profiling: "false" + terminated-pod-gc-threshold: "25" + use-service-account-credentials: "true" + dns: {} + kubernetesVersion: v1.25.2 + etcd: + local: + dataDir: "/etc/kubernetes/etcd" + extraArgs: + listen-client-urls: "https://0.0.0.0:2379" + networking: + podSubnet: 192.168.0.0/16 + serviceSubnet: 192.169.0.0/16 + scheduler: + extraArgs: + profiling: "false" + initConfiguration: + localAPIEndpoint: {} + nodeRegistration: + kubeletExtraArgs: + event-qps: "0" + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: "true" + read-only-port: "0" + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + joinConfiguration: + discovery: {} + nodeRegistration: + kubeletExtraArgs: + event-qps: "0" + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: "true" + read-only-port: "0" + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + +stages: + initramfs: + - sysctl: + vm.overcommit_memory: 1 + kernel.panic: 10 + kernel.panic_on_oops: 1 + commands: + - "ln -s /etc/kubernetes/admin.conf /run/kubeconfig" + files: + - path: /etc/hosts + permission: "0644" + content: | + 127.0.0.1 localhost + - path: "/etc/kubernetes/audit-policy.yaml" + owner_string: "root" + permission: 0600 + content: | + apiVersion: audit.k8s.io/v1 + kind: Policy + rules: + - level: None + users: ["system:kube-proxy"] + verbs: ["watch"] + resources: + - group: "" # core + resources: ["endpoints", "services", "services/status"] + - level: None + users: ["system:unsecured"] + namespaces: ["kube-system"] + verbs: ["get"] + resources: + - group: "" # core + resources: ["configmaps"] + - level: None + users: ["kubelet"] # legacy kubelet identity + verbs: ["get"] + resources: + - group: "" # core + resources: ["nodes", "nodes/status"] + - level: None + userGroups: ["system:nodes"] + verbs: ["get"] + resources: + - group: "" # core + resources: ["nodes", "nodes/status"] + - level: None + users: + - system:kube-controller-manager + - system:kube-scheduler + - system:serviceaccount:kube-system:endpoint-controller + verbs: ["get", "update"] + namespaces: ["kube-system"] + resources: + - group: "" # core + resources: ["endpoints"] + - level: None + users: ["system:apiserver"] + verbs: ["get"] + resources: + - group: "" # core + resources: ["namespaces", "namespaces/status", "namespaces/finalize"] + - level: None + users: ["cluster-autoscaler"] + verbs: ["get", "update"] + namespaces: ["kube-system"] + resources: + - group: "" # core + resources: ["configmaps", "endpoints"] + # Don't log HPA fetching metrics. + - level: None + users: + - system:kube-controller-manager + verbs: ["get", "list"] + resources: + - group: "metrics.k8s.io" + # Don't log these read-only URLs. + - level: None + nonResourceURLs: + - /healthz* + - /version + - /swagger* + # Don't log events requests. + - level: None + resources: + - group: "" # core + resources: ["events"] + # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes + - level: Request + users: ["kubelet", "system:node-problem-detector", "system:serviceaccount:kube-system:node-problem-detector"] + verbs: ["update","patch"] + resources: + - group: "" # core + resources: ["nodes/status", "pods/status"] + omitStages: + - "RequestReceived" + - level: Request + userGroups: ["system:nodes"] + verbs: ["update","patch"] + resources: + - group: "" # core + resources: ["nodes/status", "pods/status"] + omitStages: + - "RequestReceived" + # deletecollection calls can be large, don't log responses for expected namespace deletions + - level: Request + users: ["system:serviceaccount:kube-system:namespace-controller"] + verbs: ["deletecollection"] + omitStages: + - "RequestReceived" + # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data, + # so only log at the Metadata level. + - level: Metadata + resources: + - group: "" # core + resources: ["secrets", "configmaps"] + - group: authentication.k8s.io + resources: ["tokenreviews"] + omitStages: + - "RequestReceived" + # Get repsonses can be large; skip them. + - level: Request + verbs: ["get", "list", "watch"] + resources: + - group: "" # core + - group: "admissionregistration.k8s.io" + - group: "apiextensions.k8s.io" + - group: "apiregistration.k8s.io" + - group: "apps" + - group: "authentication.k8s.io" + - group: "authorization.k8s.io" + - group: "autoscaling" + - group: "batch" + - group: "certificates.k8s.io" + - group: "extensions" + - group: "metrics.k8s.io" + - group: "networking.k8s.io" + - group: "policy" + - group: "rbac.authorization.k8s.io" + - group: "settings.k8s.io" + - group: "storage.k8s.io" + omitStages: + - "RequestReceived" + # Default level for known APIs + - level: RequestResponse + resources: + - group: "" # core + - group: "admissionregistration.k8s.io" + - group: "apiextensions.k8s.io" + - group: "apiregistration.k8s.io" + - group: "apps" + - group: "authentication.k8s.io" + - group: "authorization.k8s.io" + - group: "autoscaling" + - group: "batch" + - group: "certificates.k8s.io" + - group: "extensions" + - group: "metrics.k8s.io" + - group: "networking.k8s.io" + - group: "policy" + - group: "rbac.authorization.k8s.io" + - group: "settings.k8s.io" + - group: "storage.k8s.io" + omitStages: + - "RequestReceived" + # Default level for all other requests. + - level: Metadata + omitStages: + - "RequestReceived" + pack: + palette: + config: + oidc: + identityProvider: palette + ``` + +### Configure OIDC Identity Provider + +The OIDC IDP feature offers the convenience of managing OIDC at the Kubernetes layer. The OIDC IDP feature is particularly useful for environments that do not have their own IDP configured. In this scenario, you can leverage Palette as an IDP without having to configure a third-party IDP. We also support the ability to take advantage of other OIDC providers by making it possible for you to configure OIDC at the tenant level. For additional flexibility, if you wish to use a different IDP than the one configured at the tenant level, you can select a different IDP by adding the OIDC configuration to your cluster profile. + +When you add the PXK-E pack to a cluster profile, Palette displays the OIDC IDP options listed below. + +All the options require you to map a set of users or groups to a Kubernetes RBAC role. To learn how to map a Kubernetes role to users and groups, refer to [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings). + +You can create a role binding that maps individual users or groups assigned within the OIDC provider's configuration to a role. To learn more, review [Use RBAC with OIDC](kubernetes-edge.md#use-rbac-with-oidc). You can also configure OIDC for virtual clusters. For guidance, refer to [Configure OIDC for a Virtual Cluster](../clusters/palette-virtual-clusters/configure-oidc-virtual-cluster.md). + + +- **None**: This is the default setting and there is nothing to configure. This setting displays in the YAML file as `noauth`. + + +- **Custom**: This setting allows you to specify a third-party OIDC provider by configuring OIDC statements in the Kubeadm configuration file as described in [Configure Custom OIDC](kubernetes-edge.md#configure-custom-oidc). This setting displays in the YAML file as `none`. + + +- **Palette**: This setting makes Palette the IDP. Any user with a Palette account in the tenant and the proper permissions to view and access the project's resources is able to log into the Kubernetes dashboard. This setting displays in the YAML file as `palette`. + + +- **Inherit from Tenant**: This setting allows you to apply RBAC to multiple clusters and requires you to configure OpenID Connect (OIDC) in **Tenant Settings**. In Tenant Admin scope, navigate to **Tenant Settings** > **SSO**, choose **OIDC**, and provide your third-party IDP details. This setting displays in the YAML file as `tenant`. For more information, check out the [SSO Setup](../user-management/saml-sso/enable-saml.md) guide. + +:::info + +If your IDP uses Security Assertion Markup Language (SAML) authentication, then the **Inherit from Tenant** option will not work, and you will need to use the **Custom** option instead. This is because Kubernetes supports only OIDC authentication and not SAML authentication. + +::: + +### Configure Custom OIDC + +Follow these steps to configure a third-party OIDC IDP. + +1. Add the following OIDC parameters to the `apiServer.extraArgs` section of your Kubernetes YAML file when creating a cluster profile. + + +```yaml +cluster: + config: + clusterConfiguration: + apiServer: + extraArgs: + oidc-issuer-url: "provider URL" + oidc-client-id: "client-id" + oidc-groups-claim: "groups" + oidc-username-claim: "email" +``` + +2. Add the following `kubeadmconfig.clientConfig` section that contains OIDC parameters to your Kubernetes YAML file. + + +```yaml +kubeadmconfig: + clientConfig: + oidc-issuer-url: "" + oidc-client-id: "" + oidc-client-secret: "" + oidc-extra-scope: profile,email,openid +``` + +3. Provide third-party OIDC IDP details. + +### Use RBAC with OIDC + +You can create a role binding that uses individual users as the subject or specify a group name as the subject to map many users to a role. The group name is the group assigned in the OIDC provider's configuration. Below is an example. To learn more, review [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings). + +Assume you created a group named `dev-east-2` within an OIDC provider. If you configure the host cluster's Kubernetes pack with all the correct OIDC settings, you could then create a role binding for the `dev-east-2` group. + +In this example, Palette is used as the IDP, and all users in the `dev-east-2` would inherit the `cluster-admin` role. + +![A subject of the type group is assigned as the subject in a RoleBinding](/clusters_cluster-management_cluster-rbac_cluster-subject-group.png) + + + + + + + +:::warning + +All versions less than v1.25.x are considered deprecated. Upgrade to a newer version to take advantage of new features. + +::: + + + + + + +## Terraform + +You can reference Kubernetes in Terraform with the following code snippet. + +
+ +```hcl +data "spectrocloud_registry" "public_registry" { + name = "Public Repo" +} + +data "spectrocloud_pack_simple" "edge-k8s" { + name = "edge-k8s" + version = "1.26.4" + type = "helm" + registry_uid = data.spectrocloud_registry.public_registry.id +} +``` + +## Resources + +- [Kubernetes](https://kubernetes.io/) + + + +- [Kubernetes Documentation](https://kubernetes.io/docs/concepts/overview/) + + + +- [Image Swap with Palette](../clusters/cluster-management/image-swap.md) diff --git a/docs/docs-content/integrations/kubernetes-generic.md b/docs/docs-content/integrations/kubernetes-generic.md index 835dedcd7c..1ed233b4bf 100644 --- a/docs/docs-content/integrations/kubernetes-generic.md +++ b/docs/docs-content/integrations/kubernetes-generic.md @@ -10,7 +10,7 @@ logoUrl: "https://registry.spectrocloud.com/v1/k8s-dashboard/blobs/sha256:2de5d8 tags: ["packs", "kubernetes"] --- -The Kubernetes pack supports several cloud and data center infrastructure providers. This pack defines the default properties we use to deploy Kubernetes clusters and enables most of the Kubernetes hardening standards that the Center for Internet Security (CIS) recommends. +The Cloud Native Computing Foundation's (CNCF) Kubernetes pack supports several cloud and data center infrastructure providers. This pack defines the default properties we use to deploy Kubernetes clusters and enables most of the Kubernetes hardening standards that the Center for Internet Security (CIS) recommends. We also support managed Kubernetes distributions for Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), Google Kubernetes Engine (GKE), and Tencent Kubernetes Engine (TKE). @@ -20,6 +20,10 @@ Review the [Maintenance Policy](maintenance-policy.md) to learn about pack updat
+### Support Lifecycle + +We support CNCF Kubernetes for N-3 Kubernetes minor versions for a duration of 14 months. The duration exceeds the official EOL by four months. Once we stop supporting the minor version, we initiate the deprecation process. Refer to the [Kubernetes Support Lifecycle](kubernetes-support.md#palette-extended-kubernetes-support) guide to learn more. + ## Versions Supported diff --git a/docs/docs-content/integrations/kubernetes-support.md b/docs/docs-content/integrations/kubernetes-support.md new file mode 100644 index 0000000000..8605baf501 --- /dev/null +++ b/docs/docs-content/integrations/kubernetes-support.md @@ -0,0 +1,69 @@ +--- +sidebar_label: "Kubernetes Support Lifecycle" +title: "Kubernetes Support Lifecycle" +description: "Learn about the Kubernetes versions we support and how we manage Kubernetes support." +icon: "" +hide_table_of_contents: false +sidebar_position: 10 +tags: ["packs", "support", "kubernetes", "lifecycle"] +--- + + +## Overview + +We follow a similar lifecycle as established by the Kubernetes community. You can learn more about the official supported Cloud Native Computing Foundation (CNCF) Kubernetes release cycle in the [Kubernetes Releases](https://kubernetes.io/releases/) page. This means we support three minor Kubernetes versions at any given time. We support the current release and the three previous minor version releases, also known as N-3. For example, if the current release is 1.29, we support 1.28, 1.27, and 1.26. + +The table below lists the Kubernetes distributions we support and the duration of support. + +| Kubernetes Distribution | Supported Minor Versions | Support Duration | Notes | +| ----------------------- | ------------------ | ---------------- | ----- | +| CNCF Kubernetes | N-3 | 14 months | The official open-source version of [Kubernetes](https://kubernetes.io/). | +| Palette eXtended Kubernetes (PXK) | N-3 | 14 months | Additional support may be extended. Discuss this with our support team if you need additional support. | +| Palette eXtended Kubernetes - Edge (PXK-E) | N-3 | 14 months | Additional support may be extended. Discuss this with our support team if you need additional support. | +| Other | N-3 | EOL | Other distributions available in Palette such as K3s, Microk8s, and RKE2, we only support until their official EOL. The EOL is set by the respective owner. | + + +:::info + +Kubernetes follows the [semantic version schema](https://semver.org/). Versions are annotated as x.y.z, where x is the major version, y is the minor version, and z is the patch version. +::: + +We support N-3 Kubernetes minor versions until the official End-Of-Life (EOL). Once we stop supporting the minor version, we initiate the deprecation process. You can learn more about our deprecation process in the [Pack Deprecation](./maintenance-policy.md#pack-deprecations) section. + + +## Palette eXtended Kubernetes Support + +We support CNCF Kubernetes, [Palette eXtended Kubernetes (PXK)](./kubernetes.md) and [Palette eXtended Kubernetes Edge (PXK-E)](./kubernetes-edge.md) for N-3 Kubernetes minor versions for a duration of 14 months. The duration exceeds the official EOL by four months. Once we stop supporting the minor version, we initiate the deprecation process. + +The diagram below illustrates the support lifecycle of a Kubernetes version. A deprecated Kubernetes version will no longer receive updates. + +![Diagram of the Kubernetes Support Lifecycle. PXK and CNCF is supported for 14 months. After that, normal deprecation flow is initiated.](/integrations_kubernetes-support_support-cycle.png) + +Additional support may be extended for PXK and PXK-E. Discuss this with our support team if you need extended support. You can contact our support team at [support@spectrocloud.com](mailto:support@spectrocloud.com). + +## Other Kubernetes Distributions + +We support other Kubernetes distributions such as K3s, Microk8s, and RKE2 until their official EOL. The EOL is set by the respective owner. Once we stop supporting the minor version, we initiate the deprecation process. You can learn more about our deprecation process in the [Pack Deprecation](./maintenance-policy.md#pack-deprecations) section. + + +![Diagram of other Kubernetes Support Lifecycle. Other distros are supported until their EOL. After that, normal deprecation flow is initiated.](/integrations_kubernetes-support_support-cycle_other.png) + + +## Kubernetes Upgrades + +The official guidelines for Kubernetes upgrades recommend upgrading one minor version at a time. For example, if you are using Kubernetes version 1.26, you should upgrade to 1.27, before upgrading to version 1.28. You can learn more about the official Kubernetes upgrade guidelines in the [Version Skew Policy](https://kubernetes.io/releases/version-skew-policy/) page. We +recommend following the official guidelines for all Kubernetes upgrades, including PXK and PXK-E. + +:::tip + +Use cluster profile versions to manage your Kubernetes upgrades. Create a new cluster profile version for each Kubernetes upgrade. You can then use the new cluster profile version to upgrade your cluster. + +::: + +To learn more about upgrading your cluster and cluster profiles, check out the resources below: + +- [Update a Cluster](../clusters/cluster-management/cluster-updates.md) +- [Version a Cluster Profile](../profiles/cluster-profiles/modify-cluster-profiles/version-cluster-profile.md) + + +We also recommend you check out [Deploy Cluster Profile Updates](../clusters/cluster-management/update-k8s-cluster.md) tutorial to learn how to update your cluster profile. \ No newline at end of file diff --git a/docs/docs-content/integrations/kubernetes.md b/docs/docs-content/integrations/kubernetes.md index 8b72db0891..4502ee3b36 100644 --- a/docs/docs-content/integrations/kubernetes.md +++ b/docs/docs-content/integrations/kubernetes.md @@ -34,6 +34,17 @@ We also offer Palette eXtended Kubernetes Edge (PXK-E) for Edge deployments. Ref ::: +### PXK and Palette VerteX + +The PXK used in [Palette VerteX](../vertex/vertex.md) is compiled and linked with our [NIST-certified FIPS crypto module](../compliance.md#fips-140-2). PXK is by default enabled with [Ubuntu Pro](https://ubuntu.com/pro) with FIPS mode enabled. Additionally, the Operating System (OS) is hardened based on the NIST-800 standard. However, if you use a different OS through the [BYOOS](./byoos.md) pack, then you are responsible for ensuring FIPS compliance and hardening of the OS. + +The combined usage of PXK and Palette VerteX provides a secure and FIPS-compliant experience as the Kubernetes distribution, OS, and management platform VerteX is FIPS-compliant. + + +### Support Lifecycle + +We support PXK for N-3 Kubernetes minor versions for a duration of 14 months. The duration exceeds the official EOL by four months. Once we stop supporting the minor version, we initiate the deprecation process. Refer to the [Kubernetes Support Lifecycle](kubernetes-support.md#palette-extended-kubernetes-support) guide to learn more. + ## Versions Supported diff --git a/docs/docs-content/integrations/maintenance-policy.md b/docs/docs-content/integrations/maintenance-policy.md index a12ede0e0d..17509d1227 100644 --- a/docs/docs-content/integrations/maintenance-policy.md +++ b/docs/docs-content/integrations/maintenance-policy.md @@ -139,7 +139,7 @@ For important guidelines on updating pack versions, review [Update the Pack Vers ### Kubernetes Packs -A minor Kubernetes version is deprecated in Palette when the Kubernetes community announces the version is entering End of Life (EOL). +Refer to the [Kubernetes Support Lifecycle](kubernetes-support.md) page to learn about the Kubernetes versions we support and the duration of support. ### CNI / CSI / Add-on Packs diff --git a/docs/docs-content/integrations/microk8s.md b/docs/docs-content/integrations/microk8s.md index 7520fce2b1..d4563be515 100644 --- a/docs/docs-content/integrations/microk8s.md +++ b/docs/docs-content/integrations/microk8s.md @@ -12,6 +12,10 @@ tags: ['packs', 'microk8s', 'kubernetes'] MicroK8s is a Cloud Native Computing Foundation (CNCF) certified upstream Kubernetes deployment that runs entirely on your workstation or edge device. It runs all Kubernetes services natively without virtual machines and packs all the required libraries and binaries. +### Support Lifecycle + +We support other Kubernetes distributions such as K3s, Microk8s, and RKE2 until their official EOL. The EOL is set by the respective owner. Once we stop supporting the minor version, we initiate the deprecation process. Refer to the [Kubernetes Support Lifecycle](kubernetes-support.md#palette-extended-kubernetes-support) guide to learn more. + ## Prerequisites - One of the following Ubuntu environments to run commands: diff --git a/docs/docs-content/integrations/rke2.md b/docs/docs-content/integrations/rke2.md index a31c7dbbb9..0228de32c5 100644 --- a/docs/docs-content/integrations/rke2.md +++ b/docs/docs-content/integrations/rke2.md @@ -27,13 +27,11 @@ RKE2 launches control plane components as static pods, managed by the kubelet in You can deploy RKE2 by adding this pack to a cluster profile. Once the cluster profile is created, you can deploy the RKE2-based Kubernetes clusters through Palette. -
+### Support Lifecycle -:::warning +We support other Kubernetes distributions such as K3s, Microk8s, and RKE2 until their official EOL. The EOL is set by the respective owner. Once we stop supporting the minor version, we initiate the deprecation process. Refer to the [Kubernetes Support Lifecycle](kubernetes-support.md#palette-extended-kubernetes-support) guide to learn more. -RKE2 is only available for Edge host deployments. Refer to the [Edge](../clusters/edge/edge.md) documentation to learn more about Edge. -::: ## Versions Supported diff --git a/static/assets/docs/images/integrations_kubernetes-support_support-cycle.png b/static/assets/docs/images/integrations_kubernetes-support_support-cycle.png new file mode 100644 index 0000000000..df28ef1034 Binary files /dev/null and b/static/assets/docs/images/integrations_kubernetes-support_support-cycle.png differ diff --git a/static/assets/docs/images/integrations_kubernetes-support_support-cycle_other.png b/static/assets/docs/images/integrations_kubernetes-support_support-cycle_other.png new file mode 100644 index 0000000000..1acbaff1c5 Binary files /dev/null and b/static/assets/docs/images/integrations_kubernetes-support_support-cycle_other.png differ diff --git a/vale/styles/spectrocloud/spectrocloud.yml b/vale/styles/spectrocloud/spectrocloud.yml index 73892e84c6..67e26f8164 100644 --- a/vale/styles/spectrocloud/spectrocloud.yml +++ b/vale/styles/spectrocloud/spectrocloud.yml @@ -1,16 +1,18 @@ extends: existence message: "Avoid using 'spectrocloud' outside of a URL format. Use 'Spectro Cloud' instead." level: error -ignorecase: true +ignorecase: false nonword: true tokens: - - 'spectrocloud' - 'spectro cloud' - 'SpectroCloud' - 'SpectroCloud' + - 'Spectrocloud' - 'Spectro cloud' - 'spectro Cloud' scope: sentence exceptions: + - '\[mailto:[^@]*@spectrocloud\.com\]' - '\w*\.spectrocloud\.\w*' - - '\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+(?