From 21eea746dcf5aa2bd408bf23b12bc0176b4bb588 Mon Sep 17 00:00:00 2001 From: Karl Cardenas <29551334+karl-cardenas-coding@users.noreply.github.com> Date: Tue, 29 Oct 2024 17:55:57 -0700 Subject: [PATCH] docs: DOC-928 DOC-929 DOC-930 DOC-931 DOC-927 (#4272) (#4508) * docs: DOC-928 DOC-929 DOC-930 DOC-931 DOC-927 * docs: static IaaS * docs: iaas dynamic * chore: fix headings * docs: aks static placement * docs: aks dynamic placement * docs: vale suggestions * docs: apply suggestions from code review * docs: applied feedback * docs: feedback * docs: feedback * docs: upadated IAM policies * chore: updated warning * docs: apply suggestions from code review --------- Co-authored-by: Matt Welke Co-authored-by: Lenny Chen <55669665+lennessyy@users.noreply.github.com> --- .../azure/required-permissions.md | 967 +++++++ .../public-cloud/aws/required-iam-policies.md | 366 +-- .../azure/required-permissions.md | 2282 +++++++++++------ 3 files changed, 2636 insertions(+), 979 deletions(-) create mode 100644 docs/deprecated/clusters/public-cloud/azure/required-permissions.md diff --git a/docs/deprecated/clusters/public-cloud/azure/required-permissions.md b/docs/deprecated/clusters/public-cloud/azure/required-permissions.md new file mode 100644 index 0000000000..5b074a5b03 --- /dev/null +++ b/docs/deprecated/clusters/public-cloud/azure/required-permissions.md @@ -0,0 +1,967 @@ +--- +sidebar_label: "Required Permissions" +title: "Required Permissions" +description: "Review the required permissions for deploying clusters on Azure" +icon: "" +hide_table_of_contents: false +tags: ["public cloud", "azure", "permissions"] +sidebar_position: 100 +--- + +Palette requires a set of permissions to properly deploy and manage the lifecycle of clusters deployed to Azure. We +recommend creating a +[role assignment](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments) that has the +service principal you want to use with Palette and the appropriate +[scope level](https://learn.microsoft.com/en-us/azure/role-based-access-control/scope-overview). To ensure that Palette +and VerteX can deploy and manage clusters on Azure in all use cases, use a subscription as the scope level for the role +assignment. + +:::warning + +We recommend against assigning the built-in Azure +[Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#contributor) role to the +service principal you want to use in Palette because its permission scope exceeds our requirements. Instead, create a +custom role to provide only those permissions that Palette requires. + +::: + +We support the following use cases: + +- [Dynamic Placement](#dynamic-placement) - Palette creates network resources required for your cluster. + +- [Static Placement](#static-placement) - Palette deploys clusters on the pre-existing network resources you specify. + +:::tip + +You can use [Validator](https://github.com/spectrocloud-labs/validator) with the +[Azure plugin](https://github.com/spectrocloud-labs/validator-plugin-azure) to verify you have setup the correct +permissions. The Validator Azure plugin requires the following permissions: + +- Microsoft.Authorization/denyAssignments/read +- Microsoft.Authorization/roleAssignments/read +- Microsoft.Authorization/roleDefinitions/read + +You can use Validator with the Palette CLI. Check out the Palette CLI +[Validator](../../../automation/palette-cli/commands/ec.md#validate-environment) reference page for more information. + +::: + +Review the sections below to learn how to create a custom role in Azure and assign it to the service principal you want +to use with Palette. + +## Dynamic Placement + +Dynamic placement is the default method for deploying Azure clusters through Palette. With dynamic placement, Palette +dynamically creates the network resources required for your cluster. + +### Prerequisites + +- Azure CLI installed on your local machine. Refer to the + [Azure CLI Install Guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) for installation instructions. + +- The Azure subscription ID you want to use with Palette. + +- The Security Principal Object ID you want to use with Palette. You can retrieve it from the Entra ID section of the + Azure Portal. + +- A terminal or command prompt to issue the Azure CLI commands. + +- The Azure Security Principal Object ID you want to use with Palette. The Security Principal Object ID can represent a + user, group, or service principal. + +### Create Role and Assign Permissions + +1. Export your Azure subscription ID to a variable. + + ```shell + export SUBSCRIPTION_ID= + ``` + +2. Issue the following command to create a JSON file containing all the required permissions to deploy clusters + dynamically in Palette. + + ```shell + cat << EOF > dynamic-permissions.json + { + "Name": "Dynamic Placement Palette Deployer", + "IsCustom": true, + "Description": "Can deploy Azure IaaS clusters using dynamic placement with Palette.", + "Actions": [ + "Microsoft.Compute/disks/delete", + "Microsoft.Compute/disks/read", + "Microsoft.Compute/disks/write", + "Microsoft.Compute/galleries/images/read", + "Microsoft.Compute/galleries/images/versions/read", + "Microsoft.Compute/galleries/images/versions/write", + "Microsoft.Compute/galleries/images/write", + "Microsoft.Compute/galleries/read", + "Microsoft.Compute/galleries/write", + "Microsoft.Compute/images/read", + "Microsoft.Compute/images/write", + "Microsoft.Compute/virtualMachines/delete", + "Microsoft.Compute/virtualMachines/extensions/delete", + "Microsoft.Compute/virtualMachines/extensions/read", + "Microsoft.Compute/virtualMachines/extensions/write", + "Microsoft.Compute/virtualMachines/read", + "Microsoft.Compute/virtualMachines/write", + "Microsoft.Network/loadBalancers/backendAddressPools/join/action", + "Microsoft.Network/loadBalancers/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/join/action", + "Microsoft.Network/loadBalancers/inboundNatRules/read", + "Microsoft.Network/loadBalancers/inboundNatRules/write", + "Microsoft.Network/loadBalancers/read", + "Microsoft.Network/loadBalancers/write", + "Microsoft.Network/networkInterfaces/delete", + "Microsoft.Network/networkInterfaces/join/action", + "Microsoft.Network/networkInterfaces/read", + "Microsoft.Network/networkInterfaces/write", + "Microsoft.Network/networkSecurityGroups/read", + "Microsoft.Network/networkSecurityGroups/securityRules/delete", + "Microsoft.Network/networkSecurityGroups/securityRules/read", + "Microsoft.Network/networkSecurityGroups/securityRules/write", + "Microsoft.Network/privateDnsZones/A/delete", + "Microsoft.Network/privateDnsZones/A/read", + "Microsoft.Network/privateDnsZones/A/write", + "Microsoft.Network/privateDnsZones/delete", + "Microsoft.Network/privateDnsZones/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", + "Microsoft.Network/privateDnsZones/write", + "Microsoft.Network/publicIPAddresses/delete", + "Microsoft.Network/publicIPAddresses/join/action", + "Microsoft.Network/publicIPAddresses/read", + "Microsoft.Network/publicIPAddresses/write", + "Microsoft.Network/routeTables/delete", + "Microsoft.Network/routeTables/read", + "Microsoft.Network/routeTables/write", + "Microsoft.Network/virtualNetworks/delete", + "Microsoft.Network/virtualNetworks/join/action", + "Microsoft.Network/virtualNetworks/join/action", + "Microsoft.Network/virtualNetworks/joinLoadBalancer/action", + "Microsoft.Network/virtualNetworks/peer/action", + "Microsoft.Network/virtualNetworks/read", + "Microsoft.Network/virtualNetworks/subnets/delete", + "Microsoft.Network/virtualNetworks/subnets/join/action", + "Microsoft.Network/virtualNetworks/subnets/joinLoadBalancer/action", + "Microsoft.Network/virtualNetworks/subnets/read", + "Microsoft.Network/virtualNetworks/subnets/virtualMachines/read", + "Microsoft.Network/virtualNetworks/subnets/write", + "Microsoft.Network/virtualNetworks/virtualMachines/read", + "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/delete", + "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read", + "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write", + "Microsoft.Network/virtualNetworks/write", + "Microsoft.Resources/subscriptions/resourceGroups/read", + "Microsoft.Storage/storageAccounts/blobServices/containers/read", + "Microsoft.Storage/storageAccounts/blobServices/containers/write", + "Microsoft.Storage/storageAccounts/read", + "Microsoft.Storage/storageAccounts/write" + ], + "NotActions": [], + "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] + } + EOF + + ``` + + +
+ Expand to view the raw JSON for the Dynamic Placement policy + + ```json + { + "Name": "Dynamic Placement Palette Deployer", + "IsCustom": true, + "Description": "Can deploy Azure IaaS clusters using dynamic placement with Palette.", + "Actions": [ + "Microsoft.Compute/disks/delete", + "Microsoft.Compute/disks/read", + "Microsoft.Compute/disks/write", + "Microsoft.Compute/galleries/images/read", + "Microsoft.Compute/galleries/images/versions/read", + "Microsoft.Compute/galleries/images/versions/write", + "Microsoft.Compute/galleries/images/write", + "Microsoft.Compute/galleries/read", + "Microsoft.Compute/galleries/write", + "Microsoft.Compute/images/read", + "Microsoft.Compute/images/write", + "Microsoft.Compute/virtualMachines/delete", + "Microsoft.Compute/virtualMachines/extensions/delete", + "Microsoft.Compute/virtualMachines/extensions/read", + "Microsoft.Compute/virtualMachines/extensions/write", + "Microsoft.Compute/virtualMachines/read", + "Microsoft.Compute/virtualMachines/write", + "Microsoft.Network/loadBalancers/backendAddressPools/join/action", + "Microsoft.Network/loadBalancers/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/join/action", + "Microsoft.Network/loadBalancers/inboundNatRules/read", + "Microsoft.Network/loadBalancers/inboundNatRules/write", + "Microsoft.Network/loadBalancers/read", + "Microsoft.Network/loadBalancers/write", + "Microsoft.Network/networkInterfaces/delete", + "Microsoft.Network/networkInterfaces/join/action", + "Microsoft.Network/networkInterfaces/read", + "Microsoft.Network/networkInterfaces/write", + "Microsoft.Network/networkSecurityGroups/read", + "Microsoft.Network/networkSecurityGroups/securityRules/delete", + "Microsoft.Network/networkSecurityGroups/securityRules/read", + "Microsoft.Network/networkSecurityGroups/securityRules/write", + "Microsoft.Network/privateDnsZones/A/delete", + "Microsoft.Network/privateDnsZones/A/read", + "Microsoft.Network/privateDnsZones/A/write", + "Microsoft.Network/privateDnsZones/delete", + "Microsoft.Network/privateDnsZones/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", + "Microsoft.Network/privateDnsZones/write", + "Microsoft.Network/publicIPAddresses/delete", + "Microsoft.Network/publicIPAddresses/join/action", + "Microsoft.Network/publicIPAddresses/read", + "Microsoft.Network/publicIPAddresses/write", + "Microsoft.Network/routeTables/delete", + "Microsoft.Network/routeTables/read", + "Microsoft.Network/routeTables/write", + "Microsoft.Network/virtualNetworks/delete", + "Microsoft.Network/virtualNetworks/join/action", + "Microsoft.Network/virtualNetworks/join/action", + "Microsoft.Network/virtualNetworks/joinLoadBalancer/action", + "Microsoft.Network/virtualNetworks/peer/action", + "Microsoft.Network/virtualNetworks/read", + "Microsoft.Network/virtualNetworks/subnets/delete", + "Microsoft.Network/virtualNetworks/subnets/join/action", + "Microsoft.Network/virtualNetworks/subnets/joinLoadBalancer/action", + "Microsoft.Network/virtualNetworks/subnets/read", + "Microsoft.Network/virtualNetworks/subnets/virtualMachines/read", + "Microsoft.Network/virtualNetworks/subnets/write", + "Microsoft.Network/virtualNetworks/virtualMachines/read", + "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/delete", + "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read", + "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write", + "Microsoft.Network/virtualNetworks/write", + "Microsoft.Resources/subscriptions/resourceGroups/read", + "Microsoft.Storage/storageAccounts/blobServices/containers/read", + "Microsoft.Storage/storageAccounts/blobServices/containers/write", + "Microsoft.Storage/storageAccounts/read", + "Microsoft.Storage/storageAccounts/write" + ], + "NotActions": [], + "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] + } + ``` + +
+ +3. Create the custom role, _Dynamic Placement Palette Deployer_, in Azure using the JSON file you created in the + previous step. Issue the following command to create the role. + + ```shell + az role definition create --role-definition @dynamic-permissions.json --output table + ``` + +4. Export the + [security principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/overview#security-principal) + object ID you want to use with Palette to a variable. + + ```shell + export ASSIGNEE="" + ``` + +5. Assign the role by creating a role assignment referencing the role definition _Dynamic Placement Palette Deployer_. + Use the command below to assign the role to the service principal. + + ```shell + az role assignment create --assignee $ASSIGNEE \ + --role "Dynamic Placement Palette Deployer" \ + --scope "/subscriptions/$SUBSCRIPTION_ID" + ``` + +### Validate + +1. Log in to the Azure portal. + +2. Navigate to the Microsoft Entra ID section. + +3. Review the role assignments to ensure the service principal has the `Dynamic Placement Palette Deployer` role + assigned. + +## Static Placement + +Choose static placement when you want Palette to use pre-existing network resource groups, VNets, subnets, and security +groups. For your convenience, we have provided the required permissions to deploy clusters using static placement in +various scenarios. + +### Prerequisites + +- Azure CLI installed on your local machine. Refer to the + [Azure CLI Install Guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) for installation instructions. + +- The Azure subscription ID you want to use with Palette. + +- The Security Principal Object ID you want to use with Palette. You can retrieve it from the Entra ID section of the + Azure Portal. + +- A terminal or command prompt to issue the Azure CLI commands. + +- The Azure Security Principal Object ID you want to use with Palette. The Security Principal Object ID can represent a + user, group, or service principal. + +### Create Role and Assign Permissions + +Select the tab below to view the required permissions and steps for the static placement scenario you want to use. + + + + + +1. Export your Azure subscription ID to a variable. + + ```shell + export SUBSCRIPTION_ID= + ``` + +2. Issue the following command to create a JSON file containing all the required permissions to deploy a cluster + statically in Palette. + + ```shell + cat << EOF > static-permissions.json + { + "Name": "Static Placement Palette Deployer", + "IsCustom": true, + "Description": "Can deploy Azure IaaS clusters using static placement with Palette.", + "Actions": [ + "Microsoft.Compute/disks/delete", + "Microsoft.Compute/disks/read", + "Microsoft.Compute/disks/write", + "Microsoft.Compute/galleries/images/versions/write", + "Microsoft.Compute/galleries/images/write", + "Microsoft.Compute/galleries/read", + "Microsoft.Compute/galleries/write", + "Microsoft.Compute/images/read", + "Microsoft.Compute/images/write", + "Microsoft.Compute/virtualMachines/delete", + "Microsoft.Compute/virtualMachines/extensions/delete", + "Microsoft.Compute/virtualMachines/extensions/read", + "Microsoft.Compute/virtualMachines/extensions/write", + "Microsoft.Compute/virtualMachines/read", + "Microsoft.Compute/virtualMachines/write", + "Microsoft.Network/loadBalancers/backendAddressPools/join/action", + "Microsoft.Network/loadBalancers/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/join/action", + "Microsoft.Network/loadBalancers/inboundNatRules/read", + "Microsoft.Network/loadBalancers/inboundNatRules/write", + "Microsoft.Network/loadBalancers/read", + "Microsoft.Network/loadBalancers/write", + "Microsoft.Network/networkInterfaces/delete", + "Microsoft.Network/networkInterfaces/join/action", + "Microsoft.Network/networkInterfaces/read", + "Microsoft.Network/networkInterfaces/write", + "Microsoft.Network/networkSecurityGroups/read", + "Microsoft.Network/networkSecurityGroups/securityRules/delete", + "Microsoft.Network/networkSecurityGroups/securityRules/read", + "Microsoft.Network/networkSecurityGroups/securityRules/write", + "Microsoft.Network/privateDnsZones/A/delete", + "Microsoft.Network/privateDnsZones/A/read", + "Microsoft.Network/privateDnsZones/A/write", + "Microsoft.Network/privateDnsZones/delete", + "Microsoft.Network/privateDnsZones/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", + "Microsoft.Network/privateDnsZones/write", + "Microsoft.Network/publicIPAddresses/delete", + "Microsoft.Network/publicIPAddresses/join/action", + "Microsoft.Network/publicIPAddresses/read", + "Microsoft.Network/publicIPAddresses/write", + "Microsoft.Network/routeTables/delete", + "Microsoft.Network/routeTables/read", + "Microsoft.Network/routeTables/write", + "Microsoft.Network/virtualNetworks/delete", + "Microsoft.Network/virtualNetworks/join/action", + "Microsoft.Network/virtualNetworks/join/action", + "Microsoft.Network/virtualNetworks/joinLoadBalancer/action", + "Microsoft.Network/virtualNetworks/peer/action", + "Microsoft.Network/virtualNetworks/subnets/delete", + "Microsoft.Network/virtualNetworks/subnets/joinLoadBalancer/action", + "Microsoft.Network/virtualNetworks/subnets/virtualMachines/read", + "Microsoft.Network/virtualNetworks/subnets/write", + "Microsoft.Network/virtualNetworks/virtualMachines/read", + "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/delete", + "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read", + "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write", + "Microsoft.Network/virtualNetworks/write", + "Microsoft.Resources/subscriptions/resourceGroups/read", + "Microsoft.Storage/storageAccounts/blobServices/containers/read", + "Microsoft.Storage/storageAccounts/blobServices/containers/write", + "Microsoft.Storage/storageAccounts/read", + "Microsoft.Storage/storageAccounts/write" + ], + "NotActions": [], + "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] + } + EOF + ``` + +3. Next, create a JSON file for the compute gallery scope permissions. Issue the following command to create the JSON + file. + + ```shell + cat << EOF > static-gallery-scope-permissions.json + { + "Name": "Static Placement Palette Deployer - Compute Gallery Scope", + "IsCustom": true, + "Description": "Can deploy the compute gallery level components of Azure IaaS clusters using static placement with Palette.", + "Actions": [ + "Microsoft.Compute/galleries/images/read", + "Microsoft.Compute/galleries/images/versions/read" + ], + "NotActions": [], + "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] + } + EOF + ``` + +4. Create a JSON file for the subnet scope permissions. + + ```shell + cat << EOF > static-subnet-scope-permissions.json + { + "Name": "Static Placement Palette Deployer - Subnet Scope", + "IsCustom": true, + "Description": "Can deploy the subnet level components of Azure IaaS clusters using static placement with Palette.", + "Actions": [ + "Microsoft.Network/virtualNetworks/subnets/join/action", + "Microsoft.Network/virtualNetworks/subnets/read" + ], + "NotActions": [], + "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] + } + EOF + ``` + +5. The last JSON file is for the virtual network scope permissions. Issue the following command to create the JSON + file. + + ```shell + cat << EOF > static-virtual-network-scope-permissions.json + { + "Name": "Static Placement Palette Deployer - Virtual Network Scope", + "IsCustom": true, + "Description": "Can deploy the virtual network level components of Azure IaaS clusters using static placement with Palette.", + "Actions": ["Microsoft.Network/virtualNetworks/read"], + "NotActions": [], + "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] + } + EOF + ``` + + +
+ Click here to view the raw JSON policies. + ```json + { + "Name": "Static Placement Palette Deployer", + "IsCustom": true, + "Description": "Can deploy Azure IaaS clusters using static placement with Palette.", + "Actions": [ + "Microsoft.Compute/disks/delete", + "Microsoft.Compute/disks/read", + "Microsoft.Compute/disks/write", + "Microsoft.Compute/galleries/images/versions/write", + "Microsoft.Compute/galleries/images/write", + "Microsoft.Compute/galleries/read", + "Microsoft.Compute/galleries/write", + "Microsoft.Compute/images/read", + "Microsoft.Compute/images/write", + "Microsoft.Compute/virtualMachines/delete", + "Microsoft.Compute/virtualMachines/extensions/delete", + "Microsoft.Compute/virtualMachines/extensions/read", + "Microsoft.Compute/virtualMachines/extensions/write", + "Microsoft.Compute/virtualMachines/read", + "Microsoft.Compute/virtualMachines/write", + "Microsoft.Network/loadBalancers/backendAddressPools/join/action", + "Microsoft.Network/loadBalancers/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/join/action", + "Microsoft.Network/loadBalancers/inboundNatRules/read", + "Microsoft.Network/loadBalancers/inboundNatRules/write", + "Microsoft.Network/loadBalancers/read", + "Microsoft.Network/loadBalancers/write", + "Microsoft.Network/networkInterfaces/delete", + "Microsoft.Network/networkInterfaces/join/action", + "Microsoft.Network/networkInterfaces/read", + "Microsoft.Network/networkInterfaces/write", + "Microsoft.Network/networkSecurityGroups/read", + "Microsoft.Network/networkSecurityGroups/securityRules/delete", + "Microsoft.Network/networkSecurityGroups/securityRules/read", + "Microsoft.Network/networkSecurityGroups/securityRules/write", + "Microsoft.Network/privateDnsZones/A/delete", + "Microsoft.Network/privateDnsZones/A/read", + "Microsoft.Network/privateDnsZones/A/write", + "Microsoft.Network/privateDnsZones/delete", + "Microsoft.Network/privateDnsZones/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", + "Microsoft.Network/privateDnsZones/write", + "Microsoft.Network/publicIPAddresses/delete", + "Microsoft.Network/publicIPAddresses/join/action", + "Microsoft.Network/publicIPAddresses/read", + "Microsoft.Network/publicIPAddresses/write", + "Microsoft.Network/routeTables/delete", + "Microsoft.Network/routeTables/read", + "Microsoft.Network/routeTables/write", + "Microsoft.Network/virtualNetworks/delete", + "Microsoft.Network/virtualNetworks/join/action", + "Microsoft.Network/virtualNetworks/join/action", + "Microsoft.Network/virtualNetworks/joinLoadBalancer/action", + "Microsoft.Network/virtualNetworks/peer/action", + "Microsoft.Network/virtualNetworks/subnets/delete", + "Microsoft.Network/virtualNetworks/subnets/joinLoadBalancer/action", + "Microsoft.Network/virtualNetworks/subnets/virtualMachines/read", + "Microsoft.Network/virtualNetworks/subnets/write", + "Microsoft.Network/virtualNetworks/virtualMachines/read", + "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/delete", + "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read", + "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write", + "Microsoft.Network/virtualNetworks/write", + "Microsoft.Resources/subscriptions/resourceGroups/read", + "Microsoft.Storage/storageAccounts/blobServices/containers/read", + "Microsoft.Storage/storageAccounts/blobServices/containers/write", + "Microsoft.Storage/storageAccounts/read", + "Microsoft.Storage/storageAccounts/write" + ], + "NotActions": [], + "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] + } + ``` + ```json + { + "Name": "Static Placement Palette Deployer - Compute Gallery Scope", + "IsCustom": true, + "Description": "Can deploy the compute gallery level components of Azure IaaS clusters using static placement with Palette.", + "Actions": [ + "Microsoft.Compute/galleries/images/read", + "Microsoft.Compute/galleries/images/versions/read" + ], + "NotActions": [], + "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] + } + ``` + + ```json + { + "Name": "Static Placement Palette Deployer - Subnet Scope", + "IsCustom": true, + "Description": "Can deploy the subnet level components of Azure IaaS clusters using static placement with Palette.", + "Actions": [ + "Microsoft.Network/virtualNetworks/subnets/join/action", + "Microsoft.Network/virtualNetworks/subnets/read" + ], + "NotActions": [], + "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] + } + ``` + + ```json + { + "Name": "Static Placement Palette Deployer - Virtual Network Scope", + "IsCustom": true, + "Description": "Can deploy the virtual network level components of Azure IaaS clusters using static placement with Palette.", + "Actions": ["Microsoft.Network/virtualNetworks/read"], + "NotActions": [], + "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] + } + ``` + +
+ +6. Create a role for each of the JSON files you created in the previous steps. Issue the following commands to create + the roles. + + ```shell + az role definition create --role-definition @static-permissions.json --output table + az role definition create --role-definition @static-gallery-scope-permissions.json --output table + az role definition create --role-definition @static-subnet-scope-permissions.json --output table + az role definition create --role-definition @static-virtual-network-scope-permissions.json --output table + ``` + +7. Export resource group, virtual network, subnet, and compute gallery names you desire Palette to use to a set of + variables. + + ```shell + export RESOURCE_GROUP_NAME= + export VIRTUAL_NETWORK_NAME= + export SUBNET_NAME= + export COMPUTE_GALLERY_NAME= + ``` + +8. Export the + [security principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/overview#security-principal) + object ID you want to use with Palette to a variable. + + ```shell + export ASSIGNEE="" + ``` + +9. Assign the roles to the service principal. Use the following commands to assign the roles. + + ```json + az role assignment create --assignee $ASSIGNEE \ + --role "Static Placement Palette Deployer - Resource Group Scope" \ + --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME" + ``` + + ```json + az role assignment create --assignee $ASSIGNEE \ + --role "Static Placement Palette Deployer - Virtual Network Scope" \ + --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME/providers/Microsoft.Network/virtualNetworks/$VIRTUAL_NETWORK_NAME" + ``` + + ```json + az role assignment create --assignee $ASSIGNEE \ + --role "Static Placement Palette Deployer - Subnet Scope" \ + --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME/providers/Microsoft.Network/virtualNetworks/$VIRTUAL_NETWORK_NAME/subnets/$SUBNET_NAME" + ``` + + ```json + az role assignment create --assignee $ASSIGNEE \ + --role "Static Placement Palette Deployer - Compute Gallery Scope" \ + --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME/providers/Microsoft.Compute/galleries/$COMPUTE_GALLERY_NAME" + ``` + +
+ + +1. Export your Azure subscription ID to a variable. + + ```shell + export SUBSCRIPTION_ID= + ``` + +2. Issue the following command to create a JSON file containing all the required permissions to deploy a cluster + statically in Palette. + + ```shell + cat << EOF > static-placement-permissions.json + { + "Name": "Static Placement Palette Deployer", + "IsCustom": true, + "Description": "Can deploy Azure IaaS clusters using static placement with Palette.", + "Actions": [ + "Microsoft.Compute/disks/delete", + "Microsoft.Compute/disks/read", + "Microsoft.Compute/disks/write", + "Microsoft.Compute/virtualMachines/delete", + "Microsoft.Compute/virtualMachines/extensions/delete", + "Microsoft.Compute/virtualMachines/extensions/read", + "Microsoft.Compute/virtualMachines/extensions/write", + "Microsoft.Compute/virtualMachines/read", + "Microsoft.Compute/virtualMachines/write", + "Microsoft.Network/loadBalancers/backendAddressPools/join/action", + "Microsoft.Network/loadBalancers/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/join/action", + "Microsoft.Network/loadBalancers/inboundNatRules/read", + "Microsoft.Network/loadBalancers/inboundNatRules/write", + "Microsoft.Network/loadBalancers/read", + "Microsoft.Network/loadBalancers/write", + "Microsoft.Network/networkInterfaces/delete", + "Microsoft.Network/networkInterfaces/join/action", + "Microsoft.Network/networkInterfaces/read", + "Microsoft.Network/networkInterfaces/write", + "Microsoft.Network/networkSecurityGroups/read", + "Microsoft.Network/networkSecurityGroups/securityRules/delete", + "Microsoft.Network/networkSecurityGroups/securityRules/read", + "Microsoft.Network/networkSecurityGroups/securityRules/write", + "Microsoft.Network/privateDnsZones/A/delete", + "Microsoft.Network/privateDnsZones/A/read", + "Microsoft.Network/privateDnsZones/A/write", + "Microsoft.Network/privateDnsZones/delete", + "Microsoft.Network/privateDnsZones/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", + "Microsoft.Network/privateDnsZones/write", + "Microsoft.Network/publicIPAddresses/delete", + "Microsoft.Network/publicIPAddresses/join/action", + "Microsoft.Network/publicIPAddresses/read", + "Microsoft.Network/publicIPAddresses/write", + "Microsoft.Network/routeTables/delete", + "Microsoft.Network/routeTables/read", + "Microsoft.Network/routeTables/write", + "Microsoft.Network/virtualNetworks/join/action", + "Microsoft.Resources/subscriptions/resourceGroups/read", + "Microsoft.Network/virtualNetworks/read", + "Microsoft.Network/virtualNetworks/subnets/join/action", + "Microsoft.Network/virtualNetworks/subnets/read", + "Microsoft.Compute/galleries/images/read", + "Microsoft.Compute/galleries/images/versions/read" + ], + "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] + } + EOF + ``` + + +
+ Click here to view the raw JSON policy. + ```json + { + "Name": "Static Placement Palette Deployer", + "IsCustom": true, + "Description": "Can deploy Azure IaaS clusters using static placement with Palette.", + "Actions": [ + "Microsoft.Compute/disks/delete", + "Microsoft.Compute/disks/read", + "Microsoft.Compute/disks/write", + "Microsoft.Compute/virtualMachines/delete", + "Microsoft.Compute/virtualMachines/extensions/delete", + "Microsoft.Compute/virtualMachines/extensions/read", + "Microsoft.Compute/virtualMachines/extensions/write", + "Microsoft.Compute/virtualMachines/read", + "Microsoft.Compute/virtualMachines/write", + "Microsoft.Network/loadBalancers/backendAddressPools/join/action", + "Microsoft.Network/loadBalancers/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/join/action", + "Microsoft.Network/loadBalancers/inboundNatRules/read", + "Microsoft.Network/loadBalancers/inboundNatRules/write", + "Microsoft.Network/loadBalancers/read", + "Microsoft.Network/loadBalancers/write", + "Microsoft.Network/networkInterfaces/delete", + "Microsoft.Network/networkInterfaces/join/action", + "Microsoft.Network/networkInterfaces/read", + "Microsoft.Network/networkInterfaces/write", + "Microsoft.Network/networkSecurityGroups/read", + "Microsoft.Network/networkSecurityGroups/securityRules/delete", + "Microsoft.Network/networkSecurityGroups/securityRules/read", + "Microsoft.Network/networkSecurityGroups/securityRules/write", + "Microsoft.Network/privateDnsZones/A/delete", + "Microsoft.Network/privateDnsZones/A/read", + "Microsoft.Network/privateDnsZones/A/write", + "Microsoft.Network/privateDnsZones/delete", + "Microsoft.Network/privateDnsZones/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", + "Microsoft.Network/privateDnsZones/write", + "Microsoft.Network/publicIPAddresses/delete", + "Microsoft.Network/publicIPAddresses/join/action", + "Microsoft.Network/publicIPAddresses/read", + "Microsoft.Network/publicIPAddresses/write", + "Microsoft.Network/routeTables/delete", + "Microsoft.Network/routeTables/read", + "Microsoft.Network/routeTables/write", + "Microsoft.Network/virtualNetworks/join/action", + "Microsoft.Resources/subscriptions/resourceGroups/read", + "Microsoft.Network/virtualNetworks/read", + "Microsoft.Network/virtualNetworks/subnets/join/action", + "Microsoft.Network/virtualNetworks/subnets/read", + "Microsoft.Compute/galleries/images/read", + "Microsoft.Compute/galleries/images/versions/read" + ], + "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] + } + ``` +
+ +3. Create a role for the JSON file you created in the previous step. Issue the following command to create the role. + + ```shell + az role definition create --role-definition @static-placement-permissions.json --output table + ``` + +4. Export the + [security principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/overview#security-principal) + object ID and resource group name you want to use with Palette to a set of variables. + + ```shell + export ASSIGNEE="" + export RESOURCE_GROUP_NAME= + ``` + +5. Assign the role to the service principal. Use the following command to assign the role. + + ```json + az role assignment create --assignee $ASSIGNEE \ + --role "Static Placement Palette Deployer" \ + --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME" + ``` + +
+ + + +1. Export your Azure subscription ID to a variable. + + ```shell + export SUBSCRIPTION_ID= + ``` + +2. Issue the following command to create a JSON file containing all the required permissions to deploy a cluster + statically in Palette. + + ```shell + cat << EOF > static-placement-permissions.json + { + "Name": "Static Placement Palette Deployer", + "IsCustom": true, + "Description": "Can deploy Azure IaaS clusters using static placement with Palette.", + "Actions": [ + "Microsoft.Compute/disks/delete", + "Microsoft.Compute/disks/read", + "Microsoft.Compute/disks/write", + "Microsoft.Compute/virtualMachines/delete", + "Microsoft.Compute/virtualMachines/extensions/delete", + "Microsoft.Compute/virtualMachines/extensions/read", + "Microsoft.Compute/virtualMachines/extensions/write", + "Microsoft.Compute/virtualMachines/read", + "Microsoft.Compute/virtualMachines/write", + "Microsoft.Network/loadBalancers/backendAddressPools/join/action", + "Microsoft.Network/loadBalancers/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/join/action", + "Microsoft.Network/loadBalancers/inboundNatRules/read", + "Microsoft.Network/loadBalancers/inboundNatRules/write", + "Microsoft.Network/loadBalancers/read", + "Microsoft.Network/loadBalancers/write", + "Microsoft.Network/networkInterfaces/delete", + "Microsoft.Network/networkInterfaces/join/action", + "Microsoft.Network/networkInterfaces/read", + "Microsoft.Network/networkInterfaces/write", + "Microsoft.Network/networkSecurityGroups/read", + "Microsoft.Network/networkSecurityGroups/securityRules/delete", + "Microsoft.Network/networkSecurityGroups/securityRules/read", + "Microsoft.Network/networkSecurityGroups/securityRules/write", + "Microsoft.Network/privateDnsZones/A/delete", + "Microsoft.Network/privateDnsZones/A/read", + "Microsoft.Network/privateDnsZones/A/write", + "Microsoft.Network/privateDnsZones/delete", + "Microsoft.Network/privateDnsZones/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", + "Microsoft.Network/privateDnsZones/write", + "Microsoft.Network/publicIPAddresses/delete", + "Microsoft.Network/publicIPAddresses/join/action", + "Microsoft.Network/publicIPAddresses/read", + "Microsoft.Network/publicIPAddresses/write", + "Microsoft.Network/routeTables/delete", + "Microsoft.Network/routeTables/read", + "Microsoft.Network/routeTables/write", + "Microsoft.Network/virtualNetworks/join/action", + "Microsoft.Resources/subscriptions/resourceGroups/read", + "Microsoft.Network/virtualNetworks/read", + "Microsoft.Network/virtualNetworks/subnets/join/action", + "Microsoft.Network/virtualNetworks/subnets/read", + "Microsoft.Compute/galleries/images/read", + "Microsoft.Compute/galleries/images/versions/read" + ], + "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] + } + EOF + ``` + + + +
+ Click here to view the raw JSON policy. + ```json + { + "Name": "Static Placement Palette Deployer", + "IsCustom": true, + "Description": "Can deploy Azure IaaS clusters using static placement with Palette.", + "Actions": [ + "Microsoft.Compute/disks/delete", + "Microsoft.Compute/disks/read", + "Microsoft.Compute/disks/write", + "Microsoft.Compute/virtualMachines/delete", + "Microsoft.Compute/virtualMachines/extensions/delete", + "Microsoft.Compute/virtualMachines/extensions/read", + "Microsoft.Compute/virtualMachines/extensions/write", + "Microsoft.Compute/virtualMachines/read", + "Microsoft.Compute/virtualMachines/write", + "Microsoft.Network/loadBalancers/backendAddressPools/join/action", + "Microsoft.Network/loadBalancers/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/join/action", + "Microsoft.Network/loadBalancers/inboundNatRules/read", + "Microsoft.Network/loadBalancers/inboundNatRules/write", + "Microsoft.Network/loadBalancers/read", + "Microsoft.Network/loadBalancers/write", + "Microsoft.Network/networkInterfaces/delete", + "Microsoft.Network/networkInterfaces/join/action", + "Microsoft.Network/networkInterfaces/read", + "Microsoft.Network/networkInterfaces/write", + "Microsoft.Network/networkSecurityGroups/read", + "Microsoft.Network/networkSecurityGroups/securityRules/delete", + "Microsoft.Network/networkSecurityGroups/securityRules/read", + "Microsoft.Network/networkSecurityGroups/securityRules/write", + "Microsoft.Network/privateDnsZones/A/delete", + "Microsoft.Network/privateDnsZones/A/read", + "Microsoft.Network/privateDnsZones/A/write", + "Microsoft.Network/privateDnsZones/delete", + "Microsoft.Network/privateDnsZones/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", + "Microsoft.Network/privateDnsZones/write", + "Microsoft.Network/publicIPAddresses/delete", + "Microsoft.Network/publicIPAddresses/join/action", + "Microsoft.Network/publicIPAddresses/read", + "Microsoft.Network/publicIPAddresses/write", + "Microsoft.Network/routeTables/delete", + "Microsoft.Network/routeTables/read", + "Microsoft.Network/routeTables/write", + "Microsoft.Network/virtualNetworks/join/action", + "Microsoft.Resources/subscriptions/resourceGroups/read", + "Microsoft.Network/virtualNetworks/read", + "Microsoft.Network/virtualNetworks/subnets/join/action", + "Microsoft.Network/virtualNetworks/subnets/read", + "Microsoft.Compute/galleries/images/read", + "Microsoft.Compute/galleries/images/versions/read" + ], + "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] + } + ``` +
+ +3. Create a role using the JSON file you created in the previous step. Issue the following command to create the role. + + ```shell + az role definition create --role-definition @static-placement-permissions.json --output table + ``` + +4. Export the + [security principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/overview#security-principal) + object ID you want to use with Palette to a variable. + + ```shell + export ASSIGNEE="" + ``` + +5. Assign the role to the service principal. Use the following command to assign the role. + + ```shell + az role assignment create --assignee $ASSIGNEE \ + --role "Static Placement Palette Deployer" \ + --scope "/subscriptions/$SUBSCRIPTION_ID" + ``` + +
+ +
+ +### Validate + +1. Log in to the Azure portal. + +2. Navigate to the Microsoft Entra ID section. + +3. Review the role, or roles if you created multiple. Review the role assignments to ensure the service principal has + the correct permissions assigned. diff --git a/docs/docs-content/clusters/public-cloud/aws/required-iam-policies.md b/docs/docs-content/clusters/public-cloud/aws/required-iam-policies.md index 75b8bfac0a..16aa1b8cf7 100644 --- a/docs/docs-content/clusters/public-cloud/aws/required-iam-policies.md +++ b/docs/docs-content/clusters/public-cloud/aws/required-iam-policies.md @@ -8,7 +8,7 @@ sidebar_position: 50 --- Palette requires proper Amazon Web Services (AWS) permissions to operate and perform actions on your behalf. The -following policies include all the permissions needed for cluster provisioning with Palette. +following policies include all the permissions needed for Palette to deploy and manage clusters on AWS. - **PaletteControllerPolicy** @@ -31,6 +31,16 @@ into a custom-managed policy. You can learn more about AWS IAM limits in the ::: +If you want to narrow down the IAM permissions, you can use the [Minimum Permissions](#minimum-permissions) policies. +These policies are designed to follow the +[principle of least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege). +You can also use the Static Policy for deploying clusters within an existing VPC without provisioning or deleting +foundational network resources. + +## Core IAM Policies + +The four core IAM policies are required for Palette to operate. + @@ -368,7 +378,7 @@ into a custom-managed policy. You can learn more about AWS IAM limits in the -## Controllers EKS Policy +### Controllers EKS Policy If you plan to deploy host clusters to AWS EKS, make sure to attach the **PaletteControllersEKSPolicy**. @@ -512,124 +522,153 @@ If you plan to deploy host clusters to AWS EKS, make sure to attach the **Palett } ``` -## Restricting Palette VPC Permissions +## Minimum Permissions + +The following policies are designed from the +[principle of least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege). +You can use these policies to narrow the permissions Palette requires to operate instead of using the +[Core IAM Policies](#core-iam-policies). + +The **Minimum Dynamic Permissions** grants broad permissions across EC2, VPC, and Elastic Container Registry (ECR), +allowing complete control over the lifecycle of virtual network and compute resources. This includes creating, +modifying, and deleting VPCs, subnets, internet gateways, and route tables, as well as managing Docker images in ECR. -You can choose to have Palette operate in a static or dynamic environment. You can configure Palette to perform an AWS -cluster creation into an existing VPC. The following policy allows Palette to operate but restricts its access to the -[Principle of Least Privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege). +In contrast, the **Static Policy** is more restrictive, omitting permissions for creating and deleting core VPC +components and excluding ECR access. However, it includes additional permissions for managing EC2 volumes and accessing +S3 bucket encryption configurations. + +Both policies provide similar access to CloudFormation, Elastic Load Balancing, IAM roles, and Secrets Manager. However, +the **Dynamic Policy** supports full EC2 and network resource control, while the **Static Policy** is tailored for +managing existing infrastructure without provisioning or deleting foundational network resources. -This is a policy for those who want to restrict Palette to a single VPC and not give Palette access to create or delete -VPCs. - -### Minimum Dynamic Permissions +The following policy allows Palette to operate and create VPC resources as needed while retaining minimal permissions +for deploying clusters through Palette. ```json { "Version": "2012-10-17", "Statement": [ { - "Sid": "VisualEditor0", + "Sid": "PalettedynamicPermissions", "Effect": "Allow", "Action": [ - "ec2:AuthorizeSecurityGroupIngress", - "ec2:DescribeInstances", - "iam:RemoveRoleFromInstanceProfile", + "autoscaling:StartInstanceRefresh", + "cloudformation:CreateStack", + "cloudformation:DescribeStacks", + "cloudformation:UpdateStack", + "ec2:AllocateAddress", + "ec2:AssociateAddress", "ec2:AttachInternetGateway", - "iam:AddRoleToInstanceProfile", + "ec2:AuthorizeSecurityGroupIngress", + "ec2:CreateInternetGateway", + "ec2:CreateNatGateway", + "ec2:CreateRoute", + "ec2:CreateRouteTable", + "ec2:CreateSecurityGroup", + "ec2:CreateSnapshot", + "ec2:CreateSubnet", + "ec2:CreateTags", + "ec2:CreateVpc", + "ec2:DeleteInternetGateway", + "ec2:DeleteNatGateway", + "ec2:DeleteNetworkInterface", + "ec2:DeleteRoute", "ec2:DeleteRouteTable", - "ec2:AssociateRouteTable", + "ec2:DeleteSecurityGroup", + "ec2:DeleteSnapshot", + "ec2:DeleteSubnet", + "ec2:DeleteTags", + "ec2:DeleteVpc", + "ec2:DescribeAddresses", + "ec2:DescribeAvailabilityZones", + "ec2:DescribeDhcpOptions", + "ec2:DescribeImages", + "ec2:DescribeInstances", "ec2:DescribeInternetGateways", - "ec2:CreateRoute", - "ec2:CreateInternetGateway", - "ec2:DescribeVolumes", + "ec2:DescribeIpv6Pools", "ec2:DescribeKeyPairs", + "ec2:DescribeNatGateways", "ec2:DescribeNetworkAcls", + "ec2:DescribeNetworkInterfaceAttribute", + "ec2:DescribeNetworkInterfaces", + "ec2:DescribeRegions", "ec2:DescribeRouteTables", - "ec2:CreateTags", - "ec2:CreateRouteTable", - "ec2:RunInstances", - "ec2:ModifyInstanceAttribute", - "ec2:TerminateInstances", - "ec2:DetachInternetGateway", - "ec2:DisassociateRouteTable", - "ec2:RevokeSecurityGroupIngress", - "ec2:DescribeIpv6Pools", - "ec2:DeleteVpc", - "ec2:CreateSubnet", + "ec2:DescribeSecurityGroups", + "ec2:DescribeSnapshots", "ec2:DescribeSubnets", - "iam:CreateInstanceProfile", - "ec2:DisassociateAddress", - "ec2:DescribeAddresses", - "ec2:CreateNatGateway", - "ec2:DescribeRegions", - "ec2:CreateVpc", - "ec2:DescribeDhcpOptions", + "ec2:DescribeTags", + "ec2:DescribeVolumes", + "ec2:DescribeVolumesModifications", "ec2:DescribeVpcAttribute", - "ec2:DescribeNetworkInterfaces", - "ec2:DescribeAvailabilityZones", - "ec2:DescribeNetworkInterfaceAttribute", - "ec2:CreateSecurityGroup", + "ec2:DescribeVpcs", + "ec2:DisassociateAddress", + "ec2:DisassociateRouteTable", + "ec2:ModifyInstanceAttribute", + "ec2:ModifyNetworkInterfaceAttribute", + "ec2:ModifySubnetAttribute", "ec2:ModifyVpcAttribute", - "iam:DeleteInstanceProfile", "ec2:ReleaseAddress", - "iam:GetInstanceProfile", - "ec2:DescribeTags", - "ec2:DeleteRoute", - "ec2:DescribeNatGateways", - "ec2:DescribeIpamPools", - "ec2:AllocateAddress", - "ec2:DescribeSecurityGroups", - "ec2:DescribeImages", - "ec2:DescribeVpcs", + "ec2:ReplaceRoute", + "ec2:RevokeSecurityGroupIngress", + "ec2:RunInstances", + "ec2:TerminateInstances", + "ecr:BatchCheckLayerAvailability", + "ecr:BatchGetImage", + "ecr:DescribeRepositories", + "ecr:GetAuthorizationToken", + "ecr:GetDownloadUrlForLayer", + "ecr:GetRepositoryPolicy", + "ecr:ListImages", + "eks:DescribeCluster", + "eks:ListClusters", + "elasticloadbalancing:ConfigureHealthCheck", + "elasticloadbalancing:CreateLoadBalancer", "elasticloadbalancing:DeleteLoadBalancer", - "elasticloadbalancing:DescribeLoadBalancers", + "elasticloadbalancing:DeregisterInstancesFromLoadBalancer", "elasticloadbalancing:DescribeLoadBalancerAttributes", - "elasticloadbalancing:CreateLoadBalancer", - "elasticloadbalancing:ModifyLoadBalancerAttributes", + "elasticloadbalancing:DescribeLoadBalancers", + "elasticloadbalancing:DescribeTargetHealth", "elasticloadbalancing:DescribeTags", + "elasticloadbalancing:ModifyLoadBalancerAttributes", + "elasticloadbalancing:RegisterInstancesWithLoadBalancer", + "iam:AddRoleToInstanceProfile", + "iam:AddUserToGroup", + "iam:AttachGroupPolicy", + "iam:CreateGroup", + "iam:CreateInstanceProfile", + "iam:CreatePolicy", + "iam:CreatePolicyVersion", + "iam:CreateUser", + "iam:DeleteGroup", + "iam:DeleteInstanceProfile", + "iam:DeletePolicy", + "iam:DeletePolicyVersion", + "iam:DetachGroupPolicy", + "iam:GetGroup", + "iam:GetInstanceProfile", + "iam:GetPolicy", + "iam:GetUser", + "iam:ListPolicies", + "iam:ListPolicyVersions", + "iam:PassRole", + "iam:RemoveRoleFromInstanceProfile", + "iam:RemoveUserFromGroup", + "pricing:GetProducts", "secretsmanager:CreateSecret", "secretsmanager:DeleteSecret", - "secretsmanager:TagResource", "secretsmanager:GetSecretValue", - "autoscaling:StartInstanceRefresh", - "elasticloadbalancing:DeregisterInstancesFromLoadBalancer", - "elasticloadbalancing:RegisterInstancesWithLoadBalancer", - "eks:DescribeCluster", - "eks:ListClusters", - "cloudformation:CreateStack", - "cloudformation:DescribeStacks", - "cloudformation:UpdateStack", - "ecr:GetAuthorizationToken", - "iam:PassRole", - "elasticloadbalancing:ConfigureHealthCheck", - "elasticloadbalancing:DescribeTargetHealth", - "ecr:BatchCheckLayerAvailability", - "ecr:GetDownloadUrlForLayer", - "ecr:GetRepositoryPolicy", - "ecr:DescribeRepositories", - "ecr:ListImages", - "ecr:BatchGetImage", - "ec2:DeleteInternetGateway", - "ec2:DeleteNatGateway", - "ec2:DeleteNetworkInterface", - "ec2:DeleteSecurityGroup", - "ec2:DeleteSubnet", - "ec2:DeleteTags", + "secretsmanager:TagResource", "ssm:UpdateInstanceInformation", "ssmmessages:CreateControlChannel", "ssmmessages:CreateDataChannel", "ssmmessages:OpenControlChannel", "ssmmessages:OpenDataChannel", - "pricing:GetProducts", "sts:AssumeRole", - "ec2:ReplaceRoute", - "ec2:ModifyNetworkInterfaceAttribute", - "ec2:AssociateAddress", - "tag:GetResources", - "ec2:ModifySubnetAttribute" + "sts:GetServiceBearerToken", + "tag:GetResources" ], "Resource": "*" }, @@ -646,92 +685,109 @@ VPCs. -This is a policy for those who want to restrict Palette to a single VPC and not give Palette access to create or delete -VPCs. - -### Minimum Static Permissions +The following policy allows Palette to operate within an existing VPC while retaining minimal permissions for deploying +clusters through Palette. ```json { "Version": "2012-10-17", "Statement": [ { - "Sid": "VisualEditor0", + "Sid": "PaletteStaticPermissions", "Effect": "Allow", "Action": [ + "autoscaling:StartInstanceRefresh", + "cloudformation:CreateStack", + "cloudformation:DescribeStacks", + "cloudformation:UpdateStack", + "ec2:AllocateAddress", + "ec2:AssociateAddress", + "ec2:AttachVolume", "ec2:AuthorizeSecurityGroupIngress", + "ec2:CreateSecurityGroup", + "ec2:CreateSnapshot", + "ec2:CreateTags", + "ec2:DeleteNetworkInterface", + "ec2:DeleteSecurityGroup", + "ec2:DeleteSnapshot", + "ec2:DeleteTags", + "ec2:DescribeAddresses", + "ec2:DescribeAvailabilityZones", + "ec2:DescribeImages", "ec2:DescribeInstances", - "iam:RemoveRoleFromInstanceProfile", - "pricing:GetProducts", - "sts:AssumeRole", - "ec2:DescribeRegions", "ec2:DescribeKeyPairs", - "ec2:DescribeVpcs", - "ec2:DescribeVpcAttribute", - "ec2:DescribeSubnets", - "cloudformation:DescribeStacks", - "cloudformation:CreateStack", - "cloudformation:UpdateStack", - "ec2:DescribeRouteTables", "ec2:DescribeNatGateways", + "ec2:DescribeNetworkInterfaceAttribute", + "ec2:DescribeNetworkInterfaces", + "ec2:DescribeRegions", + "ec2:DescribeRouteTables", "ec2:DescribeSecurityGroups", + "ec2:DescribeSnapshots", + "ec2:DescribeSubnets", + "ec2:DescribeTags", + "ec2:DescribeVpcs", + "ec2:DescribeVolumes", + "ec2:DescribeVolumesModifications", + "ec2:DisassociateAddress", + "ec2:ModifyInstanceAttribute", + "ec2:ModifyNetworkInterfaceAttribute", + "ec2:ModifyVpcAttribute", + "ec2:ModifyVolume", + "ec2:RevokeSecurityGroupIngress", + "ec2:ReleaseAddress", + "ec2:RunInstances", + "ec2:TerminateInstances", + "ec2:DetachVolume", + "ec2:DetachInternetGateway", + "eks:DescribeCluster", + "eks:ListClusters", + "elasticloadbalancing:AddTags", + "elasticloadbalancing:ConfigureHealthCheck", + "elasticloadbalancing:CreateLoadBalancer", + "elasticloadbalancing:DeleteLoadBalancer", + "elasticloadbalancing:DeregisterInstancesFromLoadBalancer", "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:DescribeLoadBalancerAttributes", "elasticloadbalancing:DescribeTags", - "secretsmanager:CreateSecret", - "secretsmanager:TagResource", - "secretsmanager:GetSecretValue", - "secretsmanager:DeleteSecret", - "iam:GetInstanceProfile", + "elasticloadbalancing:DescribeTargetHealth", + "elasticloadbalancing:ModifyLoadBalancerAttributes", + "elasticloadbalancing:RegisterInstancesWithLoadBalancer", + "elasticloadbalancing:DetachLoadBalancerFromSubnets", "iam:AddRoleToInstanceProfile", + "iam:AddUserToGroup", + "iam:AttachGroupPolicy", + "iam:CreateGroup", "iam:CreateInstanceProfile", + "iam:CreatePolicy", + "iam:CreatePolicyVersion", + "iam:CreateUser", + "iam:DeleteGroup", "iam:DeleteInstanceProfile", - "ec2:RunInstances", - "ec2:ModifyInstanceAttribute", - "ec2:TerminateInstances", - "autoscaling:StartInstanceRefresh", - "elasticloadbalancing:DeregisterInstancesFromLoadBalancer", - "elasticloadbalancing:RegisterInstancesWithLoadBalancer", - "elasticloadbalancing:AddTags", + "iam:DeletePolicy", + "iam:DeletePolicyVersion", + "iam:DetachGroupPolicy", + "iam:GetGroup", + "iam:GetInstanceProfile", + "iam:GetPolicy", + "iam:GetUser", + "iam:ListPolicies", + "iam:ListPolicyVersions", + "iam:RemoveRoleFromInstanceProfile", + "iam:RemoveUserFromGroup", + "pricing:GetProducts", + "s3:GetEncryptionConfiguration", + "secretsmanager:CreateSecret", + "secretsmanager:DeleteSecret", + "secretsmanager:GetSecretValue", + "secretsmanager:TagResource", "ssm:UpdateInstanceInformation", - "ec2:DescribeAvailabilityZones", - "ec2:DescribeTags", - "eks:DescribeCluster", - "eks:ListClusters", - "ec2:CreateSecurityGroup", - "ec2:DeleteSecurityGroup", - "ec2:DeleteTags", - "ec2:RevokeSecurityGroupIngress", "ssmmessages:CreateControlChannel", "ssmmessages:CreateDataChannel", "ssmmessages:OpenControlChannel", "ssmmessages:OpenDataChannel", - "elasticloadbalancing:ConfigureHealthCheck", - "elasticloadbalancing:DescribeTargetHealth", - "ec2:CreateTags", - "ec2:DescribeNetworkInterfaces", - "elasticloadbalancing:DeleteLoadBalancer", - "elasticloadbalancing:CreateLoadBalancer", - "elasticloadbalancing:ModifyLoadBalancerAttributes", - "ec2:DisassociateAddress", - "ec2:DescribeAddresses", - "ec2:DescribeVolumes", - "ec2:DescribeImages", - "ec2:ModifyVpcAttribute", - "s3:GetEncryptionConfiguration", - "ec2:ModifyVolume", - "ec2:AttachVolume", - "ec2:DescribeVolumesModifications", - "ec2:DetachVolume", - "elasticloadbalancing:DetachLoadBalancerFromSubnets", - "ec2:DetachInternetGateway", - "ec2:DeleteNetworkInterface", - "tag:GetResources", - "ec2:ReleaseAddress", - "ec2:ModifyNetworkInterfaceAttribute", - "ec2:DescribeNetworkInterfaceAttribute", - "ec2:AllocateAddress", - "ec2:AssociateAddress" + "sts:AssumeRole", + "sts:GetServiceBearerToken", + "tag:GetResources" ], "Resource": "*" }, @@ -748,17 +804,23 @@ VPCs. -:::info +:::warning The following are important points to be aware of. -- Ensure that the role created contain all the policies defined above. +- These permissions specified do not include all the permissions required for all possible use cases and for taking full + advantage of all Palette features. Additional permissions may be required based on the specific use case. + +- Ensure that the IAM Role or IAM User created contain all the core policies defined above, or one of the minimum + permissions policies. - These IAM policies cannot be used as an inline policy, as it exceeds the 2048 non-whitespace character limit by AWS. + Break policy into multiple inline policies or create new managed policies. + +- The following IAM warning is expected and can be ignored: -- The following warning is expected and can be ignored: These policies define some actions, resources, or conditions - that do not provide permissions. To grant access, policies must have an action that has an applicable resource or - condition. +> These policies define some actions, resources, or conditions that do not provide permissions. _To grant access, +> policies must have an action that has an applicable resource or condition_. ::: diff --git a/docs/docs-content/clusters/public-cloud/azure/required-permissions.md b/docs/docs-content/clusters/public-cloud/azure/required-permissions.md index c64ea7db20..484a22baf4 100644 --- a/docs/docs-content/clusters/public-cloud/azure/required-permissions.md +++ b/docs/docs-content/clusters/public-cloud/azure/required-permissions.md @@ -27,9 +27,15 @@ custom role to provide only those permissions that Palette requires. We support the following use cases: -- [Dynamic Placement](#dynamic-placement) - Palette creates network resources required for your cluster. +- [IaaS Static Placement](#iaas-static-placement) - Palette deploys clusters on the pre-existing network resources you + specify. -- [Static Placement](#static-placement) - Palette deploys clusters on the pre-existing network resources you specify. +- [IaaS Dynamic Placement](#iaas-dynamic-placement) - Palette creates network resources required for your cluster. + +- [AKS Static Placement](#aks-static-placement) - Palette deploys AKS clusters on the pre-existing network resources you + specify. + +- [AKS Dynamic Placement](#aks-dynamic-placement) - Palette creates resources required for your AKS cluster. :::tip @@ -49,12 +55,20 @@ You can use the Validator with the Palette CLI. Check out the Palette CLI Review the sections below to learn how to create a custom role in Azure and assign it to the service principal you want to use with Palette. -## Dynamic Placement +## IaaS + +Palette can deploy Virtual Machines (VMs) on Azure that collectively will be used to form the cluster. Depending on the +use case, you can deploy VMs to a pre-existing virtual network or let Palette create the required network resources. + +Select the appropriate section below to learn how to create a custom role in Azure and assign it to the service +principal you want to use with Palette. -Dynamic placement is the default method for deploying Azure clusters through Palette. With dynamic placement, Palette -dynamically creates the network resources required for your cluster. +### Static Placement {#iaas-static-placement} -### Prerequisites +Palette requires a set of permissions at the subscription level to deploy IaaS clusters using static placement. The +remainder of the permissions required by IaaS can be assigned at the resource group level. + +#### Prerequisites - Azure CLI installed on your local machine. Refer to the [Azure CLI Install Guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) for installation instructions. @@ -69,57 +83,26 @@ dynamically creates the network resources required for your cluster. - The Azure Security Principal Object ID you want to use with Palette. The Security Principal Object ID can represent a user, group, or service principal. -### Create Role and Assign Permissions +#### Create Role and Assign Permissions -1. Export your Azure subscription ID to a variable. +1. Open a terminal or command prompt. Export your Azure subscription ID to a variable. - ```shell - export SUBSCRIPTION_ID= - ``` + ```shell + export SUBSCRIPTION_ID= + ``` -2. Issue the following command to create a JSON file containing all the required permissions to deploy clusters - dynamically in Palette. +2. Issue the following command to create a JSON file containing all the permissions that must be applied at the virtual + network scope level. - ```shell - cat << EOF > dynamic-permissions.json - { - "Name": "Dynamic Placement Palette Deployer", + ```shell + cat << EOF > iaas_static_vnet_role.json + { + "Name": "Palette Static Placement IaaS Cluster Deployer (vnet)", "IsCustom": true, - "Description": "Can deploy Azure IaaS clusters using dynamic placement with Palette.", + "Description": "Can use Palette to deploy IaaS clusters using static placement. This role contains the permissions that must be applied at the virtual network scope level.", "Actions": [ - "Microsoft.Compute/disks/delete", - "Microsoft.Compute/disks/read", - "Microsoft.Compute/disks/write", - "Microsoft.Compute/galleries/images/read", - "Microsoft.Compute/galleries/images/versions/read", - "Microsoft.Compute/galleries/images/versions/write", - "Microsoft.Compute/galleries/images/write", - "Microsoft.Compute/galleries/read", - "Microsoft.Compute/galleries/write", - "Microsoft.Compute/images/read", - "Microsoft.Compute/images/write", - "Microsoft.Compute/virtualMachines/delete", - "Microsoft.Compute/virtualMachines/extensions/delete", - "Microsoft.Compute/virtualMachines/extensions/read", - "Microsoft.Compute/virtualMachines/extensions/write", - "Microsoft.Compute/virtualMachines/read", - "Microsoft.Compute/virtualMachines/write", - "Microsoft.Network/loadBalancers/backendAddressPools/join/action", - "Microsoft.Network/loadBalancers/delete", - "Microsoft.Network/loadBalancers/inboundNatRules/delete", - "Microsoft.Network/loadBalancers/inboundNatRules/join/action", - "Microsoft.Network/loadBalancers/inboundNatRules/read", - "Microsoft.Network/loadBalancers/inboundNatRules/write", - "Microsoft.Network/loadBalancers/read", - "Microsoft.Network/loadBalancers/write", - "Microsoft.Network/networkInterfaces/delete", - "Microsoft.Network/networkInterfaces/join/action", - "Microsoft.Network/networkInterfaces/read", - "Microsoft.Network/networkInterfaces/write", "Microsoft.Network/networkSecurityGroups/read", - "Microsoft.Network/networkSecurityGroups/securityRules/delete", "Microsoft.Network/networkSecurityGroups/securityRules/read", - "Microsoft.Network/networkSecurityGroups/securityRules/write", "Microsoft.Network/privateDnsZones/A/delete", "Microsoft.Network/privateDnsZones/A/read", "Microsoft.Network/privateDnsZones/A/write", @@ -129,172 +112,144 @@ dynamically creates the network resources required for your cluster. "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", "Microsoft.Network/privateDnsZones/write", - "Microsoft.Network/publicIPAddresses/delete", - "Microsoft.Network/publicIPAddresses/join/action", - "Microsoft.Network/publicIPAddresses/read", - "Microsoft.Network/publicIPAddresses/write", "Microsoft.Network/routeTables/delete", + "Microsoft.Network/routeTables/join/action", "Microsoft.Network/routeTables/read", "Microsoft.Network/routeTables/write", - "Microsoft.Network/virtualNetworks/delete", - "Microsoft.Network/virtualNetworks/join/action", "Microsoft.Network/virtualNetworks/join/action", "Microsoft.Network/virtualNetworks/joinLoadBalancer/action", "Microsoft.Network/virtualNetworks/peer/action", "Microsoft.Network/virtualNetworks/read", - "Microsoft.Network/virtualNetworks/subnets/delete", "Microsoft.Network/virtualNetworks/subnets/join/action", "Microsoft.Network/virtualNetworks/subnets/joinLoadBalancer/action", "Microsoft.Network/virtualNetworks/subnets/read", "Microsoft.Network/virtualNetworks/subnets/virtualMachines/read", - "Microsoft.Network/virtualNetworks/subnets/write", "Microsoft.Network/virtualNetworks/virtualMachines/read", - "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/delete", - "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read", - "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write", - "Microsoft.Network/virtualNetworks/write", - "Microsoft.Resources/subscriptions/resourceGroups/read", - "Microsoft.Storage/storageAccounts/blobServices/containers/read", - "Microsoft.Storage/storageAccounts/blobServices/containers/write", - "Microsoft.Storage/storageAccounts/read", - "Microsoft.Storage/storageAccounts/write" + "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read" ], "NotActions": [], - "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] + "AssignableScopes": [ + "/subscriptions/$SUBSCRIPTION_ID" + ] } - EOF - - ``` - - -
- Expand to view the raw JSON for the Dynamic Placement policy - - ```json - { - "Name": "Dynamic Placement Palette Deployer", - "IsCustom": true, - "Description": "Can deploy Azure IaaS clusters using dynamic placement with Palette.", - "Actions": [ - "Microsoft.Compute/disks/delete", - "Microsoft.Compute/disks/read", - "Microsoft.Compute/disks/write", - "Microsoft.Compute/galleries/images/read", - "Microsoft.Compute/galleries/images/versions/read", - "Microsoft.Compute/galleries/images/versions/write", - "Microsoft.Compute/galleries/images/write", - "Microsoft.Compute/galleries/read", - "Microsoft.Compute/galleries/write", - "Microsoft.Compute/images/read", - "Microsoft.Compute/images/write", - "Microsoft.Compute/virtualMachines/delete", - "Microsoft.Compute/virtualMachines/extensions/delete", - "Microsoft.Compute/virtualMachines/extensions/read", - "Microsoft.Compute/virtualMachines/extensions/write", - "Microsoft.Compute/virtualMachines/read", - "Microsoft.Compute/virtualMachines/write", - "Microsoft.Network/loadBalancers/backendAddressPools/join/action", - "Microsoft.Network/loadBalancers/delete", - "Microsoft.Network/loadBalancers/inboundNatRules/delete", - "Microsoft.Network/loadBalancers/inboundNatRules/join/action", - "Microsoft.Network/loadBalancers/inboundNatRules/read", - "Microsoft.Network/loadBalancers/inboundNatRules/write", - "Microsoft.Network/loadBalancers/read", - "Microsoft.Network/loadBalancers/write", - "Microsoft.Network/networkInterfaces/delete", - "Microsoft.Network/networkInterfaces/join/action", - "Microsoft.Network/networkInterfaces/read", - "Microsoft.Network/networkInterfaces/write", - "Microsoft.Network/networkSecurityGroups/read", - "Microsoft.Network/networkSecurityGroups/securityRules/delete", - "Microsoft.Network/networkSecurityGroups/securityRules/read", - "Microsoft.Network/networkSecurityGroups/securityRules/write", - "Microsoft.Network/privateDnsZones/A/delete", - "Microsoft.Network/privateDnsZones/A/read", - "Microsoft.Network/privateDnsZones/A/write", - "Microsoft.Network/privateDnsZones/delete", - "Microsoft.Network/privateDnsZones/read", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", - "Microsoft.Network/privateDnsZones/write", - "Microsoft.Network/publicIPAddresses/delete", - "Microsoft.Network/publicIPAddresses/join/action", - "Microsoft.Network/publicIPAddresses/read", - "Microsoft.Network/publicIPAddresses/write", - "Microsoft.Network/routeTables/delete", - "Microsoft.Network/routeTables/read", - "Microsoft.Network/routeTables/write", - "Microsoft.Network/virtualNetworks/delete", - "Microsoft.Network/virtualNetworks/join/action", - "Microsoft.Network/virtualNetworks/join/action", - "Microsoft.Network/virtualNetworks/joinLoadBalancer/action", - "Microsoft.Network/virtualNetworks/peer/action", - "Microsoft.Network/virtualNetworks/read", - "Microsoft.Network/virtualNetworks/subnets/delete", - "Microsoft.Network/virtualNetworks/subnets/join/action", - "Microsoft.Network/virtualNetworks/subnets/joinLoadBalancer/action", - "Microsoft.Network/virtualNetworks/subnets/read", - "Microsoft.Network/virtualNetworks/subnets/virtualMachines/read", - "Microsoft.Network/virtualNetworks/subnets/write", - "Microsoft.Network/virtualNetworks/virtualMachines/read", - "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/delete", - "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read", - "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write", - "Microsoft.Network/virtualNetworks/write", - "Microsoft.Resources/subscriptions/resourceGroups/read", - "Microsoft.Storage/storageAccounts/blobServices/containers/read", - "Microsoft.Storage/storageAccounts/blobServices/containers/write", - "Microsoft.Storage/storageAccounts/read", - "Microsoft.Storage/storageAccounts/write" - ], - "NotActions": [], - "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] - } - ``` - -
- -3. Create the custom role, _Dynamic Placement Palette Deployer_, in Azure using the JSON file you created in the - previous step. Issue the following command to create the role. - - ```shell - az role definition create --role-definition @dynamic-permissions.json --output table - ``` - -4. Export the - [security principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/overview#security-principal) - object ID you want to use with Palette to a variable. - - ```shell - export ASSIGNEE="" - ``` - -5. Assign the role by creating a role assignment referencing the role definition _Dynamic Placement Palette Deployer_. - Use the command below to assign the role to the service principal. - - ```shell - az role assignment create --assignee $ASSIGNEE \ - --role "Dynamic Placement Palette Deployer" \ - --scope "/subscriptions/$SUBSCRIPTION_ID" - ``` - -### Validate + EOF + ``` + +3. Next, create a JSON file for the permissions that must be applied at the resource group or subscription scope level. + Issue the following command to create the JSON file. + + ```shell + cat << EOF > iaas_static_rg_sub_role.json + { + "Name": "Palette Static Placement IaaS Cluster Deployer (rg/sub)", + "IsCustom": true, + "Description": "Can use Palette to deploy IaaS clusters using static placement. This role contains the permissions that must be applied at the resource group scope level. If deploying multiple clusters in a variety of resource groups within a subscription, apply the role with the subscription as scope instead of the resource group as scope.", + "Actions": [ + "Microsoft.Compute/availabilitySets/delete", + "Microsoft.Compute/availabilitySets/read", + "Microsoft.Compute/availabilitySets/write", + "Microsoft.Compute/disks/delete", + "Microsoft.Compute/disks/read", + "Microsoft.Compute/disks/write", + "Microsoft.Compute/galleries/images/read", + "Microsoft.Compute/galleries/images/versions/read", + "Microsoft.Compute/galleries/images/versions/write", + "Microsoft.Compute/galleries/images/write", + "Microsoft.Compute/galleries/read", + "Microsoft.Compute/galleries/write", + "Microsoft.Compute/images/read", + "Microsoft.Compute/images/write", + "Microsoft.Compute/virtualMachines/delete", + "Microsoft.Compute/virtualMachines/extensions/delete", + "Microsoft.Compute/virtualMachines/extensions/read", + "Microsoft.Compute/virtualMachines/extensions/write", + "Microsoft.Compute/virtualMachines/read", + "Microsoft.Compute/virtualMachines/write", + "Microsoft.Network/loadBalancers/backendAddressPools/join/action", + "Microsoft.Network/loadBalancers/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/join/action", + "Microsoft.Network/loadBalancers/inboundNatRules/read", + "Microsoft.Network/loadBalancers/inboundNatRules/write", + "Microsoft.Network/loadBalancers/read", + "Microsoft.Network/loadBalancers/write", + "Microsoft.Network/networkInterfaces/delete", + "Microsoft.Network/networkInterfaces/join/action", + "Microsoft.Network/networkInterfaces/read", + "Microsoft.Network/networkInterfaces/write", + "Microsoft.Network/publicIPAddresses/delete", + "Microsoft.Network/publicIPAddresses/join/action", + "Microsoft.Network/publicIPAddresses/read", + "Microsoft.Network/publicIPAddresses/write", + "Microsoft.Resources/subscriptions/resourceGroups/read", + "Microsoft.Resources/subscriptions/resourceGroups/write", + "Microsoft.Storage/storageAccounts/blobServices/containers/read", + "Microsoft.Storage/storageAccounts/blobServices/containers/write", + "Microsoft.Storage/storageAccounts/listKeys/action", + "Microsoft.Storage/storageAccounts/read", + "Microsoft.Storage/storageAccounts/write" + ], + "NotActions": [], + "AssignableScopes": [ + "/subscriptions/$SUBSCRIPTION_ID" + ] + } + EOF + ``` + +4. Create a role for each of the JSON files you created in the previous steps. Issue the following commands to create + the roles. + + ```shell + az role definition create --role-definition @iaas_static_vnet_role.json --output table + az role definition create --role-definition @iaas_static_rg_sub_role.json --output table + ``` + +5. Export the + [security principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/overview#security-principal) + object ID you want to use with Palette to a variable. + + ```shell + export ASSIGNEE="" + ``` + +6. Export the resource group name and virtual network name to a variable. + + ```shell + export RESOURCE_GROUP_NAME="" + export VNET_NAME="" + ``` + +7. Assign the roles to the service principal. Use the following commands to assign the roles. + + ```json + az role assignment create --assignee $ASSIGNEE \ + --role "Palette Static Placement IaaS Cluster Deployer (rg/sub)" \ + --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME" + ``` + + ```json + az role assignment create --assignee $ASSIGNEE \ + --role "Palette Static Placement IaaS Cluster Deployer (vnet)" \ + --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME/providers/Microsoft.Network/virtualNetworks/$VNET_NAME" + ``` + +#### Validate 1. Log in to the Azure portal. 2. Navigate to the Microsoft Entra ID section. -3. Review the role assignments to ensure the service principal has the `Dynamic Placement Palette Deployer` role - assigned. +3. Review the role, or roles if you created multiple. Review the role assignments to ensure the service principal has + the correct permissions assigned. -## Static Placement +### Dynamic Placement {#iaas-dynamic-placement} -Choose static placement when you want Palette to use pre-existing network resource groups, VNets, subnets, and security -groups. For your convenience, we have provided the required permissions to deploy clusters using static placement in -various scenarios. +The permission requirements for IaaS dynamic placement can be provided at the resource group or subscription level. If +you are deploying multiple clusters in a variety of resource groups within a subscription, apply the role with the +subscription as scope instead of the resource group as scope. -### Prerequisites +#### Prerequisites - Azure CLI installed on your local machine. Refer to the [Azure CLI Install Guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) for installation instructions. @@ -309,655 +264,749 @@ various scenarios. - The Azure Security Principal Object ID you want to use with Palette. The Security Principal Object ID can represent a user, group, or service principal. -### Create Role and Assign Permissions +#### Create Role and Assign Permissions + +1. Open a terminal or command prompt. Export your Azure subscription ID to a variable. + + ```shell + export SUBSCRIPTION_ID= + ``` + +2. Issue the following command to create a JSON file containing all the required permissions to create resources. + + ```shell + cat << EOF > iaas_dynamic_rg_sub_role.json + { + "Name": "Palette Dynamic Placement IaaS Cluster Deployer (rg/sub)", + "IsCustom": true, + "Description": "Can use Palette to deploy IaaS clusters using dynamic placement. This role contains the permissions that must be applied at the resource group scope level. If deploying multiple clusters in a variety of resource groups within a subscription, apply the role with the subscription as scope instead of the resource group as scope.", + "Actions": [ + "Microsoft.Compute/disks/delete", + "Microsoft.Compute/disks/read", + "Microsoft.Compute/disks/write", + "Microsoft.Compute/galleries/images/read", + "Microsoft.Compute/galleries/images/versions/read", + "Microsoft.Compute/galleries/images/versions/write", + "Microsoft.Compute/galleries/images/write", + "Microsoft.Compute/galleries/read", + "Microsoft.Compute/galleries/write", + "Microsoft.Compute/images/read", + "Microsoft.Compute/images/write", + "Microsoft.Compute/virtualMachines/delete", + "Microsoft.Compute/virtualMachines/extensions/delete", + "Microsoft.Compute/virtualMachines/extensions/read", + "Microsoft.Compute/virtualMachines/extensions/write", + "Microsoft.Compute/virtualMachines/read", + "Microsoft.Compute/virtualMachines/write", + "Microsoft.Network/loadBalancers/backendAddressPools/join/action", + "Microsoft.Network/loadBalancers/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/delete", + "Microsoft.Network/loadBalancers/inboundNatRules/join/action", + "Microsoft.Network/loadBalancers/inboundNatRules/read", + "Microsoft.Network/loadBalancers/inboundNatRules/write", + "Microsoft.Network/loadBalancers/read", + "Microsoft.Network/loadBalancers/write", + "Microsoft.Network/networkInterfaces/delete", + "Microsoft.Network/networkInterfaces/join/action", + "Microsoft.Network/networkInterfaces/read", + "Microsoft.Network/networkInterfaces/write", + "Microsoft.Network/networkSecurityGroups/read", + "Microsoft.Network/networkSecurityGroups/securityRules/delete", + "Microsoft.Network/networkSecurityGroups/securityRules/read", + "Microsoft.Network/networkSecurityGroups/securityRules/write", + "Microsoft.Network/privateDnsZones/A/delete", + "Microsoft.Network/privateDnsZones/A/read", + "Microsoft.Network/privateDnsZones/A/write", + "Microsoft.Network/privateDnsZones/delete", + "Microsoft.Network/privateDnsZones/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", + "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", + "Microsoft.Network/privateDnsZones/write", + "Microsoft.Network/publicIPAddresses/delete", + "Microsoft.Network/publicIPAddresses/join/action", + "Microsoft.Network/publicIPAddresses/read", + "Microsoft.Network/publicIPAddresses/write", + "Microsoft.Network/routeTables/delete", + "Microsoft.Network/routeTables/read", + "Microsoft.Network/routeTables/write", + "Microsoft.Network/virtualNetworks/delete", + "Microsoft.Network/virtualNetworks/join/action", + "Microsoft.Network/virtualNetworks/join/action", + "Microsoft.Network/virtualNetworks/joinLoadBalancer/action", + "Microsoft.Network/virtualNetworks/peer/action", + "Microsoft.Network/virtualNetworks/read", + "Microsoft.Network/virtualNetworks/subnets/delete", + "Microsoft.Network/virtualNetworks/subnets/join/action", + "Microsoft.Network/virtualNetworks/subnets/joinLoadBalancer/action", + "Microsoft.Network/virtualNetworks/subnets/read", + "Microsoft.Network/virtualNetworks/subnets/virtualMachines/read", + "Microsoft.Network/virtualNetworks/subnets/write", + "Microsoft.Network/virtualNetworks/virtualMachines/read", + "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/delete", + "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read", + "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write", + "Microsoft.Network/virtualNetworks/write", + "Microsoft.Resources/subscriptions/resourceGroups/read", + "Microsoft.Storage/storageAccounts/blobServices/containers/read", + "Microsoft.Storage/storageAccounts/blobServices/containers/write", + "Microsoft.Storage/storageAccounts/read", + "Microsoft.Storage/storageAccounts/write" + ], + "NotActions": [], + "AssignableScopes": [ + "/subscriptions/$SUBSCRIPTION_ID" + ] + } + EOF + ``` + +3. Create a role using the JSON file you created in the previous step. Issue the following command to create the role. + + ```shell + az role definition create --role-definition @iaas_dynamic_rg_sub_role.json --output table + ``` + +4. Export the + [security principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/overview#security-principal) + object ID you want to use with Palette to a variable. + + ```shell + export ASSIGNEE="" + ``` + +5. Export the resource group name to a variable. + + ```shell + export RESOURCE_GROUP_NAME="" + ``` + +6. Assign the role to the service principal. Use the following command to assign the role. + + ```shell + az role assignment create --assignee $ASSIGNEE \ + --role "Palette Dynamic Placement IaaS Cluster Deployer (rg/sub)" \ + --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME" + ``` + +#### Validate -Select the tab below to view the required permissions and steps for the static placement scenario you want to use. +1. Log in to the Azure portal. - +2. Navigate to the Microsoft Entra ID section. - +3. Review the role, or roles if you created multiple. Review the role assignments to ensure the service principal has + the correct permissions assigned. -1. Export your Azure subscription ID to a variable. +## AKS - ```shell - export SUBSCRIPTION_ID= - ``` +Palette can deploy [AKS](https://learn.microsoft.com/en-us/azure/aks/what-is-aks) clusters on Azure. You can deploy AKS +clusters on pre-existing network resources or let Palette create the required network resources. Select the appropriate +section below to learn how to create a custom role in Azure and assign it to the service principal you want to use with +Palette. -2. Issue the following command to create a JSON file containing all the required permissions to deploy a cluster - statically in Palette. +### Static Placement {#aks-static-placement} - ```shell - cat << EOF > static-permissions.json - { - "Name": "Static Placement Palette Deployer", - "IsCustom": true, - "Description": "Can deploy Azure IaaS clusters using static placement with Palette.", - "Actions": [ - "Microsoft.Compute/disks/delete", - "Microsoft.Compute/disks/read", - "Microsoft.Compute/disks/write", - "Microsoft.Compute/galleries/images/versions/write", - "Microsoft.Compute/galleries/images/write", - "Microsoft.Compute/galleries/read", - "Microsoft.Compute/galleries/write", - "Microsoft.Compute/images/read", - "Microsoft.Compute/images/write", - "Microsoft.Compute/virtualMachines/delete", - "Microsoft.Compute/virtualMachines/extensions/delete", - "Microsoft.Compute/virtualMachines/extensions/read", - "Microsoft.Compute/virtualMachines/extensions/write", - "Microsoft.Compute/virtualMachines/read", - "Microsoft.Compute/virtualMachines/write", - "Microsoft.Network/loadBalancers/backendAddressPools/join/action", - "Microsoft.Network/loadBalancers/delete", - "Microsoft.Network/loadBalancers/inboundNatRules/delete", - "Microsoft.Network/loadBalancers/inboundNatRules/join/action", - "Microsoft.Network/loadBalancers/inboundNatRules/read", - "Microsoft.Network/loadBalancers/inboundNatRules/write", - "Microsoft.Network/loadBalancers/read", - "Microsoft.Network/loadBalancers/write", - "Microsoft.Network/networkInterfaces/delete", - "Microsoft.Network/networkInterfaces/join/action", - "Microsoft.Network/networkInterfaces/read", - "Microsoft.Network/networkInterfaces/write", - "Microsoft.Network/networkSecurityGroups/read", - "Microsoft.Network/networkSecurityGroups/securityRules/delete", - "Microsoft.Network/networkSecurityGroups/securityRules/read", - "Microsoft.Network/networkSecurityGroups/securityRules/write", - "Microsoft.Network/privateDnsZones/A/delete", - "Microsoft.Network/privateDnsZones/A/read", - "Microsoft.Network/privateDnsZones/A/write", - "Microsoft.Network/privateDnsZones/delete", - "Microsoft.Network/privateDnsZones/read", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", - "Microsoft.Network/privateDnsZones/write", - "Microsoft.Network/publicIPAddresses/delete", - "Microsoft.Network/publicIPAddresses/join/action", - "Microsoft.Network/publicIPAddresses/read", - "Microsoft.Network/publicIPAddresses/write", - "Microsoft.Network/routeTables/delete", - "Microsoft.Network/routeTables/read", - "Microsoft.Network/routeTables/write", - "Microsoft.Network/virtualNetworks/delete", - "Microsoft.Network/virtualNetworks/join/action", - "Microsoft.Network/virtualNetworks/join/action", - "Microsoft.Network/virtualNetworks/joinLoadBalancer/action", - "Microsoft.Network/virtualNetworks/peer/action", - "Microsoft.Network/virtualNetworks/subnets/delete", - "Microsoft.Network/virtualNetworks/subnets/joinLoadBalancer/action", - "Microsoft.Network/virtualNetworks/subnets/virtualMachines/read", - "Microsoft.Network/virtualNetworks/subnets/write", - "Microsoft.Network/virtualNetworks/virtualMachines/read", - "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/delete", - "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read", - "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write", - "Microsoft.Network/virtualNetworks/write", - "Microsoft.Resources/subscriptions/resourceGroups/read", - "Microsoft.Storage/storageAccounts/blobServices/containers/read", - "Microsoft.Storage/storageAccounts/blobServices/containers/write", - "Microsoft.Storage/storageAccounts/read", - "Microsoft.Storage/storageAccounts/write" - ], - "NotActions": [], - "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] - } - EOF - ``` +Palette requires a set of permissions at the subscription level to deploy AKS clusters using static placement. The +remainder of the permissions required by AKS can be assigned at the subscription or resource group level. -3. Next, create a JSON file for the compute gallery scope permissions. Issue the following command to create the JSON - file. - - ```shell - cat << EOF > static-gallery-scope-permissions.json - { - "Name": "Static Placement Palette Deployer - Compute Gallery Scope", - "IsCustom": true, - "Description": "Can deploy the compute gallery level components of Azure IaaS clusters using static placement with Palette.", - "Actions": [ - "Microsoft.Compute/galleries/images/read", - "Microsoft.Compute/galleries/images/versions/read" - ], - "NotActions": [], - "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] - } - EOF - ``` +#### Prerequisites -4. Create a JSON file for the subnet scope permissions. +- Azure CLI installed on your local machine. Refer to the + [Azure CLI Install Guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) for installation instructions. - ```shell - cat << EOF > static-subnet-scope-permissions.json - { - "Name": "Static Placement Palette Deployer - Subnet Scope", - "IsCustom": true, - "Description": "Can deploy the subnet level components of Azure IaaS clusters using static placement with Palette.", - "Actions": [ - "Microsoft.Network/virtualNetworks/subnets/join/action", - "Microsoft.Network/virtualNetworks/subnets/read" - ], - "NotActions": [], - "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] - } - EOF - ``` +- The Azure subscription ID you want to use with Palette. -5. The last JSON file is for the virtual network scope permissions. Issue the following command to create the JSON - file. +- The Security Principal Object ID you want to use with Palette. You can retrieve it from the Entra ID section of the + Azure Portal. - ```shell - cat << EOF > static-virtual-network-scope-permissions.json - { - "Name": "Static Placement Palette Deployer - Virtual Network Scope", - "IsCustom": true, - "Description": "Can deploy the virtual network level components of Azure IaaS clusters using static placement with Palette.", - "Actions": ["Microsoft.Network/virtualNetworks/read"], - "NotActions": [], - "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] - } - EOF - ``` - - -
- Click here to view the raw JSON policies. - ```json - { - "Name": "Static Placement Palette Deployer", - "IsCustom": true, - "Description": "Can deploy Azure IaaS clusters using static placement with Palette.", - "Actions": [ - "Microsoft.Compute/disks/delete", - "Microsoft.Compute/disks/read", - "Microsoft.Compute/disks/write", - "Microsoft.Compute/galleries/images/versions/write", - "Microsoft.Compute/galleries/images/write", - "Microsoft.Compute/galleries/read", - "Microsoft.Compute/galleries/write", - "Microsoft.Compute/images/read", - "Microsoft.Compute/images/write", - "Microsoft.Compute/virtualMachines/delete", - "Microsoft.Compute/virtualMachines/extensions/delete", - "Microsoft.Compute/virtualMachines/extensions/read", - "Microsoft.Compute/virtualMachines/extensions/write", - "Microsoft.Compute/virtualMachines/read", - "Microsoft.Compute/virtualMachines/write", - "Microsoft.Network/loadBalancers/backendAddressPools/join/action", - "Microsoft.Network/loadBalancers/delete", - "Microsoft.Network/loadBalancers/inboundNatRules/delete", - "Microsoft.Network/loadBalancers/inboundNatRules/join/action", - "Microsoft.Network/loadBalancers/inboundNatRules/read", - "Microsoft.Network/loadBalancers/inboundNatRules/write", - "Microsoft.Network/loadBalancers/read", - "Microsoft.Network/loadBalancers/write", - "Microsoft.Network/networkInterfaces/delete", - "Microsoft.Network/networkInterfaces/join/action", - "Microsoft.Network/networkInterfaces/read", - "Microsoft.Network/networkInterfaces/write", - "Microsoft.Network/networkSecurityGroups/read", - "Microsoft.Network/networkSecurityGroups/securityRules/delete", - "Microsoft.Network/networkSecurityGroups/securityRules/read", - "Microsoft.Network/networkSecurityGroups/securityRules/write", - "Microsoft.Network/privateDnsZones/A/delete", - "Microsoft.Network/privateDnsZones/A/read", - "Microsoft.Network/privateDnsZones/A/write", - "Microsoft.Network/privateDnsZones/delete", - "Microsoft.Network/privateDnsZones/read", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", - "Microsoft.Network/privateDnsZones/write", - "Microsoft.Network/publicIPAddresses/delete", - "Microsoft.Network/publicIPAddresses/join/action", - "Microsoft.Network/publicIPAddresses/read", - "Microsoft.Network/publicIPAddresses/write", - "Microsoft.Network/routeTables/delete", - "Microsoft.Network/routeTables/read", - "Microsoft.Network/routeTables/write", - "Microsoft.Network/virtualNetworks/delete", - "Microsoft.Network/virtualNetworks/join/action", - "Microsoft.Network/virtualNetworks/join/action", - "Microsoft.Network/virtualNetworks/joinLoadBalancer/action", - "Microsoft.Network/virtualNetworks/peer/action", - "Microsoft.Network/virtualNetworks/subnets/delete", - "Microsoft.Network/virtualNetworks/subnets/joinLoadBalancer/action", - "Microsoft.Network/virtualNetworks/subnets/virtualMachines/read", - "Microsoft.Network/virtualNetworks/subnets/write", - "Microsoft.Network/virtualNetworks/virtualMachines/read", - "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/delete", - "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read", - "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write", - "Microsoft.Network/virtualNetworks/write", - "Microsoft.Resources/subscriptions/resourceGroups/read", - "Microsoft.Storage/storageAccounts/blobServices/containers/read", - "Microsoft.Storage/storageAccounts/blobServices/containers/write", - "Microsoft.Storage/storageAccounts/read", - "Microsoft.Storage/storageAccounts/write" - ], - "NotActions": [], - "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] - } - ``` - ```json - { - "Name": "Static Placement Palette Deployer - Compute Gallery Scope", - "IsCustom": true, - "Description": "Can deploy the compute gallery level components of Azure IaaS clusters using static placement with Palette.", - "Actions": [ - "Microsoft.Compute/galleries/images/read", - "Microsoft.Compute/galleries/images/versions/read" - ], - "NotActions": [], - "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] - } - ``` +- A terminal or command prompt to issue the Azure CLI commands. - ```json - { - "Name": "Static Placement Palette Deployer - Subnet Scope", - "IsCustom": true, - "Description": "Can deploy the subnet level components of Azure IaaS clusters using static placement with Palette.", - "Actions": [ - "Microsoft.Network/virtualNetworks/subnets/join/action", - "Microsoft.Network/virtualNetworks/subnets/read" - ], - "NotActions": [], - "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] - } - ``` +- The Azure Security Principal Object ID you want to use with Palette. The Security Principal Object ID can represent a + user, group, or service principal. - ```json - { - "Name": "Static Placement Palette Deployer - Virtual Network Scope", - "IsCustom": true, - "Description": "Can deploy the virtual network level components of Azure IaaS clusters using static placement with Palette.", - "Actions": ["Microsoft.Network/virtualNetworks/read"], - "NotActions": [], - "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] - } - ``` - -
- -6. Create a role for each of the JSON files you created in the previous steps. Issue the following commands to create - the roles. - - ```shell - az role definition create --role-definition @static-permissions.json --output table - az role definition create --role-definition @static-gallery-scope-permissions.json --output table - az role definition create --role-definition @static-subnet-scope-permissions.json --output table - az role definition create --role-definition @static-virtual-network-scope-permissions.json --output table - ``` - -7. Export resource group, virtual network, subnet, and compute gallery names you desire Palette to use to a set of - variables. - - ```shell - export RESOURCE_GROUP_NAME= - export VIRTUAL_NETWORK_NAME= - export SUBNET_NAME= - export COMPUTE_GALLERY_NAME= - ``` - -8. Export the - [security principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/overview#security-principal) - object ID you want to use with Palette to a variable. - - ```shell - export ASSIGNEE="" - ``` - -9. Assign the roles to the service principal. Use the following commands to assign the roles. - - ```json - az role assignment create --assignee $ASSIGNEE \ - --role "Static Placement Palette Deployer - Resource Group Scope" \ - --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME" - ``` - - ```json - az role assignment create --assignee $ASSIGNEE \ - --role "Static Placement Palette Deployer - Virtual Network Scope" \ - --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME/providers/Microsoft.Network/virtualNetworks/$VIRTUAL_NETWORK_NAME" - ``` - - ```json - az role assignment create --assignee $ASSIGNEE \ - --role "Static Placement Palette Deployer - Subnet Scope" \ - --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME/providers/Microsoft.Network/virtualNetworks/$VIRTUAL_NETWORK_NAME/subnets/$SUBNET_NAME" - ``` - - ```json - az role assignment create --assignee $ASSIGNEE \ - --role "Static Placement Palette Deployer - Compute Gallery Scope" \ - --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME/providers/Microsoft.Compute/galleries/$COMPUTE_GALLERY_NAME" - ``` - -
- - -1. Export your Azure subscription ID to a variable. - - ```shell - export SUBSCRIPTION_ID= - ``` - -2. Issue the following command to create a JSON file containing all the required permissions to deploy a cluster - statically in Palette. - - ```shell - cat << EOF > static-placement-permissions.json - { - "Name": "Static Placement Palette Deployer", - "IsCustom": true, - "Description": "Can deploy Azure IaaS clusters using static placement with Palette.", - "Actions": [ - "Microsoft.Compute/disks/delete", - "Microsoft.Compute/disks/read", - "Microsoft.Compute/disks/write", - "Microsoft.Compute/virtualMachines/delete", - "Microsoft.Compute/virtualMachines/extensions/delete", - "Microsoft.Compute/virtualMachines/extensions/read", - "Microsoft.Compute/virtualMachines/extensions/write", - "Microsoft.Compute/virtualMachines/read", - "Microsoft.Compute/virtualMachines/write", - "Microsoft.Network/loadBalancers/backendAddressPools/join/action", - "Microsoft.Network/loadBalancers/delete", - "Microsoft.Network/loadBalancers/inboundNatRules/delete", - "Microsoft.Network/loadBalancers/inboundNatRules/join/action", - "Microsoft.Network/loadBalancers/inboundNatRules/read", - "Microsoft.Network/loadBalancers/inboundNatRules/write", - "Microsoft.Network/loadBalancers/read", - "Microsoft.Network/loadBalancers/write", - "Microsoft.Network/networkInterfaces/delete", - "Microsoft.Network/networkInterfaces/join/action", - "Microsoft.Network/networkInterfaces/read", - "Microsoft.Network/networkInterfaces/write", - "Microsoft.Network/networkSecurityGroups/read", - "Microsoft.Network/networkSecurityGroups/securityRules/delete", - "Microsoft.Network/networkSecurityGroups/securityRules/read", - "Microsoft.Network/networkSecurityGroups/securityRules/write", - "Microsoft.Network/privateDnsZones/A/delete", - "Microsoft.Network/privateDnsZones/A/read", - "Microsoft.Network/privateDnsZones/A/write", - "Microsoft.Network/privateDnsZones/delete", - "Microsoft.Network/privateDnsZones/read", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", - "Microsoft.Network/privateDnsZones/write", - "Microsoft.Network/publicIPAddresses/delete", - "Microsoft.Network/publicIPAddresses/join/action", - "Microsoft.Network/publicIPAddresses/read", - "Microsoft.Network/publicIPAddresses/write", - "Microsoft.Network/routeTables/delete", - "Microsoft.Network/routeTables/read", - "Microsoft.Network/routeTables/write", - "Microsoft.Network/virtualNetworks/join/action", - "Microsoft.Resources/subscriptions/resourceGroups/read", - "Microsoft.Network/virtualNetworks/read", - "Microsoft.Network/virtualNetworks/subnets/join/action", - "Microsoft.Network/virtualNetworks/subnets/read", - "Microsoft.Compute/galleries/images/read", - "Microsoft.Compute/galleries/images/versions/read" - ], - "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] - } - EOF - ``` - - -
- Click here to view the raw JSON policy. - ```json - { - "Name": "Static Placement Palette Deployer", - "IsCustom": true, - "Description": "Can deploy Azure IaaS clusters using static placement with Palette.", - "Actions": [ - "Microsoft.Compute/disks/delete", - "Microsoft.Compute/disks/read", - "Microsoft.Compute/disks/write", - "Microsoft.Compute/virtualMachines/delete", - "Microsoft.Compute/virtualMachines/extensions/delete", - "Microsoft.Compute/virtualMachines/extensions/read", - "Microsoft.Compute/virtualMachines/extensions/write", - "Microsoft.Compute/virtualMachines/read", - "Microsoft.Compute/virtualMachines/write", - "Microsoft.Network/loadBalancers/backendAddressPools/join/action", - "Microsoft.Network/loadBalancers/delete", - "Microsoft.Network/loadBalancers/inboundNatRules/delete", - "Microsoft.Network/loadBalancers/inboundNatRules/join/action", - "Microsoft.Network/loadBalancers/inboundNatRules/read", - "Microsoft.Network/loadBalancers/inboundNatRules/write", - "Microsoft.Network/loadBalancers/read", - "Microsoft.Network/loadBalancers/write", - "Microsoft.Network/networkInterfaces/delete", - "Microsoft.Network/networkInterfaces/join/action", - "Microsoft.Network/networkInterfaces/read", - "Microsoft.Network/networkInterfaces/write", - "Microsoft.Network/networkSecurityGroups/read", - "Microsoft.Network/networkSecurityGroups/securityRules/delete", - "Microsoft.Network/networkSecurityGroups/securityRules/read", - "Microsoft.Network/networkSecurityGroups/securityRules/write", - "Microsoft.Network/privateDnsZones/A/delete", - "Microsoft.Network/privateDnsZones/A/read", - "Microsoft.Network/privateDnsZones/A/write", - "Microsoft.Network/privateDnsZones/delete", - "Microsoft.Network/privateDnsZones/read", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", - "Microsoft.Network/privateDnsZones/write", - "Microsoft.Network/publicIPAddresses/delete", - "Microsoft.Network/publicIPAddresses/join/action", - "Microsoft.Network/publicIPAddresses/read", - "Microsoft.Network/publicIPAddresses/write", - "Microsoft.Network/routeTables/delete", - "Microsoft.Network/routeTables/read", - "Microsoft.Network/routeTables/write", - "Microsoft.Network/virtualNetworks/join/action", - "Microsoft.Resources/subscriptions/resourceGroups/read", - "Microsoft.Network/virtualNetworks/read", - "Microsoft.Network/virtualNetworks/subnets/join/action", - "Microsoft.Network/virtualNetworks/subnets/read", - "Microsoft.Compute/galleries/images/read", - "Microsoft.Compute/galleries/images/versions/read" - ], - "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] - } - ``` -
- -3. Create a role for the JSON file you created in the previous step. Issue the following command to create the role. - - ```shell - az role definition create --role-definition @static-placement-permissions.json --output table - ``` - -4. Export the - [security principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/overview#security-principal) - object ID and resource group name you want to use with Palette to a set of variables. - - ```shell - export ASSIGNEE="" - export RESOURCE_GROUP_NAME= - ``` - -5. Assign the role to the service principal. Use the following command to assign the role. - - ```json - az role assignment create --assignee $ASSIGNEE \ - --role "Static Placement Palette Deployer" \ - --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME" - ``` - -
- - - -1. Export your Azure subscription ID to a variable. - - ```shell - export SUBSCRIPTION_ID= - ``` - -2. Issue the following command to create a JSON file containing all the required permissions to deploy a cluster - statically in Palette. - - ```shell - cat << EOF > static-placement-permissions.json - { - "Name": "Static Placement Palette Deployer", - "IsCustom": true, - "Description": "Can deploy Azure IaaS clusters using static placement with Palette.", - "Actions": [ - "Microsoft.Compute/disks/delete", - "Microsoft.Compute/disks/read", - "Microsoft.Compute/disks/write", - "Microsoft.Compute/virtualMachines/delete", - "Microsoft.Compute/virtualMachines/extensions/delete", - "Microsoft.Compute/virtualMachines/extensions/read", - "Microsoft.Compute/virtualMachines/extensions/write", - "Microsoft.Compute/virtualMachines/read", - "Microsoft.Compute/virtualMachines/write", - "Microsoft.Network/loadBalancers/backendAddressPools/join/action", - "Microsoft.Network/loadBalancers/delete", - "Microsoft.Network/loadBalancers/inboundNatRules/delete", - "Microsoft.Network/loadBalancers/inboundNatRules/join/action", - "Microsoft.Network/loadBalancers/inboundNatRules/read", - "Microsoft.Network/loadBalancers/inboundNatRules/write", - "Microsoft.Network/loadBalancers/read", - "Microsoft.Network/loadBalancers/write", - "Microsoft.Network/networkInterfaces/delete", - "Microsoft.Network/networkInterfaces/join/action", - "Microsoft.Network/networkInterfaces/read", - "Microsoft.Network/networkInterfaces/write", - "Microsoft.Network/networkSecurityGroups/read", - "Microsoft.Network/networkSecurityGroups/securityRules/delete", - "Microsoft.Network/networkSecurityGroups/securityRules/read", - "Microsoft.Network/networkSecurityGroups/securityRules/write", - "Microsoft.Network/privateDnsZones/A/delete", - "Microsoft.Network/privateDnsZones/A/read", - "Microsoft.Network/privateDnsZones/A/write", - "Microsoft.Network/privateDnsZones/delete", - "Microsoft.Network/privateDnsZones/read", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", - "Microsoft.Network/privateDnsZones/write", - "Microsoft.Network/publicIPAddresses/delete", - "Microsoft.Network/publicIPAddresses/join/action", - "Microsoft.Network/publicIPAddresses/read", - "Microsoft.Network/publicIPAddresses/write", - "Microsoft.Network/routeTables/delete", - "Microsoft.Network/routeTables/read", - "Microsoft.Network/routeTables/write", - "Microsoft.Network/virtualNetworks/join/action", - "Microsoft.Resources/subscriptions/resourceGroups/read", - "Microsoft.Network/virtualNetworks/read", - "Microsoft.Network/virtualNetworks/subnets/join/action", - "Microsoft.Network/virtualNetworks/subnets/read", - "Microsoft.Compute/galleries/images/read", - "Microsoft.Compute/galleries/images/versions/read" - ], - "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] - } - EOF - ``` - - - -
- Click here to view the raw JSON policy. - ```json - { - "Name": "Static Placement Palette Deployer", - "IsCustom": true, - "Description": "Can deploy Azure IaaS clusters using static placement with Palette.", - "Actions": [ - "Microsoft.Compute/disks/delete", - "Microsoft.Compute/disks/read", - "Microsoft.Compute/disks/write", - "Microsoft.Compute/virtualMachines/delete", - "Microsoft.Compute/virtualMachines/extensions/delete", - "Microsoft.Compute/virtualMachines/extensions/read", - "Microsoft.Compute/virtualMachines/extensions/write", - "Microsoft.Compute/virtualMachines/read", - "Microsoft.Compute/virtualMachines/write", - "Microsoft.Network/loadBalancers/backendAddressPools/join/action", - "Microsoft.Network/loadBalancers/delete", - "Microsoft.Network/loadBalancers/inboundNatRules/delete", - "Microsoft.Network/loadBalancers/inboundNatRules/join/action", - "Microsoft.Network/loadBalancers/inboundNatRules/read", - "Microsoft.Network/loadBalancers/inboundNatRules/write", - "Microsoft.Network/loadBalancers/read", - "Microsoft.Network/loadBalancers/write", - "Microsoft.Network/networkInterfaces/delete", - "Microsoft.Network/networkInterfaces/join/action", - "Microsoft.Network/networkInterfaces/read", - "Microsoft.Network/networkInterfaces/write", - "Microsoft.Network/networkSecurityGroups/read", - "Microsoft.Network/networkSecurityGroups/securityRules/delete", - "Microsoft.Network/networkSecurityGroups/securityRules/read", - "Microsoft.Network/networkSecurityGroups/securityRules/write", - "Microsoft.Network/privateDnsZones/A/delete", - "Microsoft.Network/privateDnsZones/A/read", - "Microsoft.Network/privateDnsZones/A/write", - "Microsoft.Network/privateDnsZones/delete", - "Microsoft.Network/privateDnsZones/read", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/read", - "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write", - "Microsoft.Network/privateDnsZones/write", - "Microsoft.Network/publicIPAddresses/delete", - "Microsoft.Network/publicIPAddresses/join/action", - "Microsoft.Network/publicIPAddresses/read", - "Microsoft.Network/publicIPAddresses/write", - "Microsoft.Network/routeTables/delete", - "Microsoft.Network/routeTables/read", - "Microsoft.Network/routeTables/write", - "Microsoft.Network/virtualNetworks/join/action", - "Microsoft.Resources/subscriptions/resourceGroups/read", - "Microsoft.Network/virtualNetworks/read", - "Microsoft.Network/virtualNetworks/subnets/join/action", - "Microsoft.Network/virtualNetworks/subnets/read", - "Microsoft.Compute/galleries/images/read", - "Microsoft.Compute/galleries/images/versions/read" - ], - "AssignableScopes": ["/subscriptions/$SUBSCRIPTION_ID"] - } - ``` -
- -3. Create a role using the JSON file you created in the previous step. Issue the following command to create the role. - - ```shell - az role definition create --role-definition @static-placement-permissions.json --output table - ``` - -4. Export the - [security principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/overview#security-principal) - object ID you want to use with Palette to a variable. - - ```shell - export ASSIGNEE="" - ``` - -5. Assign the role to the service principal. Use the following command to assign the role. - - ```shell - az role assignment create --assignee $ASSIGNEE \ - --role "Static Placement Palette Deployer" \ - --scope "/subscriptions/$SUBSCRIPTION_ID" - ``` - -
- -
- -### Validate +#### Create Role and Assign Permissions + +1. Open a terminal or command prompt. Export your Azure subscription ID to a variable. + + ```shell + export SUBSCRIPTION_ID= + ``` + +2. Issue the following command to create a JSON file containing all the permissions that must be applied at the virtual + network scope level. + + ```shell + cat << EOF > aks_static_vnet_role.json + { + "Name": "Palette Static Placement AKS Cluster Deployer (vnet)", + "IsCustom": true, + "Description": "Can use Palette to deploy AKS clusters using static placement. This role contains the permissions that must be applied at the virtual network scope level.", + "Actions": [ + "Microsoft.Network/virtualNetworks/read", + "Microsoft.Network/virtualNetworks/subnets/join/action", + "Microsoft.Network/virtualNetworks/subnets/read" + ], + "NotActions": [], + "AssignableScopes": [ + "/subscriptions/$SUBSCRIPTION_ID" + ] + } + EOF + ``` + +3. Create a JSON file for the permissions that must be applied at the subscription scope level. Issue the following + command to create the JSON file. + + ```shell + cat << EOF > aks_static_sub_role.json + { + "Name": "Palette Static Placement AKS Cluster Deployer (sub)", + "IsCustom": true, + "Description": "Can use Palette to deploy AKS clusters using static placement. This role contains the permissions that must be applied at the subscription scope level.", + "Actions": [ + "Microsoft.Compute/virtualMachineScaleSets/extensions/read", + "Microsoft.Compute/virtualMachineScaleSets/extensions/roles/read", + "Microsoft.Compute/virtualMachineScaleSets/instanceView/read", + "Microsoft.Compute/virtualMachineScaleSets/networkInterfaces/read", + "Microsoft.Compute/virtualMachineScaleSets/osUpgradeHistory/read", + "Microsoft.Compute/virtualMachineScaleSets/providers/Microsoft.Insights/diagnosticSettings/read", + "Microsoft.Compute/virtualMachineScaleSets/providers/Microsoft.Insights/logDefinitions/read", + "Microsoft.Compute/virtualMachineScaleSets/providers/Microsoft.Insights/metricDefinitions/read", + "Microsoft.Compute/virtualMachineScaleSets/publicIPAddresses/read", + "Microsoft.Compute/virtualMachineScaleSets/read", + "Microsoft.Compute/virtualMachineScaleSets/rollingUpgrades/read", + "Microsoft.Compute/virtualMachineScaleSets/skus/read", + "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/extensions/read", + "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/instanceView/read", + "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/networkInterfaces/ipConfigurations/publicIPAddresses/read", + "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/networkInterfaces/ipConfigurations/read", + "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/networkInterfaces/read", + "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/providers/Microsoft.Insights/metricDefinitions/read", + "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/read", + "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/runCommands/read", + "Microsoft.Compute/virtualMachineScaleSets/vmSizes/read" + ], + "NotActions": [], + "AssignableScopes": [ + "/subscriptions/$SUBSCRIPTION_ID" + ] + } + EOF + ``` + +4. Create another JSON file for the remaining permissions required by AKS. + + ```shell + cat << EOF > aks_static_rg_sub_role.json + { + "Name": "Palette Static Placement AKS Cluster Deployer (rg/sub)", + "IsCustom": true, + "Description": "Can use Palette to deploy AKS clusters using static placement. This role contains the permissions that must be applied at the resource group scope level. If deploying multiple clusters in a variety of resource groups within a subscription, apply the role with the subscription as scope instead of the resource group as scope.", + "Actions": [ + "Microsoft.Authorization/classicAdministrators/operationstatuses/read", + "Microsoft.Authorization/classicAdministrators/read", + "Microsoft.Authorization/denyAssignments/read", + "Microsoft.Authorization/diagnosticSettings/read", + "Microsoft.Authorization/diagnosticSettingsCategories/read", + "Microsoft.Authorization/locks/read", + "Microsoft.Authorization/operations/read", + "Microsoft.Authorization/permissions/read", + "Microsoft.Authorization/policyAssignments/privateLinkAssociations/read", + "Microsoft.Authorization/policyAssignments/read", + "Microsoft.Authorization/policyAssignments/resourceManagementPrivateLinks/privateEndpointConnectionProxies/read", + "Microsoft.Authorization/policyAssignments/resourceManagementPrivateLinks/privateEndpointConnections/read", + "Microsoft.Authorization/policyAssignments/resourceManagementPrivateLinks/read", + "Microsoft.Authorization/policyDefinitions/read", + "Microsoft.Authorization/policyExemptions/read", + "Microsoft.Authorization/policySetDefinitions/read", + "Microsoft.Authorization/providerOperations/read", + "Microsoft.Authorization/roleAssignments/read", + "Microsoft.Authorization/roleAssignments/write", + "Microsoft.Authorization/roleAssignmentScheduleInstances/read", + "Microsoft.Authorization/roleAssignmentScheduleRequests/read", + "Microsoft.Authorization/roleAssignmentSchedules/read", + "Microsoft.Authorization/roleDefinitions/read", + "Microsoft.Authorization/roleEligibilityScheduleInstances/read", + "Microsoft.Authorization/roleEligibilityScheduleRequests/read", + "Microsoft.Authorization/roleEligibilitySchedules/read", + "Microsoft.Authorization/roleManagementPolicies/read", + "Microsoft.Authorization/roleManagementPolicyAssignments/read", + "Microsoft.ContainerService/locations/guardrailsVersions/read", + "Microsoft.ContainerService/locations/kubernetesversions/read", + "Microsoft.ContainerService/locations/meshRevisionProfiles/read", + "Microsoft.ContainerService/locations/nodeimageversions/read", + "Microsoft.ContainerService/locations/operationresults/read", + "Microsoft.ContainerService/locations/operations/read", + "Microsoft.ContainerService/locations/orchestrators/read", + "Microsoft.ContainerService/locations/osOptions/read", + "Microsoft.ContainerService/locations/safeguardsVersions/read", + "Microsoft.ContainerService/locations/usages/read", + "Microsoft.ContainerService/managedClusters/abort/action", + "Microsoft.ContainerService/managedClusters/accessProfiles/listCredential/action", + "Microsoft.ContainerService/managedClusters/accessProfiles/read", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/initializerconfigurations/delete", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/initializerconfigurations/read", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/initializerconfigurations/write", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/mutatingwebhookconfigurations/delete", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/mutatingwebhookconfigurations/read", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/mutatingwebhookconfigurations/write", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/validatingwebhookconfigurations/delete", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/validatingwebhookconfigurations/read", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/validatingwebhookconfigurations/write", + "Microsoft.ContainerService/managedClusters/agentPools/abort/action", + "Microsoft.ContainerService/managedClusters/agentPools/delete", + "Microsoft.ContainerService/managedClusters/agentPools/read", + "Microsoft.ContainerService/managedClusters/agentPools/upgradeNodeImageVersion/action", + "Microsoft.ContainerService/managedClusters/agentPools/upgradeNodeImageVersion/write", + "Microsoft.ContainerService/managedClusters/agentPools/upgradeProfiles/read", + "Microsoft.ContainerService/managedClusters/agentPools/write", + "Microsoft.ContainerService/managedClusters/api/read", + "Microsoft.ContainerService/managedClusters/api/v1/read", + "Microsoft.ContainerService/managedClusters/apiextensions.k8s.io/customresourcedefinitions/delete", + "Microsoft.ContainerService/managedClusters/apiextensions.k8s.io/customresourcedefinitions/read", + "Microsoft.ContainerService/managedClusters/apiextensions.k8s.io/customresourcedefinitions/write", + "Microsoft.ContainerService/managedClusters/apiregistration.k8s.io/apiservices/delete", + "Microsoft.ContainerService/managedClusters/apiregistration.k8s.io/apiservices/read", + "Microsoft.ContainerService/managedClusters/apiregistration.k8s.io/apiservices/write", + "Microsoft.ContainerService/managedClusters/apis/admissionregistration.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/admissionregistration.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/admissionregistration.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/apiextensions.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/apiextensions.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/apiextensions.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/apiregistration.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/apiregistration.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/apiregistration.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/apps/read", + "Microsoft.ContainerService/managedClusters/apis/apps/v1/read", + "Microsoft.ContainerService/managedClusters/apis/apps/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/apps/v1beta2/read", + "Microsoft.ContainerService/managedClusters/apis/authentication.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/authentication.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/authentication.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/authorization.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/authorization.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/authorization.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/autoscaling/read", + "Microsoft.ContainerService/managedClusters/apis/autoscaling/v1/read", + "Microsoft.ContainerService/managedClusters/apis/autoscaling/v2beta1/read", + "Microsoft.ContainerService/managedClusters/apis/autoscaling/v2beta2/read", + "Microsoft.ContainerService/managedClusters/apis/batch/read", + "Microsoft.ContainerService/managedClusters/apis/batch/v1/read", + "Microsoft.ContainerService/managedClusters/apis/batch/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/certificates.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/certificates.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/coordination.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/coordination.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/coordination.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/events.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/events.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/extensions/read", + "Microsoft.ContainerService/managedClusters/apis/extensions/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/metrics.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/metrics.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/networking.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/networking.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/networking.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/node.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/node.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/policy/read", + "Microsoft.ContainerService/managedClusters/apis/policy/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/rbac.authorization.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/rbac.authorization.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/rbac.authorization.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/read", + "Microsoft.ContainerService/managedClusters/apis/scheduling.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/scheduling.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/scheduling.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/storage.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/storage.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/storage.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apps/controllerrevisions/delete", + "Microsoft.ContainerService/managedClusters/apps/controllerrevisions/read", + "Microsoft.ContainerService/managedClusters/apps/controllerrevisions/write", + "Microsoft.ContainerService/managedClusters/apps/daemonsets/delete", + "Microsoft.ContainerService/managedClusters/apps/daemonsets/read", + "Microsoft.ContainerService/managedClusters/apps/daemonsets/write", + "Microsoft.ContainerService/managedClusters/apps/deployments/delete", + "Microsoft.ContainerService/managedClusters/apps/deployments/read", + "Microsoft.ContainerService/managedClusters/apps/deployments/write", + "Microsoft.ContainerService/managedClusters/apps/replicasets/delete", + "Microsoft.ContainerService/managedClusters/apps/replicasets/read", + "Microsoft.ContainerService/managedClusters/apps/replicasets/write", + "Microsoft.ContainerService/managedClusters/apps/statefulsets/delete", + "Microsoft.ContainerService/managedClusters/apps/statefulsets/read", + "Microsoft.ContainerService/managedClusters/apps/statefulsets/write", + "Microsoft.ContainerService/managedClusters/authentication.k8s.io/tokenreviews/write", + "Microsoft.ContainerService/managedClusters/authentication.k8s.io/userextras/impersonate/action", + "Microsoft.ContainerService/managedClusters/authorization.k8s.io/localsubjectaccessreviews/write", + "Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectaccessreviews/write", + "Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectrulesreviews/write", + "Microsoft.ContainerService/managedClusters/authorization.k8s.io/subjectaccessreviews/write", + "Microsoft.ContainerService/managedClusters/autoscaling/horizontalpodautoscalers/delete", + "Microsoft.ContainerService/managedClusters/autoscaling/horizontalpodautoscalers/read", + "Microsoft.ContainerService/managedClusters/autoscaling/horizontalpodautoscalers/write", + "Microsoft.ContainerService/managedClusters/availableAgentPoolVersions/read", + "Microsoft.ContainerService/managedClusters/batch/cronjobs/delete", + "Microsoft.ContainerService/managedClusters/batch/cronjobs/read", + "Microsoft.ContainerService/managedClusters/batch/cronjobs/write", + "Microsoft.ContainerService/managedClusters/batch/jobs/delete", + "Microsoft.ContainerService/managedClusters/batch/jobs/read", + "Microsoft.ContainerService/managedClusters/batch/jobs/write", + "Microsoft.ContainerService/managedClusters/bindings/write", + "Microsoft.ContainerService/managedClusters/certificates.k8s.io/certificatesigningrequests/delete", + "Microsoft.ContainerService/managedClusters/certificates.k8s.io/certificatesigningrequests/read", + "Microsoft.ContainerService/managedClusters/certificates.k8s.io/certificatesigningrequests/write", + "Microsoft.ContainerService/managedClusters/cluster.kubernetes-fleet.io/internalmemberclusters/delete", + "Microsoft.ContainerService/managedClusters/cluster.kubernetes-fleet.io/internalmemberclusters/read", + "Microsoft.ContainerService/managedClusters/cluster.kubernetes-fleet.io/internalmemberclusters/write", + "Microsoft.ContainerService/managedClusters/cluster.kubernetes-fleet.io/memberclusters/delete", + "Microsoft.ContainerService/managedClusters/cluster.kubernetes-fleet.io/memberclusters/read", + "Microsoft.ContainerService/managedClusters/cluster.kubernetes-fleet.io/memberclusters/write", + "Microsoft.ContainerService/managedClusters/commandResults/read", + "Microsoft.ContainerService/managedClusters/componentstatuses/delete", + "Microsoft.ContainerService/managedClusters/componentstatuses/read", + "Microsoft.ContainerService/managedClusters/componentstatuses/write", + "Microsoft.ContainerService/managedClusters/configmaps/delete", + "Microsoft.ContainerService/managedClusters/configmaps/read", + "Microsoft.ContainerService/managedClusters/configmaps/write", + "Microsoft.ContainerService/managedClusters/coordination.k8s.io/leases/delete", + "Microsoft.ContainerService/managedClusters/coordination.k8s.io/leases/read", + "Microsoft.ContainerService/managedClusters/coordination.k8s.io/leases/write", + "Microsoft.ContainerService/managedClusters/delete", + "Microsoft.ContainerService/managedClusters/detectors/read", + "Microsoft.ContainerService/managedClusters/diagnosticsState/read", + "Microsoft.ContainerService/managedClusters/discovery.k8s.io/endpointslices/delete", + "Microsoft.ContainerService/managedClusters/discovery.k8s.io/endpointslices/read", + "Microsoft.ContainerService/managedClusters/discovery.k8s.io/endpointslices/write", + "Microsoft.ContainerService/managedClusters/endpoints/delete", + "Microsoft.ContainerService/managedClusters/endpoints/read", + "Microsoft.ContainerService/managedClusters/endpoints/write", + "Microsoft.ContainerService/managedClusters/eventGridFilters/delete", + "Microsoft.ContainerService/managedClusters/eventGridFilters/read", + "Microsoft.ContainerService/managedClusters/eventGridFilters/write", + "Microsoft.ContainerService/managedClusters/events.k8s.io/events/delete", + "Microsoft.ContainerService/managedClusters/events.k8s.io/events/read", + "Microsoft.ContainerService/managedClusters/events.k8s.io/events/write", + "Microsoft.ContainerService/managedClusters/events/delete", + "Microsoft.ContainerService/managedClusters/events/read", + "Microsoft.ContainerService/managedClusters/events/write", + "Microsoft.ContainerService/managedClusters/extensionaddons/delete", + "Microsoft.ContainerService/managedClusters/extensionaddons/read", + "Microsoft.ContainerService/managedClusters/extensionaddons/write", + "Microsoft.ContainerService/managedClusters/extensions/daemonsets/delete", + "Microsoft.ContainerService/managedClusters/extensions/daemonsets/read", + "Microsoft.ContainerService/managedClusters/extensions/daemonsets/write", + "Microsoft.ContainerService/managedClusters/extensions/deployments/delete", + "Microsoft.ContainerService/managedClusters/extensions/deployments/read", + "Microsoft.ContainerService/managedClusters/extensions/deployments/write", + "Microsoft.ContainerService/managedClusters/extensions/ingresses/delete", + "Microsoft.ContainerService/managedClusters/extensions/ingresses/read", + "Microsoft.ContainerService/managedClusters/extensions/ingresses/write", + "Microsoft.ContainerService/managedClusters/extensions/networkpolicies/delete", + "Microsoft.ContainerService/managedClusters/extensions/networkpolicies/read", + "Microsoft.ContainerService/managedClusters/extensions/networkpolicies/write", + "Microsoft.ContainerService/managedClusters/extensions/podsecuritypolicies/delete", + "Microsoft.ContainerService/managedClusters/extensions/podsecuritypolicies/read", + "Microsoft.ContainerService/managedClusters/extensions/podsecuritypolicies/write", + "Microsoft.ContainerService/managedClusters/extensions/replicasets/delete", + "Microsoft.ContainerService/managedClusters/extensions/replicasets/read", + "Microsoft.ContainerService/managedClusters/extensions/replicasets/write", + "Microsoft.ContainerService/managedClusters/flowcontrol.apiserver.k8s.io/flowschemas/delete", + "Microsoft.ContainerService/managedClusters/flowcontrol.apiserver.k8s.io/flowschemas/read", + "Microsoft.ContainerService/managedClusters/flowcontrol.apiserver.k8s.io/flowschemas/write", + "Microsoft.ContainerService/managedClusters/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/delete", + "Microsoft.ContainerService/managedClusters/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/read", + "Microsoft.ContainerService/managedClusters/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/write", + "Microsoft.ContainerService/managedClusters/groups/impersonate/action", + "Microsoft.ContainerService/managedClusters/healthz/autoregister-completion/read", + "Microsoft.ContainerService/managedClusters/healthz/etcd/read", + "Microsoft.ContainerService/managedClusters/healthz/log/read", + "Microsoft.ContainerService/managedClusters/healthz/ping/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/apiservice-openapi-controller/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/apiservice-registration-controller/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/apiservice-status-available-controller/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/bootstrap-controller/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/ca-registration/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/crd-informer-synced/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/generic-apiserver-start-informers/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/kube-apiserver-autoregistration/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/rbac/bootstrap-roles/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/scheduling/bootstrap-system-priority-classes/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/start-apiextensions-controllers/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/start-apiextensions-informers/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/start-kube-aggregator-informers/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/start-kube-apiserver-admission-initializer/read", + "Microsoft.ContainerService/managedClusters/healthz/read", + "Microsoft.ContainerService/managedClusters/limitranges/delete", + "Microsoft.ContainerService/managedClusters/limitranges/read", + "Microsoft.ContainerService/managedClusters/limitranges/write", + "Microsoft.ContainerService/managedClusters/listClusterAdminCredential/action", + "Microsoft.ContainerService/managedClusters/listClusterMonitoringUserCredential/action", + "Microsoft.ContainerService/managedClusters/listClusterUserCredential/action", + "Microsoft.ContainerService/managedClusters/livez/autoregister-completion/read", + "Microsoft.ContainerService/managedClusters/livez/etcd/read", + "Microsoft.ContainerService/managedClusters/livez/log/read", + "Microsoft.ContainerService/managedClusters/livez/ping/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/apiservice-openapi-controller/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/apiservice-registration-controller/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/apiservice-status-available-controller/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/bootstrap-controller/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/ca-registration/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/crd-informer-synced/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/generic-apiserver-start-informers/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/kube-apiserver-autoregistration/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/rbac/bootstrap-roles/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/scheduling/bootstrap-system-priority-classes/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/start-apiextensions-controllers/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/start-apiextensions-informers/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/start-kube-aggregator-informers/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/start-kube-apiserver-admission-initializer/read", + "Microsoft.ContainerService/managedClusters/livez/read", + "Microsoft.ContainerService/managedClusters/loadBalancers/delete", + "Microsoft.ContainerService/managedClusters/loadBalancers/read", + "Microsoft.ContainerService/managedClusters/loadBalancers/write", + "Microsoft.ContainerService/managedClusters/logs/read", + "Microsoft.ContainerService/managedClusters/maintenanceConfigurations/delete", + "Microsoft.ContainerService/managedClusters/maintenanceConfigurations/read", + "Microsoft.ContainerService/managedClusters/maintenanceConfigurations/write", + "Microsoft.ContainerService/managedClusters/meshUpgradeProfiles/read", + "Microsoft.ContainerService/managedClusters/metrics.k8s.io/nodes/read", + "Microsoft.ContainerService/managedClusters/metrics.k8s.io/pods/read", + "Microsoft.ContainerService/managedClusters/metrics/read", + "Microsoft.ContainerService/managedClusters/namespaces/delete", + "Microsoft.ContainerService/managedClusters/namespaces/read", + "Microsoft.ContainerService/managedClusters/namespaces/write", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/ingressclasses/delete", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/ingressclasses/read", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/ingressclasses/write", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/ingresses/delete", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/ingresses/read", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/ingresses/write", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/networkpolicies/delete", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/networkpolicies/read", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/networkpolicies/write", + "Microsoft.ContainerService/managedClusters/networkSecurityPerimeterAssociationProxies/delete", + "Microsoft.ContainerService/managedClusters/networkSecurityPerimeterAssociationProxies/read", + "Microsoft.ContainerService/managedClusters/networkSecurityPerimeterAssociationProxies/write", + "Microsoft.ContainerService/managedClusters/networkSecurityPerimeterConfigurations/read", + "Microsoft.ContainerService/managedClusters/node.k8s.io/runtimeclasses/delete", + "Microsoft.ContainerService/managedClusters/node.k8s.io/runtimeclasses/read", + "Microsoft.ContainerService/managedClusters/node.k8s.io/runtimeclasses/write", + "Microsoft.ContainerService/managedClusters/nodes/delete", + "Microsoft.ContainerService/managedClusters/nodes/read", + "Microsoft.ContainerService/managedClusters/nodes/write", + "Microsoft.ContainerService/managedClusters/openapi/v2/read", + "Microsoft.ContainerService/managedClusters/persistentvolumeclaims/delete", + "Microsoft.ContainerService/managedClusters/persistentvolumeclaims/read", + "Microsoft.ContainerService/managedClusters/persistentvolumeclaims/write", + "Microsoft.ContainerService/managedClusters/persistentvolumes/delete", + "Microsoft.ContainerService/managedClusters/persistentvolumes/read", + "Microsoft.ContainerService/managedClusters/persistentvolumes/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourcebindings/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourcebindings/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourcebindings/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceoverrides/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceoverrides/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceoverrides/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceoverridesnapshots/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceoverridesnapshots/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceoverridesnapshots/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceplacements/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceplacements/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceplacements/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourcesnapshots/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourcesnapshots/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourcesnapshots/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterschedulingpolicysnapshots/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterschedulingpolicysnapshots/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterschedulingpolicysnapshots/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/resourceoverrides/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/resourceoverrides/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/resourceoverrides/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/resourceoverridesnapshots/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/resourceoverridesnapshots/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/resourceoverridesnapshots/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/works/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/works/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/works/write", + "Microsoft.ContainerService/managedClusters/pods/delete", + "Microsoft.ContainerService/managedClusters/pods/exec/action", + "Microsoft.ContainerService/managedClusters/pods/read", + "Microsoft.ContainerService/managedClusters/pods/write", + "Microsoft.ContainerService/managedClusters/podtemplates/delete", + "Microsoft.ContainerService/managedClusters/podtemplates/read", + "Microsoft.ContainerService/managedClusters/podtemplates/write", + "Microsoft.ContainerService/managedClusters/policy/poddisruptionbudgets/delete", + "Microsoft.ContainerService/managedClusters/policy/poddisruptionbudgets/read", + "Microsoft.ContainerService/managedClusters/policy/poddisruptionbudgets/write", + "Microsoft.ContainerService/managedClusters/policy/podsecuritypolicies/delete", + "Microsoft.ContainerService/managedClusters/policy/podsecuritypolicies/read", + "Microsoft.ContainerService/managedClusters/policy/podsecuritypolicies/use/action", + "Microsoft.ContainerService/managedClusters/policy/podsecuritypolicies/write", + "Microsoft.ContainerService/managedClusters/privateEndpointConnections/delete", + "Microsoft.ContainerService/managedClusters/privateEndpointConnections/read", + "Microsoft.ContainerService/managedClusters/privateEndpointConnections/write", + "Microsoft.ContainerService/managedClusters/privateEndpointConnectionsApproval/action", + "Microsoft.ContainerService/managedClusters/providers/Microsoft.Insights/diagnosticSettings/read", + "Microsoft.ContainerService/managedClusters/providers/Microsoft.Insights/diagnosticSettings/write", + "Microsoft.ContainerService/managedClusters/providers/Microsoft.Insights/logDefinitions/read", + "Microsoft.ContainerService/managedClusters/providers/Microsoft.Insights/metricDefinitions/read", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/clusterrolebindings/delete", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/clusterrolebindings/read", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/clusterrolebindings/write", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/clusterroles/bind/action", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/clusterroles/delete", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/clusterroles/escalate/action", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/clusterroles/read", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/clusterroles/write", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/rolebindings/delete", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/rolebindings/read", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/rolebindings/write", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/roles/bind/action", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/roles/delete", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/roles/escalate/action", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/roles/read", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/roles/write", + "Microsoft.ContainerService/managedClusters/read", + "Microsoft.ContainerService/managedClusters/readyz/autoregister-completion/read", + "Microsoft.ContainerService/managedClusters/readyz/etcd/read", + "Microsoft.ContainerService/managedClusters/readyz/log/read", + "Microsoft.ContainerService/managedClusters/readyz/ping/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/apiservice-openapi-controller/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/apiservice-registration-controller/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/apiservice-status-available-controller/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/bootstrap-controller/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/ca-registration/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/crd-informer-synced/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/generic-apiserver-start-informers/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/kube-apiserver-autoregistration/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/rbac/bootstrap-roles/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/scheduling/bootstrap-system-priority-classes/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/start-apiextensions-controllers/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/start-apiextensions-informers/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/start-kube-aggregator-informers/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/start-kube-apiserver-admission-initializer/read", + "Microsoft.ContainerService/managedClusters/readyz/read", + "Microsoft.ContainerService/managedClusters/readyz/shutdown/read", + "Microsoft.ContainerService/managedClusters/replicationcontrollers/delete", + "Microsoft.ContainerService/managedClusters/replicationcontrollers/read", + "Microsoft.ContainerService/managedClusters/replicationcontrollers/write", + "Microsoft.ContainerService/managedClusters/resetAADProfile/action", + "Microsoft.ContainerService/managedClusters/resetMetrics/read", + "Microsoft.ContainerService/managedClusters/resetServicePrincipalProfile/action", + "Microsoft.ContainerService/managedClusters/resolvePrivateLinkServiceId/action", + "Microsoft.ContainerService/managedClusters/resourcequotas/delete", + "Microsoft.ContainerService/managedClusters/resourcequotas/read", + "Microsoft.ContainerService/managedClusters/resourcequotas/write", + "Microsoft.ContainerService/managedClusters/rotateClusterCertificates/action", + "Microsoft.ContainerService/managedClusters/runCommand/action", + "Microsoft.ContainerService/managedClusters/scheduling.k8s.io/priorityclasses/delete", + "Microsoft.ContainerService/managedClusters/scheduling.k8s.io/priorityclasses/read", + "Microsoft.ContainerService/managedClusters/scheduling.k8s.io/priorityclasses/write", + "Microsoft.ContainerService/managedClusters/secrets/delete", + "Microsoft.ContainerService/managedClusters/secrets/read", + "Microsoft.ContainerService/managedClusters/secrets/write", + "Microsoft.ContainerService/managedClusters/serviceaccounts/delete", + "Microsoft.ContainerService/managedClusters/serviceaccounts/impersonate/action", + "Microsoft.ContainerService/managedClusters/serviceaccounts/read", + "Microsoft.ContainerService/managedClusters/serviceaccounts/write", + "Microsoft.ContainerService/managedClusters/services/delete", + "Microsoft.ContainerService/managedClusters/services/read", + "Microsoft.ContainerService/managedClusters/services/write", + "Microsoft.ContainerService/managedClusters/start/action", + "Microsoft.ContainerService/managedClusters/stop/action", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csidrivers/delete", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csidrivers/read", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csidrivers/write", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csinodes/delete", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csinodes/read", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csinodes/write", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csistoragecapacities/delete", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csistoragecapacities/read", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csistoragecapacities/write", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/storageclasses/delete", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/storageclasses/read", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/storageclasses/write", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/volumeattachments/delete", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/volumeattachments/read", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/volumeattachments/write", + "Microsoft.ContainerService/managedClusters/swagger-api/read", + "Microsoft.ContainerService/managedClusters/swagger-ui/read", + "Microsoft.ContainerService/managedClusters/trustedAccessRoleBindings/delete", + "Microsoft.ContainerService/managedClusters/trustedAccessRoleBindings/read", + "Microsoft.ContainerService/managedClusters/trustedAccessRoleBindings/write", + "Microsoft.ContainerService/managedClusters/ui/read", + "Microsoft.ContainerService/managedClusters/unpinManagedCluster/action", + "Microsoft.ContainerService/managedClusters/upgradeProfiles/read", + "Microsoft.ContainerService/managedClusters/users/impersonate/action", + "Microsoft.ContainerService/managedClusters/version/read", + "Microsoft.ContainerService/managedClusters/write", + "Microsoft.ContainerService/managedclustersnapshots/delete", + "Microsoft.ContainerService/managedclustersnapshots/read", + "Microsoft.ContainerService/managedclustersnapshots/write", + "Microsoft.ContainerService/snapshots/delete", + "Microsoft.ContainerService/snapshots/read", + "Microsoft.ContainerService/snapshots/write", + "Microsoft.ManagedIdentity/userAssignedIdentities/assign/action", + "Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials/read", + "Microsoft.ResourceHealth/availabilityStatuses/read", + "Microsoft.Resources/subscriptions/resourceGroups/read" + ], + "NotActions": [], + "AssignableScopes": [ + "/subscriptions/$SUBSCRIPTION_ID" + ] + } + EOF + ``` + +5. Create a role for each of the JSON files you created in the previous steps. Issue the following commands to create + the roles. + + ```shell + az role definition create --role-definition @aks_static_vnet_role.json --output table + az role definition create --role-definition @aks_static_sub_role.json --output table + az role definition create --role-definition @aks_static_rg_sub_role.json --output table + ``` + +6. Export the + [security principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/overview#security-principal) + object ID you want to use with Palette to a variable. + + ```shell + export ASSIGNEE="" + ``` + +7. Export the resource group name and virtual network name to a variable. + + ```shell + export RESOURCE_GROUP_NAME="" + export VNET_NAME="" + ``` + +8. Assign the roles to the service principal. Use the following commands to assign the roles. + + ```json + az role assignment create --assignee $ASSIGNEE \ + --role "Palette Static Placement AKS Cluster Deployer (vnet)" \ + --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME/providers/Microsoft.Network/virtualNetworks/$VNET_NAME" + ``` + + ```json + az role assignment create --assignee $ASSIGNEE \ + --role "Palette Static Placement AKS Cluster Deployer (sub)" \ + --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME" + ``` + + ```json + az role assignment create --assignee $ASSIGNEE \ + --role "Palette Static Placement AKS Cluster Deployer (rg/sub)" \ + --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME" + ``` + +#### Validate 1. Log in to the Azure portal. @@ -965,3 +1014,582 @@ Select the tab below to view the required permissions and steps for the static p 3. Review the role, or roles if you created multiple. Review the role assignments to ensure the service principal has the correct permissions assigned. + +### Dynamic Placement {#aks-dynamic-placement} + +The permission requirements for AKS dynamic placement are similar to the AKS static placement requirements. The +difference is that a set of permissions are required at the subscription level. The remaining permissions can be applied +at the resource group level, or at the subscription level if you are deploying multiple clusters in a variety of +resource groups within a subscription. + +#### Prerequisites + +- Azure CLI installed on your local machine. Refer to the + [Azure CLI Install Guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) for installation instructions. + +- The Azure subscription ID you want to use with Palette. + +- The Security Principal Object ID you want to use with Palette. You can retrieve it from the Entra ID section of the + Azure Portal. + +- A terminal or command prompt to issue the Azure CLI commands. + +- The Azure Security Principal Object ID you want to use with Palette. The Security Principal Object ID can represent a + user, group, or service principal. + +#### Create Role and Assign Permissions + +1. Open a terminal or command prompt. Export your Azure subscription ID to a variable. + + ```shell + export SUBSCRIPTION_ID= + ``` + +2. Issue the following command to create a JSON file containing all the required subscription level permissions. + + ```shell + cat << EOF > aks_dynamic_sub_role.json + { + "Name": "Palette Dynamic Placement AKS Cluster Deployer (sub)", + "IsCustom": true, + "Description": "Can use Palette to deploy AKS clusters using dynamic placement. This role has the permissions required at the subscription scope level.", + "Actions": [ + "Microsoft.Compute/virtualMachineScaleSets/extensions/read", + "Microsoft.Compute/virtualMachineScaleSets/extensions/roles/read", + "Microsoft.Compute/virtualMachineScaleSets/instanceView/read", + "Microsoft.Compute/virtualMachineScaleSets/networkInterfaces/read", + "Microsoft.Compute/virtualMachineScaleSets/osUpgradeHistory/read", + "Microsoft.Compute/virtualMachineScaleSets/providers/Microsoft.Insights/diagnosticSettings/read", + "Microsoft.Compute/virtualMachineScaleSets/providers/Microsoft.Insights/logDefinitions/read", + "Microsoft.Compute/virtualMachineScaleSets/providers/Microsoft.Insights/metricDefinitions/read", + "Microsoft.Compute/virtualMachineScaleSets/publicIPAddresses/read", + "Microsoft.Compute/virtualMachineScaleSets/read", + "Microsoft.Compute/virtualMachineScaleSets/rollingUpgrades/read", + "Microsoft.Compute/virtualMachineScaleSets/skus/read", + "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/extensions/read", + "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/instanceView/read", + "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/networkInterfaces/ipConfigurations/publicIPAddresses/read", + "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/networkInterfaces/ipConfigurations/read", + "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/networkInterfaces/read", + "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/providers/Microsoft.Insights/metricDefinitions/read", + "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/read", + "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/runCommands/read", + "Microsoft.Compute/virtualMachineScaleSets/vmSizes/read" + ], + "NotActions": [], + "AssignableScopes": [ + "/subscriptions/$SUBSCRIPTION_ID" + ] + } + EOF + ``` + +3. Create another JSON file containing the remaining permissions required for AKS. + + ```shell + cat << EOF > aks_dynamic_rg_sub_role.json + { + "Name": "Palette Dynamic Placement AKS Cluster Deployer (rg/sub)", + "IsCustom": true, + "Description": "Can use Palette to deploy AKS clusters using dynamic placement. This role has the permissions required at the resource group scope level. If deploying multiple clusters in a variety of resource groups within a subscription, apply the role with the subscription as scope instead of the resource group as scope.", + "Actions": [ + "Microsoft.Authorization/classicAdministrators/operationstatuses/read", + "Microsoft.Authorization/classicAdministrators/read", + "Microsoft.Authorization/denyAssignments/read", + "Microsoft.Authorization/diagnosticSettings/read", + "Microsoft.Authorization/diagnosticSettingsCategories/read", + "Microsoft.Authorization/locks/read", + "Microsoft.Authorization/operations/read", + "Microsoft.Authorization/permissions/read", + "Microsoft.Authorization/policyAssignments/privateLinkAssociations/read", + "Microsoft.Authorization/policyAssignments/read", + "Microsoft.Authorization/policyAssignments/resourceManagementPrivateLinks/privateEndpointConnectionProxies/read", + "Microsoft.Authorization/policyAssignments/resourceManagementPrivateLinks/privateEndpointConnections/read", + "Microsoft.Authorization/policyAssignments/resourceManagementPrivateLinks/read", + "Microsoft.Authorization/policyDefinitions/read", + "Microsoft.Authorization/policyExemptions/read", + "Microsoft.Authorization/policySetDefinitions/read", + "Microsoft.Authorization/providerOperations/read", + "Microsoft.Authorization/roleAssignments/read", + "Microsoft.Authorization/roleAssignmentScheduleInstances/read", + "Microsoft.Authorization/roleAssignmentScheduleRequests/read", + "Microsoft.Authorization/roleAssignmentSchedules/read", + "Microsoft.Authorization/roleDefinitions/read", + "Microsoft.Authorization/roleEligibilityScheduleInstances/read", + "Microsoft.Authorization/roleEligibilityScheduleRequests/read", + "Microsoft.Authorization/roleEligibilitySchedules/read", + "Microsoft.Authorization/roleManagementPolicies/read", + "Microsoft.Authorization/roleManagementPolicyAssignments/read", + "Microsoft.ContainerService/locations/guardrailsVersions/read", + "Microsoft.ContainerService/locations/kubernetesversions/read", + "Microsoft.ContainerService/locations/meshRevisionProfiles/read", + "Microsoft.ContainerService/locations/nodeimageversions/read", + "Microsoft.ContainerService/locations/operationresults/read", + "Microsoft.ContainerService/locations/operations/read", + "Microsoft.ContainerService/locations/orchestrators/read", + "Microsoft.ContainerService/locations/osOptions/read", + "Microsoft.ContainerService/locations/safeguardsVersions/read", + "Microsoft.ContainerService/locations/usages/read", + "Microsoft.ContainerService/managedClusters/abort/action", + "Microsoft.ContainerService/managedClusters/accessProfiles/listCredential/action", + "Microsoft.ContainerService/managedClusters/accessProfiles/read", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/initializerconfigurations/delete", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/initializerconfigurations/read", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/initializerconfigurations/write", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/mutatingwebhookconfigurations/delete", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/mutatingwebhookconfigurations/read", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/mutatingwebhookconfigurations/write", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/validatingwebhookconfigurations/delete", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/validatingwebhookconfigurations/read", + "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/validatingwebhookconfigurations/write", + "Microsoft.ContainerService/managedClusters/agentPools/abort/action", + "Microsoft.ContainerService/managedClusters/agentPools/delete", + "Microsoft.ContainerService/managedClusters/agentPools/read", + "Microsoft.ContainerService/managedClusters/agentPools/upgradeNodeImageVersion/action", + "Microsoft.ContainerService/managedClusters/agentPools/upgradeNodeImageVersion/write", + "Microsoft.ContainerService/managedClusters/agentPools/upgradeProfiles/read", + "Microsoft.ContainerService/managedClusters/agentPools/write", + "Microsoft.ContainerService/managedClusters/api/read", + "Microsoft.ContainerService/managedClusters/api/v1/read", + "Microsoft.ContainerService/managedClusters/apiextensions.k8s.io/customresourcedefinitions/delete", + "Microsoft.ContainerService/managedClusters/apiextensions.k8s.io/customresourcedefinitions/read", + "Microsoft.ContainerService/managedClusters/apiextensions.k8s.io/customresourcedefinitions/write", + "Microsoft.ContainerService/managedClusters/apiregistration.k8s.io/apiservices/delete", + "Microsoft.ContainerService/managedClusters/apiregistration.k8s.io/apiservices/read", + "Microsoft.ContainerService/managedClusters/apiregistration.k8s.io/apiservices/write", + "Microsoft.ContainerService/managedClusters/apis/admissionregistration.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/admissionregistration.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/admissionregistration.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/apiextensions.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/apiextensions.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/apiextensions.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/apiregistration.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/apiregistration.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/apiregistration.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/apps/read", + "Microsoft.ContainerService/managedClusters/apis/apps/v1/read", + "Microsoft.ContainerService/managedClusters/apis/apps/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/apps/v1beta2/read", + "Microsoft.ContainerService/managedClusters/apis/authentication.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/authentication.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/authentication.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/authorization.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/authorization.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/authorization.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/autoscaling/read", + "Microsoft.ContainerService/managedClusters/apis/autoscaling/v1/read", + "Microsoft.ContainerService/managedClusters/apis/autoscaling/v2beta1/read", + "Microsoft.ContainerService/managedClusters/apis/autoscaling/v2beta2/read", + "Microsoft.ContainerService/managedClusters/apis/batch/read", + "Microsoft.ContainerService/managedClusters/apis/batch/v1/read", + "Microsoft.ContainerService/managedClusters/apis/batch/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/certificates.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/certificates.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/coordination.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/coordination.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/coordination.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/events.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/events.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/extensions/read", + "Microsoft.ContainerService/managedClusters/apis/extensions/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/metrics.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/metrics.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/networking.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/networking.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/networking.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/node.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/node.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/policy/read", + "Microsoft.ContainerService/managedClusters/apis/policy/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/rbac.authorization.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/rbac.authorization.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/rbac.authorization.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/read", + "Microsoft.ContainerService/managedClusters/apis/scheduling.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/scheduling.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/scheduling.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apis/storage.k8s.io/read", + "Microsoft.ContainerService/managedClusters/apis/storage.k8s.io/v1/read", + "Microsoft.ContainerService/managedClusters/apis/storage.k8s.io/v1beta1/read", + "Microsoft.ContainerService/managedClusters/apps/controllerrevisions/delete", + "Microsoft.ContainerService/managedClusters/apps/controllerrevisions/read", + "Microsoft.ContainerService/managedClusters/apps/controllerrevisions/write", + "Microsoft.ContainerService/managedClusters/apps/daemonsets/delete", + "Microsoft.ContainerService/managedClusters/apps/daemonsets/read", + "Microsoft.ContainerService/managedClusters/apps/daemonsets/write", + "Microsoft.ContainerService/managedClusters/apps/deployments/delete", + "Microsoft.ContainerService/managedClusters/apps/deployments/read", + "Microsoft.ContainerService/managedClusters/apps/deployments/write", + "Microsoft.ContainerService/managedClusters/apps/replicasets/delete", + "Microsoft.ContainerService/managedClusters/apps/replicasets/read", + "Microsoft.ContainerService/managedClusters/apps/replicasets/write", + "Microsoft.ContainerService/managedClusters/apps/statefulsets/delete", + "Microsoft.ContainerService/managedClusters/apps/statefulsets/read", + "Microsoft.ContainerService/managedClusters/apps/statefulsets/write", + "Microsoft.ContainerService/managedClusters/authentication.k8s.io/tokenreviews/write", + "Microsoft.ContainerService/managedClusters/authentication.k8s.io/userextras/impersonate/action", + "Microsoft.ContainerService/managedClusters/authorization.k8s.io/localsubjectaccessreviews/write", + "Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectaccessreviews/write", + "Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectrulesreviews/write", + "Microsoft.ContainerService/managedClusters/authorization.k8s.io/subjectaccessreviews/write", + "Microsoft.ContainerService/managedClusters/autoscaling/horizontalpodautoscalers/delete", + "Microsoft.ContainerService/managedClusters/autoscaling/horizontalpodautoscalers/read", + "Microsoft.ContainerService/managedClusters/autoscaling/horizontalpodautoscalers/write", + "Microsoft.ContainerService/managedClusters/availableAgentPoolVersions/read", + "Microsoft.ContainerService/managedClusters/batch/cronjobs/delete", + "Microsoft.ContainerService/managedClusters/batch/cronjobs/read", + "Microsoft.ContainerService/managedClusters/batch/cronjobs/write", + "Microsoft.ContainerService/managedClusters/batch/jobs/delete", + "Microsoft.ContainerService/managedClusters/batch/jobs/read", + "Microsoft.ContainerService/managedClusters/batch/jobs/write", + "Microsoft.ContainerService/managedClusters/bindings/write", + "Microsoft.ContainerService/managedClusters/certificates.k8s.io/certificatesigningrequests/delete", + "Microsoft.ContainerService/managedClusters/certificates.k8s.io/certificatesigningrequests/read", + "Microsoft.ContainerService/managedClusters/certificates.k8s.io/certificatesigningrequests/write", + "Microsoft.ContainerService/managedClusters/cluster.kubernetes-fleet.io/internalmemberclusters/delete", + "Microsoft.ContainerService/managedClusters/cluster.kubernetes-fleet.io/internalmemberclusters/read", + "Microsoft.ContainerService/managedClusters/cluster.kubernetes-fleet.io/internalmemberclusters/write", + "Microsoft.ContainerService/managedClusters/cluster.kubernetes-fleet.io/memberclusters/delete", + "Microsoft.ContainerService/managedClusters/cluster.kubernetes-fleet.io/memberclusters/read", + "Microsoft.ContainerService/managedClusters/cluster.kubernetes-fleet.io/memberclusters/write", + "Microsoft.ContainerService/managedClusters/commandResults/read", + "Microsoft.ContainerService/managedClusters/componentstatuses/delete", + "Microsoft.ContainerService/managedClusters/componentstatuses/read", + "Microsoft.ContainerService/managedClusters/componentstatuses/write", + "Microsoft.ContainerService/managedClusters/configmaps/delete", + "Microsoft.ContainerService/managedClusters/configmaps/read", + "Microsoft.ContainerService/managedClusters/configmaps/write", + "Microsoft.ContainerService/managedClusters/coordination.k8s.io/leases/delete", + "Microsoft.ContainerService/managedClusters/coordination.k8s.io/leases/read", + "Microsoft.ContainerService/managedClusters/coordination.k8s.io/leases/write", + "Microsoft.ContainerService/managedClusters/delete", + "Microsoft.ContainerService/managedClusters/detectors/read", + "Microsoft.ContainerService/managedClusters/diagnosticsState/read", + "Microsoft.ContainerService/managedClusters/discovery.k8s.io/endpointslices/delete", + "Microsoft.ContainerService/managedClusters/discovery.k8s.io/endpointslices/read", + "Microsoft.ContainerService/managedClusters/discovery.k8s.io/endpointslices/write", + "Microsoft.ContainerService/managedClusters/endpoints/delete", + "Microsoft.ContainerService/managedClusters/endpoints/read", + "Microsoft.ContainerService/managedClusters/endpoints/write", + "Microsoft.ContainerService/managedClusters/eventGridFilters/delete", + "Microsoft.ContainerService/managedClusters/eventGridFilters/read", + "Microsoft.ContainerService/managedClusters/eventGridFilters/write", + "Microsoft.ContainerService/managedClusters/events.k8s.io/events/delete", + "Microsoft.ContainerService/managedClusters/events.k8s.io/events/read", + "Microsoft.ContainerService/managedClusters/events.k8s.io/events/write", + "Microsoft.ContainerService/managedClusters/events/delete", + "Microsoft.ContainerService/managedClusters/events/read", + "Microsoft.ContainerService/managedClusters/events/write", + "Microsoft.ContainerService/managedClusters/extensionaddons/delete", + "Microsoft.ContainerService/managedClusters/extensionaddons/read", + "Microsoft.ContainerService/managedClusters/extensionaddons/write", + "Microsoft.ContainerService/managedClusters/extensions/daemonsets/delete", + "Microsoft.ContainerService/managedClusters/extensions/daemonsets/read", + "Microsoft.ContainerService/managedClusters/extensions/daemonsets/write", + "Microsoft.ContainerService/managedClusters/extensions/deployments/delete", + "Microsoft.ContainerService/managedClusters/extensions/deployments/read", + "Microsoft.ContainerService/managedClusters/extensions/deployments/write", + "Microsoft.ContainerService/managedClusters/extensions/ingresses/delete", + "Microsoft.ContainerService/managedClusters/extensions/ingresses/read", + "Microsoft.ContainerService/managedClusters/extensions/ingresses/write", + "Microsoft.ContainerService/managedClusters/extensions/networkpolicies/delete", + "Microsoft.ContainerService/managedClusters/extensions/networkpolicies/read", + "Microsoft.ContainerService/managedClusters/extensions/networkpolicies/write", + "Microsoft.ContainerService/managedClusters/extensions/podsecuritypolicies/delete", + "Microsoft.ContainerService/managedClusters/extensions/podsecuritypolicies/read", + "Microsoft.ContainerService/managedClusters/extensions/podsecuritypolicies/write", + "Microsoft.ContainerService/managedClusters/extensions/replicasets/delete", + "Microsoft.ContainerService/managedClusters/extensions/replicasets/read", + "Microsoft.ContainerService/managedClusters/extensions/replicasets/write", + "Microsoft.ContainerService/managedClusters/flowcontrol.apiserver.k8s.io/flowschemas/delete", + "Microsoft.ContainerService/managedClusters/flowcontrol.apiserver.k8s.io/flowschemas/read", + "Microsoft.ContainerService/managedClusters/flowcontrol.apiserver.k8s.io/flowschemas/write", + "Microsoft.ContainerService/managedClusters/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/delete", + "Microsoft.ContainerService/managedClusters/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/read", + "Microsoft.ContainerService/managedClusters/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/write", + "Microsoft.ContainerService/managedClusters/groups/impersonate/action", + "Microsoft.ContainerService/managedClusters/healthz/autoregister-completion/read", + "Microsoft.ContainerService/managedClusters/healthz/etcd/read", + "Microsoft.ContainerService/managedClusters/healthz/log/read", + "Microsoft.ContainerService/managedClusters/healthz/ping/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/apiservice-openapi-controller/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/apiservice-registration-controller/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/apiservice-status-available-controller/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/bootstrap-controller/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/ca-registration/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/crd-informer-synced/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/generic-apiserver-start-informers/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/kube-apiserver-autoregistration/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/rbac/bootstrap-roles/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/scheduling/bootstrap-system-priority-classes/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/start-apiextensions-controllers/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/start-apiextensions-informers/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/start-kube-aggregator-informers/read", + "Microsoft.ContainerService/managedClusters/healthz/poststarthook/start-kube-apiserver-admission-initializer/read", + "Microsoft.ContainerService/managedClusters/healthz/read", + "Microsoft.ContainerService/managedClusters/limitranges/delete", + "Microsoft.ContainerService/managedClusters/limitranges/read", + "Microsoft.ContainerService/managedClusters/limitranges/write", + "Microsoft.ContainerService/managedClusters/listClusterAdminCredential/action", + "Microsoft.ContainerService/managedClusters/listClusterMonitoringUserCredential/action", + "Microsoft.ContainerService/managedClusters/listClusterUserCredential/action", + "Microsoft.ContainerService/managedClusters/livez/autoregister-completion/read", + "Microsoft.ContainerService/managedClusters/livez/etcd/read", + "Microsoft.ContainerService/managedClusters/livez/log/read", + "Microsoft.ContainerService/managedClusters/livez/ping/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/apiservice-openapi-controller/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/apiservice-registration-controller/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/apiservice-status-available-controller/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/bootstrap-controller/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/ca-registration/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/crd-informer-synced/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/generic-apiserver-start-informers/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/kube-apiserver-autoregistration/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/rbac/bootstrap-roles/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/scheduling/bootstrap-system-priority-classes/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/start-apiextensions-controllers/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/start-apiextensions-informers/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/start-kube-aggregator-informers/read", + "Microsoft.ContainerService/managedClusters/livez/poststarthook/start-kube-apiserver-admission-initializer/read", + "Microsoft.ContainerService/managedClusters/livez/read", + "Microsoft.ContainerService/managedClusters/loadBalancers/delete", + "Microsoft.ContainerService/managedClusters/loadBalancers/read", + "Microsoft.ContainerService/managedClusters/loadBalancers/write", + "Microsoft.ContainerService/managedClusters/logs/read", + "Microsoft.ContainerService/managedClusters/maintenanceConfigurations/delete", + "Microsoft.ContainerService/managedClusters/maintenanceConfigurations/read", + "Microsoft.ContainerService/managedClusters/maintenanceConfigurations/write", + "Microsoft.ContainerService/managedClusters/meshUpgradeProfiles/read", + "Microsoft.ContainerService/managedClusters/metrics.k8s.io/nodes/read", + "Microsoft.ContainerService/managedClusters/metrics.k8s.io/pods/read", + "Microsoft.ContainerService/managedClusters/metrics/read", + "Microsoft.ContainerService/managedClusters/namespaces/delete", + "Microsoft.ContainerService/managedClusters/namespaces/read", + "Microsoft.ContainerService/managedClusters/namespaces/write", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/ingressclasses/delete", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/ingressclasses/read", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/ingressclasses/write", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/ingresses/delete", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/ingresses/read", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/ingresses/write", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/networkpolicies/delete", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/networkpolicies/read", + "Microsoft.ContainerService/managedClusters/networking.k8s.io/networkpolicies/write", + "Microsoft.ContainerService/managedClusters/networkSecurityPerimeterAssociationProxies/delete", + "Microsoft.ContainerService/managedClusters/networkSecurityPerimeterAssociationProxies/read", + "Microsoft.ContainerService/managedClusters/networkSecurityPerimeterAssociationProxies/write", + "Microsoft.ContainerService/managedClusters/networkSecurityPerimeterConfigurations/read", + "Microsoft.ContainerService/managedClusters/node.k8s.io/runtimeclasses/delete", + "Microsoft.ContainerService/managedClusters/node.k8s.io/runtimeclasses/read", + "Microsoft.ContainerService/managedClusters/node.k8s.io/runtimeclasses/write", + "Microsoft.ContainerService/managedClusters/nodes/delete", + "Microsoft.ContainerService/managedClusters/nodes/read", + "Microsoft.ContainerService/managedClusters/nodes/write", + "Microsoft.ContainerService/managedClusters/openapi/v2/read", + "Microsoft.ContainerService/managedClusters/persistentvolumeclaims/delete", + "Microsoft.ContainerService/managedClusters/persistentvolumeclaims/read", + "Microsoft.ContainerService/managedClusters/persistentvolumeclaims/write", + "Microsoft.ContainerService/managedClusters/persistentvolumes/delete", + "Microsoft.ContainerService/managedClusters/persistentvolumes/read", + "Microsoft.ContainerService/managedClusters/persistentvolumes/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourcebindings/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourcebindings/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourcebindings/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceoverrides/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceoverrides/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceoverrides/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceoverridesnapshots/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceoverridesnapshots/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceoverridesnapshots/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceplacements/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceplacements/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourceplacements/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourcesnapshots/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourcesnapshots/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterresourcesnapshots/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterschedulingpolicysnapshots/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterschedulingpolicysnapshots/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/clusterschedulingpolicysnapshots/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/resourceoverrides/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/resourceoverrides/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/resourceoverrides/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/resourceoverridesnapshots/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/resourceoverridesnapshots/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/resourceoverridesnapshots/write", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/works/delete", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/works/read", + "Microsoft.ContainerService/managedClusters/placement.kubernetes-fleet.io/works/write", + "Microsoft.ContainerService/managedClusters/pods/delete", + "Microsoft.ContainerService/managedClusters/pods/exec/action", + "Microsoft.ContainerService/managedClusters/pods/read", + "Microsoft.ContainerService/managedClusters/pods/write", + "Microsoft.ContainerService/managedClusters/podtemplates/delete", + "Microsoft.ContainerService/managedClusters/podtemplates/read", + "Microsoft.ContainerService/managedClusters/podtemplates/write", + "Microsoft.ContainerService/managedClusters/policy/poddisruptionbudgets/delete", + "Microsoft.ContainerService/managedClusters/policy/poddisruptionbudgets/read", + "Microsoft.ContainerService/managedClusters/policy/poddisruptionbudgets/write", + "Microsoft.ContainerService/managedClusters/policy/podsecuritypolicies/delete", + "Microsoft.ContainerService/managedClusters/policy/podsecuritypolicies/read", + "Microsoft.ContainerService/managedClusters/policy/podsecuritypolicies/use/action", + "Microsoft.ContainerService/managedClusters/policy/podsecuritypolicies/write", + "Microsoft.ContainerService/managedClusters/privateEndpointConnections/delete", + "Microsoft.ContainerService/managedClusters/privateEndpointConnections/read", + "Microsoft.ContainerService/managedClusters/privateEndpointConnections/write", + "Microsoft.ContainerService/managedClusters/privateEndpointConnectionsApproval/action", + "Microsoft.ContainerService/managedClusters/providers/Microsoft.Insights/diagnosticSettings/read", + "Microsoft.ContainerService/managedClusters/providers/Microsoft.Insights/diagnosticSettings/write", + "Microsoft.ContainerService/managedClusters/providers/Microsoft.Insights/logDefinitions/read", + "Microsoft.ContainerService/managedClusters/providers/Microsoft.Insights/metricDefinitions/read", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/clusterrolebindings/delete", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/clusterrolebindings/read", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/clusterrolebindings/write", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/clusterroles/bind/action", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/clusterroles/delete", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/clusterroles/escalate/action", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/clusterroles/read", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/clusterroles/write", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/rolebindings/delete", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/rolebindings/read", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/rolebindings/write", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/roles/bind/action", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/roles/delete", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/roles/escalate/action", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/roles/read", + "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/roles/write", + "Microsoft.ContainerService/managedClusters/read", + "Microsoft.ContainerService/managedClusters/readyz/autoregister-completion/read", + "Microsoft.ContainerService/managedClusters/readyz/etcd/read", + "Microsoft.ContainerService/managedClusters/readyz/log/read", + "Microsoft.ContainerService/managedClusters/readyz/ping/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/apiservice-openapi-controller/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/apiservice-registration-controller/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/apiservice-status-available-controller/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/bootstrap-controller/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/ca-registration/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/crd-informer-synced/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/generic-apiserver-start-informers/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/kube-apiserver-autoregistration/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/rbac/bootstrap-roles/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/scheduling/bootstrap-system-priority-classes/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/start-apiextensions-controllers/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/start-apiextensions-informers/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/start-kube-aggregator-informers/read", + "Microsoft.ContainerService/managedClusters/readyz/poststarthook/start-kube-apiserver-admission-initializer/read", + "Microsoft.ContainerService/managedClusters/readyz/read", + "Microsoft.ContainerService/managedClusters/readyz/shutdown/read", + "Microsoft.ContainerService/managedClusters/replicationcontrollers/delete", + "Microsoft.ContainerService/managedClusters/replicationcontrollers/read", + "Microsoft.ContainerService/managedClusters/replicationcontrollers/write", + "Microsoft.ContainerService/managedClusters/resetAADProfile/action", + "Microsoft.ContainerService/managedClusters/resetMetrics/read", + "Microsoft.ContainerService/managedClusters/resetServicePrincipalProfile/action", + "Microsoft.ContainerService/managedClusters/resolvePrivateLinkServiceId/action", + "Microsoft.ContainerService/managedClusters/resourcequotas/delete", + "Microsoft.ContainerService/managedClusters/resourcequotas/read", + "Microsoft.ContainerService/managedClusters/resourcequotas/write", + "Microsoft.ContainerService/managedClusters/rotateClusterCertificates/action", + "Microsoft.ContainerService/managedClusters/runCommand/action", + "Microsoft.ContainerService/managedClusters/scheduling.k8s.io/priorityclasses/delete", + "Microsoft.ContainerService/managedClusters/scheduling.k8s.io/priorityclasses/read", + "Microsoft.ContainerService/managedClusters/scheduling.k8s.io/priorityclasses/write", + "Microsoft.ContainerService/managedClusters/secrets/delete", + "Microsoft.ContainerService/managedClusters/secrets/read", + "Microsoft.ContainerService/managedClusters/secrets/write", + "Microsoft.ContainerService/managedClusters/serviceaccounts/delete", + "Microsoft.ContainerService/managedClusters/serviceaccounts/impersonate/action", + "Microsoft.ContainerService/managedClusters/serviceaccounts/read", + "Microsoft.ContainerService/managedClusters/serviceaccounts/write", + "Microsoft.ContainerService/managedClusters/services/delete", + "Microsoft.ContainerService/managedClusters/services/read", + "Microsoft.ContainerService/managedClusters/services/write", + "Microsoft.ContainerService/managedClusters/start/action", + "Microsoft.ContainerService/managedClusters/stop/action", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csidrivers/delete", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csidrivers/read", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csidrivers/write", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csinodes/delete", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csinodes/read", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csinodes/write", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csistoragecapacities/delete", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csistoragecapacities/read", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/csistoragecapacities/write", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/storageclasses/delete", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/storageclasses/read", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/storageclasses/write", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/volumeattachments/delete", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/volumeattachments/read", + "Microsoft.ContainerService/managedClusters/storage.k8s.io/volumeattachments/write", + "Microsoft.ContainerService/managedClusters/swagger-api/read", + "Microsoft.ContainerService/managedClusters/swagger-ui/read", + "Microsoft.ContainerService/managedClusters/trustedAccessRoleBindings/delete", + "Microsoft.ContainerService/managedClusters/trustedAccessRoleBindings/read", + "Microsoft.ContainerService/managedClusters/trustedAccessRoleBindings/write", + "Microsoft.ContainerService/managedClusters/ui/read", + "Microsoft.ContainerService/managedClusters/unpinManagedCluster/action", + "Microsoft.ContainerService/managedClusters/upgradeProfiles/read", + "Microsoft.ContainerService/managedClusters/users/impersonate/action", + "Microsoft.ContainerService/managedClusters/version/read", + "Microsoft.ContainerService/managedClusters/write", + "Microsoft.ContainerService/managedclustersnapshots/delete", + "Microsoft.ContainerService/managedclustersnapshots/read", + "Microsoft.ContainerService/managedclustersnapshots/write", + "Microsoft.ContainerService/snapshots/delete", + "Microsoft.ContainerService/snapshots/read", + "Microsoft.ContainerService/snapshots/write", + "Microsoft.ManagedIdentity/userAssignedIdentities/assign/action", + "Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials/read", + "Microsoft.Network/virtualNetworks/delete", + "Microsoft.Network/virtualNetworks/read", + "Microsoft.Network/virtualNetworks/subnets/delete", + "Microsoft.Network/virtualNetworks/subnets/join/action", + "Microsoft.Network/virtualNetworks/subnets/read", + "Microsoft.Network/virtualNetworks/subnets/write", + "Microsoft.Network/virtualNetworks/write", + "Microsoft.ResourceHealth/availabilityStatuses/read", + "Microsoft.Resources/subscriptions/resourceGroups/read" + ], + "NotActions": [], + "AssignableScopes": [ + "/subscriptions/$SUBSCRIPTION_ID" + ] + } + EOF + ``` + +4. Create a role for each of the JSON files you created in the previous steps. Issue the following commands to create + the roles. + + ```shell + az role definition create --role-definition @aks_dynamic_sub_role.json --output table + az role definition create --role-definition @aks_dynamic_rg_sub_role.json --output table + ``` + +5. Export the + [security principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/overview#security-principal) + object ID you want to use with Palette to a variable. + + ```shell + export ASSIGNEE="" + ``` + +6. Export the resource group name to a variable. + + ```shell + export RESOURCE_GROUP_NAME="" + export VNET_NAME="" + ``` + +7. Assign the roles to the service principal. Use the following commands to assign the roles. + + ```json + az role assignment create --assignee $ASSIGNEE \ + --role "Palette Dynamic Placement AKS Cluster Deployer (sub)" \ + --scope "/subscriptions/$SUBSCRIPTION_ID" + ``` + + ```json + az role assignment create --assignee $ASSIGNEE \ + --role "Palette Dynamic Placement AKS Cluster Deployer (rg/sub)" \ + --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP_NAME" + ```