-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for Azure Default Credentials in the Scaler #68
base: main
Are you sure you want to change the base?
Conversation
Signed-off-by: Piotr Karpala <[email protected]>
Signed-off-by: Piotr Karpala <[email protected]>
Signed-off-by: Piotr Karpala <[email protected]>
Signed-off-by: Piotr Karpala <[email protected]>
Signed-off-by: Piotr Karpala <[email protected]>
@@ -25,7 +26,11 @@ public Worker(CosmosDbConfig cosmosDbConfig, ILogger<Worker> logger) | |||
|
|||
public override async Task StartAsync(CancellationToken cancellationToken) | |||
{ | |||
Database leaseDatabase = await new CosmosClient(_cosmosDbConfig.LeaseConnection) | |||
var cosmosClient = _cosmosDbConfig.Connection.Contains("AccountKey") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @karpikpl, the PR looks good overall. This particular cosmosClient
should be based on LeaseConnection
configuration as the sample code aims to work even when the Cosmos DB accounts are different for the monitored and lease containers. I have a couple of minor changes planned and should be able to fix this one too in the new PR if that's okay with you.
I have a few questions on AAD identity-based access. Would be helpful if you have answer to some or all of them.
- Were you able to test the demo code with default credentials? I had tough time trying to set the permissions as none of the existing role (and with custom role) seem to give me access to create database, etc.
- Do we have steps to allow use of default credentials inside locally running Docker container?
- Do we know the minimum (or a superset of) permissions required by the Cosmos DB scaler to be able to check change feeds and generate necessary KEDA metrics? If so, I can document and publish it in this repo.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@karpikpl - Reminder on this one. Let me know if you know answers to any of the questions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@JatinSanghvi I am highly interested in this one as well.
Were you able to test the demo code with default credentials? I had tough time trying to set the permissions as none of the existing role (and with custom role) seem to give me access to create database, etc.
Do you want me to take a stab at this?
You would most likely need a control plane role here:
Do we have steps to allow use of default credentials inside locally running Docker container?
I believe you would use Account Key based authentication there - there's no robust need or support for that imo.
Do we know the minimum (or a superset of) permissions required by the Cosmos DB scaler to be able to check change feeds and generate necessary KEDA metrics? If so, I can document and publish it in this repo.
here's a link to the built-in data plane roles
I believe you are looking for: Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/readChangeFeed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cmeyertons , one of the teams in Microsoft is looking for adding support for accessing Cosmos DB account using Azure Kubernetes Service's managed identity. However, we noticed that supporting managed identities requires referencing TriggerAuthentication
from the ScaledObject
and that is currently not supported for external scalers AFAIK.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note: The feature described in here is not generally supported and unless publicly documented, they should not be used on production deployments. There are based on internal documentation, and I haven't tested them to verify accuracy.
The roles to create/replace databases and containers are respectively as follows:
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/write
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/write
The following glob-expansion actions are also allowed:
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/*
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/*
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/storedProcedures/*
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/triggers/*
Microsoft.DocumentDB/databaseAccounts/throughputSettings/*
With glob expansion, the minimal set of actions needed to get access to all resources under a CosmosDB account are as follows:
Microsoft.DocumentDB/databaseAccounts/readMetadata
Microsoft.DocumentDB/databaseAccounts/sqlDatabases/*
Microsoft.DocumentDB/databaseAccounts/throughputSettings/*
These are expanded RBAC actions, so they will need to be stored in JSON file as follows to be able to apply them:
{
"RoleName": "ExpandedRBACActions",
"Type": "CustomRole",
"AssignableScopes": ["/"],
"Permissions": [{
"DataActions": [
"Microsoft.DocumentDB/databaseAccounts/readMetadata",
"Microsoft.DocumentDB/databaseAccounts/sqlDatabases/*",
"Microsoft.DocumentDB/databaseAccounts/throughputSettings/*"
]
}]
}
PowerShell commands to create role using Azure CLI:
# Create RoleDefinition.
az cosmosdb sql role definition create --account-name $accountName --resource-group $resourceGroupName --body expandedActions.json
# Create RoleAssignment.
az cosmosdb sql role assignment create --account-name $accountName --resource-group $resourceGroupName --role-definition-name "ExpandedRBACActions" --scope "/" --principal-id $principalId
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note: The feature described in here is not generally supported and unless publicly documented, they should not be used on production deployments.
The external scaler is effectively a pod running outside of KEDA that KEDA talks to right? so the only steps necessary are just making sure that pod is appropriately decorated with the correct pod labels to run under the right service account as described in the workload identity docs - why is TriggerAuthentication
necessary?
Is it non-trivial to provide an example of setting up the external autoscaler with the correct labels + service account to use workload identity? What's the gap I'm missing here after the code correctly supports it?
It sounds like you did the hard part already of defining the correct permissions needed for the role, which is great!
I agree that we cannot use the generic TriggerAuthentication
here, but an Azure-first / AKS example will cover majority of use cases (I think they will be high overlap with AKS + Cosmos usage) - going cross-cloud is most likely a more niche use case where non-Entra authentication works fine.
Support for Azure Default Credentials in the Scaler
Checklist
Fixes #49