Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Azure Default Credentials in the Scaler #68

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

karpikpl
Copy link

Support for Azure Default Credentials in the Scaler

Checklist

  • Commits are signed with Developer Certificate of Origin (DCO)
  • A PR is opened to update the documentation on our docs repo

Fixes #49

karpikpl added 5 commits June 12, 2024 21:51
Signed-off-by: Piotr Karpala <[email protected]>
Signed-off-by: Piotr Karpala <[email protected]>
Signed-off-by: Piotr Karpala <[email protected]>
@@ -25,7 +26,11 @@ public Worker(CosmosDbConfig cosmosDbConfig, ILogger<Worker> logger)

public override async Task StartAsync(CancellationToken cancellationToken)
{
Database leaseDatabase = await new CosmosClient(_cosmosDbConfig.LeaseConnection)
var cosmosClient = _cosmosDbConfig.Connection.Contains("AccountKey")
Copy link
Collaborator

@JatinSanghvi JatinSanghvi Jul 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @karpikpl, the PR looks good overall. This particular cosmosClient should be based on LeaseConnection configuration as the sample code aims to work even when the Cosmos DB accounts are different for the monitored and lease containers. I have a couple of minor changes planned and should be able to fix this one too in the new PR if that's okay with you.

I have a few questions on AAD identity-based access. Would be helpful if you have answer to some or all of them.

  1. Were you able to test the demo code with default credentials? I had tough time trying to set the permissions as none of the existing role (and with custom role) seem to give me access to create database, etc.
  2. Do we have steps to allow use of default credentials inside locally running Docker container?
  3. Do we know the minimum (or a superset of) permissions required by the Cosmos DB scaler to be able to check change feeds and generate necessary KEDA metrics? If so, I can document and publish it in this repo.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@karpikpl - Reminder on this one. Let me know if you know answers to any of the questions.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JatinSanghvi I am highly interested in this one as well.

Were you able to test the demo code with default credentials? I had tough time trying to set the permissions as none of the existing role (and with custom role) seem to give me access to create database, etc.

Do you want me to take a stab at this?

You would most likely need a control plane role here:

Do we have steps to allow use of default credentials inside locally running Docker container?

I believe you would use Account Key based authentication there - there's no robust need or support for that imo.

Do we know the minimum (or a superset of) permissions required by the Cosmos DB scaler to be able to check change feeds and generate necessary KEDA metrics? If so, I can document and publish it in this repo.

here's a link to the built-in data plane roles

I believe you are looking for: Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/readChangeFeed

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cmeyertons , one of the teams in Microsoft is looking for adding support for accessing Cosmos DB account using Azure Kubernetes Service's managed identity. However, we noticed that supporting managed identities requires referencing TriggerAuthentication from the ScaledObject and that is currently not supported for external scalers AFAIK.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: The feature described in here is not generally supported and unless publicly documented, they should not be used on production deployments. There are based on internal documentation, and I haven't tested them to verify accuracy.

The roles to create/replace databases and containers are respectively as follows:

  • Microsoft.DocumentDB/databaseAccounts/sqlDatabases/write
  • Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/write

The following glob-expansion actions are also allowed:

  • Microsoft.DocumentDB/databaseAccounts/sqlDatabases/*
  • Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/*
  • Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/storedProcedures/*
  • Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/triggers/*
  • Microsoft.DocumentDB/databaseAccounts/throughputSettings/*

With glob expansion, the minimal set of actions needed to get access to all resources under a CosmosDB account are as follows:

  • Microsoft.DocumentDB/databaseAccounts/readMetadata
  • Microsoft.DocumentDB/databaseAccounts/sqlDatabases/*
  • Microsoft.DocumentDB/databaseAccounts/throughputSettings/*

These are expanded RBAC actions, so they will need to be stored in JSON file as follows to be able to apply them:

{
    "RoleName": "ExpandedRBACActions",
    "Type": "CustomRole",
    "AssignableScopes": ["/"],
    "Permissions": [{
        "DataActions": [
            "Microsoft.DocumentDB/databaseAccounts/readMetadata",
            "Microsoft.DocumentDB/databaseAccounts/sqlDatabases/*",
            "Microsoft.DocumentDB/databaseAccounts/throughputSettings/*"
        ]
    }]
}

PowerShell commands to create role using Azure CLI:

# Create RoleDefinition.
az cosmosdb sql role definition create --account-name $accountName --resource-group $resourceGroupName --body expandedActions.json

# Create RoleAssignment.
az cosmosdb sql role assignment create --account-name $accountName --resource-group $resourceGroupName  --role-definition-name "ExpandedRBACActions" --scope "/" --principal-id $principalId

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: The feature described in here is not generally supported and unless publicly documented, they should not be used on production deployments.

The external scaler is effectively a pod running outside of KEDA that KEDA talks to right? so the only steps necessary are just making sure that pod is appropriately decorated with the correct pod labels to run under the right service account as described in the workload identity docs - why is TriggerAuthentication necessary?

Is it non-trivial to provide an example of setting up the external autoscaler with the correct labels + service account to use workload identity? What's the gap I'm missing here after the code correctly supports it?

It sounds like you did the hard part already of defining the correct permissions needed for the role, which is great!

I agree that we cannot use the generic TriggerAuthentication here, but an Azure-first / AKS example will cover majority of use cases (I think they will be high overlap with AKS + Cosmos usage) - going cross-cloud is most likely a more niche use case where non-Entra authentication works fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support AAD Pod Identity
3 participants