error establishing Datadog connection: error connecting to Datadog API endpoint: 403 Forbidden

[Anand03]

I have installed Keda in a dedicated keda namespace using helm. The Keda scaledObject, Deployment and datadog secret are in the same namespace. I am isung a ClusterTriggerAuthentication.

Getting 403 error when I try to use Datadog scaler following this steps https://keda.sh/docs/2.7/scalers/datadog/

2024-07-29T13:35:22Z ERROR scale_handler error resolving auth params {"type": "ScaledObject", "namespace": "keda", "name": "datadog-scaledobject", "triggerIndex": 0, "error": "error establishing Datadog connection: error connecting to Datadog API endpoint: 403 Forbidden"} github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers /workspace/pkg/scaling/scalers_builder.go:99 github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache /workspace/pkg/scaling/scale_handler.go:357 github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache /workspace/pkg/scaling/scale_handler.go:282 github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs /workspace/controllers/keda/hpa.go:217 github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject /workspace/controllers/keda/hpa.go:72 2024-07-29T13:35:22Z ERROR Error getting scalers {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"datadog-scaledobject","namespace":"keda"}, "namespace": "keda", "name": "datadog-scaledobject", "reconcileID": "9c0b7e60-bf84-4472-9acf-7f183f9302a3", "error": "error establishing Datadog connection: error connecting to Datadog API endpoint: 403 Forbidden"}

The secrets are valid as curl command works when using the same secrets.

KEDA version: 2.14.0
Kubectl version: 1.28
Platform: EKS
datadogSite: datadoghq.eu

[arapulido]

Hello @Anand03 . The datadog-secrets secret needs to be in the same namespace as keda, as explained in the second paragraph of this section of the docs: https://keda.sh/docs/2.14/concepts/authentication/#authentication-scopes-namespace-vs-cluster

"By default, Secrets loaded from a secretTargetRef must be in the same namespace as KEDA is deployed in (usually keda). This can be overridden by setting a KEDA_CLUSTER_OBJECT_NAMESPACE environment variable for the keda-operator container."

[arapulido]

Someone in Slack had the same issue, and solved it moving the secret to the keda namespace: https://kubernetes.slack.com/archives/CKZJ36A5D/p1720498658146869

[Anand03]

Hi @arapulido .. Thanks for the prompt reply. As mentioned in the issue description, the secret, scaledobject and the deployment all are in the same namespace keda. Keda is also deployed in the same namespace keda.

[arapulido]

@Anand03 ah, I thought you mentioned that the deployment, scaledobject and secret were all in the same namespace, not that they were also in the keda namespace.

Can you please double check that that's the case? Can you report the output of kubectl get secret,deployment,pods,scaledobject -n keda?

[Anand03]

Sure.

[arapulido]

hey! not sure if relevant, but in your first set of screenshots, the secret is called datadog-secrets and in this last one the secret is called datadog-secret (without the s)

[Anand03]

It was relevant, I was hoping it would go un-noticed. I had updated the files with correct value datadog-secret .. did not update the screenshot. So datadog-secret is the correct value and updated everywhere.

[arapulido]

and does still fail? can you recreate all objects after fixing the reference?

[Anand03]

The screenshots were old.. error still exists after updating everything. I have updated the screenshot in description.

[arapulido]

I am puzzled. With the current information, it should work. It is clear that it is something with how the secrets are being read.

How did you create the datadog-secret ?

[Anand03]

I am using kubectl apply command to create the secret

[arapulido]

Again, not sure if relevant, but the keys in that secret are called api-key and app-key, but in the ClusterTriggerAuthentication object you reference different keys (apiKey and appKey)

[Anand03]

That was a silly mistake from my side. Thank you so much for identifying the issue.

[arapulido]

Awesome! Glad you got it sorted!