Keda & Datadog scaler - tls: failed to verify certificate: x509: certificate signed by unknown authority

[senthilprasanth]

Hello Community,

I am new to KEDA and working on a POC with the following setup:

Kubernetes Cluster: Docker Desktop
Kubernetes Version: v1.30.2
KEDA Version: 2.15.1
Authentication type: REST ENDPOINT

I am encountering errors related to Datadog integration.

I am hitting with the following error at Docker-desktop K8 cluster(local)

From scaledObject events

error establishing Datadog connection: error connecting to Datadog API endpoint: Get "https://api.us5.datadoghq.com/api/v1/validate": tls: failed to verify certificate: x509: certificate signed by unknown authority

From keda-operator pod log

ERROR Reconciler error: error establishing Datadog connection: error connecting to Datadog API endpoint: Get "https://api.us5.datadoghq.com/api/v1/validate": tls: failed to verify certificate: x509: certificate signed by unknown authority

The issue seems to be related to certificate verification for Datadog's API endpoint. I am unable to proceed further. Looking to bypass this scenario / fix it permanently.

Referred the below documentation, but stuck (brain-faded how to approach)

1. https://keda.sh/docs/2.15/concepts/authentication/
2. https://keda.sh/docs/2.15/scalers/datadog/
3. https://keda.sh/docs/2.15/operate/security/#use-your-own-tls-certificates

I tried the same setup at "Killer-koda" environment. It worked.
Having this tls certificate issue at docker desktop.

Your help is much appreciated

My setup is as follows

---
apiVersion: v1
kind: Namespace
metadata:
  name: datadog-keda
---
apiVersion: v1
kind: Secret
metadata:
  name: datadog-secrets
  namespace: datadog-keda
type: Opaque
data:
  apiKey: XXX=    # Required: base64 encoded value of Datadog apiKey
  appKey: XXX==  # Required: base64 encoded value of Datadog appKey
  datadogSite: dXM1LmRhdGFkb2docS5jb20=  # Optional: base64 encoded value of Datadog site
---
apiVersion: [keda.sh/v1alpha1](http://keda.sh/v1alpha1)
kind: TriggerAuthentication
metadata:
  name: datadog-trigger-auth
  namespace: datadog-keda
spec:
  secretTargetRef:
    - parameter: apiKey
      name: datadog-secrets
      key: apiKey
    - parameter: appKey
      name: datadog-secrets
      key: appKey
    - parameter: datadogSite
      name: datadog-secrets
      key: datadogSite
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-keda-deploy
  namespace: datadog-keda
  labels:
    app: nginx-keda-deploy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-keda
  template:
    metadata:
      labels:
        app: nginx-keda
      annotations:
        [ad.datadoghq.com/nginx-keda.check_names](http://ad.datadoghq.com/nginx-keda.check_names): '["nginx"]'
        [ad.datadoghq.com/nginx-keda.init_configs](http://ad.datadoghq.com/nginx-keda.init_configs): '[{}]'
        [ad.datadoghq.com/nginx-keda.instances](http://ad.datadoghq.com/nginx-keda.instances): '[{"nginx_status_url": "http://%%host%%/nginx_status"}]'
        [ad.datadoghq.com/nginx-keda.logs](http://ad.datadoghq.com/nginx-keda.logs): '[{"type": "docker", "image": "nginx", "service": "nginx", "source": "nginx"}]'
    spec:
      containers:
        - name: nginx-keda
          image: nginx:latest
          ports:
            - name: http
              containerPort: 80
          volumeMounts:
            - name: "config"
              mountPath: "/etc/nginx/nginx.conf"
              subPath: "nginx.conf"
      volumes:
        - name: "config"
          configMap:
            name: "nginxconfig-keda"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginxconfig-keda
  namespace: datadog-keda
data:
  nginx.conf: |
    worker_processes 5;
    events {
      worker_connections 4096;
    }
    http {
      server {
        location /nginx_status {
          stub_status on;
          access_log /dev/stdout;
          allow all;
        }
      }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-keda-svc
  namespace: datadog-keda
  labels:
    app: nginx-keda-svc
spec:
  selector:
    app: nginx-keda
  ports:
    - name: http
      port: 8091
      targetPort: 80
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: datadog-scaledobject
  namespace: datadog-keda
spec:
  scaleTargetRef:
    name: nginx-keda-deploy
  maxReplicaCount: 10
  minReplicaCount: 1
  pollingInterval: 30
  triggers:
    - type: datadog
      metricType: "AverageValue"
      metadata:
        query: "avg:nginx.net.request_per_s{kube_container_name:nginx-keda}"
        queryValue: "2"
      authenticationRef:
        name: datadog-trigger-auth

[senthilprasanth]

Followed this knowledge base https://keda.sh/docs/2.15/operate/security/

Tried passing certificates.operator.caDirs at helm values.yaml. keda-operator deployment reflects with --caDir option. But the scaledObject still throws the same error.

operator
  caDirs:
  - /home/<custom_path_certs/

Again any pointers might help me to learn more

[zroubalik]

Have you tried unsafeSsl https://keda.sh/docs/2.15/scalers/datadog/#authentication?

[senthilprasanth]

@zroubalik : Apologies for the delayed response. Yeah we have tried the option. But it fails.
Just a curious question. We are trying to authN via REST api method. Will the "unsafeSsl" option work here