Prometheus ScaledObject x509: certificate signed by unknown authority

[ejsa13]

I am trying to connect KEDA to the built-in prometheus server of openshift. By default the endpoint of the built-in prometheus server is TLS enabled. The prometheus service has this annotation:

service.alpha.openshift.io/serving-cert-signed-by: openshift-service-serving-signer@1645736028

I was able to find the CA cert in signing-key secret resource.

I created another secret key, cluster trigger authentication in the keda namespace and scaledobject in workload namespace...
kind: Secret
apiVersion: v1
metadata:
name: prometheus-k8s-ca
namespace: keda
stringData:
ca: >-

apiVersion: keda.sh/v1alpha1
kind: ClusterTriggerAuthentication
metadata:
name: cluster-prometheus-k8s-trigger-authentication
namespace: keda
spec:
secretTargetRef:
- name: prometheus-k8s-ca
parameter: ca
key: ca

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: simple-client-test
spec:
maxReplicaCount: 5
scaleTargetRef:
name: simple-client-test
triggers:
- type: prometheus
metadata:
serverAddress: https://prometheus-k8s.openshift-monitoring.svc:9091
metricName:
threshold: '3'
query:
authenticationRef:
name: cluster-prometheus-k8s-trigger-authentication
kind: ClusterTriggerAuthentication

Checking the logs of the keda-metrics-apiserver I am seeing this error

"x509: certificate signed by unknown authority"

and getting the status of the scaledobject, I got this

status:
conditions:

• message: Triggers defined in ScaledObject are not working correctly
  reason: TriggerError
  status: "False"
  type: Ready
• message: Scaling is not performed because triggers are not active
  reason: ScalerNotActive
  status: "False"
  type: Active
• message: No fallbacks are active on this scaled object
  reason: NoFallbackFound
  status: "False"
  type: Fallback

[mrethers]

@ejsa13 did you ever figure it out? I'm running into the same problem.

[ejsa13]

You may want to look at the example repo of zroubalik https://github.com/zroubalik/keda-openshift-examples/tree/main/prometheus/ocp-monitoring

[mrethers]

Thanks I did already, I opened an issue here: zroubalik/keda-openshift-examples#3

…

On Tue, Oct 4, 2022 at 9:27 AM Erwin Sta Ana ***@***.***> wrote: You may want to look at the example repo of zroubalik https://github.com/zroubalik/keda-openshift-examples/tree/main/prometheus/ocp-monitoring — Reply to this email directly, view it on GitHub <#2966 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBZ37KZ6PZ36KMQKKDFQ7TWBQWELANCNFSM5U3I6DYA> . You are receiving this because you commented.Message ID: ***@***.***>

[ejsa13]

I read issue #3 that you opened and saw your TriggerAuthentication

kind: TriggerAuthentication
metadata:
name: thanos
namespace: sample-ns
spec:
secretTargetRef:
- key: token
name: keda-token
parameter: bearerToken
- key: ca.crt
name: keda-token
parameter: ca

Is the "keda-token", one of the tokens from the service account that you created or repurposed? If yes, did you also deploy the necessary role.yaml and rolebinding.yaml manifests that was part of the ocp-monitoring example?

[mrethers]

It is yes, I just edited for simplicity's sake. I'm actually able to make it work by using the service-ca.crt key instead of the ca.crt key. It looks almost identical except the last certificate in the chain. I'm not quite sure why but it might be because my cluster is using a self-signed cert.

…

On Tue, Oct 4, 2022 at 11:25 AM Erwin Sta Ana ***@***.***> wrote: I read issue #3 <#3> that you opened and saw your TriggerAuthentication kind: TriggerAuthentication metadata: name: thanos namespace: sample-ns spec: secretTargetRef: - key: token name: keda-token parameter: bearerToken - key: ca.crt name: keda-token parameter: ca Is the "keda-token", one of the tokens from the service account that you created or repurposed? If yes, did you also deploy the necessary role.yaml and rolebinding.yaml manifests that was part of the ocp-monitoring example? — Reply to this email directly, view it on GitHub <#2966 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBZ37INP3VYBFZ7R73UYM3WBRD63ANCNFSM5U3I6DYA> . You are receiving this because you commented.Message ID: ***@***.***>