How to monitor KEDA operator?

[amirschw]

KEDA is an important component in our clusters that a lot of other applications depend upon, and it's critical for us to monitor it and ensure it's working properly.

We recently moved to using KEDA v2 and the new metrics adapter Prometheus metrics provide good visibility and help detecting errors related to the adapter and the scalers / scaled objects. However, we do not know of a good way to detect issues in the operator or the controllers.

Are there any recommendations on how or what to monitor in the operator? Or is it possible to add Prometheus metrics to the operator?

Thanks!

[tomkerkhove]

We should document this on https://keda.sh/docs/2.3/operate/cluster, that's a good question!

I'd say make sure to check the logs, but I'll add @zroubalik for a complete answer.

[amirschw]

Thanks!

I see that the controller-runtime already instruments some useful metrics:

• Total number of reconcilation errors per controller
• Length of reconcile queue per controller
• Reconcilation latency
• Usual resource metrics such as CPU, memory usage, file descriptor usage
• Go runtime metrics such as number of Go routines, GC duration

@zroubalik, I'd be happy to try but I will need some guidance as I am not too familiar with the internals of the Operator. Do you have some sort of plan or thoughts as to what additional metrics should be exposed?

[zroubalik]

I'd add something related to ScaledObjects - number of watched SO, number of errors, number of active SO.

And another missing thing are metrics related to ScaleJob (because SJs don't you Metrics Adapater). So would be awesome if you could add these as well (similar to the SO metrics that we are just exposing).

@tomkerkhove would you add anything else?

Thanks!

[tomkerkhove]

Might be interesting to add trigger authentication then as well? Were we able to authenticate or not? But it might be a nice follow-up though.

[amirschw]

Metrics for HPA operations (create, patch, delete) and their status could also be helpful.

[tomkerkhove]

Feel free to open a feature request for those please, would love to hear your use case.