Logstash Exporter and multiple logstash pods

[alekslebedev]

Hi! I have multiple Logstash pods within my Kubernetes cluster. I've set up a ClusterIP service for them and configured Logstash exporter on this service. However, I have some doubts. Will the metrics be duplicated, and will they be correct in this setup? Moreover, will the Grafana dashboard function correctly? Essentially, am I configuring the dashboard for Logstash exporter or for the Logstash instances themselves?

[kuskoman]

Hello @alekslebedev
since service is going to load balance the requests, you will essentially get metrics from a random pod each time.
What you can do is either to deploy exporter as a sidecar to each of the containers and tell prometheus/victoriametrics agent to scrape them (using labels if using k8s operator), or to deploy logstash pods with predictable dns names (statefulset does this out of the box) and configure logstash exporter to scrape all of them. please note, that the second option is only possible using unstable (master) version of the exporter, while sidecar can be created with either version, but requires custom helm configuration. currently i am thinking about creating a k8s operator to simplify the process of monitoring multiple logstash instances in kubernetes, but it is pretty complex task and is currently blocked by another issue (config hot reload). since the exporter is my side project, i can't provide you with the expected time of the operator being ready

[kuskoman]

i just noticed you mentioned grafana dashboard. to be honest, you have to test it yourself, as i personally don't use it and it was created by another user, based on v1 version of the exporter. judging by screenshots shared by other people, it is supposed to work properly with multiple logstash instances

[alekslebedev]

Thanks for the quick answer! Now it is clear.

[kuskoman]

• Document logstash-exporter usage with a Helm chart Document logstash-exporter usage with a Helm chart #297
• Document logstash-exporter usage as a standalone application Document logstash-exporter usage as a standalone application #298
• Implement Kubernetes controller to automatically find pods in a cluster based on labels/annotations Implement Kubernetes controller to automatically find pods in a cluster based on labels/annotations #299
• Implement config hot reload Config hot-reload #281