Mismatch Between Prometheus Metrics in Goflow2 and Grafana Dashboard

[firasrasmy-thg]

The prometheus metrics exposed by goflow2 (as seen in the source code) do not match most of the target metrics in the Grafana dashboard at https://github.com/netsampler/goflow2/blob/main/compose/kcg/grafana/dashboards/perfs.json. This is in addition to the fact that target metrics in perfs.json lacks the prefix namespace (goflow2).

[lspgn]

@firasrasmy-thg thank you for reporting, could you have a look at #361?

[firasrasmy-thg]

@lspgn thank you for your prompt response. Based on this PR, it looks much better now:

Metrics exposed by goflow2:
goflow2_flow_dropped_packets_total
goflow2_flow_dropped_bytes_total
goflow2_flow_traffic_bytes_total
goflow2_flow_traffic_packets_total
goflow2_flow_traffic_size_bytes
goflow2_flow_decoder_error_total
goflow2_flow_decoding_time_seconds
goflow2_flow_process_nf_total
goflow2_flow_process_nf_errors_total
goflow2_flow_process_nf_flowset_records_total
goflow2_flow_process_nf_flowset_total
goflow2_flow_process_nf_delay_seconds
goflow2_flow_process_nf_templates_total
goflow2_flow_process_sf_total
goflow2_flow_process_sf_samples_total
goflow2_flow_process_sf_samples_records_total

Metrics targeted by Grafana dashboard:
goflow2_flow_decoding_time_seconds
goflow2_flow_process_nf_delay_seconds
goflow2_flow_process_nf_errors_total
goflow2_flow_process_nf_flowset_records_total
goflow2_flow_process_nf_templates_total
goflow2_flow_process_sf_samples_records_total
goflow2_flow_process_sf_samples_total
goflow2_flow_traffic_bytes_total
goflow2_flow_traffic_packets_total
goflow2_flow_traffic_size_bytes_count
goflow2_flow_traffic_size_bytes_sum

I noticed the last two metrics (goflow2_flow_traffic_size_bytes_count and goflow2_flow_traffic_size_bytes_sum) are not exposed by goflow2. Additionally, these goflow2 metrics are not targeted by the Grafana dashboard: goflow2_flow_dropped_packets_total, goflow2_flow_dropped_bytes_total, goflow2_flow_traffic_size_bytes, goflow2_flow_decoder_error_total, goflow2_flow_process_nf_total, goflow2_flow_process_nf_flowset_total, goflow2_flow_process_sf_total

I'd like to take this opportunity to ask you the following: I have goflow2 deployed in two setups, both receiving IMON flows. One of them is exposing all the goflow2_flow_process_nf_ metrics while the other one is not. I haven't had the chance to check the source code to determine whether this behavior is intentional. I would greatly appreciate your thoughts on this.

[lspgn]

I noticed the last two metrics (goflow2_flow_traffic_size_bytes_count and goflow2_flow_traffic_size_bytes_sum) are not exposed by goflow2

Unless I'm mistaken, that's a summary which provides _sum and _count metrics:

goflow2/metrics/metrics.go

Lines 43 to 50 in c2fe351

	MetricPacketSizeSum = prometheus.NewSummaryVec(
	prometheus.SummaryOpts{
	Name: "flow_traffic_size_bytes",
	Help: "Summary of packet size.",
	Namespace: NAMESPACE, Objectives: map[float64]float64{0.5: 0.05, 0.9: 0.01, 0.99: 0.001},
	},
	[]string{"remote_ip", "local_ip", "local_port", "type"},
	)

Additionally, these goflow2 metrics are not targeted by the Grafana dashboard

Ah yes, the compose directory is to provide some boilerplate but it's far from being a fully maintained suite.

One of them is exposing all the goflow2_flow_process_nf_ metrics while the other one is not

Running the same versions? If it's missing, it did not process any NetFlow packet. Was there any output?

[firasrasmy-thg]

Unless I'm mistaken, that's a summary which provides _sum and _count metrics:

Ah yes you are right, my mistake! I just quickly searched for prometheus.NewCounterVec in metrics.go and missed prometheus.NewSummaryVec. I see both my goflow2 instances are exposing the '_sum' and '_count' metrics.

Running the same versions? If it's missing, it did not process any NetFlow packet. Was there any output?

Yes, both of them are running the same version (GoFlow2 v2.2.1-4-gbf8b794) and receiving exactly the same IMON flows. Both setups produce exactly the same results. The goflow2 instance exposing the goflow2_flow_process_nf_ metrics is running in a Docker container, while the other instance is deployed in a Kubernetes cluster utilizing an OpenStack Octavia load balancer to expose the goflow2 port externally.

[lspgn]

Hmm unless each instance received part of the traffic and in the case of NetFlow/IPFIX, if one of the two nodes does not have the templates, it won't be able to decode and the metric won't be created.

[firasrasmy-thg]

It turned out that the template packets were being forwarded to another goflow2 pod. After modifying the loadbalancer algorithm, they are now delivered to the same pod and I see all metrics are available now. Thanks for your support on this, Louis!