Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NH-57036: Fix metrics relation to Nodes on Fargate #381

Merged
merged 8 commits into from
Oct 9, 2023
Merged

NH-57036: Fix metrics relation to Nodes on Fargate #381

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
42278c2
NH-57036: Fix metrics relation to Nodes on Fargate
pstranak-sw Oct 4, 2023
f8ab64b
NH-57036: Avoid using the 'transform' processor when possible
pstranak-sw Oct 4, 2023
8662b3b
NH-57036: Using only last 5 jsons out of all so that integration test…
gantrior Oct 6, 2023
c5d4458
Merge remote-tracking branch 'origin/master' into feature/NH-57036-fa…
gantrior Oct 6, 2023
fb787d6
NH-57036: Updating expected output
gantrior Oct 6, 2023
cf8222d
NH-57036: I reworked the merge script of integration tests to be more…
gantrior Oct 6, 2023
03e66d5
NH-57036: Minor adjustment
gantrior Oct 6, 2023
7cf531a
NH-57036: Further stabilization of integration tests
gantrior Oct 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 0 additions & 13 deletions build/docker/wiremockFiles/redirectKubeStateMetricsResponse.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,11 +199,6 @@ kube_pod_container_resource_limits{container="test-container",endpoint="http",in
kube_pod_container_resource_limits{container="test-container",endpoint="http",instance="test-node",job="test-job",namespace="test-namespace",node="test-node",pod="test-pod",resource="memory",service="test-service",uid="bafeef2c-1292-4a5e-a92c-d709480b04b6",unit="byte",prometheus="prometheus-system/kube-prometheus-kube-prome-prometheus",prometheus_replica="prometheus-kube-prometheus-kube-prome-prometheus-0"} 3.221225472e+09 1675856675021
kube_pod_container_resource_limits{container="test-container",endpoint="tcp-model",instance="test-node",job="test-job",namespace="test-namespace",node="test-node",pod="test-pod",resource="memory",service="test-service",uid="bafeef2c-1292-4a5e-a92c-d709480b04b6",unit="byte",prometheus="prometheus-system/kube-prometheus-kube-prome-prometheus",prometheus_replica="prometheus-kube-prometheus-kube-prome-prometheus-0"} 3.221225472e+09 1675856675021
kube_pod_container_resource_limits{container="test-container",endpoint="tcp-model",instance="test-node",job="test-job",namespace="test-namespace",node="test-node",pod="test-pod",resource="cpu",service="test-service",uid="bafeef2c-1292-4a5e-a92c-d709480b04b6",unit="core",prometheus="prometheus-system/kube-prometheus-kube-prome-prometheus",prometheus_replica="prometheus-kube-prometheus-kube-prome-prometheus-0"} 0.1 1675856675021
# TYPE kube_pod_container_resource_limits_cpu_cores untyped
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These metrics existed only in kube-state-metrics v2.0.0-alpha and were removed in v2.0.0-alpha.2. I'm not even sure how they appeared in the mocked data.

kube_pod_container_resource_limits_cpu_cores{container="test-container",endpoint="tcp-model",instance="test-node",job="test-job",namespace="test-namespace",node="test-node",pod="test-pod",service="test-service",uid="71057e83-7723-4db5-a7ca-52ad7904e34d",prometheus="prometheus-system/kube-prometheus-kube-prome-prometheus",prometheus_replica="prometheus-kube-prometheus-kube-prome-prometheus-0"} 1 1675856675021
# TYPE kube_pod_container_resource_limits_memory_bytes untyped
kube_pod_container_resource_limits_memory_bytes{container="test-container",endpoint="tcp-model",instance="test-node",job="test-job",namespace="test-namespace",node="test-node",pod="test-pod",service="test-service",uid="03d2b55c-b225-476b-9178-c74f8e5eaba2",prometheus="prometheus-system/kube-prometheus-kube-prome-prometheus",prometheus_replica="prometheus-kube-prometheus-kube-prome-prometheus-0"} 2.68435456e+08 1675856675021
kube_pod_container_resource_limits_memory_bytes{container="test-container",endpoint="tcp-model",instance="test-node",job="test-job",namespace="test-namespace",node="test-node",pod="test-pod",service="test-service",uid="bafeef2c-1292-4a5e-a92c-d709480b04b6",prometheus="prometheus-system/kube-prometheus-kube-prome-prometheus",prometheus_replica="prometheus-kube-prometheus-kube-prome-prometheus-0"} 3.221225472e+09 1675856675021
# TYPE kube_pod_container_resource_requests untyped
kube_pod_container_resource_requests{container="test-container",endpoint="http",instance="test-node",job="test-job",namespace="test-namespace",node="test-node",pod="test-pod",resource="cpu",service="test-service",uid="f15ca7ca-af33-4f43-a793-ec3176b31842",unit="core",prometheus="prometheus-system/kube-prometheus-kube-prome-prometheus",prometheus_replica="prometheus-kube-prometheus-kube-prome-prometheus-0"} 0.1 1675856675021
kube_pod_container_resource_requests{container="test-container",endpoint="http",instance="test-node",job="test-job",namespace="test-namespace",node="test-node",pod="test-pod",resource="memory",service="test-service",uid="bafeef2c-1292-4a5e-a92c-d709480b04b6",unit="byte",prometheus="prometheus-system/kube-prometheus-kube-prome-prometheus",prometheus_replica="prometheus-kube-prometheus-kube-prome-prometheus-0"} 3.221225472e+09 1675856675021
Expand Down Expand Up @@ -242,14 +237,6 @@ kube_pod_info{container="test-container",created_by_kind="ReplicaSet",created_by
kube_pod_init_container_info{container="test-container",container_id="docker://f1c98663d614379552d6c9aae831b4eb3e149c469d4d589b62ea077cf3dad807",endpoint="http",image="busybox:1.29.2",image_id="docker-pullable://busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd",instance="test-node",job="test-job",namespace="test-namespace",pod="test-pod",service="test-service",uid="9cab81cb-9da1-4029-ac51-c7c3024c6fbf",prometheus="prometheus-system/kube-prometheus-kube-prome-prometheus",prometheus_replica="prometheus-kube-prometheus-kube-prome-prometheus-0"} 1 1675856675021
kube_pod_init_container_info{container="test-container",container_id="docker://f55a97f615b1a90f15f114def871a7d4f0ed8a32d9a329cc7e2f94a24d4780c1",endpoint="http",image="fullstorydev/grpcurl:latest",image_id="docker-pullable://fullstorydev/grpcurl@sha256:d42ef512419560776bee5bb51e338a1734a0edb99f450b6d98fd98bcc93796f3",instance="test-node",job="test-job",namespace="test-namespace",pod="test-pod",service="test-service",uid="bafeef2c-1292-4a5e-a92c-d709480b04b6",prometheus="prometheus-system/kube-prometheus-kube-prome-prometheus",prometheus_replica="prometheus-kube-prometheus-kube-prome-prometheus-0"} 1 1675856675021
kube_pod_init_container_info{container="test-container",container_id="docker://6fb07210b86971939dbfb16ae47f98afdb8214a52a21dd5fc26f21ae35c09d9e",endpoint="http",image="busybox:latest",image_id="docker-pullable://busybox@sha256:b5d6fe0712636ceb7430189de28819e195e8966372edfc2d9409d79402a0dc16",instance="test-node",job="test-job",namespace="test-namespace",pod="test-pod",service="test-service",uid="bafeef2c-1292-4a5e-a92c-d709480b04b6",prometheus="prometheus-system/kube-prometheus-kube-prome-prometheus",prometheus_replica="prometheus-kube-prometheus-kube-prome-prometheus-0"} 1 1675856675021
# TYPE kube_pod_init_container_resource_limits_cpu_cores untyped
kube_pod_init_container_resource_limits_cpu_cores{container="test-container",endpoint="http",instance="test-node",job="test-job",namespace="test-namespace",pod="test-pod",service="test-service",uid="2a82ed12-a31a-427a-adb9-d14cf6a4a063",prometheus="prometheus-system/kube-prometheus-kube-prome-prometheus",prometheus_replica="prometheus-kube-prometheus-kube-prome-prometheus-0"} 0.1 1675856675021
# TYPE kube_pod_init_container_resource_limits_memory_bytes untyped
kube_pod_init_container_resource_limits_memory_bytes{container="test-container",endpoint="http",instance="test-node",job="test-job",namespace="test-namespace",pod="test-pod",service="test-service",uid="a920ed2f-477d-4ad7-93d6-3222aabfece2",prometheus="prometheus-system/kube-prometheus-kube-prome-prometheus",prometheus_replica="prometheus-kube-prometheus-kube-prome-prometheus-0"} 1.073741824e+09 1675856675021
# TYPE kube_pod_init_container_resource_requests_cpu_cores untyped
kube_pod_init_container_resource_requests_cpu_cores{container="test-container",endpoint="http",instance="test-node",job="test-job",namespace="test-namespace",pod="test-pod",service="test-service",uid="c7f7a05a-a1b5-4fd0-a611-8e2a0e7acbf6",prometheus="prometheus-system/kube-prometheus-kube-prome-prometheus",prometheus_replica="prometheus-kube-prometheus-kube-prome-prometheus-0"} 0.25 1675856675021
# TYPE kube_pod_init_container_resource_requests_memory_bytes untyped
kube_pod_init_container_resource_requests_memory_bytes{container="test-container",endpoint="http",instance="test-node",job="test-job",namespace="test-namespace",pod="test-pod",service="test-service",uid="c7f7a05a-a1b5-4fd0-a611-8e2a0e7acbf6",prometheus="prometheus-system/kube-prometheus-kube-prome-prometheus",prometheus_replica="prometheus-kube-prometheus-kube-prome-prometheus-0"} 6.7108864e+07 1675856675021
# TYPE kube_pod_init_container_status_ready untyped
kube_pod_init_container_status_ready{container="test-container",endpoint="http",instance="test-node",job="test-job",namespace="test-namespace",pod="test-pod",service="test-service",uid="64b6d309-44e3-4a41-8883-c15c7cc9bc4a",prometheus="prometheus-system/kube-prometheus-kube-prome-prometheus",prometheus_replica="prometheus-kube-prometheus-kube-prome-prometheus-0"} 1 1675856675021
kube_pod_init_container_status_ready{container="test-container",endpoint="http",instance="test-node",job="test-job",namespace="test-namespace",pod="test-pod",service="test-service",uid="bafeef2c-1292-4a5e-a92c-d709480b04b6",prometheus="prometheus-system/kube-prometheus-kube-prome-prometheus",prometheus_replica="prometheus-kube-prometheus-kube-prome-prometheus-0"} 1 1675856675021
Expand Down
5 changes: 4 additions & 1 deletion deploy/helm/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## Unreleased

### Fixed

- Detection of Node name for Fargate Nodes's metrics

## [2.8.0-alpha.3] - 2023-10-06

### Changed
Expand All @@ -18,7 +22,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added

- Add monitoring windows node logs

## [2.8.0-alpha.1] - 2023-09-11

### Added
Expand Down
12 changes: 8 additions & 4 deletions deploy/helm/metrics-collector-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,6 @@ processors:
datapoint:
- 'attributes["container"] == "POD" and IsMatch(metric.name, "container_network_.*|k8s.container.*") == true'


# unify attributes
attributes/unify_node_attribute:
include:
Expand All @@ -72,9 +71,13 @@ processors:
from_attribute: kubernetes_io_hostname
action: insert
{{- end }}
- key: k8s.node.name
from_attribute: service.instance.id
action: insert

transform/unify_node_attribute:
metric_statements:
- context: datapoint
statements:
# use "service.instance.id" for Node name when the attributes/unify_node_attribute processor failed to provide it
- set(attributes["k8s.node.name"], resource.attributes["service.instance.id"]) where IsMatch(metric.name, "^(container_.*)|(kube_node_.*)|(kube_pod_info)|(kube_pod_container_resource_requests)|(kube_pod_container_resource_limits)|(kube_pod_init_container_resource_requests)|(kube_pod_init_container_resource_limits)$") == true and attributes["k8s.node.name"] == nil

attributes/unify_volume_attribute:
include:
Expand Down Expand Up @@ -1484,6 +1487,7 @@ service:
- transform
- filter/remove_internal
- attributes/unify_node_attribute
- transform/unify_node_attribute
- attributes/unify_volume_attribute
- attributes/identify_init_container
- attributes/identify_standard_container
Expand Down
11 changes: 8 additions & 3 deletions deploy/helm/tests/__snapshot__/metrics-collector-config-map-fargate_test.yaml.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -339,9 +339,6 @@ Metrics config should match snapshot when using default values:
- action: insert
from_attribute: node
key: k8s.node.name
- action: insert
from_attribute: service.instance.id
key: k8s.node.name
include:
match_type: regexp
metric_names:
Expand Down Expand Up @@ -1926,6 +1923,13 @@ Metrics config should match snapshot when using default values:
- delete_key(resource.attributes, "sw.k8s.job.found")
- delete_key(resource.attributes, "sw.k8s.cronjob.found")
- delete_key(resource.attributes, "sw.k8s.node.found")
transform/unify_node_attribute:
metric_statements:
- context: datapoint
statements:
- set(attributes["k8s.node.name"], resource.attributes["service.instance.id"])
where IsMatch(metric.name, "^(container_.*)|(kube_node_.*)|(kube_pod_info)|(kube_pod_container_resource_requests)|(kube_pod_container_resource_limits)|(kube_pod_init_container_resource_requests)|(kube_pod_init_container_resource_limits)$")
== true and attributes["k8s.node.name"] == nil
receivers:
k8s_events: null
prometheus/kube-state-metrics:
Expand Down Expand Up @@ -1989,6 +1993,7 @@ Metrics config should match snapshot when using default values:
- transform
- filter/remove_internal
- attributes/unify_node_attribute
- transform/unify_node_attribute
- attributes/unify_volume_attribute
- attributes/identify_init_container
- attributes/identify_standard_container
Expand Down
11 changes: 8 additions & 3 deletions deploy/helm/tests/__snapshot__/metrics-collector-config-map_test.yaml.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -342,9 +342,6 @@ Metrics config should match snapshot when using default values:
- action: insert
from_attribute: kubernetes_io_hostname
key: k8s.node.name
- action: insert
from_attribute: service.instance.id
key: k8s.node.name
include:
match_type: regexp
metric_names:
Expand Down Expand Up @@ -1929,6 +1926,13 @@ Metrics config should match snapshot when using default values:
- delete_key(resource.attributes, "sw.k8s.job.found")
- delete_key(resource.attributes, "sw.k8s.cronjob.found")
- delete_key(resource.attributes, "sw.k8s.node.found")
transform/unify_node_attribute:
metric_statements:
- context: datapoint
statements:
- set(attributes["k8s.node.name"], resource.attributes["service.instance.id"])
where IsMatch(metric.name, "^(container_.*)|(kube_node_.*)|(kube_pod_info)|(kube_pod_container_resource_requests)|(kube_pod_container_resource_limits)|(kube_pod_init_container_resource_requests)|(kube_pod_init_container_resource_limits)$")
== true and attributes["k8s.node.name"] == nil
receivers:
k8s_events: null
prometheus/kube-state-metrics:
Expand Down Expand Up @@ -1992,6 +1996,7 @@ Metrics config should match snapshot when using default values:
- transform
- filter/remove_internal
- attributes/unify_node_attribute
- transform/unify_node_attribute
- attributes/unify_volume_attribute
- attributes/identify_init_container
- attributes/identify_standard_container
Expand Down
Loading