Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Port from Agent] Update kafka-exporter dependency #562

Merged
merged 2 commits into from
Apr 12, 2024
Merged

[Port from Agent] Update kafka-exporter dependency #562

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2c55321
update kafka-exporter dependency to use grafana fork instead of david…
wildum Apr 12, 2024
51898d3
fix docker file in integration test
wildum Apr 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,24 @@ This document contains a historical list of changes between releases. Only
changes that impact end-user behavior are listed; changes to documentation or
internal API changes are not present.

Main (unreleased)
-----------------

### Enhancements

- Update `prometheus.exporter.kafka` with the following functionalities (@wildum):
* GSSAPI config
* enable/disable PA_FX_FAST
* set a TLS server name
* show the offset/lag for all consumer group or only the connected ones
* set the minimum number of topics to monitor
* enable/disable auto-creation of requested topics if they don't already exist
* regex to exclude topics / groups
* added metric kafka_broker_info

- In `prometheus.exporter.kafka`, the interpolation table used to compute estimated lag metrics is now pruned
on `metadata_refresh_interval` instead of `prune_interval_seconds`. (@wildum)

v1.0.0 (2024-04-09)
-------------------

Expand Down
62 changes: 37 additions & 25 deletions docs/sources/reference/components/prometheus.exporter.kafka.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ title: prometheus.exporter.kafka
# prometheus.exporter.kafka

The `prometheus.exporter.kafka` component embeds
[kafka_exporter](https://github.com/davidmparrott/kafka_exporter) for collecting metrics from a kafka server.
[kafka_exporter](https://github.com/grafana/kafka_exporter) for collecting metrics from a kafka server.

## Usage

Expand All @@ -22,30 +22,42 @@ prometheus.exporter.kafka "LABEL" {
You can use the following arguments to configure the exporter's behavior.
Omitted fields take their default values.

| Name | Type | Description | Default | Required |
| --------------------------- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | -------- |
| `kafka_uris` | `array(string)` | Address array (host:port) of Kafka server. | | yes |
| `instance` | `string` | The`instance`label for metrics, default is the hostname:port of the first kafka_uris. You must manually provide the instance value if there is more than one string in kafka_uris. | | no |
| `use_sasl` | `bool` | Connect using SASL/PLAIN. | | no |
| `use_sasl_handshake` | `bool` | Only set this to false if using a non-Kafka SASL proxy. | `false` | no |
| `sasl_username` | `string` | SASL user name. | | no |
| `sasl_password` | `string` | SASL user password. | | no |
| `sasl_mechanism` | `string` | The SASL SCRAM SHA algorithm sha256 or sha512 as mechanism. | | no |
| `use_tls` | `bool` | Connect using TLS. | | no |
| `ca_file` | `string` | The optional certificate authority file for TLS client authentication. | | no |
| `cert_file` | `string` | The optional certificate file for TLS client authentication. | | no |
| `key_file` | `string` | The optional key file for TLS client authentication. | | no |
| `insecure_skip_verify` | `bool` | If set to true, the server's certificate will not be checked for validity. This makes your HTTPS connections insecure. | | no |
| `kafka_version` | `string` | Kafka broker version. | `2.0.0` | no |
| `use_zookeeper_lag` | `bool` | If set to true, use a group from zookeeper. | | no |
| `zookeeper_uris` | `array(string)` | Address array (hosts) of zookeeper server. | | no |
| `kafka_cluster_name` | `string` | Kafka cluster name. | | no |
| `metadata_refresh_interval` | `duration` | Metadata refresh interval. | `1m` | no |
| `allow_concurrency` | `bool` | If set to true, all scrapes trigger Kafka operations. Otherwise, they will share results. WARNING: Disable this on large clusters. | `true` | no |
| `max_offsets` | `int` | The maximum number of offsets to store in the interpolation table for a partition. | `1000` | no |
| `prune_interval_seconds` | `int` | How frequently should the interpolation table be pruned, in seconds. | `30` | no |
| `topics_filter_regex` | `string` | Regex filter for topics to be monitored. | `.*` | no |
| `groups_filter_regex` | `string` | Regex filter for consumer groups to be monitored. | `.*` | no |
| Name | Type | Description | Default | Required |
| ----------------------------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | -------- |
| `kafka_uris` | `array(string)` | Address array (host:port) of Kafka server. | | yes |
| `instance` | `string` | The`instance`label for metrics, default is the hostname:port of the first kafka_uris. You must manually provide the instance value if there is more than one string in kafka_uris. | | no |
| `use_sasl` | `bool` | Connect using SASL/PLAIN. | | no |
| `use_sasl_handshake` | `bool` | Only set this to false if using a non-Kafka SASL proxy. | `true` | no |
| `sasl_username` | `string` | SASL user name. | | no |
| `sasl_password` | `string` | SASL user password. | | no |
| `sasl_mechanism` | `string` | The SASL SCRAM SHA algorithm sha256 or sha512 as mechanism. | | no |
| `sasl_disable_pafx_fast` | `bool` | Configure the Kerberos client to not use PA_FX_FAST. | | no |
| `use_tls` | `bool` | Connect using TLS. | | no |
| `tls_server_name` | `string` | Used to verify the hostname on the returned certificates unless tls.insecure-skip-tls-verify is given. If you don't provide the Kafka server name, the hostname is taken from the URL. | | no |
| `ca_file` | `string` | The optional certificate authority file for TLS client authentication. | | no |
| `cert_file` | `string` | The optional certificate file for TLS client authentication. | | no |
| `key_file` | `string` | The optional key file for TLS client authentication. | | no |
| `insecure_skip_verify` | `bool` | If set to true, the server's certificate will not be checked for validity. This makes your HTTPS connections insecure. | | no |
| `kafka_version` | `string` | Kafka broker version. | `2.0.0` | no |
| `use_zookeeper_lag` | `bool` | If set to true, use a group from zookeeper. | | no |
| `zookeeper_uris` | `array(string)` | Address array (hosts) of zookeeper server. | | no |
| `kafka_cluster_name` | `string` | Kafka cluster name. | | no |
| `metadata_refresh_interval` | `duration` | Metadata refresh interval. | `1m` | no |
| `gssapi_service_name` | `string` | Service name when using Kerberos Authorization | | no |
| `gssapi_kerberos_config_path` | `string` | Kerberos config path. | | no |
| `gssapi_realm` | `string` | Kerberos realm. | | no |
| `gssapi_key_tab_path` | `string` | Kerberos keytab file path. | | no |
| `gssapi_kerberos_auth_type` | `string` | Kerberos auth type. Either 'keytabAuth' or 'userAuth'. | | no |
| `offset_show_all` | `bool` | If true, the broker may auto-create topics that we requested which do not already exist. | `true` | no |
| `topic_workers` | `int` | Minimum number of topics to monitor. | `100` | no |
| `allow_concurrency` | `bool` | If set to true, all scrapes trigger Kafka operations. Otherwise, they will share results. WARNING: Disable this on large clusters. | `true` | no |
| `allow_auto_topic_creation` | `bool` | If true, the broker may auto-create topics that we requested which do not already exist. | | no |
| `max_offsets` | `int` | The maximum number of offsets to store in the interpolation table for a partition. | `1000` | no |
| `prune_interval_seconds` | `int` | Deprecated (no-op), use `metadata_refresh_interval` instead. | `30` | no |
| `topics_filter_regex` | `string` | Regex filter for topics to be monitored. | `.*` | no |
| `topics_exclude_regex` | `string` | Regex that determines which topics to exclude. | `^$` | no |
| `groups_filter_regex` | `string` | Regex filter for consumer groups to be monitored. | `.*` | no |
| `groups_exclude_regex` | `string` | Regex that determines which consumer groups to exclude. | `^$` | no |

## Exported fields

Expand Down
11 changes: 7 additions & 4 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ require (
github.com/coreos/go-systemd v0.0.0-20191104093116-d3cd4ed1dbcf
github.com/coreos/go-systemd/v22 v22.5.0
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/davidmparrott/kafka_exporter/v2 v2.0.1
github.com/dimchansky/utfbom v1.1.1
github.com/docker/docker v24.0.9+incompatible
github.com/docker/go-connections v0.5.0
Expand Down Expand Up @@ -258,6 +257,11 @@ require (
sigs.k8s.io/yaml v1.4.0
)

require (
github.com/Shopify/sarama v1.38.1
github.com/grafana/kafka_exporter v0.0.0-20240409084445-5e3488ad9f9a
)

require (
cloud.google.com/go v0.112.0 // indirect
cloud.google.com/go/compute v1.23.3 // indirect
Expand Down Expand Up @@ -300,7 +304,6 @@ require (
github.com/Microsoft/hcsshim v0.11.4 // indirect
github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5 // indirect
github.com/ProtonMail/go-crypto v0.0.0-20230828082145-3c4c8a2d2371 // indirect
github.com/Shopify/sarama v1.38.1 // indirect
github.com/Showmax/go-fqdn v1.0.0 // indirect
github.com/Workiva/go-datastructures v1.1.0 // indirect
github.com/alecthomas/participle/v2 v2.1.1 // indirect
Expand Down Expand Up @@ -638,8 +641,8 @@ require (
github.com/xanzy/ssh-agent v0.3.3 // indirect
github.com/xdg-go/pbkdf2 v1.0.0 // indirect
github.com/xdg-go/stringprep v1.0.4 // indirect
github.com/xdg/scram v0.0.0-20180814205039-7eeb5667e42c // indirect
github.com/xdg/stringprep v1.0.0 // indirect
github.com/xdg/scram v1.0.3 // indirect
github.com/xdg/stringprep v1.0.3 // indirect
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
github.com/xeipuuv/gojsonschema v1.2.0 // indirect
Expand Down
Loading
Loading