Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix topics in loki.source.kafka #2351

Merged
merged 7 commits into from
Jan 8, 2025
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
thampiotr marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,8 @@ Main (unreleased)

- Change the log level in the `eventlogmessage` stage of the `loki.process` component from `warn` to `debug`. (@wildum)

- Fix a bug in `loki.source.kafka` where the `topics` argument incorrectly used regex matching instead of exact matches. (@wildum)

### Other changes

- Change the stability of the `livedebugging` feature from "experimental" to "generally available". (@wildum)
Expand Down
2 changes: 2 additions & 0 deletions docs/sources/reference/components/loki/loki.source.kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ loki.source.kafka "LABEL" {

`assignor` values can be either `"range"`, `"roundrobin"`, or `"sticky"`.

If a topic starts with a '^', it is treated as a regex and may match multiple topics.
wildum marked this conversation as resolved.
Show resolved Hide resolved

Labels from the `labels` argument are applied to every message that the component reads.

The `relabel_rules` field can make use of the `rules` export value from a
Expand Down
8 changes: 8 additions & 0 deletions docs/sources/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,14 @@ For a complete list of changes to {{< param "FULL_PRODUCT_NAME" >}}, with links

[Changelog]: https://github.com/grafana/alloy/blob/main/CHANGELOG.md

## v1.6

### Breaking change: The `topics` argument in the component `loki.source.kafka` does not use regex by default anymore

A bug in `loki.source.kafka` caused the component to treat all topics as regex. For example, setting the topic value to "foo" would match any topic containing the substring "foo".
wildum marked this conversation as resolved.
Show resolved Hide resolved
wildum marked this conversation as resolved.
Show resolved Hide resolved
With the fix introduced in this version, topic values are now treated as exact matches by default.
Regex matching is still supported by prefixing a topic with "^", allowing it to match multiple topics.
wildum marked this conversation as resolved.
Show resolved Hide resolved

## v1.5

### Breaking change: Change default value of `max_streams` in `otelcol.processor.deltatocumulative`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ func newTopicManager(client topicClient, topics []string) (*topicManager, error)
}
if t[0] != '^' {
matches = append(matches, t)
continue
}
re, err := regexp.Compile(t)
if err != nil {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,16 @@ func Test_Topics(t *testing.T) {
[]string{"foo", "foobar"},
false,
},
{
mustNewTopicsManager(&mockKafkaClient{topics: []string{"foo", "foobar", "buzz"}}, []string{"^foo$"}),
[]string{"foo"},
false,
},
{
mustNewTopicsManager(&mockKafkaClient{topics: []string{"foo", "foobar", "buzz"}}, []string{"foo"}),
[]string{"foo"},
false,
},
{
mustNewTopicsManager(&mockKafkaClient{topics: []string{"foo", "foobar", "buzz"}}, []string{"^foo.*", "buzz"}),
[]string{"buzz", "foo", "foobar"},
Expand Down
Loading