Fix topics in loki.source.kafka (#2351)

* fix topics in loki.source.kafka * add release notes * Update docs/sources/reference/components/loki/loki.source.kafka.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/release-notes.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/release-notes.md Co-authored-by: Clayton Cornell <[email protected]> * replace foo by telemetry in release notes --------- Co-authored-by: Clayton Cornell <[email protected]>
grafana · Jan 8, 2025 · 0a3e6df · 0a3e6df
1 parent 28e4209
commit 0a3e6df
Show file tree

Hide file tree

Showing 5 changed files with 21 additions and 0 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -90,6 +90,8 @@ Main (unreleased)
 
 - Change the log level in the `eventlogmessage` stage of the `loki.process` component from `warn` to `debug`. (@wildum)
 
+- Fix a bug in `loki.source.kafka` where the `topics` argument incorrectly used regex matching instead of exact matches. (@wildum)
+
 ### Other changes
 
 - Change the stability of the `livedebugging` feature from "experimental" to "generally available". (@wildum)

diff --git a/docs/sources/reference/components/loki/loki.source.kafka.md b/docs/sources/reference/components/loki/loki.source.kafka.md
@@ -49,6 +49,8 @@ loki.source.kafka "LABEL" {
 
 `assignor` values can be either `"range"`, `"roundrobin"`, or `"sticky"`.
 
+If a topic starts with a '^', it is treated as a regular expression and may match multiple topics.
+
 Labels from the `labels` argument are applied to every message that the component reads.
 
 The `relabel_rules` field can make use of the `rules` export value from a

diff --git a/docs/sources/release-notes.md b/docs/sources/release-notes.md
@@ -16,6 +16,12 @@ For a complete list of changes to {{< param "FULL_PRODUCT_NAME" >}}, with links
 
 ## v1.6
 
+### Breaking change: The `topics` argument in the component `loki.source.kafka` does not use regex by default anymore
+
+A bug in `loki.source.kafka` caused the component to treat all topics as regular expressions. For example, setting the topic value to "telemetry" would match any topic containing the substring "telemetry".
+With the fix introduced in this version, topic values are now treated as exact matches by default.
+Regular expression matching is still supported by prefixing a topic with "^", allowing it to match multiple topics.
+
 ### Breaking change: Change decision precedence in `otelcol.processor.tail_sampling` when using `and_sub_policy` and `invert_match` 
 
 Alloy v1.5 upgraded to [OpenTelemetry Collector v0.104.0][otel-v0_104], which included a [fix][#33671] to the tail sampling processor:

diff --git a/internal/component/loki/source/internal/kafkatarget/topics.go b/internal/component/loki/source/internal/kafkatarget/topics.go
@@ -36,6 +36,7 @@ func newTopicManager(client topicClient, topics []string) (*topicManager, error)
 		}
 		if t[0] != '^' {
 			matches = append(matches, t)
+			continue
 		}
 		re, err := regexp.Compile(t)
 		if err != nil {

diff --git a/internal/component/loki/source/internal/kafkatarget/topics_test.go b/internal/component/loki/source/internal/kafkatarget/topics_test.go
@@ -91,6 +91,16 @@ func Test_Topics(t *testing.T) {
 			[]string{"foo", "foobar"},
 			false,
 		},
+		{
+			mustNewTopicsManager(&mockKafkaClient{topics: []string{"foo", "foobar", "buzz"}}, []string{"^foo$"}),
+			[]string{"foo"},
+			false,
+		},
+		{
+			mustNewTopicsManager(&mockKafkaClient{topics: []string{"foo", "foobar", "buzz"}}, []string{"foo"}),
+			[]string{"foo"},
+			false,
+		},
 		{
 			mustNewTopicsManager(&mockKafkaClient{topics: []string{"foo", "foobar", "buzz"}}, []string{"^foo.*", "buzz"}),
 			[]string{"buzz", "foo", "foobar"},