Kafka scaler - "error creating kafka client: kafka: client has run out of available brokers to talk to: EOF"

[nkfirdigi]

Hi
I'm using kafka scaler
Keda version 2.9.1
using kafka confluent cloud
Getting error "error creating kafka client: kafka: client has run out of available brokers to talk to: EOF" in keda operator
I used various versions "0.10.x.0" up to "3.3.1", got the same error.
Appreciate any suggestion,

my kafka scaler:

  triggers:
    - type: kafka
      metadata:
        bootstrapServers: pkc-yyyyyy.aws.confluent.cloud:9092
        consumerGroup: typed-consumer      
        topic: keda-test
        lagThreshold: "2"
        activationLagThreshold: "1"
        offsetResetPolicy: "latest"
        scaleToZeroOnInvalidOffset: "true"
        allowIdleConsumers: "true"
        version: "0.10.2.0"
      authenticationRef:
        name: keda-trigger-auth-kafka-credential

[nkfirdigi]

Just for reference
I was using the wrong property name
I used authMode (from keda 1.4) instead of "sasl" which is the correct property name for keda 2.9 I'm using.
so I was able to connect to confluent cloud with following settings

apiVersion: v1
kind: Secret
metadata:
  name: keda-kafka-secrets
data:
  sasl: "cGxhaW50ZXh0" # plaintext base 64 encoded
  tls: "ZW5hYmxl" # enable base 64 encoded
  username: $(secrets.kafka.username)
  password: $(secrets.kafka.password

[zroubalik]

Great to hear that!

[vchaudhary-vn]

Hey @nkfirdigi
please don't mind me asking!
very new to Keda and SASL stuff, what are these values you provided for sasl and tls above? are these static values ?

[nkfirdigi]

@vchaudhary-vn
Note that sasl and tls are not related to Keda, but are Kafka authorization configuration.
Depending on your Kafka cluster configuration / Kafka cloud U need to set these parameters and others.
see Kafka doc
https://docs.confluent.io/platform/current/kafka/overview-authentication-methods.html

[Elias-Swais]

please @nkfirdigi can you help me , ive been stuck on this issue for a while now
heres my yaml file:

apiVersion: v1
kind: Secret
metadata:
name: keda-kafka-secrets
namespace: karavan
data:
sasl: "cGxhaW50ZXh0"
username: "V1VDM0ZZUFVZWUdQWVE3Rw=="
password: "YWZDOFhodVlHais1SmdqWklnemJHMEZDZ0xmUkY3dkg1WjRlV05KUzN4U3FIQmx5bTFDYXZIUzlaNjJ3eElhQQ=="
tls: "ZW5hYmxl"

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: keda-trigger-auth-kafka-credential
namespace: karavan
spec:
secretTargetRef:

• parameter: sasl
  name: keda-kafka-secrets
  key: sasl
• parameter: username
  name: keda-kafka-secrets
  key: username
• parameter: password
  name: keda-kafka-secrets
  key: password
• parameter: tls
  name: keda-kafka-secrets
  key: tls

---

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: kafka-scaledobject
namespace: karavan
spec:
scaleTargetRef:
name: s9wl00s
pollingInterval: 30
triggers:

• type: kafka
  metadata:
  bootstrapServers: pkc-q9z57.uaenorth.azure.confluent.cloud:9092
  consumerGroup: my-group # Make sure that this consumer group name is the same one as the one that is consuming topics
  topic: test-topic

  Optional

  lagThreshold: "5"
  offsetResetPolicy: latest
  authenticationRef:
  name: keda-trigger-auth-kafka-credential

[nkfirdigi]

@Elias-Swais

1. Dont put your plain user passwords on github ...

2. Example of working yaml - (user and pass are taken from k8s secrets)
   `
   apiVersion: v1
   kind: Secret
   metadata:
   name: s2k-keda-kafka-secrets
   data:
   sasl: {{ "plaintext" | b64enc }}
   tls: {{ "enable" | b64enc }}

---

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: s2k-keda-trigger-auth-kafka-credential
spec:
secretTargetRef:
- parameter: tls
name: s2k-keda-kafka-secrets
key: tls
- parameter: sasl
name: s2k-keda-kafka-secrets
key: sasl
- parameter: username
name: {{ .Values.secretName }}
key: kafka_confluentcloud_username
- parameter: password
name: {{ .Values.secretName }}
key: kafka_confluentcloud_password
`

[javaDer]

Hello everyone, this is how I solved this problem:
for reference only:
I downgraded sarama to v1.19.0. The code is as follows:
go get github.com/Shopify/[email protected]
It will work normally