Kafka Scaler - lagThreshold

[arnoldyahad]

Hey guys,
We want to implement KEDA in our organization and we have a lot of kafka consumers, so using the Apache Kafka scaler would be our main use-case.

I was going through the documentation, especially this page: https://keda.sh/docs/2.7/scalers/apache-kafka/
I'm trying to understand how can the lagThreshold be configured if i have a topic with 100M-500M messages, and about 50K messages in a minute.
is the lagThreshold basically how much lag there is now in the consumer group? for example if the offset is 1,000,000 and the consumers only reached 950,000 and i dont want to be in a larger lag - should i put lagThreshold on 50K ?
(basically what im asking is an explanation to the meaning of lagThreshold, an example would be great)

currently, we have alerts and manual scaling if we reach lags of 20 minutes, we know the information of how much time we are in lag due to kafka-lag-exporter that exposes this metric :prometheus_kafka_consumergroup_group_lag_seconds - which tells us exactly how much time we are lagging behind - is something like this exist in KEDA?

(link to kafka-lag-exporter: https://github.com/seglo/kafka-lag-exporter#overview)

Thanks a lot,
Arnold.

[javierlga]

I'm having to understand this as well, so I took a look at the code, and from what I could understand first it gets the producer and consumer offsets in a partition from a specific topic, if topic is unspecified it calculate the lag for all the topics from a consumer group.

After getting all topic partitions, it will start calculating the total lag for every partition by subtracting the latest and current consumer offsets.

Hopefully someone from the Keda team can clarify this and improve the documentation as it's pretty vague.

[kamilglod]

I was asking myself the same question and I how I understand it after reading the docs:

we have hpa (kubectl get hpa) that controlling how and when to scale pods (more details here: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/). So generally it bases on TARGETS value - we can set lagThreshold and it becomes the desire value for TARGET. When the external metric (kafka returns sum of lags of all partitions in the topic) return value then hpa is trying to scale up to totalLag / lagThreshold. So for example with:

lagThreshold = 100
totalLag = 600
target = 600
desire number of pods = 6

So we can treat lagThreshold value as number of messages that one pod should handle. When the lag grows, keda is scaling new pod that should help to reduce the totalLag.