keda kafka scaled object frequently scaling pods

[k8sdevopsuser]

EKS version v1.18.9
MSK version
We have Keda version 2.2 installed
AME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
keda keda 1 2021-05-12 12:15:45.61748888 +0530 IST deployed keda-2.2.2 2.2.0
We have defined a scaled object as follows

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: myapp-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp 
  pollingInterval: 5
  minReplicaCount: 1 
  maxReplicaCount: 5  
  cooldownPeriod: 30
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: kafka-broker:9092 
      consumerGroup: myappCG 
      topic: myappTopic
      lagThreshold: "10"

We usually see the message in the keda operator log wherein we get lag values for the 3 partitions and everything is good .

021-05-13T12:49:08.945Z	DEBUG	kafka_scaler	Group myappTopicConsumerGroup has a lag of 0 for topic myappTopic and partition 0
2021-05-13T12:49:08.946Z	DEBUG	kafka_scaler	Group myappTopicConsumerGroup has a lag of 0 for topic myappTopic and partition 2
2021-05-13T12:49:08.947Z	DEBUG	kafka_scaler	Group myappTopicConsumerGroup has a lag of 0 for topic myappTopic and partition 1
2021-05-13T12:49:08.947Z	DEBUG	scaleexecutor	ScaleTarget no change	{"scaledobject.Name": "myapp-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "myapp"}

Sometimes we get response from only one partition , log entry similar to below and we are seeing the pods being scaled as expected as our target is set to 10 .

2021-05-13T12:49:13.965Z	DEBUG	scaleexecutor	ScaleTarget no change	{"scaledobject.Name": "myapp-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "myapp"}
2021-05-13T12:49:18.981Z	DEBUG	kafka_scaler	Group myappTopicConsumerGroup has a lag of 12 for topic myappTopic and partition 0

2021-05-13T12:49:18.981Z	DEBUG	scalehandler	Scaler for scaledObject is active	{"Metrics Name ": "kafka-myappTopic-myappCG"}

Our application processes the messages in batches so the lag threshold goes up and then as the batch is processed the lag goes to 0 . Keda currently only looks at the instantaneous lag value it polls every 5 seconds and sends it to hpa . HPA does the average of the instantaneous values and decides whether to scale out or scale in .

Is there a way keda can do overage of the values received over a predefined period and then send it to HPA .

Or is there a way to configure number of polling intervals the lag is more than threshold before the scale out or scale in happens .
In our case we see the scaling happens very frequently and whenever we check the hpa the current value is 0 but we still see 3 pods running because of the way the HPA is scaling the pods .

[zroubalik]

KEDA is stateless so unfortunately we don't keep any state or record of previous metrics, on which we can try to do some analysis as you suggest. But that's interesting feature so we are looking to see some specific proposals!

Because 1<->N scaling is done by HPA, so you can try to twek HPA settings, it might help you with that.

You can try to look at the scaling behavior:
https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior

or --horizontal-pod-autoscaler-downscale-stabilization
https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-cooldown-delay