Why the number of pods running is reduced while the number of messages in the queue is still high?

[ChayanBansal]

I am running a KEDA enabled k8s cluster with Storage Queue Azure Function which executes 2000 messages at once having variable execution times (between 30 seconds to 10 minutes).
The system works fine for a certain duration but when the number of messages in the queue starts decreasing, the number of pods also start decreasing.

Question: Why the number of pods running is reduced while the number of messages in the queue is still high?

Example: The maximum replica count is 120 and the number of messages in the queue is around 250. So why the HPA scales down the pods. Ideally, it should still be using all the 120 pods to complete all messages in the queue.

Supplement Questions:

1. Does queueLength affects the behavior of HPA?
2. Few messages were not executed fully. It seems they were shut down abruptly during the execution though terminationGracePeriodSeconds is 10 minutes. Any comments?

Following is the scaledobject.yaml file:

apiVersion: v1
kind: Secret
metadata:
  name: queue-connection-secret
data:
  connection-string: ####
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: azure-queue-auth
spec:
  secretTargetRef:
  - parameter: connection
    name: queue-connection-secret
    key: connection-string
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: queuetrigfuncpod
spec:
  scaleTargetRef:
    name: queuetrigfuncpod
  minReplicaCount: 0
  maxReplicaCount: 120
  pollingInterval: 1
  cooldownPeriod: 900

  triggers:
  - type: azure-queue
    metadata:
      queueName: k8s-poc-queue
      queueLength: "1"
    authenticationRef:
        name: azure-queue-auth

Deployments.yaml file:

apiVersion : apps/v1
kind: Deployment
metadata:
  name: queuetrigfuncpod
  labels:
    app: queuetrigfuncpod
spec:
  selector:
    matchLabels:
      app: queuetrigfuncpod
  template:
    metadata:
      labels:
        app: queuetrigfuncpod
    spec:
      containers:
        - image: #######.azurecr.io/#######
          name: queuetrigcontainer
          ports:
          - containerPort: 80
          resources:
            requests:
              memory: "500Mi"
              cpu: "700m"
            limits:
              memory: "600Mi"
              cpu: "700m"
      nodeSelector:
        agentpool: testuserpool
      terminationGracePeriodSeconds: 600
      imagePullSecrets:
      - name: regcred

[zroubalik]

I think that HPA controller does that based on the queue length (coming from KEDA metrics) and the current pod countm, you can check the HPA algorithm for details: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details

You might want to check https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior as well, there are scaling policies, that could help you with tweaking how fast the HPA does the scale up/down operation.

[ChayanBansal]

Thanks for your quick reply!
I was going through the links you shared but was wondering that when I try to describe the hpa using kubectl command the following is shown:

Can you please explain what does the target average value 4167m / 5 mean?

[zroubalik]

I recommend you to check HPA design docs: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details