EKS 1.21 - SQS Scalar - Metric calculation by KEDA

[rangarb885]

In EKS 1.21 i have deployed KEDA 2.3 and trying to scale my deployment based on SQS.

kubectl get hpa -n sbx --watch                                                                                                                                                
NAME               REFERENCE            TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-adapter   Deployment/adapter   12334m/30 (avg)   1         10        3          9m22s
keda-hpa-adapter   Deployment/adapter   3/30 (avg)        1         10        3          9m28s
keda-hpa-adapter   Deployment/adapter   2667m/30 (avg)    1         10        3          9m43s
keda-hpa-adapter   Deployment/adapter   6667m/30 (avg)    1         10        3          9m58s

What does this TARGETS value of 12334m/30 (avg) really means?
Usually the queue length is returned as whole number in AWS API. How should we convert this metric value 12334m to a whole number in order to validate the thresholds & target scaling?

[tomkerkhove]

The current target value on the Horizontal Pod Autoscaler (HPA) often does not match with the metrics on the system you are scaling on.

You can learn more about it on https://keda.sh/docs/2.4/faq/#kubernetes

[rangarb885]

Thanks @tomkerkhove . I agree that HPA decides the # of replicas using its own algorithm.

Here the metric value reported is from keda metric server so how does it calculate this value?

Reason i am asking is:

I paused my SQS consuming application and when i push 5000 message to SQS and look at this metric using the metric API, it shows metric value as 5k. But HPA shows the target metric value as 165500m,40875m....500 (at the end).

k get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/sbx/aws-sqs-queue-adapter?labelSelector=scaledobject.keda.sh%2Fname%3Dadapter" |  jq .
{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metricName": "aws-sqs-queue-adapter",
      "metricLabels": null,
      "timestamp": "2021-08-26T00:00:41Z",
      "value": "5k"
    }
  ]
}

 kubectl get hpa -n sbx --watch
NAME               REFERENCE            TARGETS      MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-adapter   Deployment/adapter   0/10 (avg)   1         10        1          3h19m
keda-hpa-adapter   Deployment/adapter   83/10 (avg)   1         10        1          3h19m
keda-hpa-adapter   Deployment/adapter   165500m/10 (avg)   1         10        4          3h19m
keda-hpa-adapter   Deployment/adapter   40875m/10 (avg)    1         10        8          3h20m
keda-hpa-adapter   Deployment/adapter   169500m/10 (avg)   1         10        10         3h20m
keda-hpa-adapter   Deployment/adapter   227100m/10 (avg)   1         10        10         3h20m
keda-hpa-adapter   Deployment/adapter   176900m/10 (avg)   1         10        10         3h20m
keda-hpa-adapter   Deployment/adapter   227100m/10 (avg)   1         10        10         3h21m
keda-hpa-adapter   Deployment/adapter   340100m/10 (avg)   1         10        10         3h21m
keda-hpa-adapter   Deployment/adapter   393200m/10 (avg)   1         10        10         3h21m
keda-hpa-adapter   Deployment/adapter   388200m/10 (avg)   1         10        10         3h21m
keda-hpa-adapter   Deployment/adapter   437800m/10 (avg)   1         10        10         3h22m
keda-hpa-adapter   Deployment/adapter   500/10 (avg)   1         10        10         3h22m

[JorTurFer]

Hi @rangarb885
That's becase the majority of the KEDA's scalers use AverageValue metric type. This means that the value returned by KEDA will be divided between all the instances by the HPA. If KEDA returns 5k (like you can see in the response from metrics), the HPA will take that 5000 and divide between the number of instances (10 in your case).
That's the reason because you see 500 instead of 5000