AWS Cloudwatch Scaler: Not able to fetch the metrics from cloudwatch through query expression and dimentions also #5310

sushama-kothawale · 2023-12-22T04:57:35Z

sushama-kothawale
Dec 22, 2023

I have KEDA running on EKS cluster. EKS version 1.25 , Keda version 2.11.1
I am using AWS cloudwatch scaler https://keda.sh/docs/2.12/scalers/aws-cloudwatch/
Here is my scaledobject definition:

spec:
  cooldownPeriod: 300
  maxReplicaCount: 10
  minReplicaCount: 2
  pollingInterval: 5
  scaleTargetRef:
    name: cnt-dlm-cmw
  triggers:
  - authenticationRef:
      name: keda-trigger-auth-aws-credentials-irsa
    metadata:
      awsRegion: ap-south-1
      dimensionName: QueueName
      dimensionValue: event_queue.CMW.picasso_media_queue
      metricName: MessageCount
      metricStatPeriod: "5"
      minMetricValue: "1"
      namespace: AWS/AmazonMQ
      targetMetricValue: "10"
    type: aws-cloudwatch
status:
  conditions:
  - message: ScaledObject is defined correctly and is ready for scaling
    reason: ScaledObjectReady
    status: "True"
    type: Ready
  - message: Scaling is performed because triggers are active
    reason: ScalerActive
    status: "True"
    type: Active
  - message: No fallbacks are active on this scaled object
    reason: NoFallbackFound
    status: "False"
    type: Fallback
  - status: Unknown
    type: Paused
  externalMetricNames:
  - s0-aws-cloudwatch
  health:
    s0-aws-cloudwatch:
      numberOfFailures: 0
      status: Happy
  lastActiveTime: "2023-12-22T04:49:45Z"
  originalReplicaCount: 10
  scaleTargetGVKR:
    group: apps
    kind: Deployment
    resource: deployments
    version: v1
  scaleTargetKind: apps/v1.Deployment

I wanted to fetch the MessageCount metric of AWS/AmazonMQ service for above queue name. If the MessageCount exceeds the targetvalue then we wanted to scale the deployment.

We tried clouwatch query added as expression inside scaledobject as below:
expression: SELECT SUM("MessageCount") FROM "AWS/AmazonMQ" WHERE Queue = 'event_queue.CMW.picasso_callback_queue' AND Broker = 'content-rmq-integration' metricName: MessageCount

Expected behaviour:
Keda should scrape the metrics from AWS cloudwatch and it should scale if MessageCount reached the threashold value.

Actual Behavior:
We are seeing below logs on keda-operator:
Reconciling ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"cnt-dlm-cmw","namespace":"integration"}, "namespace": "integration", "name": "cnt-dlm-cmw", "reconcileID": "bff91f8d-0d6d-43eb-82d3-42ed2138ecee"} 2023-12-22T04:45:35Z INFO aws_cloudwatch_scaler empty metric data received, returning minMetricValue {"type": "ScaledObject", "namespace": "integration", "name": "cnt-dlm-cmw"} 2023-12-22T04:45:40Z INFO aws_cloudwatch_scaler empty metric data received, returning minMetricValue {"type": "ScaledObject", "namespace": "integration", "name": "cnt-dlm-cmw"} 2023-12-22T04:45:41Z INFO

When we check the current usage through HPA it shows as below:
k get hpa -n integration | grep keda-hpa-cnt-dlm-cmw keda-hpa-cnt-dlm-cmw Deployment/cnt-dlm-cmw 500m/10 (avg) 2 10 2
HPA spec is as below:

spec:
  maxReplicas: 10
  metrics:
  - external:
      metric:
        name: s0-aws-cloudwatch
        selector:
          matchLabels:
            scaledobject.keda.sh/name: cnt-dlm-cmw
      target:
        averageValue: "10"
        type: AverageValue
    type: External
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: cnt-dlm-cmw
status:
  conditions:
  - lastTransitionTime: "2023-12-21T10:10:51Z"
    message: recommended size matches current size
    reason: ReadyForNewScale
    status: "True"
    type: AbleToScale
  - lastTransitionTime: "2023-12-21T15:29:11Z"
    message: 'the HPA was able to successfully calculate a replica count from external
      metric s0-aws-cloudwatch(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name:
      cnt-dlm-cmw,},MatchExpressions:[]LabelSelectorRequirement{},})'
    reason: ValidMetricFound
    status: "True"
    type: ScalingActive
  - lastTransitionTime: "2023-12-21T05:17:53Z"
    message: the desired replica count is less than the minimum replica count
    reason: TooFewReplicas
    status: "True"
    type: ScalingLimited
  currentMetrics:
  - external:
      current:
        averageValue: 500m
        value: "0"
      metric:
        name: s0-aws-cloudwatch
        selector:
          matchLabels:
            scaledobject.keda.sh/name: cnt-dlm-cmw
    type: External
  currentReplicas: 2
  desiredReplicas: 2
  lastScaleTime: "2023-12-21T10:10:51Z"

So wanted to understand what is this 500m value. Even when we push the 1000 msgs on the queue it does not go beyond this 500m value.

Can someone please help on this?
cc: @JorTurFer

JorTurFer · 2023-12-22T10:17:02Z

JorTurFer
Dec 22, 2023
Maintainer

Hello @sushama-kothawale

The m means mili. 500m is 0,5. and it's the result of splitting minMetricValue between current instances. (1/2=0,5)

About your expression, the log you sent says empty metric data received, returning minMetricValue. You have to review your expression/query because it returns an empty response. I'm not an expert in AWS Cloudwatch to I can't help you so much in that side, sorry.

2 replies

sushama-kothawale Dec 22, 2023
Author

AWS Cloudwatch metrics are showing correct MessageCount. But when we try to fetch it from keda it return empty metric.

2023-12-22T11:38:31Z INFO aws_cloudwatch_scaler empty metric data received, returning minMetricValue {"type": "ScaledObject", "namespace": "integration", "name": "cnt-dlm-cmw"} 2023-12-22T11:38:36Z INFO aws_cloudwatch_scaler empty metric data received, returning minMetricValue {"type": "ScaledObject", "namespace": "integration", "name": "cnt-dlm-cmw"} 2023-12-22T11:38:38Z INFO Reconciling ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"umg-svc-gkpr","namespace":"integration"}, "namespace": "integration", "name": "umg-svc-gkpr", "reconcileID": "b7e11a38-dbcc-4ecf-92b5-38119fac541c"} 2023-12-22T11:38:38Z INFO "metricName"

Tried with dimensionName and dimensionValue parameters instead of expression,

    - authenticationRef:
        name: keda-trigger-auth-aws-credentials-irsa
      metadata:
        activationTargetMetricValue: '0'
        namespace: AmazonMQ
        awsRegion: ap-south-1
        metricName: MessageCount
        dimensionName: QueueName
        dimensionValue: event_queue.CMW.picasso_media_queue
        metricStatPeriod: '5'
        minMetricValue: '1'
        targetMetricValue: '10'
      type: aws-cloudwatch

everytime it shows the value as 500m only.

k get hpa -n integration keda-hpa-cnt-dlm-cmw  
NAME                   REFERENCE                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-cnt-dlm-cmw   Deployment/cnt-dlm-cmw   500m/10 (avg)   2         10        2          44h

Scaler is also active this mensa keda is able to conenct to cloudwatch but not able scrape the metrics:

k get scaledobject -n integration cnt-dlm-cmw  
NAME          SCALETARGETKIND      SCALETARGETNAME   MIN   MAX   TRIGGERS         AUTHENTICATION                           READY   ACTIVE   FALLBACK   PAUSED    AGE
cnt-dlm-cmw   apps/v1.Deployment   cnt-dlm-cmw       2     10    aws-cloudwatch   keda-trigger-auth-aws-credentials-irsa   True    True     False      Unknown   46h

Can you confirm if I am missing any configurations ins scaledobject? So that I tweak that .

JorTurFer Dec 23, 2023
Maintainer

I don't see anything weird in your config, maybe the time window can be producing the issue? Could you share the query that you do on CloudWatch directly?

sushama-kothawale · 2023-12-25T07:44:45Z

sushama-kothawale
Dec 25, 2023
Author

Thanks @JorTurFer for looking into this. I am using below query to check the MessageCount on queue event_queue.CMW.picasso_media_queue

expression: SELECT SUM("MessageCount") FROM "AWS/AmazonMQ" WHERE Queue = 'event_queue.CMW.picaso_media_queue' AND Broker = 'content-rmq-integration'
        metricName: MessageCount

5 replies

JorTurFer Dec 25, 2023
Maintainer

And that exact query executed directly in AWS console works, right? Which is the time window in AWS Console?

sushama-kothawale Dec 27, 2023
Author

Yes @JorTurFer , same query on AWS console gives output. Attaching the snapshot of AWS console here.

JorTurFer Dec 29, 2023
Maintainer

Do you see any extra option to configure which is missing in trigger? I mean, from Options tab

sushama-kothawale Dec 30, 2023
Author

These are some options but those are related to graph plotting on AWS clouwatch console. (Attached screenshot). Can you please check it?

As we are seeing metrics on cloudwatch not sure why keda is returning empty metrics. Can you please look into this?

JorTurFer Dec 30, 2023
Maintainer

I'm not an AWS expert, so I can't help you so far with this topic. I'm just checking the SO you shared in the beginning, and the dimensionName is QueueName but in your pictures I see just Queue.
Using an expression in the scaler do you see any other different message in KEDA logs?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS Cloudwatch Scaler: Not able to fetch the metrics from cloudwatch through query expression and dimentions also #5310

{{title}}

Replies: 2 comments 7 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

AWS Cloudwatch Scaler: Not able to fetch the metrics from cloudwatch through query expression and dimentions also #5310

sushama-kothawale Dec 22, 2023

Replies: 2 comments · 7 replies

JorTurFer Dec 22, 2023 Maintainer

sushama-kothawale Dec 22, 2023 Author

JorTurFer Dec 23, 2023 Maintainer

sushama-kothawale Dec 25, 2023 Author

JorTurFer Dec 25, 2023 Maintainer

sushama-kothawale Dec 27, 2023 Author

JorTurFer Dec 29, 2023 Maintainer

sushama-kothawale Dec 30, 2023 Author

JorTurFer Dec 30, 2023 Maintainer

sushama-kothawale
Dec 22, 2023

Replies: 2 comments 7 replies

JorTurFer
Dec 22, 2023
Maintainer

sushama-kothawale Dec 22, 2023
Author

JorTurFer Dec 23, 2023
Maintainer

sushama-kothawale
Dec 25, 2023
Author

JorTurFer Dec 25, 2023
Maintainer

sushama-kothawale Dec 27, 2023
Author

JorTurFer Dec 29, 2023
Maintainer

sushama-kothawale Dec 30, 2023
Author

JorTurFer Dec 30, 2023
Maintainer