Error from server: exactly one ScaledObject should match label

[hajdukda]

Hey,

I have few scaled object like this:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: some-scaler-{uniqueIdentifierName}
spec:
  scaleTargetRef:
    name: some-deployment-{differentDeployment}
  minReplicaCount: 1
  maxReplicaCount: 10
  advanced:
    restoreToOriginalReplicaCount: true
    horizontalPodAutoscalerConfig:
      name: some-hpa-{SomethingUnique}
      behavior:
         .... doesnt matter whats here
  triggers:
    - type: cpu
      metricType: Utilization
      metadata:
        value: "50"
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: my_metric_name{uniqueName}
        metricType: AverageValue
        threshold: "500"
        query: max(rabbitmq_messages{queue="something{uniqueName}"})
        ignoreNullValues: "false"

When Im quering API to get the latest value of my metric - api returns whats in title...

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/something/s1-prometheus-my_metric_name"
Error from server: exactly one ScaledObject should match label

MetricsServer ofcourse reports the same, "soemthing something should match label"

Funny enough that my HPA reports:

ValidMetricFound    the HPA was able to successfully calculate a replica count from external metric s1-prometheus-my_metric_name

But getting metric value through API still fails.. Does anyone has any idea why that might be the case ?

ps. Status of my scaler - for which Im quering API is:

Status:
  Conditions:
    Message:  ScaledObject is defined correctly and is ready for scaling
    Reason:   ScaledObjectReady
    Status:   True
    Type:     Ready
    Message:  Scaling is performed because triggers are active
    Reason:   ScalerActive
    Status:   True
    Type:     Active
    Message:  No fallbacks are active on this scaled object
    Reason:   NoFallbackFound
    Status:   False
    Type:     Fallback
  External Metric Names:
    s1-prometheus-my_metric_name..
  Health:
    s1-prometheus-my_metric_name..:
      Number Of Failures:  0
      Status:              Happy
  Hpa Name:                my-hpa

[zroubalik]

https://keda.sh/docs/2.8/operate/metrics-server/#how-to-get-metric-when-multiple-scaledobjects-have-the-same-metric-name

[hajdukda]

What in case all of them have completely different metricName (unique between all ScaledObjects)?
metricName: {queue_name}_queue_size

I read this doc, still I find it weird that for different metricNames it still requires the label.

[zroubalik]

that's weird then, could you post here your scaledobjects (or just statuses with metric names)?

[psladek-seekr]

Did a fresh install of 2.8.2.. Only one namespace. The only scaled objects we have.. metric names are different. labels are different. Fake queries still give the same error.

kind: ScaledObject
metadata:
annotations:
helm.sh/hook: post-install,post-upgrade
creationTimestamp: "2023-08-10T18:51:58Z"
finalizers:

• finalizer.keda.sh
  generation: 2
  labels:
  deploymentName: selenium-chrome-node
  scaledobject.keda.sh/name: selenium-grid-chrome-scaledobject

labels:
deploymentName: selenium-edge-node
scaledobject.keda.sh/name: selenium-grid-edge-scaledobject

externalMetricNames:

• s0-seleniumgrid-chrome

externalMetricNames:

• s0-seleniumgrid-firefox

externalMetricNames:

• s0-seleniumgrid-MicrosoftEdge

NAMESPACE NAME SCALETARGETKIND SCALETARGETNAME MIN MAX TRIGGERS AUTHENTICATION READY ACTIVE FALLBACK AGE
selenium-grid selenium-grid-chrome-scaledobject apps/v1.Deployment selenium-chrome-node 1 8 selenium-grid True False False 43d
selenium-grid selenium-grid-edge-scaledobject apps/v1.Deployment selenium-edge-node 1 8 selenium-grid True False False 43d
selenium-grid selenium-grid-firefox-scaledobject apps/v1.Deployment selenium-firefox-node 1 8 selenium-grid True False False 43d

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/selenium-grid-chrome/s0-seleniumgrid-chrome?labelSelector=scaledobject.keda.sh%2Fname%3Dselenium-grid-chrome-scaledobject" Error from server: exactly one ScaledObject should match label scaledobject.keda.sh/name=selenium-grid-chrome-scaledobject

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/selenium-grid-chrome/s0-seleniumgrid-chrome" Error from server: exactly one ScaledObject should match label

$APICURL https://xxxxxx:6443/"/apis/external.metrics.k8s.io/v1beta1/namespaces//flux-query?labelSelector=query-name%3Dhttp_requests"
HTTP/1.1 500 Internal Server Error
Audit-Id: xxxxxxxxxxxxxxxxx
Cache-Control: no-cache, private
Cache-Control: no-cache, private
Content-Length: 185
Content-Type: application/json
Date: Thu, 19 Oct 2023 12:51:23 GMT
X-Kubernetes-Pf-Flowschema-Uid: xxxx
X-Kubernetes-Pf-Prioritylevel-Uid: xxxx

{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "exactly one ScaledObject should match label query-name=http_requests",
"code": 500
}

@zroubalik Any help would be appreciated. Thx!