Prometheus threshold semantics

[kefik]

Hi! Thanks a lot for Keda!

I'm facing a scaling issue with the following scaledobject

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: xxx
  namespace: some-namespace
  labels:
    deploymentName: xxx
spec:
  minReplicaCount: 1
  maxReplicaCount: 20
  pollingInterval: 60
  scaleTargetRef:
    kind: Deployment
    name: xxx
  triggers:
  - type: prometheus
    metadata:
      serverAddress: correct_address
      metricName: correct_metric_name
      threshold: '1'
      query: sum(correct_metric_spec)
      namespace: some-namespace

I don't quite understand the 'threshold' semantics. My initial thought was that the
number of replicas == round_down(metric / threshold)

However that seems to hold only up to the metric value of 11, then if there are 11 running replicas and the metric is 12.1, HPA reports 1.091/1 and refuses to scale to number of replicas 12.

Appreciate any thoughts here! Thanks!

EDIT

Keda revision is 2.6.2
Here is a sequence of readings how the metric is translating to HPA value

Replicas	Metric		HPA
1		1		1/1
1		2.1		2/1 -> scale up
2		3.1		1500m/1 -> scale up
3		3.1 		1/1
4		4.1 		1334m/1 -> scale up
4 		5.1 		1200m/1 -> scale up
5 		6.1 		1200m/1 -> scale up
6		6.1 		1167m/1 -> scale up
7 		7.1 		1143m/1 -> scale up
8 		8.1 		1/1
8		9.1 		1125m/1 -> scale up
9		9.1 		1/1
9		10.1 		1112m/1 -> scale up
10 		10.1 		1/1
10 		11.1 		1100m/1 -> scale up
11  		11.1 		1/1
11 		12.1 		1091m/1 -> NOT scaling up

[kefik]

After playing with the threshold value, I found out it is the value that is set as desiredMetricValue of HorizontalPodAutoscaler (HPA)
https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

Plus, as stated in documentation Prometheus scaler is working with average values only, also mentioned here:
#1881

And the HPA scaling then follows this equation:
desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]

Where desiredMetricValue is the threshold in Prometheus scaler configuration.
currentMetricValue is then the value obtained from Prometheus (according to query) divided by the number of currently existing replicas.

[kefik]

Oh and one last thing, the documentation on HPA is mentioning tolerance to values sufficiently close to 1, which defaults to 0.1, which explains why scaling up stopped once the HPA metric values reached 1091m/1 as that value is within dead zone of 0.1.

[kefik]

Finally, if you are checking on scale down (values N-1), this is governed by HPA itself, which scales down only if the metric is indicating so for 5 minutes.

[zroubalik]

Yeah, that's correct. Please also note that with 2.7 version you can use targetValue instead of averageTargetValue: https://keda.sh/docs/2.7/concepts/scaling-deployments/#triggers -> see metricType section.

[carillonator]

Could this be documented with the prometheus scaler? To me this is not at all obvious:

currentMetricValue is then the value obtained from Prometheus (according to query) divided by the number of currently existing replicas

How would anyone know that without finding this discussion?