Scaling 0 -> 1 is istant, 1 -> X takes up to five minutes

[MaxWinterstein]

We use some Redis scaler to spin up worker pods/deployments using scaledObjects. As there are none needed sometimes, we scale them to 0.

Now when there is a need, we observe the following behavior:

• Redis list grows
• KEDA scales the deployment from 0 to 1
• It takes sometimes up to five minutes till we get more than 1 replica

While we wait for the - in this case 3 remaining ones - we can have a look at some things:

The metrics from the metrics server look fine,

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/sacle-test/s0-redis-default3?labelSelector=scaledobject.keda.sh%2Fname%3Dredis-scaledobject"

{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metricName": "s0-redis-default3",
      "metricLabels": null,
      "timestamp": "2024-11-28T14:13:34Z",
      "value": "4"
    }
  ]

The k get hpa looks correct:

keda-hpa-redis-scaledobject            Deployment/dummy2   4/1 (avg)                       1         50        4          4h5m

Even the k describe hpa shows only good stuff:

  Normal   SuccessfulRescale        2m49s (x2 over 45m)  horizontal-pod-autoscaler  New size: 4; reason: external metric s0-redis-default3(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: redis-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}) above target

And, after around 3 minutes this time, we see the pods:

NAME                      READY   STATUS    RESTARTS   AGE
dummy2-74f677bb54-4c46d   1/1     Running   0          38s
dummy2-74f677bb54-lwf6c   1/1     Running   0          3m32s
dummy2-74f677bb54-pzsp5   1/1     Running   0          38s
dummy2-74f677bb54-xs6jb   1/1     Running   0          38s

Using GCP and Autopilot as well as Classic GKE, v1.31.x.

A minimal reproduction can look like this:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: redis-scaledobject
  labels:
    deploymentName: dummy2
spec:
  maxReplicaCount: 50
  pollingInterval: 3  # Optional. Default: 30 seconds
  cooldownPeriod:  5 # Optional. Default: 300 seconds
  scaleTargetRef:
    name: dummy2
  triggers:
    - type: redis
      metadata:
        address: redis.sacle-test:6379
        listName: default3
        listLength: "1"

We also sometimes see the -9223372036854775808m/1 (avg) thing, rel #5730.

---

I can't find any discussion about that behavior, other than https://keda.sh/docs/2.14/concepts/scaling-deployments/#activating-and-scaling-thresholds mentioning that some scalers support "bang-scaling" from 0 -> 1+

Am I missing something out? Is this expected behavior?

[MaxWinterstein]

I did some local testing, and can't reproduce that issues with minikube.

Only visible difference is the issue with the -9..., that are not visible at minikube. There is looks like this:

I opened some case at google, lets see what happens...

[zroubalik]

Let's continue on #5730