Scaled Job behaviour with Redis Trigger not plausible

[QuintinBecker]

We use ScaledJobs for long running workloads.
At the moment we write events to the Redis-List we can see some jobs getting scheduled, but much less than we would expect.
In addition there are some entries in the list that stay there for very long time and cause a very delayed scaleup.

During this delay we see following output in KEDA-Operator:

2020-11-13T13:10:03.408Z	INFO	scalehandler	Active trigger	{"Scaler": {}, "isTriggerActive": true}
2020-11-13T13:10:03.409Z	INFO	scalehandler	Scaler targetAverageValue	{"Scaler": {}, "targetAverageValue": 1}
2020-11-13T13:10:03.409Z	INFO	scalehandler	QueueLength Metric value	{"Scaler": {}, "queueLength": 2}
2020-11-13T13:10:03.409Z	INFO	scalehandler	Scaler is active	{"Scaler": {}}
2020-11-13T13:10:03.409Z	INFO	scalehandler	Scaler maxValue	{"maxValue": 2}
2020-11-13T13:10:03.410Z	INFO	scaleexecutor	Scaling Jobs	{"scaledJob.Name": "JOB_NAME", "scaledJob.Namespace": "OUR_NAMESPACE", "Number of running Jobs": 32}
2020-11-13T13:10:03.422Z	INFO	scaleexecutor	Creating jobs	{"scaledJob.Name": "JOB_NAME", "scaledJob.Namespace": "OUR_NAMESPACE", "Effective number of max jobs": 0}
2020-11-13T13:10:03.423Z	INFO	scaleexecutor	Creating jobs	{"scaledJob.Name": "JOB_NAME", "scaledJob.Namespace": "OUR_NAMESPACE", "Number of jobs": 0}
2020-11-13T13:10:03.423Z	INFO	scaleexecutor	Created jobs	{"scaledJob.Name": "JOB_NAME", "scaledJob.Namespace": "OUR_NAMESPACE", "Number of jobs": 0}

Please correct me but what I see is that there are 32 running jobs and two elements waiting in the Redis-List. As our maxReplicaCount is set to 100 there should be some new jobs to be scaled.
What am I missing here ?

This is the definition of the ScaledJob:

  pollingInterval: 30 
  maxReplicaCount: 100 
  successfulJobsHistoryLimit: 10
  failedJobsHistoryLimit: 10
  #scalingStrategy: "accurate"
  jobTargetRef:
    parallelism: 1 
    completions: 1 
    activeDeadlineSeconds: 3600 
    backoffLimit: 1 
    template:
      ....
triggers:
    - type: redis
      metadata:
        addressFromEnv: REDIS_HOST_SCALER
        passwordFromEnv: REDIS_PASSWORD
        listName: REDIS_LIST_NAME
        listLength: "1" 

[zroubalik]

@TsuyoshiUshio PTAL^

[TsuyoshiUshio]

Hi @QuintinBecker

Could you share your log and find this log? The behavior is similar to "default" strategy. If the log shows following, it is surely accurate strategy is selected. If so, we need to investigate deeper, it also helpful to get the log.

"Selecting Scale Strategy", "specified", scaledJob.Spec.ScalingStrategy.Strategy, "selected", "accurate"

https://github.com/kedacore/keda/blob/main/pkg/scaling/executor/scale_jobs.go#L286-L308