[Kafka Scaler] Scaler does not scale up to maximum replicas

[chubutin]

Report

I have created a ScaledObject that scales up correctly under certain configurations, but not consistently under all circumstances.

Expected Behavior

I want KEDA to scale my pods to the maximum possible number, or perhaps receive guidance on where my ScaledObject configuration might be incorrect.

Actual Behavior

When minReplicaCount is set to 1 and maxReplicaCount is set to 10, the service starts with 1 pod, and once messages are enqueued to the Kafka topic, the Horizontal Pod Autoscaler (HPA) scales to 3 pods. However, when minReplicaCount is set to 3 and maxReplicaCount is set to 10, the service still starts with 1 pod, but the HPA does not scale.

Scenario:

I have tested this process with a script that injects 1500 messages into the target_topic. I also have a consumer with the consumer group consumer_group that reads from this topic, and after 5 minutes, all messages get consumed. I have verified that the messages are enqueued using Kafka-UI.

Issue with Metrics Visibility

One peculiar issue I've encountered is the inability to check the metric that triggers the scaling. I don't have a way to access this value, which might be part of the problem.

Configurations on my Server

• EKS server version: 1.25
• KEDA Version: 2.12
• Kubecost installed

External metrics config

kubectl get apiservices v1beta1.external.metrics.k8s.io
NAME                              SERVICE                                      AVAILABLE   AGE
v1beta1.external.metrics.k8s.io   monitoring/keda-operator-metrics-apiserver   True        6h45m

Issue with metrics not visisible

When I attempt to retrieve the metrics that trigger scaling, I don't see any metrics. I should see the KEDA ScaledObject, but it's not there. However, the HPA works correctly for scaling based on these metrics.

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "external.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "externalmetrics",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

If I attempt to retrieve the value for the HPA metrics:

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/*/s0-kafka-ingress_listing_operations" | jq .    
Error from server: scaledObject name is not specified

In my HPA configuration, I can confirm that the metrics are being read and are triggering scaling.

Current objects

ScaledObject Configuration:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: base-application-staging-hpa
  namespace: staging
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: base-application-staging
  pollingInterval:  5
  cooldownPeriod:   30
  fallback:
    failureThreshold: 2
    replicas: 2
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
    - type: kafka
      metadata:
        tls: enable
        bootstrapServers: {{api_brokers}}
        consumerGroup: consumer_group
        topic: target_topic
        offsetResetPolicy: latest
        allowIdleConsumers: 'false'
        scaleToZeroOnInvalidOffset: 'false'
        excludePersistentLag: 'false'
        activationLagThreshold: '2'
        lagThreshold: '1'

HPA Configuration

kubectl get hpa keda-hpa-base-application-staging-hpa -n staging -o yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    meta.helm.sh/release-name: base-application-staging
    meta.helm.sh/release-namespace: staging
  creationTimestamp: "2023-10-02T18:12:05Z"
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: keda-hpa-base-application-staging-hpa
    app.kubernetes.io/part-of: base-application-staging-hpa
    app.kubernetes.io/version: 2.12.0
    scaledobject.keda.sh/name: base-application-staging-hpa
  name: keda-hpa-base-application-staging-hpa
  namespace: staging
  ownerReferences:
  - apiVersion: keda.sh/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ScaledObject
    name: base-application-staging-hpa
    uid: 992355b0-7230-4546-adaa-72125bb65269
  resourceVersion: "342201543"
  uid: 0f5cec75-af01-4133-be40-deccf9dddab6
spec:
  maxReplicas: 10
  metrics:
  - external:
      metric:
        name: s0-kafka-target_topic
        selector:
          matchLabels:
            scaledobject.keda.sh/name: base-application-staging-hpa
      target:
        averageValue: "1"
        type: AverageValue
    type: External
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: base-application-staging
status:
  conditions:
  - lastTransitionTime: "2023-10-02T18:12:20Z"
    message: recommended size matches current size
    reason: ReadyForNewScale
    status: "True"
    type: AbleToScale
  - lastTransitionTime: "2023-10-02T18:12:20Z"
    message: 'the HPA was able to successfully calculate a replica count from external
      metric s0-kafka-target_topic(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name:
     base-application-staging-hpa,},MatchExpressions:[]LabelSelectorRequirement{},})'
    reason: ValidMetricFound
    status: "True"
    type: ScalingActive
  - lastTransitionTime: "2023-10-03T00:10:32Z"
    message: the desired replica count is less than the minimum replica count
    reason: TooFewReplicas
    status: "True"
    type: ScalingLimited
  currentMetrics:
  - external:
      current:
        averageValue: "0"
        value: "0"
      metric:
        name: s0-kafka_listing_operations
        selector:
          matchLabels:
            scaledobject.keda.sh/name:base-application-staging-hpa
    type: External
  currentReplicas: 1
  desiredReplicas: 1
  lastScaleTime: "2023-10-03T00:10:32Z"

Keda Operator Logs

I have reviewed the KEDA Operator logs, and there doesn't seem to be any apparent issues.

2023-10-03T00:25:04Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"base-application-staging-hpa","namespace":"staging"}, "namespace": "staging", "name": "base-application-staging-hpa", "reconcileID": "4dc3489f-0e40-41c1-82b9-d3044204b196"}
2023-10-03T00:25:19Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"base-application-staging-hpa","namespace":"staging"}, "namespace": "staging", "name": "base-application-staging-hpa", "reconcileID": "ee1d402e-6139-45ae-b3cf-4c5795bab81f"}

Any insights or assistance in resolving this issue would be greatly appreciated.