scaled up pods hangs on terminating state

[converge]

Hi,

I have a basic Minikube local k8s running:

Knative 1.11 (Knative serving, Knative eventing + default event-display sample app)
Kafka (strimzi and a topic with 50 partitions)
Keda 2.11

Keda ScaledObject is:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kafka-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    name: event-display-00001-deployment
  pollingInterval: 30
  minReplicaCount: 0
  maxReplicaCount: 10
  cooldownPeriod: 30
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: my-cluster-kafka-bootstrap.kafka:9092
        consumerGroup: knative-group
        topic: knative-demo-topic

The issue is that the Pods are starting and get killed by kubelet:

event-display-00001-deployment-9d84cf88f-22wtr   2/2     Terminating   0          12s
event-display-00001-deployment-9d84cf88f-6g6vr   1/2     Terminating   0          31s
event-display-00001-deployment-9d84cf88f-9pphc   1/2     Terminating   0          31s
event-display-00001-deployment-9d84cf88f-cw4vf   2/2     Terminating   0          12h
event-display-00001-deployment-9d84cf88f-nkmqd   1/2     Terminating   0          31s

describe one of these pods:

k describe pod event-display-00001-deployment-9d84cf88f-9tx6p
Name:                      event-display-00001-deployment-9d84cf88f-9tx6p
Namespace:                 default
Priority:                  0
Service Account:           default
Node:                      minikube/192.168.49.2
Start Time:                Tue, 26 Sep 2023 12:15:01 -0300
Labels:                    app=event-display-00001
                           pod-template-hash=9d84cf88f
                           serving.knative.dev/configuration=event-display
                           serving.knative.dev/configurationGeneration=1
                           serving.knative.dev/configurationUID=0a94134c-211b-4c71-b572-f97dc19fbbe9
                           serving.knative.dev/revision=event-display-00001
                           serving.knative.dev/revisionUID=d39ac7ab-7a35-4d17-a82a-75e9668ea543
                           serving.knative.dev/service=event-display
                           serving.knative.dev/serviceUID=4c50dd4b-7dc5-4ff2-a46c-ebca7bb1e87b
Annotations:               serving.knative.dev/creator: minikube-user
Status:                    Terminating (lasts <invalid>)
Termination Grace Period:  300s
IP:                        10.244.9.211
IPs:
  IP:           10.244.9.211
Controlled By:  ReplicaSet/event-display-00001-deployment-9d84cf88f
Containers:
  user-container:
    Container ID:   docker://8ac8877dd284a76e73e4be5525e40a715c36b043498919c5ea3ff3080047379f
    Image:          gcr.io/knative-releases/knative.dev/eventing/cmd/event_display@sha256:8aa178f98a02b9caa229f7937afec5760e989cae7cc5d680c181acaca7fb8cdd
    Image ID:       docker-pullable://gcr.io/knative-releases/knative.dev/eventing/cmd/event_display@sha256:8aa178f98a02b9caa229f7937afec5760e989cae7cc5d680c181acaca7fb8cdd
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 26 Sep 2023 12:15:01 -0300
    Ready:          True
    Restart Count:  0
    Environment:
      PORT:             8080
      K_REVISION:       event-display-00001
      K_CONFIGURATION:  event-display
      K_SERVICE:        event-display
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5fpvg (ro)
  queue-proxy:
    Container ID:    docker://051f23578c361ebd15351be7a247b8ac6c51140800191fb2c13f58d2cc203959
    Image:           gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:2b9e0783e9f7143b27d1d21974853a0af36f2029cf1dee085dcf46d1f8c5b677
    Image ID:        docker-pullable://gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:2b9e0783e9f7143b27d1d21974853a0af36f2029cf1dee085dcf46d1f8c5b677
    Ports:           8022/TCP, 9090/TCP, 9091/TCP, 8012/TCP, 8112/TCP
    Host Ports:      0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    State:           Running
      Started:       Tue, 26 Sep 2023 12:15:01 -0300
    Ready:           True
    Restart Count:   0
    Requests:
      cpu:      25m
    Readiness:  http-get http://:8012/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      SERVING_NAMESPACE:                                 default
      SERVING_SERVICE:                                   event-display
      SERVING_CONFIGURATION:                             event-display
      SERVING_REVISION:                                  event-display-00001
      QUEUE_SERVING_PORT:                                8012
      QUEUE_SERVING_TLS_PORT:                            8112
      CONTAINER_CONCURRENCY:                             0
      REVISION_TIMEOUT_SECONDS:                          300
      REVISION_RESPONSE_START_TIMEOUT_SECONDS:           0
      REVISION_IDLE_TIMEOUT_SECONDS:                     0
      SERVING_POD:                                       event-display-00001-deployment-9d84cf88f-9tx6p (v1:metadata.name)
      SERVING_POD_IP:                                     (v1:status.podIP)
      SERVING_LOGGING_CONFIG:
      SERVING_LOGGING_LEVEL:
      SERVING_REQUEST_LOG_TEMPLATE:                      {"httpRequest": {"requestMethod": "{{.Request.Method}}", "requestUrl": "{{js .Request.RequestURI}}", "requestSize": "{{.Request.ContentLength}}", "status": {{.Response.Code}}, "responseSize": "{{.Response.Size}}", "userAgent": "{{js .Request.UserAgent}}", "remoteIp": "{{js .Request.RemoteAddr}}", "serverIp": "{{.Revision.PodIP}}", "referer": "{{js .Request.Referer}}", "latency": "{{.Response.Latency}}s", "protocol": "{{.Request.Proto}}"}, "traceId": "{{index .Request.Header "X-B3-Traceid"}}"}
      SERVING_ENABLE_REQUEST_LOG:                        false
      SERVING_REQUEST_METRICS_BACKEND:                   prometheus
      SERVING_REQUEST_METRICS_REPORTING_PERIOD_SECONDS:  5
      TRACING_CONFIG_BACKEND:                            none
      TRACING_CONFIG_ZIPKIN_ENDPOINT:
      TRACING_CONFIG_DEBUG:                              false
      TRACING_CONFIG_SAMPLE_RATE:                        0.1
      USER_PORT:                                         8080
      SYSTEM_NAMESPACE:                                  knative-serving
      METRICS_DOMAIN:                                    knative.dev/internal/serving
      SERVING_READINESS_PROBE:                           {"tcpSocket":{"port":8080,"host":"127.0.0.1"},"successThreshold":1}
      ENABLE_PROFILING:                                  false
      SERVING_ENABLE_PROBE_REQUEST_LOG:                  false
      METRICS_COLLECTOR_ADDRESS:
      HOST_IP:                                            (v1:status.hostIP)
      ENABLE_HTTP2_AUTO_DETECTION:                       false
      ROOT_CA:
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5fpvg (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-5fpvg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  9s    default-scheduler  Successfully assigned default/event-display-00001-deployment-9d84cf88f-9tx6p to minikube
  Normal  Pulled     9s    kubelet            Container image "gcr.io/knative-releases/knative.dev/eventing/cmd/event_display@sha256:8aa178f98a02b9caa229f7937afec5760e989cae7cc5d680c181acaca7fb8cdd" already present on machine
  Normal  Created    9s    kubelet            Created container user-container
  Normal  Started    9s    kubelet            Started container user-container
  Normal  Pulled     9s    kubelet            Container image "gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:2b9e0783e9f7143b27d1d21974853a0af36f2029cf1dee085dcf46d1f8c5b677" already present on machine
  Normal  Created    9s    kubelet            Created container queue-proxy
  Normal  Started    9s    kubelet            Started container queue-proxy
  Normal  Killing    8s    kubelet            Stopping container user-container
  Normal  Killing    8s    kubelet            Stopping container queue-proxy

The HorizontalPodAutoscaller object contains:
sometimes TARGET is <unknown>/10 (avg), sometimes 50/10 (avg)

default            keda-hpa-kafka-scaledobject   Deployment/event-display-00001-deployment   <unknown>/10 (avg)   1         10        0          12h

describe this HPA:

k describe hpa keda-hpa-kafka-scaledobject
Name:                                                    keda-hpa-kafka-scaledobject
Namespace:                                               default
Labels:                                                  app.kubernetes.io/managed-by=keda-operator
                                                         app.kubernetes.io/name=keda-hpa-kafka-scaledobject
                                                         app.kubernetes.io/part-of=kafka-scaledobject
                                                         app.kubernetes.io/version=2.11.2
                                                         scaledobject.keda.sh/name=kafka-scaledobject
Annotations:                                             <none>
CreationTimestamp:                                       Mon, 25 Sep 2023 23:36:18 -0300
Reference:                                               Deployment/event-display-00001-deployment
Metrics:                                                 ( current / target )
  "s0-kafka-knative-demo-topic" (target average value):  <unknown> / 10
Min replicas:                                            1
Max replicas:                                            10
Deployment pods:                                         0 current / 0 desired
Conditions:
  Type            Status  Reason             Message
  ----            ------  ------             -------
  AbleToScale     True    SucceededGetScale  the HPA controller was able to get the target's current scale
  ScalingActive   False   ScalingDisabled    scaling is disabled since the replica count of the target is zero
  ScalingLimited  True    ScaleUpLimit       the desired replica count is increasing faster than the maximum scale rate
Events:
  Type    Reason             Age                  From                       Message
  ----    ------             ----                 ----                       -------
  Normal  SuccessfulRescale  28m (x555 over 12h)  horizontal-pod-autoscaler  New size: 4; reason: external metric s0-kafka-knative-demo-topic(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: kafka-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}) above target

notes:

on minikube I installed this addon to enable metrics minikube addons enable metrics-server

kubectl top nodes and kubectl top nodes returns valid metrics.

if I send messages to the topic, the event-display app start normally

Is there something I could look at to identify the issue? I have tried it with a fresh Kind installation as well, and got the same issue. Also tried on Windows and Mac.