ScaledJob - Scaling up and scaling down behavior

[thsbt]

Hi there,

I'm currently working on a PoC in order to test KEDA. Unfortunately I'm facing behaviors that I cannot understand 😢, that's why I'm asking for your help !

Use case:

In a Kubernetes cluster I have a Kafka broker and KEDA 2.1.0 deployed. What I want to do is send a messages on a specific Kafka topic and, thanks to KEDA, spawn a Job in order to read it (my code only reads 1 message, prints it and terminates). Actually, I achieved to do this 🍾

Incomprehensible behavior:

What I cannot understand is when I send several messages to my Kafka topic. In this case, a Job spawns, reads a message, finishes, then another Job spawns, reads a message, etc. until all messages are read. What I was expecting is that Jobs will spawn at the same time, not one after another. If I send 10 messages I was hoping for 10 Jobs to spawn at the same time.

Am I missing something ? Is the behavior I'm facing normal ? Do I correctly configure my ScaledJob ?

Here is the ScaledJob I deploy:

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: scaled-kafka-reader
spec:
  jobTargetRef:
    parallelism: 1
    completions: 1
    activeDeadlineSeconds: 86400
    backoffLimit: 6
    template:
      spec:
        containers:
          - name: kafka-reader
            image: <my-docker-image>
            imagePullPolicy: Always
        restartPolicy: Never
  pollingInterval: 10
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 2
  maxReplicaCount: 10
  scalingStrategy:
    strategy: "default"
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: <my-bootstrapserver-@>
        consumerGroup: my-group
        topic: <my-topic>
        lagThreshold: '1'
        offsetResetPolicy: earliest

Many thanks for your help !! 😄

[zroubalik]

I can recommend you check the job configuration options for parallelism:
https://kubernetes.io/docs/concepts/workloads/controllers/job/#parallel-jobs

[thsbt]

Hi @zroubalik,

Thanks for your help ! I already read this some time ago. What I understand from this documentation is, when configuring correctly .spec.parallelism and .spec.completions, you can have one Job creating several Pods. Here I just want a Pod per Job (I want to do simple stuff).

What I was expecting from KEDA was this behavior: If 10 messages are in my Kafka topic, KEDA will create 10 Jobs at the same time (each one creating a pod). So I will have 10 Jobs and 10 Pods reading my Kafka topic at the same time. But what I'm facing is sequential Jobs creations. The first Job is created, it launches a Pod, finishes, the second Job is created, it launches a Pod, etc.

Maybe I misunderstood how KEDA works with ScaledJob 😢