[Azure Function] Remove jobs and related pods when work is done

[andrekiba]

Report

I have an Azure Function that processes one message at a time.
The processing takes 5-10 minutes (maybe more) so we diced to use the ScaledJob approach.
This is the configuration of the ScaledJob:

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: assembler-job
spec:
  jobTargetRef:
    parallelism: 1                # [max number of desired pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
    completions: 1                # [desired number of successfully finished pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
    activeDeadlineSeconds: 3600   # Specifies the duration in seconds relative to the startTime that the job may be active before the system tries to terminate it; value must be positive integer
    backoffLimit: 1               # Specifies the number of retries before marking this job failed. Defaults to 6
    ttlSecondsAfterFinished: 100
    template:
      spec:
        containers:
          - env:
              - name: ASPNETCORE_ENVIRONMENT
                value: Development
              - name: AzureWebJobsServiceBus
                valueFrom:
                  secretKeyRef:
                    name: assembler-job
                    key: AzureWebJobsServiceBus
              - name: AzureWebJobsStorage
                valueFrom:
                  secretKeyRef:
                    name: assembler-job
                    key: AzureWebJobsStorage
            image: <azure function image here>
            name: assembler-job
            ports:
              - containerPort: 80
                protocol: TCP
            resources:
              requests:
                cpu: 100m
                memory: 128Mi
              limits:
                cpu: 200m
                memory: 256Mi
        restartPolicy: Never
  pollingInterval: 10    
  successfulJobsHistoryLimit: 0
  failedJobsHistoryLimit: 3
  maxReplicaCount: 10
  triggers:
    - type: azure-servicebus
      metadata:
        queueName: <queue name here>
        messageCount: '1'
      authenticationRef:
        name: trigger-auth-asb

The idea of messageCount=1 in the trigger is to create one job (and pod) for each new message.
The scale-up is ok...the problem is the scale-down...
Is it something related to the concept of job "completion"??

Expected Behavior

Remove the pod/s and the job when completed.

Actual Behavior

Pods related to jobs remains in Running without a reason.

Steps to Reproduce the Problem

1. Setup a ScaledJob triggered by an AzureServiceBus queue
2. Each job creates 1 pod (parallelism=1, completions=1) with an Azure Function
3. Each job processes only one message that takes 5-10 minutes
4. The message is processed correctly, in pod log everything is ok and the function terminates correctly
5. The pod remains in Running after the work is done

Logs from KEDA operator

example

KEDA Version

2.2.0

Kubernetes Version

1.18

Platform

Microsoft Azure

Scaler Details

Azure Service Bus

Anything else?

No response

[coderanger]

This sounds like you're looking for more of a functions-as-a-service tool that is hosted in k8s? Not sure Keda is the right tool here. In Keda, it doesn't know anything about task routing or other data-level configuration, just "I should have N jobs" (or N replicas) and it's up to your code to implement a worker model system that matches.

[andrekiba]

@coderanger sorry what are you saying? The same behavior works perfectly in case of ScaledObject.
So please try to explain eventually.
What I'm saying is that when the job is done Keda must remove the pods, no??

[coderanger]

Keda doesn't know anything about "when the job is done". It only sees the number of messages in your servicebus queue. And based on that and the queueLength parameter it will spawn some number of job pods. When the number of messages goes down, it will reduce the replica count and Kubernetes will eventually remove those pods. Or similarly if the pod command exits cleanly K8s will consider that job a success and not restart it so the pod will be destroyed.

[tomkerkhove]

Also note that your parallelism is 1 so you'll always have 1 instance. If your Azure Function never completes (which it doesn't do by default), it will just remain in Running status.

[ChayanBansal]

@andrekiba, Azure Function host runtime does not turn off even though there is no message in the queue. It just waits for the new message in the queue to process. So, while the azure function host may not be processing a message (or executing a function), it is still running (i.e., the host is still running) because of which the job is not considered to be complete. Hence, it is not removed. If you want to use SclaedJobs, you should rather write the code in a python file and execute the python file as a job.