How to cleanly exit Keda scaled applications

[gdippolito]

Thanks for building this product! I really like it.

I would like to better understand what is the right way to exit applications deployed with Keda in a clean way.

I'm using the service bus scaler object and my application is a simple python script that process the messages in the queue. If there are no message in the queue the application will exit almost instantly.

I think the JobScaler does take a bit of time before scaling down the deployment. During that time the application will keep restarting and K8s will flag it as Back-off restarting failed container (even though the application exit with return code 0).

What is the best way to avoid this behaviour (which trigger our monitoring system) ?

Do I need to add a sleep of 60 seconds to make sure that Keda has correctly scaled down the deployment?

[zroubalik]

You should check terminationGracePeriodSeconds setting, that should help you with the issue: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination

[gdippolito]

I think terminationGracePeriodSeconds works the other way around of what I need. The flag will set the time that the scheduler will wait to have the PreStop hook completed before it kills the container. In my case (and I think in many others) there is no custom PreStop hook defined and the container will exit immediately.

[gdippolito]

Hi @zroubalik do you have more suggestion on how I could sort out this problem? Have I misunderstood how this setting work?

[zroubalik]

KEDA doesn't have any information about your aplication and it's status, so unfortunately it can't do anything about it.

So if I understand your case correctly, that's something you need to handle from your app or deployment. Do you have any liveness/readiness probes set? You can try that or try to define the prestop hook. If that's overkill, the hardcoded delay could do the job, but it's really hard to suggest the generic fix for this.

So sorry for not helping much :/