KEDA agentpool pod failing with stopped hear from agent deployment-687dd7d466-vtmx4

[vijayakrishnar]

Report

we have ADO pipelines those are configured to use self-hosted agents and those running on AKS as a pod, we have used KEDA to auto scale based on no.of jobs from ADO and each job it triggers new pod or agent on AKS.

it works perfectly for small jobs where those jobs completed with in 10-15 or 30 mins, if job taking too long like more than 40 mins or 102 hrs, those jobs are failing with stopped hearing from agent without any reason.

we don't have time set limits to kill the pod, we have bigger AKS cluster with multiple nodes, still pods getting killed after 40 mins sometime 1-2 hours and so on.

Expected Behavior

it supposed to kill the agent or pod only when there is no activity on agent for more then 15 mins.

Actual Behavior

agent supposed run job fully without killing pod.

Steps to Reproduce the Problem

1. Setup ADO pipeline that will run job on AKS with KEDA auto scaling
2. KEDA will be using ADO PAT token to connect
3. KEDA pod should be killed if no activity for more than 15 mins

Logs from KEDA operator

##[error]We stopped hearing from agent deployment-687dd7d466-vtmx4. Verify the agent machine is running and has a healthy network connection. Anything that terminates an agent process, starves it for CPU, or blocks its network access can cause this error. For more information, see: https://go.microsoft.com/fwlink/?linkid=846610

KEDA Version

2.9.3

Kubernetes Version

1.25

Platform

Microsoft Azure

Scaler Details

Azure Pipelines

Anything else?

No response

[JorTurFer]

Hello,
KEDA doesn't kill jobs except on ScaledJob updates if the rollout.strategy is default:

KEDA doesn't manage the az agent either, KEDA just create the jobs using the given spec, so I think that the issue is in the agent side, not in KEDA.

Do you see any log in KEDA operator pod? The log that you sent isn't from KEDA components.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[JorTurFer]

I think that we can close this