chore(jobs): add ttl of 30 days to job templates

[deer-wmde]

There are currently >300 job objects floating around in our prod cluster. Are there any downsides of setting a ttl to them? I went for 30 days here

https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/

As an alternative we could add a script or something that edits this into the spec of existing jobs, so we could also opt for "cleaning" whenever we feel like its too much, instead of specifying it in beforehand (this would also come in handy to get rid of the current pile of old completed jobs)

[m90]

I could also live with a TTL of 7 days I guess.

this would also come in handy to get rid of the current pile of old completed jobs

I doubt we miss out on anything if we delete all of them as of now. Can't think of anything we'd need to retroactively debug right now.

[deer-wmde]

I could also live with a TTL of 7 days I guess.

Me too.

I doubt we miss out on anything if we delete all of them as of now. Can't think of anything we'd need to retroactively debug right now.

Yup, I think so too. I would run something like

for JOB in $(kubectl get --no-headers jobs.batch | awk '{ print $1 }'); do
  kubectl patch jobs.batch "$JOB" -p "$(kubectl get jobs.batch "$JOB" -o yaml | yq '.spec.ttlSecondsAfterFinished = 604800')";
done

[m90]

Couldn't we just really delete them ourselves, e.g.

kubectl delete jobs --field-selector status.successful=1 

[deer-wmde]

Couldn't we just really delete them ourselves, e.g.

kubectl delete jobs --field-selector status.successful=1 

lol yeah that is A LOT easier 👍