Cron job maintenance

[ksuderman]

Adds cron jobs that clean up the jobs_directory, datasets, and histories. The jobs are defined in the values.yaml file and templates generated from the cronjob-maintenance.yaml template. The cron jobs are based on the Ansible Playbook used on main

Since the /galaxy/server/database/tmp directory is cleaned up I am not sure the job that cleans the tus_upload_store is needed since it also lives in /galaxy/server/database/tmp, but I've included it for completeness.

The pgcleanup.py script writes logs to the tmp directory, but we should likely create a separate logs directory for these and then also prune the logs as needed.

Should we just call the maintenance.sh script, or a modified version of the script?

NOTE I have set the cron jobs to runAsUser: 0 but we should probably run these as the Galaxy user once we have determined why mutliqc is creating files and directories as nobody.

This is untested as of yet.

Closes #408

[nuwang]

I love the idea of providing overridable cronjob templates with unlimited cron jobs. Given however, that the default cron jobs are all run daily, what if we just have a single bash script that is invoked by a single cronjob, which will in turn invoke the 3 cleanup commands sequentially? This can be overridden or replaced as desired.

[ksuderman]

what if we just have a single bash script that is invoked by a single cronjob, which will in turn invoke the 3 cleanup commands sequentially? This can be overridden or replaced as desired.

I was going to suggest the same thing, but I wanted to ensure the syntax for defining multiple cron jobs was working correctly first. There is already a maintenance.sh script we could use as-is, or modify for our needs.

[nuwang]

Makes sense to switch to the maintenance script if we can.

[ksuderman]

Please ensure #453 has been merged before merging this.

[nuwang]

Looks good to me! Can this be taken out of draft?

[nuwang]

Better to add an enabled flag as we have for everything else?

[ksuderman]

I considered that, but in other cases when enabled: false is set we don't render the template. However, in this case it means the CronJob is not defined and not available to be run manually. By setting the schedule to a time that never occurs the CronJob is defined and is available to be run manually if desired. At least that was my thought process.

[ksuderman]

Reading more about cron jobs, maybe I should be using .spec.suspend here. I'll play around with that and see if I get the desired behavior.

[nuwang]

Good idea to investigate suspend. Maybe we should add both. Some instructions on how to manually invoke the cronjob would also be useful.

[ksuderman]

Using .spec.suspend looks to be less than ideal, particularly if one wants to run the job once but keep it suspended. In that case the user must:

1. Patch the cron job to set .spec.suspend=false
2. Run the job if it does not get triggered automatically, and
3. Patch the cron job again to re-set .spec.suspend=true

Given users can always disable a cron job with the schedule I am not sure .spec.suspend is worth it.

[nuwang]

I think that adding an enabled flag would still be useful. Users who want to run the cronjob manually can still use the trick you suggested. Just that it feels weird to have to think of an invalid date for the simplest case of not wanting the cron job.

[nuwang]

@ksuderman Looks like we have a conflict, and I've made some more minor comments, but other than that, this looks like it's good to go?

[nuwang]

I think that adding an enabled flag would still be useful. Users who want to run the cronjob manually can still use the trick you suggested. Just that it feels weird to have to think of an invalid date for the simplest case of not wanting the cron job.