feat: Add periodic job support

[dippynark]

This PR adds the scaffolding for a controller that implements the periodic jobs defined in the Lighthouse config. This is done by watching the Lighthouse ConfigMap to determine the list of periodic jobs to reconcile as described more in the README.

I have just done basic scaffolding within this PR to get an idea of what an implementation could look and to gather feedback for whether this is the right way to go. The actual reconciliation logic currently does nothing but could be implemented by either templating and managing CronJobs which periodically launch the actual LighthouseJobs or by creating a similar implementation to the CronJob controller (this is my preference): https://github.com/kubernetes-sigs/kubebuilder/blob/0b62a3ed0b7e3af53f013db4d9a31d96c7fdb7dd/docs/book/src/cronjob-tutorial/testdata/project/controllers/cronjob_controller.go

I stole the strobe name from this previous implementation. An advantage of this implementation however is that watching for changes to the Lighthouse ConfigMap is event driven.

UPDATE:

A full implementation has now been added to this PR including Docker image build and Helm chart changes

A future improvement could be to use a LighthouseJob informer to reduce the number of list requests against the API server.

[jenkins-x-bot]

Hi @dippynark. Thanks for your PR.

I'm waiting for a jenkins-x or todo member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository.

[jenkins-x-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign ankitm123
You can assign the PR to them by writing /assign @ankitm123 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ankitm123]

It would also be good to look at how prow does this, after all lighthouse is a fork of prow.
https://docs.prow.k8s.io/docs/legacy-snapshot/prow/jobs/ or https://github.com/kubernetes/test-infra/blob/master/prow/jobs.md

[dippynark]

@ankitm123 Hologram seems to work by checking every minute whether any periodic ProwJobs need running and then scheduling them accordingly:

• https://github.com/kubernetes/test-infra/blob/2dcd0ab8acd8018b03f93278d4de67e26b36fce6/prow/cmd/horologium/main.go#L134-L140
• https://github.com/kubernetes/test-infra/blob/2dcd0ab8acd8018b03f93278d4de67e26b36fce6/prow/config/config.go#L921-L926

This implementation works slightly differently by handling each periodic job individually. For example, it can individually retry a particular job with backoff if there is an issue and it will scale better as the number of the jobs increases (although for most Lighthouse users I would not expect these performance improvements to be noticeable).

I have now added a full implementation to this PR with few more details of the implementation documented.

[ankitm123]

I am not sure if it should be defined in a configmap. Normally in jx, configmap would be defined/configured in the cluster git repo, which should be normally off limits for repository owners. It should be in the same repo, so if a repository author wants to define a periodics, they should be able to define a new periodics job in the triggers yaml, wdyt?
May be a block like pre-submit/post-submit?
Also the lifecycle of lighthouse configmap and a build job are very different.

[dippynark]

@ankitm123 By triggers yaml do you mean the TriggerConfig resource? https://github.com/dippynark/kfmt/blob/main/.lighthouse/triggers.yaml

How would the controller get access to this TriggerConfig in all registered repositories? Following through the webhook code, this is done for presubmit and postsubmit jobs when a SCM event is received for a specific repository:

• lighthouse/pkg/webhook/events.go

  Line 132 in 6a37f27

  agent, err := s.CreateAgent(l, ce.Repo.Namespace, ce.Repo.Name, ce.HeadSha)

• lighthouse/pkg/webhook/create_agent.go

  Line 76 in 6a37f27

  pc.Config, pc.PluginConfig, err = inrepo.Generate(s.FileBrowsers, fc, cache, pc.Config, pc.PluginConfig, owner, repo, ref)

• lighthouse/pkg/triggerconfig/inrepo/calculate.go

  Lines 12 to 13 in 6a37f27

  	// Generate generates the in repository config if enabled for this repository otherwise return the shared config
  	func Generate(fileBrowsers *filebrowser.FileBrowsers, fc filebrowser.FetchCache, cache *ResolverCache, sharedConfig *config.Config, sharedPlugins *plugins.Configuration, owner, repo, eventRef string) (*config.Config, *plugins.Configuration, error) {

But of course periodic jobs don't have any associated SCM events since they are triggered on a timer. One option could be to add a new webhook plugin which persists periodic job configuration by syncing TriggerConfig periodic jobs to a ConfigMap which the controller then picks up? But I guess that work could be done in a subsequent PR

[dippynark]

Also sorry to reference other repositories, but would it be possible to get a review for this PR please? jenkins-x/go-scm#321
We depend on the poller component being able to see if a particular job has ever succeeded (rather than just the latest run) which is the reason for the change. We are using a custom fork using that PR at the moment

[dippynark]

Thanks for the approval! I have created another PR to bump the go-scm version used by Lighthouse: #1493

[tomhobson]

I suppose the periodics would be registered similar to how pipelines are cached (but periodics will store instead). It being registered within the cluster repo goes away from the jenkins-x ideaollogy of being repo and developer centric as it would make you have to jump through 2 repos to add a feature to your repo

[dippynark]

Hey @ankitm123, do you think any other changes are needed for this PR? I couldn't see how to get the TriggerConfig suggestion to work: #1474 (comment)

Maybe it'd be a larger piece of work to get it working which I could do in a subsequent PR?

[tomhobson]

@dippynark would you like to continue with this? I'd be happy to work with you to get this merged. It would be really cool to have!

[dippynark]

@tomhobson Yes this would be great! Sorry for the slow reply, haven't done much OSS stuff recently but would love to work more on this feature

Reading your comments it sounds like the next step would be to store the configuration for periodics whenever a webhook event is received (possibly this would require creating a ConfigMap per repository containing the periodic configuration for that repository, but perhaps a custom resource would be better for this)

What do you think about first getting this PR merged where you can only configure periodics through a ConfigMap like in the README: https://github.com/jenkins-x/lighthouse/blob/e4adcf7d2c513350dc3fc2ca6fc3efc3b192a184/cmd/strobe/README.md

And then subsequent PR(s) to add the support in the webhook for storing periodic configuration in the local cluster

[tomhobson]

@dippynark I think it would make the most sense in the triggerconfig within lighthouse.

That way lighthouse can have a registry of all of the periodic jobs defined through a repository in the proper jenkins x way. I really don't like using a configmap for this as it feels quite anti-jenkins x in the way that it works.

[msvticket]

This has been superseded by #1563

/close