Proposal : buildkit-tekton local executor project

[vdemeester]

TL;DR: adoption of buildkit-tekton in the tektoncd org.

The buildkit-tekton propose an alternative implementation of the Tekton API, dedicate to local execution. It is currently composed of two artifcats:

• a buildkit frontend
• a tkn-local command to easily consume this frontend from the comand-line

A bit more context around this can be visible in #145 and in CI/CD and the development workflow.

Since the inception of the Tekton project, we always focused our work around kubernetes, but with a higher end-goal which is, to probably in a long-term future, be able to execute Tekton Task and Pipeline elsewhere. We are trying our hardest in tektoncd/pipeline to leak as little as possible from k8s in our APIs.
This alternate implementation / executor is inline with that approach, as it provides an alternative execution flow, and should help us detect where we are leaking too much from Kubernetes.

Who will own it

• @vdemeester
• [any other takers]

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[jerop]

/lifecycle frozen

[jerop]

discussed this topic at the governance/community working group and agreed that we're interested in adopting buildkit-tekton, but we need to identify the work that needs to be done before adoption (e.g. alignment with design principles)

[aaron-prindle]

As of 1/22/2024 it might also make sense to look into leveraging dagger - eg: possibly creating a Tekton frontend over dagger.
dagger is a framework for CI/CD pipelines built to allow for executing in different enviroments (local) and to be extensible (extensible front ends, etc.). dagger leverages buildkit

[vdemeester]

@aaron-prindle can you go into more detail on what you mean by a "frontend" over dagger ? I am assuming it would be something that would take a set of Tekton Resources and translate it to the graphql that dagger uses so it can be used by it, am I right ? (If that's the case, it's very similar to what buildkit-tekton aims to do — but one level lower — and, yeah that's something that might be worth to explore, this was in my "personal backlog" to try that out someday 😬 )

[aaron-prindle]

@vdemeester exactly! The idea would be to translate a set of Tekton Resources -> dagger graph for local execution. This video shows something similar to the concept I was suggesting only for a Github Actions "frontend" for dagger:
https://www.youtube.com/watch?v=rHk3spfCHFQ

The gist being that instead of using buildkit directly and then writing the additional logic to map/pass artifacts, params, etc. across different Tasks we can likely leverage dagger which has these type of primitives for DAGs on top of buildkit already.

[vdemeester]

The gist being that instead of using buildkit directly and then writing the additional logic to map/pass artifacts, params, etc. across different Tasks we can likely leverage dagger which has these type of primitives for DAGs on top of buildkit already.

Having worked on both, it's a bit higher level yes, but in both case primitives are there. A very thin 1h "test" of this (from last august — according to the last modified date on my laptop) is there. I was essentially slightly simpler than with buildkit but not that much different.