You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@JohanLorenzo linked me to dagger.io recently. One of the things that it tries to solve is reproducability problems between local and CI executions of the same thing. It does by making it trivial to run exactly what CI does locally. We have a lot of issues with this at the moment (especially in Gecko), to the point where it is often avoided altogether, and I think taskgraph should be able to make this similarly easy for containerized tasks.
I'm envisioning something like a taskgraph run command that can be fed a label or some other unique task identifier. Taskgraph would then either build or download the image the task runs on, launch a container, and run the task.
The trickiest bit here is likely to be what to do with tasks that depend on things from other tasks. In many cases we'd probably want to fetch artifacts from existing tasks, but there may be cases where we'd want to rebuild those tasks locally. This is extra tricky in some Gecko scenarios where something like mozharness is built as part of the build task, but we probably don't want to do a full build to pick up changes to it. (Maybe we can ignore extra complicated scenarios like this to start with. The fix there could be not to build mozharness as part of builds...). Docker images are the most notable thing that nearly all containerized tasks depend on.
Other possible issues we may encounter:
run-task or other task runners currently only working on our deployed images
Tasks that use volume caches may need some adjustments to run locally
The text was updated successfully, but these errors were encountered:
For a bit of additional context, one of the things that makes it so difficult to reproduce some things locally is all of the upstream artifact downloading and unpacking that taskgraph/run-task does. These are done opaquely in Python code, with no way to simply copy/paste some commands to have the equivalent done locally.
@JohanLorenzo linked me to dagger.io recently. One of the things that it tries to solve is reproducability problems between local and CI executions of the same thing. It does by making it trivial to run exactly what CI does locally. We have a lot of issues with this at the moment (especially in Gecko), to the point where it is often avoided altogether, and I think taskgraph should be able to make this similarly easy for containerized tasks.
I'm envisioning something like a
taskgraph run
command that can be fed a label or some other unique task identifier. Taskgraph would then either build or download the image the task runs on, launch a container, and run the task.The trickiest bit here is likely to be what to do with tasks that depend on things from other tasks. In many cases we'd probably want to fetch artifacts from existing tasks, but there may be cases where we'd want to rebuild those tasks locally. This is extra tricky in some Gecko scenarios where something like
mozharness
is built as part of thebuild
task, but we probably don't want to do a full build to pick up changes to it. (Maybe we can ignore extra complicated scenarios like this to start with. The fix there could be not to build mozharness as part of builds...). Docker images are the most notable thing that nearly all containerized tasks depend on.Other possible issues we may encounter:
run-task
or other task runners currently only working on our deployed imagesThe text was updated successfully, but these errors were encountered: