Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
rearrange buildomat jobs; rewrite releng process in rust and aggressi…
…vely parallelize (#5744) (Note: documentation says `cargo xtask releng` but I am going to wire that up in a follow-up PR; the current equivalent is `cargo run --release --bin omicron-releng`.) Prior to this change we have five main "release engineering" Buildomat jobs that do operations beyond running the test suite: - a **package** job which runs omicron-package in various configurations, - a **build OS images** job which builds the host and trampoline images, - a **TUF repo** job which builds the final TUF repo *(this is the build artifact we actually want)*, - a **deploy** job which uses the single-sled packages to test that a VM boots to SSH *(this is a test we actually want)*, - and a **CI tools** job which builds common tools used by multiple jobs. This looks like: ```mermaid graph LR package --> host-image["build OS images"] package --> deploy package --> tuf-repo["TUF repo"] host-image --> tuf-repo ci-tools["CI tools"] --> deploy ci-tools --> tuf-repo ``` (There are also the currently-disabled a4x2 jobs but those are independent of this particular graph.) I think the initial idea behind this was to reuse build artifacts where possible, but this is pretty complicated and adds a lot more output upload/download overhead than expected, which slows down the time to get the end artifact we actually want. This PR changes the graph to: ```mermaid graph LR package --> deploy tuf-repo["TUF repo"] ``` And the **TUF repo** job primarily runs a new **releng** binary, which runs all of the steps required to download and build all the components of the TUF repo in a single task, using a terrible job runner I wrote. The primary goal here was to reduce the time from pushing a commit to getting a TUF repo out the other end; this drops time-to-TUF-repo from ~80 minutes to ~45. In the process this also made it much easier to build a TUF repo (and iterate on that process) locally: just run `cargo xtask releng` (TODO: soon). It also deleted a lot of Bash. One thing to note is that, in service of the mission to get time-to-TUF-repo down as much as possible, that job _only_ uploads the TUF repo (and some logs). I also put all of the outputs for the **package** job into a single tarball for the **deploy** job to unpack. There are no longer separate uploads for the OS images and each zone; these can be extracted from the repo as we normally do.
- Loading branch information