rearrange buildomat jobs; rewrite releng process in rust and aggressively parallelize #5744

iliana · 2024-05-13T15:51:09Z

Note: documentation says cargo xtask releng but I am going to wire that up in a follow-up PR; the current equivalent is cargo run --release --bin omicron-releng.

Prior to this change we have five main "release engineering" Buildomat jobs that do operations beyond running the test suite:

a package job which runs omicron-package in various configurations,
a build OS images job which builds the host and trampoline images,
a TUF repo job which builds the final TUF repo (this is the build artifact we actually want),
a deploy job which uses the single-sled packages to test that a VM boots to SSH (this is a test we actually want),
and a CI tools job which builds common tools used by multiple jobs.

This looks like:

graph LR
    package --> host-image["build OS images"]
    package --> deploy
    package --> tuf-repo["TUF repo"]
    host-image --> tuf-repo
    ci-tools["CI tools"] --> deploy
    ci-tools --> tuf-repo

(There are also the currently-disabled a4x2 jobs but those are independent of this particular graph.)

I think the initial idea behind this was to reuse build artifacts where possible, but this is pretty complicated and adds a lot more output upload/download overhead than expected, which slows down the time to get the end artifact we actually want.

This PR changes the graph to:

graph LR
    package --> deploy
    tuf-repo["TUF repo"]

And the TUF repo job primarily runs a new releng binary, which runs all of the steps required to download and build all the components of the TUF repo in a single task, using a terrible job runner I wrote.

^{(thank you to @lifning for the image)}

The primary goal here was to reduce the time from pushing a commit to getting a TUF repo out the other end; this drops time-to-TUF-repo from ~80 minutes to ~45. In the process this also made it much easier to build a TUF repo (and iterate on that process) locally: just run cargo xtask releng (TODO: soon). It also deleted a lot of Bash.

One thing to note is that, in service of the mission to get time-to-TUF-repo down as much as possible, that job only uploads the TUF repo (and some logs). I also put all of the outputs for the package job into a single tarball for the deploy job to unpack. There are no longer separate uploads for the OS images and each zone; these can be extracted from the repo as we normally do.

labbott

This looks like it means it's really easy to build a TUF repo via xtask?

labbott · 2024-05-13T16:11:55Z

dev-tools/releng/src/hubris.rs

+    client
+        .get(format!("{}/artifact/{}", base_url, hash))
+        .send()
+        .and_then(|response| response.json())
+        .await
+        .with_context(|| {
+            format!(
+                "failed to fetch hubris artifact {} from {}",
+                hash, base_url
+            )
+        })


it's unfortunate that permslip is still a private repo but I guess it's mostly just a wrapper around reqwest anyway

In an earlier revision (not pushed) I ended up copying the openapi.json out of the repo and generating a client with Progenitor, and then I realized how much that was slowing down rust-analyzer just for a single API call. So I went with this relatively-small wrapper.

dev-tools/releng/src/hubris.rs

(we haven't used this since starting to use permslip for hubris artifacts, anyway)

iliana · 2024-05-13T16:21:24Z

This looks like it means it's really easy to build a TUF repo via xtask?

That, and running it locally re-uses all the Omicron artifacts too, so iterating on this process is a lot easier. (I think it takes about 8 minutes total if the control plane is previously built; that time is mostly building the OS images.)

david-crespo · 2024-05-13T16:57:56Z

Thanks for the great description, and looking forward to docs/releng.adoc. I never really knew what these jobs were doing.

sunshowers

First-pass review, generally looks good!

sunshowers · 2024-05-13T18:14:12Z

.github/buildomat/jobs/package.sh

@@ -3,24 +3,11 @@
 #: name = "helios / package"
 #: variety = "basic"
 #: target = "helios-2.0"
-#: rust_toolchain = "1.72.1"
+#: rust_toolchain = "1.77.2"


Is this supposed to stay in sync with rust-toolchain.toml? Can we read from that file directly instead? To be honest I didn't know this existed.

Yes:

omicron/rust-toolchain.toml

Lines 1 to 7 in d2ed452

[toolchain]

# NOTE: This toolchain is also specified in various jobs in

# .github/buildomat/jobs/. If you update it here, update those files too.

#

# We choose a specific toolchain (rather than "stable") for repeatability. The

# intent is to keep this up-to-date with recently-released stable Rust.

channel = "1.77.2"

If it's not in sync it doesn't actually break anything, but the job runs a bit slower (Buildomat downloads 1.72.1 and then rustup's Cargo proxy sees you actually wanted 1.77.2 and downloads that).

Buildomat feature request: oxidecomputer/buildomat#55

.github/buildomat/jobs/package.sh

Cargo.toml

sunshowers · 2024-05-13T18:18:12Z

dev-tools/releng/src/cmd.rs

+        for (name, value) in command.get_envs() {
+            if let Some(value) = value {
+                write!(
+                    command_str,
+                    "{}={} ",
+                    shell_words::quote(&name.to_string_lossy()),
+                    shell_words::quote(&value.to_string_lossy())
+                )
+                .unwrap();
+            }
+        }


Oh good spot.

dev-tools/releng/src/hubris.rs

dev-tools/releng/src/job.rs

docs/releng.adoc

david-crespo · 2024-05-13T20:58:44Z

docs/releng.adoc

+  also manage installation and uninstallation of these zones; see
+  how-to-run.adoc.)
+. Some of the packaged artifacts are installed directly in the OS
+  images; `cargo xtask releng` unpacks these into a temporary directory


It took me a bit of thinking to understand this — I assume it refers to stuff like putting the console assets directly in the file system of the control plane zone in the host image. But the line as-is is pretty cryptic; it could stand to be two or three sentences.

I rewrote this bullet, is it better?

docs/releng.adoc

sunshowers

Looked through again, just a few more comments but pretty happy otherwise!

dev-tools/releng/src/cmd.rs

dev-tools/releng/src/hubris.rs

dev-tools/releng/src/job.rs

iliana · 2024-05-14T17:30:46Z

I think I've addressed all outstanding review feedback; I still need to figure out a pattern for external xtasks (clap is currently making me unhappy), but I think that's the last TODO here.

sunshowers

love it, thanks for doing this work!

sunshowers · 2024-05-15T01:14:11Z

dev-tools/releng/src/cmd.rs

+impl std::fmt::Display for Command {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        let command = self.inner.as_std();
+        for (name, value) in command.get_envs() {


How noisy is this? I think this would print out the entire environment, which would be a lot: on my machine, export | wc -c returns 4923.

It only prints the environment variables explicitly set in the command builder. https://doc.rust-lang.org/stable/std/process/struct.Command.html#method.get_envs

sunshowers · 2024-05-15T01:18:30Z

dev-tools/releng/src/main.rs

+struct Args {
+    /// ZFS dataset to use for `helios-build` when building the host image
+    #[clap(long, default_value_t = Self::default_dataset("host"))]
+    host_dataset: String,


nit: is there some validation we can do on the dataset names? Would that be worth it at all?

I think verifying the datasets already exist (one of the preflight checks we run) is sufficient.

iliana · 2024-05-15T02:47:07Z

Despite the documentation all saying cargo xtask releng I think I am going to go ahead and merge this (after I test it on london) so we get the CI time improvements. For now folks can use cargo run --release --bin omicron-releng. I'm still toying with a few different options for external xtasks and I think that probably deserves its own separate review.

I forgot to remove these in #5744. These scripts were adapted into `cargo xtask releng` and aren't referenced anywhere in this repo anymore.

iliana added 14 commits May 12, 2024 21:50

wip

ee79d02

bug fixes and performance improvements

d584262

i did not know you could do this!

e948b3c

various logging tweaks

eab1717

hubris bits, stamping, job limits

3b00902

tuf repo!

5dcbb21

will it blend?

e6a0129

move logfiles to end of output_rules

12d1cf0

attempt to fix deploy job

0219f87

don't separately upload OS images (it's slow)

2ae7fc5

[shakes fist at tokio]

712aef1

try to get to host-image faster

aa7506d

correctness: put recovery artifacts in a new dir

9bb1c17

actually use the stamped artifacts

6dbca3c

iliana requested review from sunshowers and smklein May 13, 2024 15:51

labbott reviewed May 13, 2024

View reviewed changes

delete caboose-util

f9186f9

(we haven't used this since starting to use permslip for hubris artifacts, anyway)

sunshowers reviewed May 13, 2024

View reviewed changes

iliana added 8 commits May 13, 2024 20:02

we no longer use tools/hubris_{checksums,version}

35ec9e6

write releng.adoc

2e0916e

oops

84be01c

num_cpus -> std::thread::available_parallelism

c3a0537

spawn job work onto tasks

b720b63

fix incorrect sha256

8b3c76d

asciidoc :(

6b6fc8c

not sure why these fell out but sure

86003af

david-crespo reviewed May 13, 2024

View reviewed changes

docs/releng.adoc Outdated Show resolved Hide resolved

david-crespo reviewed May 13, 2024

View reviewed changes

docs/releng.adoc Outdated Show resolved Hide resolved

iliana added 5 commits May 13, 2024 23:20

explain the hubris structs

18ac798

refactor out the os_image_jobs! macro

fe6adf4

Merge remote-tracking branch 'origin/main' into iliana/releng

eddff63

job runner comments

e457190

use the new-ish actually line tables only setting

61400a3

sunshowers reviewed May 14, 2024

View reviewed changes

dev-tools/releng/src/cmd.rs Outdated Show resolved Hide resolved

dev-tools/releng/src/hubris.rs Show resolved Hide resolved

dev-tools/releng/src/job.rs Outdated Show resolved Hide resolved

iliana added 10 commits May 14, 2024 03:34

honor $GIT/$OMICRON_PACKAGE

a2168a2

cancel safety in spawn_with_output

f491f2f

handle alternate target directories

bf93834

Merge remote-tracking branch 'origin/main' into iliana/releng

3b6c54b

workspace dep cargo_metadata

0453fd1

replace the terrible std::mem::replace hack with more indirection

aef238c

explain $CARGO

b7664dd

respect $CARGO/$CARGO_HOME

9e341e8

Merge remote-tracking branch 'origin/main' into iliana/releng

1f7320d

doc tweaks

d5341bb

sunshowers approved these changes May 15, 2024

View reviewed changes

Merge remote-tracking branch 'origin/main' into iliana/releng

f3c1fa8

iliana merged commit 59636c9 into main May 15, 2024
20 checks passed

iliana deleted the iliana/releng branch May 15, 2024 04:55

iliana mentioned this pull request Jun 11, 2024

remove scripts from old releng CI process #5879

Merged

iliana added a commit that referenced this pull request Jun 21, 2024

remove scripts from old releng CI process (#5879)

e9c2432

I forgot to remove these in #5744. These scripts were adapted into `cargo xtask releng` and aren't referenced anywhere in this repo anymore.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rearrange buildomat jobs; rewrite releng process in rust and aggressively parallelize #5744

rearrange buildomat jobs; rewrite releng process in rust and aggressively parallelize #5744

iliana commented May 13, 2024 •

edited

Loading

labbott left a comment

labbott May 13, 2024

iliana May 13, 2024

iliana commented May 13, 2024

david-crespo commented May 13, 2024

sunshowers left a comment

sunshowers May 13, 2024

iliana May 13, 2024

sunshowers May 13, 2024

david-crespo May 13, 2024

iliana May 14, 2024

sunshowers left a comment

iliana commented May 14, 2024

sunshowers left a comment

sunshowers May 15, 2024

iliana May 15, 2024 •

edited

Loading

sunshowers May 15, 2024

iliana May 15, 2024

iliana commented May 15, 2024 •

edited

Loading

	[toolchain]
	# NOTE: This toolchain is also specified in various jobs in
	# .github/buildomat/jobs/. If you update it here, update those files too.
	#
	# We choose a specific toolchain (rather than "stable") for repeatability. The
	# intent is to keep this up-to-date with recently-released stable Rust.
	channel = "1.77.2"

rearrange buildomat jobs; rewrite releng process in rust and aggressively parallelize #5744

rearrange buildomat jobs; rewrite releng process in rust and aggressively parallelize #5744

Conversation

iliana commented May 13, 2024 • edited Loading

labbott left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iliana commented May 13, 2024

david-crespo commented May 13, 2024

sunshowers left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunshowers left a comment

Choose a reason for hiding this comment

iliana commented May 14, 2024

sunshowers left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iliana May 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iliana commented May 15, 2024 • edited Loading

iliana commented May 13, 2024 •

edited

Loading

iliana May 15, 2024 •

edited

Loading

iliana commented May 15, 2024 •

edited

Loading