Add on-demand communication probes.

[rcgoodfellow]

This PR adds the following.

Probes

A new first-class Omicron element called a probe is introduced. A probe is similar to an instance but is underpinned by a zone instead of an HVM. They are managed much like instances in terms of lifecycle. They have network interfaces on a VPC that are externally reachable via ephemeral IP addresses. They also have network interfaces on the underlay network. The primary function of probes in this initial PR is for network testing. They come with daemons like thundermuffin pre-installed as SMF services to facilitate communications testing both through boundary services and within the rack. Probes may become more general over time to facilitate more kinds of testing beyond networking.

The idea for probes came from the desire to test the Oxide stack in an environment where the hardware is constructed by interconnected virtual machines. Nested virtualization is undesirable for a number of reasons, so probes plus virtual rack topologies give us the ability to test a significant surface area of the control plane and the underlying systems it manages, such as networking and storage. While the motivation for probes was to test in virtual environments, they're just as applicable to hardware environments. Because they're a first-class Omicron element, the same tests written using probes in virtual environments can run on real racks.

Probes currently require fleet admin privileges. This is enforced in the API handlers.

A 4-gimlet 2-sidecar CI test

A primary goal of building the probes mechanism is to have automated multi-switch multi-sled tests that run in CI. This PR also takes the first step along that path. There are two new CI jobs added. The a4x2-prepare job builds and packages omicron for each of the 4 gimlets in the topology. This includes RSS configuration, individual sled configuration, and faux-mgs configuration for each scrimlet. The omicron packages and their configuration bits are then tarred up and provided as artifacts for the a4x2-deploy job.

The a4x2-deploy job extracts the artifacts from a4x2-prepare into a set of folders with omicron configuration and deployment archives for each gimlet in the topology. This is in a folder called cargo-bay. The cargo bay contains a top-level folder for each virtual gimlet. The a4x2 falcon topology knows to look for these folders and mounts each into the corresponding gimlet via P9fs mounts. In the cargo bay for each gimlet, there is also an initialization script that installs omicron in the VM using omicron-install. This initialization script is run automatically by falcon (this is in the code for the topology itself, not something that is hard coded). Once the topology has been launched the control plane is on it's way toward coming up.

The way that we test the virtual rack is functional is through probes. There is a new testing program in this PR called commtest in the end-to-end-tests directory. This program takes an address to reach the Oxide API at as a parameter and waits for the /ping endpoint to become responsive. Once that happens, a probe is launched on each sled in the topology and a basic connectivity test is run against each probe using ICMP. The test checks for packet loss within a configurable threshold for each probe. If the test passes, then the CI job passes.

In addition to the virtual rack, the test topology also contains a pair of routers running the Arista network stack that the sidecars are connected to. The connection between the rack and these routers is facilitated by BGP. There is also a third router called the customer edge (ce) that connects to both Arista routers. This router is running Linux+FRR. It advertises a default route to both Arista routers that propagates via BGP to the Oxide rack routers. Similarly, the IP pool address block the Oxide rack advertises propagates to the customer edge router via the Arista routers through BGP. This customer edge VM is also running an iptables configuration to decouple the testing network from the host network on the lab machine the entire topology is running on. All that needs to be done on the lab machine to connect to the rack is create a route to the IP pool block the rack is using that uses the IP address of the external interface on the customer edge machine as a nexthop. This is done in the a4x2-deploy job.

Depends on

• image: bump gimlet ramdisk zfs size helios#143

[rcgoodfellow]

Threre are still a few TODOs in here to take care of, but I think this is generally ready for review.

[ahl]

Probes could be useful as a tool to narrow down problems for operators. If an end user reports a connectivity problem with a particular instance, an operator could launch a probe on the same sled, drawing an external address from the same IP pool, to help determine if the connectivity issue is at the instance level, the sled level or they could launch several probes to see if there is a broader connectivity issue.

That being said, the primary use for probes right now is to ensure working end-to-end networking for PRs in CI.

In that case, my suggestion would be to keep it out of the documented API / CLI / etc. What do you think?

[rcgoodfellow]

In that case, my suggestion would be to keep it out of the documented API / CLI / etc. What do you think?

I've moved the probes API endpoints to an "experimental" API in 4f58a4a

[sunshowers]

(Just a comment on experimental API endpoints.)

Thank you for taking this on! I've been wanting something like this for a while, and I really like this solution (my own half-baked solution was to use sets of tags, but this is easier to read I think.)

[FelixMcFelix]

Thanks Ry, from what I've looked over this is a very nice addition. I haven't yet had the chance to stand this up and test it locally, I'm hoping to do so shortly. I've left some questions throughout.

On 'what' we can test with this, I guess this gives us really good mileage for any traffic carried by a VPC (Instance<->Probe and Probe<->Rack-external). If we wanted to ask more targeted questions about the underlay itself, would we create probes and zlogin ?

[FelixMcFelix]

Thanks for handling the nits/papercuts etc., looks good!

[rcgoodfellow]

Unfortunately, I need to disable the a4x2 CI jobs in this initial commit. The zones and services within VMs come up too unreliably and would be a major source of friction for day-to-day development. We'll continue to push to make things more reliable within VMs.

The primary issue appears to be SMF service startup failures within zones. Sometimes it's CRDB, sometimes it's more basic things like ndp. These service start failures seem to correspond to periods of heavy I/O. I did try running zpool trim as a precursor to a4x2 CI runs, and that sped things up significantly - but unfortunately, executing several runs with the benefit of trim shows reliability issues remain. I see these issues much less often on my local workstation, so it does appear to be system performance related in some way.

The CI tests and machinery remain in place within this PR, but the buildomat enable property is set to false.