Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[sled-agent] Self assembling switch zone #5593

Merged
merged 60 commits into from
Jul 22, 2024
Merged
Show file tree
Hide file tree
Changes from 56 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
d0a1123
Update the zone setup CLI to take several static addresses during com…
karencfv Apr 22, 2024
444e9fb
Set MGS service
karencfv Apr 22, 2024
b82297b
Set Wicket service (almost)
karencfv Apr 22, 2024
9378f6f
Merge branch 'main' into switch-zone-self-assembling
karencfv Apr 22, 2024
b83a46c
switch zone setup
karencfv Apr 23, 2024
5062a38
Set up wicket and support users via zone-setup CLI
karencfv Apr 23, 2024
68029f1
switch setup commands
karencfv Apr 24, 2024
be0f793
modify zone setup start method
karencfv Apr 29, 2024
223fd48
Set dendrite service
karencfv Apr 29, 2024
c233daa
Set tfport service
karencfv Apr 29, 2024
11dd38d
Set lldpd service
karencfv Apr 29, 2024
80c3724
Set pumpkind service
karencfv Apr 29, 2024
ead54fb
Set up Mgd
karencfv Apr 30, 2024
04b5037
Set up mg ddm service
karencfv Apr 30, 2024
4873d47
set up link local links during switch zone set up
karencfv May 1, 2024
a5d4dc1
Set up switch zone networking configuration
karencfv May 1, 2024
ac5bbbb
Clean up
karencfv May 2, 2024
c792fc7
Switch zone set up should not depend on common networking set up
karencfv May 3, 2024
89ac45a
Fix manifest and some fixes for the switch zone set up command
karencfv May 7, 2024
95ca659
Get bootstrap address working
karencfv May 9, 2024
73f0d04
enable all services from the start
karencfv May 10, 2024
5c6d64d
It works! 😭
karencfv May 10, 2024
29b1da4
all services successful on a4x2 testbed
karencfv May 15, 2024
be38d08
restart switch services' instances after updating properties
karencfv May 16, 2024
26137dc
refresh is enough
karencfv May 16, 2024
0f9f1d3
Merge branch 'main' into switch-zone-self-assembling
karencfv Jun 6, 2024
64b8e09
small fixes after merge
karencfv Jun 6, 2024
181bebe
Add sp-sim to PropertyGroupBuilder
karencfv Jun 6, 2024
91204f9
Update dendrite hashes
karencfv Jun 10, 2024
f509c43
First round of clean up
karencfv Jun 10, 2024
0d1ef88
fmt
karencfv Jun 10, 2024
845b07a
Clean up bootstrap related code
karencfv Jun 11, 2024
0fcd764
Clean up switch zone setup
karencfv Jun 11, 2024
ab75734
extract switch zone user
karencfv Jun 11, 2024
dafe3da
fmt and add files
karencfv Jun 11, 2024
2ea71b8
Add logs from PR #5853
karencfv Jun 11, 2024
06e1614
Tidy up
karencfv Jun 12, 2024
6221c39
remove switch zone setup bash script entirely
karencfv Jun 12, 2024
19bd61c
Verify ensure default route loop
karencfv Jun 12, 2024
1bd8e85
Remove more commented out code
karencfv Jun 12, 2024
d2f7ae0
Remove wicket service's dependency on common networking service
karencfv Jun 12, 2024
30eb1ee
Make sure to set all properties on instance FMRI
karencfv Jun 13, 2024
7753318
fix typo
karencfv Jun 16, 2024
86bef4d
Remove unecessary clone
karencfv Jun 17, 2024
ebb6fce
Clean up
karencfv Jun 17, 2024
70be78e
Address comment
karencfv Jun 17, 2024
45678c3
Adding SP sim to property builder isn't necessary and is cleaner on a…
karencfv Jun 18, 2024
17c4931
Modify start for when underlay is not available yet
karencfv Jun 20, 2024
2f14648
Add default route during second run of setting property values
karencfv Jun 25, 2024
9499512
Add some logging
karencfv Jun 26, 2024
5a1d23e
Set bootstrap address and link local in switch zone setup service
karencfv Jun 26, 2024
e3d40fa
Clean up
karencfv Jun 26, 2024
ab6ddf3
Include forwarding bootstrap traffic to switch zone start up service
karencfv Jun 27, 2024
207162b
Clean up
karencfv Jun 27, 2024
36c95ca
Clean up --gateway flag
karencfv Jun 27, 2024
fae95de
noop commit
karencfv Jun 27, 2024
2153e5c
Merge main into switch-zone-self-assembling
karencfv Jul 15, 2024
2f7b313
Update Dendrite hashes
karencfv Jul 15, 2024
1048969
Merge branch 'main' into switch-zone-self-assembling
karencfv Jul 19, 2024
4a269b3
update to latest dendrite commit
karencfv Jul 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions illumos-utils/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ pub const PFEXEC: &str = "/usr/bin/pfexec";
pub struct CommandFailureInfo {
command: String,
status: std::process::ExitStatus,
stdout: String,
stderr: String,
pub stdout: String,
pub stderr: String,
}

impl std::fmt::Display for CommandFailureInfo {
Expand Down
19 changes: 19 additions & 0 deletions illumos-utils/src/route.rs
Original file line number Diff line number Diff line change
Expand Up @@ -107,4 +107,23 @@ impl Route {
};
Ok(())
}

pub fn add_bootstrap_route(
bootstrap_prefix: u16,
gz_bootstrap_addr: Ipv6Addr,
zone_vnic_name: &str,
) -> Result<(), ExecutionError> {
let mut cmd = std::process::Command::new(PFEXEC);
let cmd = cmd.args(&[
ROUTE,
"add",
"-inet6",
&format!("{bootstrap_prefix:x}::/16"),
&gz_bootstrap_addr.to_string(),
"-ifp",
zone_vnic_name,
]);
execute(cmd)?;
Ok(())
}
}
20 changes: 14 additions & 6 deletions illumos-utils/src/running_zone.rs
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ pub enum ServiceError {
pub struct RunCommandError {
zone: String,
#[source]
err: crate::ExecutionError,
pub err: crate::ExecutionError,
}

/// Errors returned from [`RunningZone::boot`].
Expand Down Expand Up @@ -510,7 +510,7 @@ impl RunningZone {
/// Note that the zone must already be configured to be booted.
pub async fn boot(zone: InstalledZone) -> Result<Self, BootError> {
// Boot the zone.
info!(zone.log, "Zone booting");
info!(zone.log, "Booting {} zone", zone.name);

Zones::boot(&zone.name).await?;

Expand All @@ -528,6 +528,9 @@ impl RunningZone {
zone: zone.name.to_string(),
})?;

// TODO https://github.com/oxidecomputer/omicron/issues/1898:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this still relevant?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep! There is a check that verifies whether the site profile xml file exists (it will always exist for a self assembling zone), and performs various networking configuration tasks if it does not. There are various bits of code like this sprinkled around, and I'd like to remove them all in a separate PR.

// Remove all non-self assembling code

// If the zone is self-assembling, then SMF service(s) inside the zone
// will be creating the listen address for the zone's service(s),
// setting the appropriate ifprop MTU, and so on. The idea behind
Expand Down Expand Up @@ -623,7 +626,6 @@ impl RunningZone {
&self,
address: Ipv6Addr,
) -> Result<(), EnsureAddressError> {
info!(self.inner.log, "Adding bootstrap address");
let vnic = self.inner.bootstrap_vnic.as_ref().ok_or_else(|| {
EnsureAddressError::MissingBootstrapVnic {
address: address.to_string(),
Expand Down Expand Up @@ -785,15 +787,16 @@ impl RunningZone {
gz_bootstrap_addr: Ipv6Addr,
zone_vnic_name: &str,
) -> Result<(), RunCommandError> {
self.run_cmd([
let args = [
"/usr/sbin/route",
"add",
"-inet6",
&format!("{bootstrap_prefix:x}::/16"),
&gz_bootstrap_addr.to_string(),
"-ifp",
zone_vnic_name,
])?;
];
self.run_cmd(args)?;
Ok(())
}

Expand Down Expand Up @@ -914,7 +917,7 @@ impl RunningZone {

/// Return a reference to the links for this zone.
pub fn links(&self) -> &Vec<Link> {
&self.inner.links
&self.inner.links()
}

/// Return a mutable reference to the links for this zone.
Expand Down Expand Up @@ -1149,6 +1152,11 @@ impl InstalledZone {
pub fn root(&self) -> Utf8PathBuf {
self.zonepath.join(Self::ROOT_FS_PATH)
}

/// Return a reference to the links for this zone.
pub fn links(&self) -> &Vec<Link> {
&self.links
}
}

#[derive(Clone)]
Expand Down
2 changes: 1 addition & 1 deletion illumos-utils/src/zone.rs
Original file line number Diff line number Diff line change
Expand Up @@ -640,7 +640,7 @@ impl Zones {
//
// Does NOT check if the address already exists.
#[allow(clippy::needless_lifetimes)]
fn create_address_internal<'a>(
pub fn create_address_internal<'a>(
zone: Option<&'a str>,
addrobj: &AddrObject,
addrtype: AddressRequest,
Expand Down
22 changes: 15 additions & 7 deletions package-manifest.toml
Original file line number Diff line number Diff line change
Expand Up @@ -439,7 +439,6 @@ service_name = "switch_zone_setup"
source.type = "local"
source.paths = [
{ from = "smf/switch_zone_setup/manifest.xml", to = "/var/svc/manifest/site/switch_zone_setup/manifest.xml" },
{ from = "smf/switch_zone_setup/switch_zone_setup", to = "/opt/oxide/bin/switch_zone_setup" },
{ from = "smf/switch_zone_setup/support_authorized_keys", to = "/opt/oxide/support/authorized_keys" },
{ from = "/opt/ooce/pgsql-13/lib/amd64", to = "/opt/ooce/pgsql-13/lib/amd64" },
]
Expand Down Expand Up @@ -628,8 +627,9 @@ only_for_targets.image = "standard"
# 2. Copy dendrite.tar.gz from dendrite/out to omicron/out
source.type = "prebuilt"
source.repo = "dendrite"
source.commit = "6334bf74fa21790c15f1c4e494ea2ec0edd1c83c"
source.sha256 = "5929f9abf0daf4bbf17d835e5d69fc842b9617b312fb5644fa99daf785203700"
# TODO: Set to images from main branch. Testing out with images from my PR for now.
source.commit = "e058ad924378bdd7e27a169dda299867472a0503"
source.sha256 = "16798a450f33800e004f915442f6b603f547a6bf16312c26faf087853aec6c4a"
output.type = "zone"
output.intermediate_only = true

Expand All @@ -653,8 +653,9 @@ only_for_targets.image = "standard"
# 2. Copy the output zone image from dendrite/out to omicron/out
source.type = "prebuilt"
source.repo = "dendrite"
source.commit = "6334bf74fa21790c15f1c4e494ea2ec0edd1c83c"
source.sha256 = "0294a1911212c4764d1034b5e0ca00cc9dfc51df482a9f6e5547b191b4481ad8"
# TODO: Set to images from main branch. Testing out with images from my PR for now.
source.commit = "e058ad924378bdd7e27a169dda299867472a0503"
source.sha256 = "8545ddd844653c8ba917ba0eaabf6ec9effc140ea1e16bd1f5f2838c083729d5"
output.type = "zone"
output.intermediate_only = true

Expand All @@ -671,8 +672,9 @@ only_for_targets.image = "standard"
# 2. Copy dendrite.tar.gz from dendrite/out to omicron/out/dendrite-softnpu.tar.gz
source.type = "prebuilt"
source.repo = "dendrite"
source.commit = "6334bf74fa21790c15f1c4e494ea2ec0edd1c83c"
source.sha256 = "1a188da01dccf565058145b43573a549a2eb4d71fe8800170152b823af27a010"
# TODO: Set to images from main branch. Testing out with images from my PR for now.
source.commit = "e058ad924378bdd7e27a169dda299867472a0503"
source.sha256 = "d9c1460eb3de99d3303f56b014278eef804cfd3c1bef585cd9b516f6670997f7"
output.type = "zone"
output.intermediate_only = true

Expand Down Expand Up @@ -719,6 +721,8 @@ source.packages = [
"switch_zone_setup.tar.gz",
"xcvradm.tar.gz",
"omicron-omdb.tar.gz",
"zone-setup.tar.gz",
"zone-network-install.tar.gz"
]
output.type = "zone"

Expand All @@ -743,6 +747,8 @@ source.packages = [
"switch_zone_setup.tar.gz",
"sp-sim-stub.tar.gz",
"omicron-omdb.tar.gz",
"zone-setup.tar.gz",
"zone-network-install.tar.gz"
]
output.type = "zone"

Expand All @@ -767,6 +773,8 @@ source.packages = [
"switch_zone_setup.tar.gz",
"sp-sim-softnpu.tar.gz",
"omicron-omdb.tar.gz",
"zone-setup.tar.gz",
"zone-network-install.tar.gz"
]
output.type = "zone"

Expand Down
3 changes: 0 additions & 3 deletions sled-agent/src/params.rs
Original file line number Diff line number Diff line change
Expand Up @@ -588,9 +588,6 @@ impl crate::smf_helper::Service for OmicronZoneType {
fn smf_name(&self) -> String {
format!("svc:/oxide/{}", self.service_name())
}
fn should_import(&self) -> bool {
true
}
}

impl From<OmicronZoneType> for sled_agent_client::types::OmicronZoneType {
Expand Down
13 changes: 7 additions & 6 deletions sled-agent/src/profile.rs
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,7 @@ impl Display for ServiceInstanceBuilder {
}
}

#[derive(Clone)]
pub struct PropertyGroupBuilder {
name: String,
/// names of the properties that were added, in the order they were added
Expand Down Expand Up @@ -233,7 +234,7 @@ impl Display for PropertyGroupBuilder {
if values.len() == 1 {
write!(
f,
r#" <propval type="{ty}" name="{name}" value="{value}"/>
smklein marked this conversation as resolved.
Show resolved Hide resolved
r#" <propval type="{ty}" name="{name}" value='{value}'/>
"#,
name = property_name,
value = &values[0],
Expand Down Expand Up @@ -302,7 +303,7 @@ mod tests {
<service_bundle type="profile" name="myprofile">
<service version="1" type="service" name="myservice">
<property_group type="application" name="mypg">
<propval type="astring" name="myprop" value="myvalue"/>
<propval type="astring" name="myprop" value='myvalue'/>
</property_group>
</service>
</service_bundle>"#,
Expand Down Expand Up @@ -384,7 +385,7 @@ mod tests {
<service version="1" type="service" name="myservice">
<instance enabled="true" name="default">
<property_group type="application" name="mypg">
<propval type="type" name="prop" value="value"/>
<propval type="type" name="prop" value='value'/>
</property_group>
</instance>
</service>
Expand Down Expand Up @@ -429,11 +430,11 @@ mod tests {
</property_group>
<instance enabled="true" name="default">
<property_group type="application" name="mypg">
<propval type="type" name="prop" value="value"/>
<propval type="type" name="prop2" value="value2"/>
<propval type="type" name="prop" value='value'/>
<propval type="type" name="prop2" value='value2'/>
</property_group>
<property_group type="application" name="mypg2">
<propval type="type" name="prop3" value="value3"/>
<propval type="type" name="prop3" value='value3'/>
</property_group>
</instance>
</service>
Expand Down
Loading