Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VPC Subnet Routing [2/2] -- Custom Routers and NIC 'transit IP' lists #5823

Merged
merged 77 commits into from
Jun 26, 2024
Merged
Show file tree
Hide file tree
Changes from 68 commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
76106b1
Clear XDE underlay when destroying virtual hardware
FelixMcFelix Apr 23, 2024
0afee1b
Merge branch 'main' into felixmcfelix/opte-underlay-lock
FelixMcFelix May 21, 2024
94e1808
Pull in latest OPTE after merge.
FelixMcFelix May 21, 2024
b67f416
Bump OPTE and related.
FelixMcFelix May 21, 2024
5f7cfa8
Compat with newer OPTE, more believable router rules
FelixMcFelix May 21, 2024
16e0107
VPC Subnet route reconcile.
FelixMcFelix May 15, 2024
22b71bb
We now have valid, sane, default system routes
FelixMcFelix May 15, 2024
160853b
Start refreshing API specs
FelixMcFelix May 15, 2024
034dd0f
Iterating.
FelixMcFelix May 16, 2024
87d9b26
Route resolution (but not actual installation)
FelixMcFelix May 17, 2024
568f44c
Wrong ID in router initialisation.
FelixMcFelix May 17, 2024
2a25d74
Rule insert/delete with OPTE.
FelixMcFelix May 17, 2024
c718c8d
Correctly instantiate router rules if existing
FelixMcFelix May 17, 2024
06eaaab
Feed Clippy.
FelixMcFelix May 20, 2024
38beadd
Comment adapted.
FelixMcFelix May 20, 2024
17489cb
The backing code for a generational RPW
FelixMcFelix May 20, 2024
006b1ca
Iterating.
FelixMcFelix May 21, 2024
c7de875
Trigger RPW in all the right spots.
FelixMcFelix May 21, 2024
40edbc8
Unpublish VPC routers API.
FelixMcFelix May 22, 2024
3b3abb1
Fixup broken tests.
FelixMcFelix May 22, 2024
be9f8ab
Accidental local state...
FelixMcFelix May 22, 2024
f433b38
Unsubscribe routes from sled when ports are removed.
FelixMcFelix May 22, 2024
62ca9f0
Migration query for subnet route creation.
FelixMcFelix May 23, 2024
2c06ff4
Rework migration slightly.
FelixMcFelix May 23, 2024
0e8d1ad
Bump OPTE to include latest perf work.
FelixMcFelix May 23, 2024
7b32e09
Merge branch 'main' into felixmcfelix/opte-underlay-lock
FelixMcFelix May 23, 2024
f7646ef
Merge branch 'felixmcfelix/opte-underlay-lock' into felixmcfelix/vpc-…
FelixMcFelix May 23, 2024
2537222
Self-review pt.1.
FelixMcFelix May 23, 2024
f217bd1
Self-review pt.2.
FelixMcFelix May 23, 2024
b549044
Test route resolution.
FelixMcFelix May 23, 2024
f02535e
Accidentally ended up on the wrong maghemite.
FelixMcFelix May 24, 2024
880378a
Hook VPC checks into sim-sled-agent, instance networking tests.
FelixMcFelix May 24, 2024
08c982e
Correctly cleanup after new tests...
FelixMcFelix May 24, 2024
821e241
Allow bind of subnet to custom router in create/update
FelixMcFelix May 24, 2024
9c68888
Fix VpcRouter field order, test usual attachment behaviour
FelixMcFelix May 28, 2024
e886d16
Fix custom router listing.
FelixMcFelix May 28, 2024
e6fe817
Merge branch 'felixmcfelix/vpc-subnet-routing' into felixmcfelix/vpc-…
FelixMcFelix May 28, 2024
e78c51f
First shot at spoof-prevention prevention.
FelixMcFelix May 29, 2024
c1dc6d0
Test router attach swap, cleanup tests.
FelixMcFelix May 29, 2024
0f0dcb6
Validate custom, dest/target pairs on route create
FelixMcFelix May 29, 2024
3d17960
Visit + test allowed routes and modification semantics in-depth.
FelixMcFelix May 29, 2024
4c40916
Don't break parsing NICs from old blueprints
FelixMcFelix May 29, 2024
536b40c
Minor fmt + lint tweaks.
FelixMcFelix May 29, 2024
83b1710
Fixup unexpected authz endpoints.
FelixMcFelix May 29, 2024
7413adf
Last big test, cutesier test data names.
FelixMcFelix May 30, 2024
bd7294c
Fixup one test.
FelixMcFelix May 30, 2024
e3b7b90
Self-review.
FelixMcFelix May 31, 2024
321cb16
Merge branch 'main' into felixmcfelix/vpc-subnet-routing
FelixMcFelix Jun 3, 2024
a8415b3
Merge branch 'felixmcfelix/vpc-subnet-routing' into felixmcfelix/vpc-…
FelixMcFelix Jun 3, 2024
f21ef51
Merge branch 'main' into felixmcfelix/vpc-subnet-routing
FelixMcFelix Jun 13, 2024
87cf88e
Merge branch 'felixmcfelix/vpc-subnet-routing' into felixmcfelix/vpc-…
FelixMcFelix Jun 13, 2024
cca764f
Bump image.
FelixMcFelix Jun 13, 2024
543cac4
Merge branch 'felixmcfelix/vpc-subnet-routing' into felixmcfelix/vpc-…
FelixMcFelix Jun 13, 2024
bf3ffb0
Handle greater specificity of rule addition.
FelixMcFelix Jun 13, 2024
2425016
Forgot some maghemite SHAs...
FelixMcFelix Jun 13, 2024
8e005a2
Merge branch 'felixmcfelix/vpc-subnet-routing' into felixmcfelix/vpc-…
FelixMcFelix Jun 13, 2024
feb455f
Merge branch 'main' into felixmcfelix/vpc-subnet-routing
FelixMcFelix Jun 21, 2024
30e4043
Minor fixes post-merge.
FelixMcFelix Jun 21, 2024
0057228
Review feedback.
FelixMcFelix Jun 21, 2024
a63b8b0
Merge branch 'felixmcfelix/vpc-subnet-routing' into felixmcfelix/vpc-…
FelixMcFelix Jun 21, 2024
28cb739
One or two UUID-types missed...
FelixMcFelix Jun 21, 2024
992b9b5
More liberal use of `#[serde(default)]`
FelixMcFelix Jun 21, 2024
ab7223b
Accidentally a line.
FelixMcFelix Jun 21, 2024
2568e2c
Especially good at EXPECTORATE-ing
FelixMcFelix Jun 21, 2024
d747d01
Again.
FelixMcFelix Jun 21, 2024
c1e5e75
Review feedback: clearer explanation of `RouteDestination`
FelixMcFelix Jun 21, 2024
15bb2ee
Merge branch 'main' into felixmcfelix/vpc-subnet-routing
FelixMcFelix Jun 24, 2024
3766244
Merge branch 'felixmcfelix/vpc-subnet-routing' into felixmcfelix/vpc-…
FelixMcFelix Jun 24, 2024
0ab6483
Slight adjustments to `destination` docstrings
FelixMcFelix Jun 25, 2024
8162a23
Review feedback: typed RouterKind instead of Option abuse
FelixMcFelix Jun 26, 2024
e1971bc
Bump Maghemite.
FelixMcFelix Jun 26, 2024
899d367
Merge branch 'main' into felixmcfelix/vpc-subnet-routing
FelixMcFelix Jun 26, 2024
70263a2
Better conflict resolution on Nexus-managed subnet route names
FelixMcFelix Jun 26, 2024
9b46a28
Merge branch 'felixmcfelix/vpc-subnet-routing' into felixmcfelix/vpc-…
FelixMcFelix Jun 26, 2024
62de75e
Unearthed a nice li'l bug during migration on london
FelixMcFelix Jun 26, 2024
93b073c
Merge branch 'felixmcfelix/vpc-subnet-routing' into felixmcfelix/vpc-…
FelixMcFelix Jun 26, 2024
2bd4bd7
Merge branch 'main' into felixmcfelix/vpc-subnets-user
FelixMcFelix Jun 26, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/buildomat/jobs/deploy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
#:
#: name = "helios / deploy"
#: variety = "basic"
#: target = "lab-2.0-opte-0.31"
#: target = "lab-2.0-opte-0.32"
#: output_rules = [
#: "%/var/svc/log/oxide-sled-agent:default.log*",
#: "%/zone/oxz_*/root/var/svc/log/oxide-*.log*",
Expand Down
17 changes: 9 additions & 8 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 4 additions & 4 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -328,8 +328,8 @@ macaddr = { version = "1.0.1", features = ["serde_std"] }
maplit = "1.0.2"
mockall = "0.12"
newtype_derive = "0.1.6"
mg-admin-client = { git = "https://github.com/oxidecomputer/maghemite", rev = "5630887d0373857f77cb264f84aa19bdec720ce3" }
ddm-admin-client = { git = "https://github.com/oxidecomputer/maghemite", rev = "5630887d0373857f77cb264f84aa19bdec720ce3" }
mg-admin-client = { git = "https://github.com/oxidecomputer/maghemite", rev = "e63f6d408908b3332d7cd89a4dd44a0f980d931d" }
ddm-admin-client = { git = "https://github.com/oxidecomputer/maghemite", rev = "e63f6d408908b3332d7cd89a4dd44a0f980d931d" }
multimap = "0.10.0"
nexus-auth = { path = "nexus/auth" }
nexus-client = { path = "clients/nexus-client" }
Expand Down Expand Up @@ -367,14 +367,14 @@ omicron-sled-agent = { path = "sled-agent" }
omicron-test-utils = { path = "test-utils" }
omicron-zone-package = "0.11.0"
oxide-client = { path = "clients/oxide-client" }
oxide-vpc = { git = "https://github.com/oxidecomputer/opte", rev = "417f74e94978c23f3892ac328c3387f3ecd9bb29", features = [ "api", "std" ] }
oxide-vpc = { git = "https://github.com/oxidecomputer/opte", rev = "915975f6d1729db95619f752148974016912412f", features = [ "api", "std" ] }
once_cell = "1.19.0"
openapi-lint = { git = "https://github.com/oxidecomputer/openapi-lint", branch = "main" }
openapiv3 = "2.0.0"
# must match samael's crate!
openssl = "0.10"
openssl-sys = "0.9"
opte-ioctl = { git = "https://github.com/oxidecomputer/opte", rev = "417f74e94978c23f3892ac328c3387f3ecd9bb29" }
opte-ioctl = { git = "https://github.com/oxidecomputer/opte", rev = "915975f6d1729db95619f752148974016912412f" }
oso = "0.27"
owo-colors = "4.0.0"
oximeter = { path = "oximeter/oximeter" }
Expand Down
5 changes: 5 additions & 0 deletions clients/sled-agent-client/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,11 @@ progenitor::generate_api!(
NetworkInterface = omicron_common::api::internal::shared::NetworkInterface,
PortFec = omicron_common::api::internal::shared::PortFec,
PortSpeed = omicron_common::api::internal::shared::PortSpeed,
RouterId = omicron_common::api::internal::shared::RouterId,
ResolvedVpcRoute = omicron_common::api::internal::shared::ResolvedVpcRoute,
ResolvedVpcRouteSet = omicron_common::api::internal::shared::ResolvedVpcRouteSet,
RouterTarget = omicron_common::api::internal::shared::RouterTarget,
RouterVersion = omicron_common::api::internal::shared::RouterVersion,
SourceNatConfig = omicron_common::api::internal::shared::SourceNatConfig,
SwitchLocation = omicron_common::api::external::SwitchLocation,
TypedUuidForInstanceKind = omicron_uuid_kinds::InstanceUuid,
Expand Down
14 changes: 11 additions & 3 deletions common/src/api/external/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1331,6 +1331,9 @@ pub enum RouteTarget {
#[display("inetgw:{0}")]
/// Forward traffic to an internet gateway
InternetGateway(Name),
#[display("drop")]
/// Drop matching traffic
Drop,
}

/// A `RouteDestination` is used to match traffic with a routing rule, on the
Expand Down Expand Up @@ -1404,14 +1407,14 @@ pub struct RouterRoute {
/// common identifying metadata
#[serde(flatten)]
pub identity: IdentityMetadata,

/// The ID of the VPC Router to which the route belongs
pub vpc_router_id: Uuid,

/// Describes the kind of router. Set at creation. `read-only`
pub kind: RouterRouteKind,

/// The location that matched packets should be forwarded to.
pub target: RouteTarget,
/// The set of destination IP addresses or subnets that this route
/// will match packets against.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry to keep nitpicking, but saying "set" and "subnets" might be confusing because this can only be one subnet. I can't think of a better wording — maybe rely on the definiton of RouteDestination to cover what is inside it, and here emphasize its role in the route. I like this bit in the doc comment on RouteDestination, though it's kind of long:

/// When traffic is to be sent to a destination that is within a given
/// `RouteDestination`, the corresponding `RouterRoute` applies, and traffic
/// will be forward to the `RouteTarget` for that rule.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the existing RouteDestination text is good there. I think if we lean into just showing its role and delegating the details to the type, I'd go for:

/// Selects which traffic this routing rule will apply to.
pub destination: RouteDestination,

pub destination: RouteDestination,
}

Expand Down Expand Up @@ -1976,6 +1979,11 @@ pub struct InstanceNetworkInterface {
/// True if this interface is the primary for the instance to which it's
/// attached.
pub primary: bool,

/// A set of additional networks that this interface may send and
/// receive traffic on.
#[serde(default)]
pub transit_ips: Vec<IpNet>,
FelixMcFelix marked this conversation as resolved.
Show resolved Hide resolved
}

#[derive(
Expand Down
76 changes: 72 additions & 4 deletions common/src/api/internal/shared.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@

use crate::{
address::NUM_SOURCE_NAT_PORTS,
api::external::{self, BfdMode, ImportExportPolicy, Name},
api::external::{self, BfdMode, ImportExportPolicy, Name, Vni},
};
use oxnet::{IpNet, Ipv4Net, Ipv6Net};
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
use std::{
collections::HashMap,
collections::{HashMap, HashSet},
fmt,
net::{IpAddr, Ipv4Addr, Ipv6Addr},
str::FromStr,
Expand Down Expand Up @@ -50,13 +50,15 @@ pub enum NetworkInterfaceKind {
pub struct NetworkInterface {
pub id: Uuid,
pub kind: NetworkInterfaceKind,
pub name: external::Name,
pub name: Name,
pub ip: IpAddr,
pub mac: external::MacAddr,
pub subnet: IpNet,
pub vni: external::Vni,
pub vni: Vni,
pub primary: bool,
pub slot: u8,
#[serde(default)]
pub transit_ips: Vec<IpNet>,
}

/// An IP address and port range used for source NAT, i.e., making
Expand Down Expand Up @@ -624,6 +626,72 @@ impl TryFrom<&[ipnetwork::IpNetwork]> for IpAllowList {
}
}

/// A VPC route resolved into a concrete target.
#[derive(
Clone, Debug, Deserialize, Serialize, JsonSchema, PartialEq, Eq, Hash,
)]
pub struct ResolvedVpcRoute {
pub dest: IpNet,
pub target: RouterTarget,
}

/// The target for a given router entry.
#[derive(
Clone, Debug, Deserialize, Serialize, JsonSchema, PartialEq, Eq, Hash,
)]
#[serde(tag = "type", rename_all = "snake_case", content = "value")]
pub enum RouterTarget {
Drop,
InternetGateway,
Ip(IpAddr),
VpcSubnet(IpNet),
}

/// Information on the current parent router (and version) of a route set
/// according to the control plane.
#[derive(
Copy, Clone, Debug, Deserialize, Serialize, JsonSchema, PartialEq, Eq, Hash,
)]
pub struct RouterVersion {
pub router_id: Uuid,
pub version: u64,
}

impl RouterVersion {
/// Return whether a new route set should be applied over the current
/// values.
///
/// This will occur when seeing a new version and a matching parent,
/// or a new parent router on the control plane.
pub fn is_replaced_by(&self, other: &Self) -> bool {
(self.router_id != other.router_id) || self.version < other.version
}
}

/// Identifier for a VPC and/or subnet.
#[derive(
Copy, Clone, Debug, Deserialize, Serialize, JsonSchema, PartialEq, Eq, Hash,
)]
pub struct RouterId {
pub vni: Vni,
pub subnet: Option<IpNet>,
}

/// Version information for routes on a given VPC subnet.
#[derive(Clone, Debug, Deserialize, Serialize, JsonSchema, PartialEq, Eq)]
pub struct ResolvedVpcRouteState {
pub id: RouterId,
pub version: Option<RouterVersion>,
}

/// An updated set of routes for a given VPC and/or subnet.
#[derive(Clone, Debug, Deserialize, Serialize, JsonSchema, PartialEq, Eq)]
pub struct ResolvedVpcRouteSet {
pub id: RouterId,
pub version: Option<RouterVersion>,
pub routes: HashSet<ResolvedVpcRoute>,
}

#[cfg(test)]
mod tests {
use crate::api::internal::shared::AllowedSourceIps;
Expand Down
12 changes: 12 additions & 0 deletions dev-tools/omdb/tests/env.out
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,10 @@ task: "v2p_manager"
manages opte v2p mappings for vpc networking


task: "vpc_route_manager"
propagates updated VPC routes to all OPTE ports


---------------------------------------------
stderr:
note: using Nexus URL http://127.0.0.1:REDACTED_PORT
Expand Down Expand Up @@ -251,6 +255,10 @@ task: "v2p_manager"
manages opte v2p mappings for vpc networking


task: "vpc_route_manager"
propagates updated VPC routes to all OPTE ports


---------------------------------------------
stderr:
note: Nexus URL not specified. Will pick one from DNS.
Expand Down Expand Up @@ -362,6 +370,10 @@ task: "v2p_manager"
manages opte v2p mappings for vpc networking


task: "vpc_route_manager"
propagates updated VPC routes to all OPTE ports


---------------------------------------------
stderr:
note: Nexus URL not specified. Will pick one from DNS.
Expand Down
11 changes: 11 additions & 0 deletions dev-tools/omdb/tests/successes.out
Original file line number Diff line number Diff line change
Expand Up @@ -328,6 +328,10 @@ task: "v2p_manager"
manages opte v2p mappings for vpc networking


task: "vpc_route_manager"
propagates updated VPC routes to all OPTE ports


---------------------------------------------
stderr:
note: using Nexus URL http://127.0.0.1:REDACTED_PORT/
Expand Down Expand Up @@ -532,6 +536,13 @@ task: "v2p_manager"
started at <REDACTED TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
warning: unknown background task: "v2p_manager" (don't know how to interpret details: Object {})

task: "vpc_route_manager"
configured period: every 30s
currently executing: no
last completed activation: <REDACTED ITERATIONS>, triggered by an explicit signal
started at <REDACTED TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
warning: unknown background task: "vpc_route_manager" (don't know how to interpret details: Object {})

---------------------------------------------
stderr:
note: using Nexus URL http://127.0.0.1:REDACTED_PORT/
Expand Down
19 changes: 2 additions & 17 deletions illumos-utils/src/opte/firewall_rules.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

//! Convert Omicron VPC firewall rules to OPTE firewall rules.

use super::net_to_cidr;
use crate::opte::params::VpcFirewallRule;
use crate::opte::Vni;
use macaddr::MacAddr6;
Expand All @@ -18,11 +19,6 @@ use oxide_vpc::api::Filters;
use oxide_vpc::api::FirewallAction;
use oxide_vpc::api::FirewallRule;
use oxide_vpc::api::IpAddr;
use oxide_vpc::api::IpCidr;
use oxide_vpc::api::Ipv4Cidr;
use oxide_vpc::api::Ipv4PrefixLen;
use oxide_vpc::api::Ipv6Cidr;
use oxide_vpc::api::Ipv6PrefixLen;
use oxide_vpc::api::Ports;
use oxide_vpc::api::ProtoFilter;
use oxide_vpc::api::Protocol;
Expand Down Expand Up @@ -68,21 +64,10 @@ impl FromVpcFirewallRule for VpcFirewallRule {
HostIdentifier::Ip(IpNet::V4(net)) if net.is_host_net() => {
Address::Ip(IpAddr::Ip4(net.addr().into()))
}
HostIdentifier::Ip(IpNet::V4(net)) => {
Address::Subnet(IpCidr::Ip4(Ipv4Cidr::new(
net.addr().into(),
Ipv4PrefixLen::new(net.width()).unwrap(),
)))
}
HostIdentifier::Ip(IpNet::V6(net)) if net.is_host_net() => {
Address::Ip(IpAddr::Ip6(net.addr().into()))
}
HostIdentifier::Ip(IpNet::V6(net)) => {
Address::Subnet(IpCidr::Ip6(Ipv6Cidr::new(
net.addr().into(),
Ipv6PrefixLen::new(net.width()).unwrap(),
)))
}
HostIdentifier::Ip(ip) => Address::Subnet(net_to_cidr(*ip)),
HostIdentifier::Vpc(vni) => {
Address::Vni(Vni::new(u32::from(*vni)).unwrap())
}
Expand Down
Loading
Loading