✨ Introduce edge subnets to support AWS Local Zones #4882
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
|
|
@@ -20,6 +20,10 @@ import ( | |||||||
| "fmt" | ||||||||
| "sort" | ||||||||
| "time" | ||||||||
|
|
||||||||
| "github.com/aws/aws-sdk-go/aws" | ||||||||
| "github.com/aws/aws-sdk-go/service/ec2" | ||||||||
| "k8s.io/utils/ptr" | ||||||||
| ) | ||||||||
|
|
||||||||
| const ( | ||||||||
|
|
@@ -37,6 +41,11 @@ const ( | |||||||
| DefaultAPIServerHealthThresholdCount = 5 | ||||||||
| // DefaultAPIServerUnhealthThresholdCount the API server unhealthy check threshold count. | ||||||||
| DefaultAPIServerUnhealthThresholdCount = 3 | ||||||||
|
|
||||||||
| // ZoneTypeAvailabilityZone defines the regular AWS zones in the Region. | ||||||||
| ZoneTypeAvailabilityZone ZoneType = "availability-zone" | ||||||||
| // ZoneTypeLocalZone defines the AWS zone type in Local Zone infrastructure. | ||||||||
| ZoneTypeLocalZone ZoneType = "local-zone" | ||||||||
| ) | ||||||||
|
|
||||||||
| // NetworkStatus encapsulates AWS networking resources. | ||||||||
|
|
@@ -508,6 +517,39 @@ type SubnetSpec struct { | |||||||
|
|
||||||||
| // Tags is a collection of tags describing the resource. | ||||||||
| Tags Tags `json:"tags,omitempty"` | ||||||||
|
|
||||||||
| // ZoneType defines the type of the zone where the subnet is created. | ||||||||
| // | ||||||||
| // The valid values are availability-zone, and local-zone. | ||||||||
| // | ||||||||
| // Subnet with zone type availability-zone (regular) is always selected to create cluster | ||||||||
| // resources, like Load Balancers, NAT Gateways, Contol Plane nodes, etc. | ||||||||
| // | ||||||||
| // Subnet with zone type local-zone is not eligible to automatically create | ||||||||
| // regular cluster resources. | ||||||||
| // | ||||||||
| // The public subnet in availability-zone or local-zone is associated with regular public | ||||||||
| // route table with default route entry to a Internet Gateway. | ||||||||
| // | ||||||||
| // The private subnet in the availability-zone is associated with a private route table with | ||||||||
| // the default route entry to a NAT Gateway created in that zone. | ||||||||
| // | ||||||||
| // The private subnet in the local-zone is associated with a private route table with | ||||||||
| // the default route entry re-using the NAT Gateway in the Region (preferred from the | ||||||||
| // parent zone, the zone type availability-zone in the region, or first table available). | ||||||||
| // | ||||||||
| // +kubebuilder:validation:Enum=availability-zone;local-zone | ||||||||
| // +optional | ||||||||
| ZoneType *ZoneType `json:"zoneType,omitempty"` | ||||||||
|
|
||||||||
| // ParentZoneName is the zone name where the current subnet's zone is tied when | ||||||||
| // the zone is a Local Zone. | ||||||||
| // | ||||||||
| // The subnets in Local Zone locations consume the ParentZoneName to determine the correct | ||||||||
| // private route table to egress traffic to the internet. | ||||||||
| // | ||||||||
| // +optional | ||||||||
nrb marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||
| ParentZoneName *string `json:"parentZoneName,omitempty"` | ||||||||
|
There was a problem hiding this comment. It would be good to have CEL to enforce the cross field validation. There was a problem hiding this comment. Or use validation in the webhook. There was a problem hiding this comment. @richardcase I am thiking the best strategy here. I think we need to prevent the ParentZoneName when not // SubnetSpec configures an AWS Subnet.
+// +kubebuilder:validation:XValidation:rule="has(self.zoneType) && self.zoneType == 'local-zone' ? (has(self.parentZoneName) && size(self.parentZoneName) > 0)
: !has(self.parentZoneName)",message="parentZoneName must be set if zoneType is 'local-zone'"
type SubnetSpec struct {
// ID defines a unique identifier to reference this resource.
....Do you think it makes sense? is it enough? There was a problem hiding this comment.
yeah, when thinking in the follow up PRs (Wavelength), it seems to be better to create a validation in the webhook. Let me do it instead of CEL. |
||||||||
| } | ||||||||
|
|
||||||||
| // GetResourceID returns the identifier for this subnet, | ||||||||
|
|
@@ -524,6 +566,39 @@ func (s *SubnetSpec) String() string { | |||||||
| return fmt.Sprintf("id=%s/az=%s/public=%v", s.GetResourceID(), s.AvailabilityZone, s.IsPublic) | ||||||||
| } | ||||||||
|
|
||||||||
| // IsEdge returns the true when the subnet is created in the edge zone, | ||||||||
| // Local Zones. | ||||||||
| func (s *SubnetSpec) IsEdge() bool { | ||||||||
| return s.ZoneType != nil && *s.ZoneType == ZoneTypeLocalZone | ||||||||
| } | ||||||||
|
|
||||||||
| // SetZoneInfo updates the subnets with zone information. | ||||||||
| func (s *SubnetSpec) SetZoneInfo(zones []*ec2.AvailabilityZone) error { | ||||||||
| zoneInfo := func(zoneName string) *ec2.AvailabilityZone { | ||||||||
| for _, zone := range zones { | ||||||||
| if aws.StringValue(zone.ZoneName) == zoneName { | ||||||||
| return zone | ||||||||
| } | ||||||||
| } | ||||||||
| return nil | ||||||||
| } | ||||||||
|
|
||||||||
| zone := zoneInfo(s.AvailabilityZone) | ||||||||
| if zone == nil { | ||||||||
| if len(s.AvailabilityZone) > 0 { | ||||||||
| return fmt.Errorf("unable to update zone information for subnet '%v' and zone '%v'", s.ID, s.AvailabilityZone) | ||||||||
| } | ||||||||
| return fmt.Errorf("unable to update zone information for subnet '%v'", s.ID) | ||||||||
| } | ||||||||
| if zone.ZoneType != nil { | ||||||||
| s.ZoneType = ptr.To(ZoneType(*zone.ZoneType)) | ||||||||
|
There was a problem hiding this comment. Maybe I'm just being paranoid, but I feel like it's safer to save the value to a local var before taking its address:
Suggested change
There was a problem hiding this comment. You are not, or I am being too! =] There was a problem hiding this comment. Oh, I didn't mean safer in the sense of the value being |
||||||||
| } | ||||||||
| if zone.ParentZoneName != nil { | ||||||||
| s.ParentZoneName = zone.ParentZoneName | ||||||||
| } | ||||||||
| return nil | ||||||||
| } | ||||||||
|
|
||||||||
| // Subnets is a slice of Subnet. | ||||||||
| // +listType=map | ||||||||
| // +listMapKey=id | ||||||||
|
|
@@ -541,6 +616,22 @@ func (s Subnets) ToMap() map[string]*SubnetSpec { | |||||||
|
|
||||||||
| // IDs returns a slice of the subnet ids. | ||||||||
| func (s Subnets) IDs() []string { | ||||||||
| res := []string{} | ||||||||
| for _, subnet := range s { | ||||||||
| // Prevent returning edge zones (Local Zone) to regular Subnet IDs. | ||||||||
| // Edge zones should not deploy control plane nodes, and does not support Nat Gateway and | ||||||||
| // Network Load Balancers. Any resource for the core infrastructure should not consume edge | ||||||||
| // zones. | ||||||||
| if subnet.IsEdge() { | ||||||||
| continue | ||||||||
| } | ||||||||
| res = append(res, subnet.GetResourceID()) | ||||||||
| } | ||||||||
| return res | ||||||||
| } | ||||||||
|
|
||||||||
| // IDsWithEdge returns a slice of the subnet ids. | ||||||||
| func (s Subnets) IDsWithEdge() []string { | ||||||||
| res := []string{} | ||||||||
| for _, subnet := range s { | ||||||||
| res = append(res, subnet.GetResourceID()) | ||||||||
|
|
@@ -581,21 +672,29 @@ func (s Subnets) FindEqual(spec *SubnetSpec) *SubnetSpec { | |||||||
| // FilterPrivate returns a slice containing all subnets marked as private. | ||||||||
| func (s Subnets) FilterPrivate() (res Subnets) { | ||||||||
| for _, x := range s { | ||||||||
| // Subnets in AWS Local Zones or Wavelength should not be used by core infrastructure. | ||||||||
| if x.IsEdge() { | ||||||||
| continue | ||||||||
| } | ||||||||
| if !x.IsPublic { | ||||||||
| res = append(res, x) | ||||||||
| } | ||||||||
| } | ||||||||
| return res | ||||||||
| } | ||||||||
|
|
||||||||
| // FilterPublic returns a slice containing all subnets marked as public. | ||||||||
| func (s Subnets) FilterPublic() (res Subnets) { | ||||||||
| for _, x := range s { | ||||||||
| // Subnets in AWS Local Zones or Wavelength should not be used by core infrastructure. | ||||||||
| if x.IsEdge() { | ||||||||
| continue | ||||||||
| } | ||||||||
| if x.IsPublic { | ||||||||
| res = append(res, x) | ||||||||
| } | ||||||||
| } | ||||||||
| return res | ||||||||
| } | ||||||||
|
|
||||||||
| // FilterByZone returns a slice containing all subnets that live in the availability zone specified. | ||||||||
|
|
@@ -605,22 +704,32 @@ func (s Subnets) FilterByZone(zone string) (res Subnets) { | |||||||
| res = append(res, x) | ||||||||
| } | ||||||||
| } | ||||||||
| return res | ||||||||
| } | ||||||||
|
|
||||||||
| // GetUniqueZones returns a slice containing the unique zones of the subnets. | ||||||||
| func (s Subnets) GetUniqueZones() []string { | ||||||||
| keys := make(map[string]bool) | ||||||||
| zones := []string{} | ||||||||
| for _, x := range s { | ||||||||
| if _, value := keys[x.AvailabilityZone]; len(x.AvailabilityZone) > 0 && !value { | ||||||||
| keys[x.AvailabilityZone] = true | ||||||||
| zones = append(zones, x.AvailabilityZone) | ||||||||
| } | ||||||||
| } | ||||||||
| return zones | ||||||||
| } | ||||||||
|
|
||||||||
| // SetZoneInfo updates the subnets with zone information. | ||||||||
| func (s Subnets) SetZoneInfo(zones []*ec2.AvailabilityZone) error { | ||||||||
| for i := range s { | ||||||||
| if err := s[i].SetZoneInfo(zones); err != nil { | ||||||||
| return err | ||||||||
| } | ||||||||
| } | ||||||||
| return nil | ||||||||
| } | ||||||||
|
|
||||||||
| // CNISpec defines configuration for CNI. | ||||||||
| type CNISpec struct { | ||||||||
| // CNIIngressRules specify rules to apply to control plane and worker node security groups. | ||||||||
|
|
@@ -835,3 +944,11 @@ func (i *IngressRule) Equals(o *IngressRule) bool { | |||||||
|
|
||||||||
| return true | ||||||||
| } | ||||||||
|
|
||||||||
| // ZoneType defines listener AWS Availability Zone type. | ||||||||
| type ZoneType string | ||||||||
|
|
||||||||
| // String returns the string representation for the zone type. | ||||||||
| func (z ZoneType) String() string { | ||||||||
| return string(z) | ||||||||
| } | ||||||||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note for the reviewer: