Using AWS Controllers for Kubernetes (ACK) for AWS resource reconciliation

[mikestef9]

Background
AWS Controllers for Kubernetes (ACK) enables users to define and use AWS service resources using Kubernetes native API resources.

Kubernetes Cluster API for AWS (CAPA) provides consistent deployment and day 2 operations of "self-managed" and EKS Kubernetes clusters on AWS.

ACK service controllers are a direct mapping to AWS service APIs, whereas CAPA operates at a higher level of abstraction than ACK, for example, creating a VPC if one is not specified as part of a cluster definition.

There is duplication of code and effort between ACK controllers and the lower level of CAPA that ensures AWS resources are kept in sync with desired state.

Proposal
ACK now has service controllers for most AWS APIs that CAPA requires:

• IAM: https://github.com/aws-controllers-k8s/iam-controller
• EC2/VPC: https://github.com/aws-controllers-k8s/ec2-controller
• KMS: https://github.com/aws-controllers-k8s/kms-controller
• Managed AWS Control Plane (EKS): https://github.com/aws-controllers-k8s/eks-controller

We propose a refactor of CAPA to import ACK controllers as packages, replacing existing CAPA ↔ AWS reconciliation code. ACK controllers are compatible as Go modules, and CAPA Go packages should be able to take a dependency on the controller packages, and then systematically declare ACK resources. We don’t believe there is anything specific about ACK code paths that requires they need to be run as ACK controllers.

Another potential approach is to generate (using the ack-generate tool) the CAPA resource manager and SDK linkage code (which is similar to how ACK does this for Crossplane's provider-aws).

More info on ACK:

• Documentation
• Launch blog

Additional Notes

• CAPA and ACK both use AWS SDK for Go v1.
• ACK controllers CAPA may require that are not built yet:
  • ELBv2 ELB / Enterprise Load Balancer service controller aws-controllers-k8s/community#481
  • EC2 Auto Scaling groups AutoScaling service controller aws-controllers-k8s/community#588
• Today, ACK runs as separate controllers, and each require an IAM role specific to that service. In this proposal, ACK code would be assuming the IAM role associated with the CAPA controller, which would have the broader service permissions needed overall by CAPA.

[richardcase]

I like the idea of adopting ACK...especially the part about removing duplication of code/effort.

A few questions/comments come to mind initially:

1. Is there any timeframe on building the ELBv2 / EC2 ASG controllers?
2. How would we integrate this with our multi-tenancy approach? I'm guessing that if we use the resource managers directly (i.e. as packages) then we can pass the AWS SDK session we create in CAPA to the manager? (as an example here)
3. Are there any benefits to using the other approach you mention (i.e. generate the CAPA resource manager)?
4. It adds another set of dependencies for us, so it would be good to explore the support model and future plans for ACK

I will think some more on this.

@sedefsavas - this could be very interesting, especially if we can use the controllers as packages.

[sedefsavas]

Excited to see an ADR for this.
This has been in our minds for a long time, it would be great to start a discussion to find out answers to Richard's points.

We can go over this during our office hours, unfortunately it overlaps with ACK office hours.

[richardcase]

I can go to the ACK office hours this Monday if you are happy to host the CAPA office hours? I can then join the CAPA meeting afterwards.

[sedefsavas]

Another think we need to consider is we only make the AWS API fields that we support available to the user. When say through ACK if we create an ELB, all the fields will be available to be changed by the user.

[richardcase]

It we use it as a package then we still control what we expose to the user via our API.

[sedefsavas]

Yes, but since AWS services are now individual CRDs, users can go and change Kubernetes CR and we don't have any control over which fields could be modified as guessing ACK has its own webhooks.

[richardcase]

Interesting, i had a different view in my head.

@mikestef9 - it would be good to dive a bit deeper into the proposal. Would it be possible to arrange a chat to discuss?

[mikestef9]

We've put some thoughts together on ACK side about making controllers more easily importable into other projects aws-controllers-k8s/community#1299

[richardcase]

Great, thanks @mikestef9 . I'll take a read