Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Karpenter CRD upgrade support via karpenter-crd chart #191

Open
armujahid opened this issue Jun 22, 2023 · 3 comments
Open

Karpenter CRD upgrade support via karpenter-crd chart #191

armujahid opened this issue Jun 22, 2023 · 3 comments
Labels
add-on enhancement New feature or request

Comments

@armujahid
Copy link

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Issue

The Karpenter helm chart does not manage the lifecycle of CRDs; it will only install the CRD during the first installation of the helm chart. Subsequent chart upgrades will not add or remove CRDs, even if the CRDs have changed.

Describe the solution you would like

Karpenter now provides a separate karpenter-crd https://github.com/aws/karpenter/tree/main/charts/karpenter-crd helm chart to manage the lifecycle of CRDs https://karpenter.sh/docs/upgrade-guide/#custom-resource-definition-crd-upgrades. I think that we should also include this in the Karpenter addon.

Alternatives

Since upgrading the CRDs is a risky process, users can manually check release notes and upgrade CRDs if required. This is what we currently do.

kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/v0.28.0/pkg/apis/crds/karpenter.sh_provisioners.yaml
kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/v0.28.0/pkg/apis/crds/karpenter.sh_machines.yaml
kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/v0.28.0/pkg/apis/crds/karpenter.k8s.aws_awsnodetemplates.yaml

Additional context

@askulkarni2
Copy link
Contributor

@armujahid thanks for the issue! We should add this.

@askulkarni2 askulkarni2 added the enhancement New feature or request label Jun 22, 2023
@mikesplain
Copy link

mikesplain commented Jul 14, 2023

Frankly, couldn't this go for any resource in this repo or users of https://github.com/aws-ia/terraform-aws-eks-blueprints-addon that use a CRD? We have a separate process that installs the karpenter crd chart for us and it works well but we don't have a good solution for the CRDs for things like AWS Load Balancer Controller.

@askulkarni2 Where's the right spot for an issue more generalized around "CRD Management" for AWS supplied Addons?

Seems like reasonable solutions would be something along the line of:

  • Skipping the CRD install within helm charts since it doesn't keep them up to date
  • Managing the install of either "a helm chart of CRDS" like karpenter or automatically pulling the correct version of CRDs from the dependencies repository or helm chart.
  • The Addons repository can supply those values for known AWS dependencies but users of the https://github.com/aws-ia/terraform-aws-eks-blueprints-addon will have to determine them for themselves (which sounds reasonable to me).

@askulkarni2
Copy link
Contributor

Besides the github issues of the individual add-on repos, https://github.com/aws/containers-roadmap/issues would be the right place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
add-on enhancement New feature or request
Projects
Status: No status
Development

No branches or pull requests

3 participants