Iterating on a core package

[tylerschultz]

Development workflow for core packages

What we're trying to accomplish

When making changes to a core package that is involved in bootstrap, it is difficult to verify the changes will have the desired effects until the changes are committed, merged, and integrated into a build. The TKR is an output of this build process, and the TKR describes what packages shall be installed into a bootstrapped cluster. Building and injecting a modified TKR into the deploy of a cluster is a non trivial process that we've not found an easy way to do. One issue is that the Core package repo is not a part of TCE. As an alternative to building a TKR, these notes attempt to document processes for iterating on changes to a core package, building and deploying. An important part to be verified is that the changes made to a package will have the desired effect during bootstrap.

There are simpler ways to experiment with changes to a core package that are not explained here i.e. pausing a PackageInstall and editing its components on a live cluster. The described workflow attempts to more completely validate changes to a package, incorporating more tooling, and testing the boostrap lifecycle. This gives increased confidence the package changes work as desired.

3 Phases of package deployment

The deployment of Packages to a new management cluster can be described in 3 phases. Included in the packages deployed are the CNI, CPI, kapp-controller, tanzu-addons-manager, and various other controllers.

Each of the phases results in the same packages and configuration on the cluster, but each phase accomplishes it in different ways. Difficultly begins when attempting to change the packages i.e a later phase can end up overwriting an attempted modification.

Phase 1

When the tanzu CLI is invoked, templates are written to ~/.config/tanzu, then values are interpolated, producing a set of yaml documents describing how the bootstrap cluster should set up the management cluster. Some of these documents are ClusterResourceSets (CRS) which contain Deployments and DaemonSets making up the CNI, CPI, kapp-controller etc.

The CRSs are applied to the bootstrap cluster, which in turn applies the CNI, CPI, kapp-controller, etc. specified by the CRSs, to the new management cluster. Templates in ~/.config/tanzu/tkg/providers/ are vendir'd from actual packages in TCE, so that these DaemonSets and Deployments are configured similarly to the way that kapp-controller will deploy them in the next phase, but they're not yet specified by PackageInstalls, nor managed by kapp-controller.

Historically, this process is similar to how core components were installed. However, packages are layered on top of these components today.

The set of yaml documents copied from the local CLI machine also contain the TKR, which contains information that will later be used to derive PackageInstalls. It may be tempting to modify this TKR by editing its template in ~/.config/tanzu before invoking the CLI, but it is ineffectual because it is overwritten in Phase 3.

Phase 2

The tanzu-addons-manager running on the management cluster, deployed in phase 1, consumes the TKR and translates the contents into PackageInstalls. At this point kapp-controller reconciles the PackageInstalls into the DaemonSets and Deployments that were deployed in phase 1. The management-cluster has now pivoted, and now kapp-controller and tanzu-addons-manager are now managing the CNI and CPI as PackageInstalls.

Phase 3

In this phase, the tkr-controller-manager is now able to start, now that the CNI is initialized and running. The tkr-controller-manager is configured to download the TKR image bundle from an image repository. Once retrieved, it overwrites the TKR written in the first phase. The downloaded TKR, of course, does not contain any of the modifications that may have been done on the machine where the CLI is running. The tanzu-addons-manager then rewrites the PackageInstalls written in phase 2 so that the cluster conforms to the downloaded TKR.

A workflow we came up with

The following is an example of the process that can be use to iterate on a modification to the antrea cni core package. This change is to add an annotation to the antrea DaemonSet. Also, for the sake of this example, the assumption is that the change to the DaemonSet will have some effect on the behavior of the package/cluster.

The workflow described takes advantage of of one the behaviors of the tanzu-addons-manager. One of its responsibilities is to write PackageInstalls. The PackageInstalls the tanzu-addons-manager writes do not have specific version information. kapp-controller reconciles the PackageInstalls and uses the highest versioned available Package matching the PackageInstall.

On an existing cluster, perform the following steps to iterate on building and deploying a modified package version on a running cluster.

1. Make changes to the antrea package in TCE, i.e. add a new annotation to the DaemonSet.
2. Build and push imgpkgBundle

   make push-package OCI_REGISTRY=repo.mycompany.com/my-team/tce PACKAGE=antrea VERSION=0.13.3`
   

3. Create a newer version of the antrea package using the imgpkgBundle created in the last step. While there is a package located in the TCE repository for Antrea, it doesn't have the same name as the actual Antrea package used in a cluster.
   1. Instead, pull down the package yaml from the running cluster.

      kubectl get packages -n tkg-system antrea.tanzu.vmware.com.0.13.3+vmware.1-tkg.1 -o yaml > antrea.yaml
      

   2. Open the antrea.yaml file.
   3. Remove the runtime information added to the package metadata by controllers.
   4. Update spec.refName and spec.version of the package with some higher dev version: e.g 0.13.4-dev. Perhaps there is a better versioning scheme to use for development packages that doesn't bump the minor, but this is what worked.
   5. Update the imgpkgBundle.image to use the image just pushed in the previous step.
4. Apply the new dev package

   kubectl apply -f antrea.yaml
   

5. Wait for the PackageInstall to reconcile and verify that the modifications are working as desired.

Deploy a cluster with the updated package

Once satisfied with the package modifications on running cluster, deploy a new management cluster with the updated package, so that the behavior of the modified package can be verified to work at bootstrap time. To get the updated Package onto a newly created cluster, create a ClusterResourceSet and Secret.

1. Ensure the providers are populated in the ~/.config/tanzu directory

   tanzu management-cluster create -p dev -i docker whatever --dry-run
   

2. Prevent the tanzu cli from overwriting changes that will be made to the providers yaml:

   export SUPPRESS_PROVIDERS_UPDATE=true
   

3. Copy/vendir sync the changes made to the package to the providers:

   ~/.config/tanzu/tkg/providers/ytt/vendir/cni/_ytt_lib
   

   This ensures that phase 1 applies the modified package.
4. Create a ClusterReourceSet and Secret combo that applies the updated package on the management cluster. This ensures phase 2/3 doesn't overwrite the modified package with the released version. Note that the contents of the Package reference the imgpkgBundle built and pushed in the previous steps. Also note the spec.version is higher than what is specified in the TKR. In this example, create the following yaml file at ~/.config/tanzu/tkg/providers/ytt/04_user_customizations/antra_dev_package.yaml with this content:

   #@ load("@ytt:data", "data")
   #@ load("@ytt:yaml", "yaml")
   
   #@ def antrea_dev_package():
   ---
   apiVersion: data.packaging.carvel.dev/v1alpha1
   kind: Package
   metadata:
     name: antrea.tanzu.vmware.com.0.13.4-dev
     namespace: tkg-system
   spec:
     licenses:
     - 'VMware’s End User License Agreement (Underlying OSS license: Apache License 2.0)'
     refName: antrea.tanzu.vmware.com
     releaseNotes: antrea 0.13.3 https:<!--github.com/antrea-io/antrea/releases/tag/v0.13.3 -->
     releasedAt: "2021-08-30T15:42:12Z"
     template:
       spec:
         deploy:
         - kapp:
             rawOptions:
             - --wait-timeout=30s
             - --kube-api-qps=20
             - --kube-api-burst=30
         fetch:
         - imgpkgBundle:
             image: harbor-repo.vmware.com/kokoni/tce/antrea@sha256:d362e2de75b4adde9eced9725a481cdf44f356c89d87a459b4c41d5013f429a3
         template:
         - ytt:
             ignoreUnknownComments: true
             paths:
             - config/
         - kbld:
             paths:
             - '-'
             - .imgpkg/images.yml
     valuesSchema:
       openAPIv3:
         title: antrea.tanzu.vmware.com.0.13.3+vmware.1-tkg.1 values schema
     version: 0.13.4-dev
   #@ end
   
   ---
   apiVersion: addons.cluster.x-k8s.io/v1alpha3
   kind: ClusterResourceSet
   metadata:
     name: #@ "{}-antrea-dev-package".format(data.values.CLUSTER_NAME)
     labels:
       cluster.x-k8s.io/cluster-name: #@ data.values.CLUSTER_NAME
     annotations:
       tkg.tanzu.vmware.com/addon-type: "cni/antrea"
   spec:
     strategy: "ApplyOnce"
     clusterSelector:
       matchLabels:
         tkg.tanzu.vmware.com/cluster-name: #@ data.values.CLUSTER_NAME
     resources:
       - name: #@ "{}-antrea-crs-dev-package".format(data.values.CLUSTER_NAME)
         kind: Secret
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: #@ "{}-antrea-crs-dev-package".format(data.values.CLUSTER_NAME)
     annotations:
       tkg.tanzu.vmware.com/addon-type: "cni/antrea"
   type: addons.cluster.x-k8s.io/resource-set
   stringData:
     value: #@ yaml.encode(antrea_dev_package())
   

5. Create the management cluster, remembering to use the SUPPRESS_PROVIDERS_UPDATE environment variable to prevent the cli from overwriting the customizations.
6. Wait for the cluster to start and verify the changes have the desired effect.

Questions we have

• Why is it using tkg core packages and not tce core packages? Is that expected to change?
• Is it safe to rely on kapp-controller picking up new versions like we are now?
• What are others doing when working on core packages today?
• Any questions or feedback about what we've come up with above? Any improvements to suggest?
• Would it be valuable to document this workflow somewhere other than this discussion?

[ankeesler]

first of all - thank you for writing this up. i work on the pinniped package and definitely have this same development workflow conundrum.

---

What are others doing when working on core packages today?

i wrote a script that pushes the in-tree pinniped package and patches it into the cluster. i do this (roughly) by:

1. pushing my own PackageRepository that points to my in-development pushed pinniped package
2. editing the TKr BOM for my cluster with the pushed PackageRepository url
3. waiting for the PackageRepository and pinniped App to reconcile

this script is extremely janky. i think i like your way better. i am going to try it out.

---

Would it be valuable to document this workflow somewhere other than this discussion?

i wonder if tce could offer a one-shot script or Makefile target to do what you outlined above 😏

[blc1996]

Thanks for the write up! The addons team will think about all the challenges faced by the developers and try to come up with something more developer friendly during code orange.

The current situation is mainly caused by the following two considerations:

• We wanted to enable the ability to customize downstream packages, i.e. downstream packages could be different from TCE packages in some ways by adding overlays (Although the upstream and downstream core packages today are the mostly the same). This introduced the two layer structure of TCE and tkg-packages.
• We need to use downstream images in the packages we built with tkg-packages. We would not know the exact info of downstream images before we have the buildinfo provided by Bolt. This introduces Bolt as another layer.

We understand the pain points and will circle back when we have something solid.

To answer the questions

• Why is it using tkg core packages and not tce core packages? Is that expected to change?
  • We want to keep the ability to customize downstream packages as mentioned above.
• Is it safe to rely on kapp-controller picking up new versions like we are now?
  • We didn't explicitly specify version info because we want to keep the info in TKR minimal and static. If we need to include version info, we would need to update TKR addons section each release. We assume users will not create packages named as packageName.tanzu.vmware.com. Personally, I think it should be fine, if users decide to break the glass, it's their own decision.
• What are others doing when working on core packages today?
  • As you mentioned in the write up, locally testing your changes could involve many manual steps, and we definitely want to improve it. Another way is to utilize Bolt, have changes in your own branch and don't merge yet, create a MR in Bolt and it will help test out. We are looking to automate the process in the near future.
• Any questions or feedback about what we've come up with above? Any improvements to suggest?
  • My initial thought would be building some package pipelines that reduces the hassle. We don't have any solid solutions yet due to the 2 reasons I mentioned above, but will definitely circle back after some discussions with the team.
• Would it be valuable to document this workflow somewhere other than this discussion?
  • Sure, I'll summarize it and include it in the README of tkg-packages.