Document port_forward usage with charts that deploy multiple pods

[nathanperkins]

I'm deploying prometheus-community/kube-prometheus-stack with multiple pods and it's not clear how I should use port_forwards to forward ports from specific pods.

[ns-rboyd]

@nathanperkins did you ever find a solution to this?

[JustinLex]

I think this is functionality missing from helm_resource().

Right now, the port_forwards argument on helm_resource() passes directly to k8s_resource()'s port_forwards argument underneath. See the PR that added the port forward functionality: #356

Because of this, there is no way to specify which pod should be port forwarded. I think k8s_resource() just picks the first pod from the set, from what I've seen happen.

The best workaround for this is to add your own k8s_resource() to forward the port, but use discovery_strategy='selectors-only' and the extra_pod_selectors argument to filter to only the pod you want.

Here's a working example for the Pyroscope operator:

helm_resource('pyroscope',
              chart='grafana/pyroscope',
              release_name='pyroscope',
              namespace="pyroscope",
              flags=[
                  '--create-namespace'
              ])
k8s_resource(workload='pyroscope',
             extra_pod_selectors={'statefulset.kubernetes.io/pod-name': 'pyroscope-0'},
             discovery_strategy='selectors-only',
             port_forwards="4040:4040")

This port-forwards the pyroscope pod, instead of picking the default pyroscope-agent pod.

Note that I also tried using k8s_resource() with the objects argument, but that doesn't work because objects doesn't work with k8s_custom_deploy(). See tilt-dev/tilt#5406. extra_pod_selectors is the only way to do this, because it is aware of k8s_custom_deploy(). See tilt-dev/tilt#5774.

Hope that helps anybody else encountering this problem!

[austin-kauble]

The above solution does result in the port being forwarded to the correct pod, but it has a secondary effect of de-associating the other pods from the resource for other functionality. In my case, the other deployments (which sometimes share an image with the main deployment, and are associated as such using image_deps and image_keys) will no longer receive live_updates and are no longer included in the merged log stream for that resource.

I've so far failed to find any solution when working with helm_resource which allows me to get port forwards landed at the correct pod while still keeping other functionality with the other pods. There doesn't seem to be any reasonable way to separate out the deployments into separate resources within the confines of k8s_custom_deploy (tilt-dev/tilt#5406)