kube-proxy extraArgs ignored

[fredgate]

Bug Report

kube-proxy pod runs without args specified in the machine configuration file with key cluster.proxy.extraArgs

Description

To change the bind address of kube-proxy for metrics, I edit the configuration file of each node as it :

cluster:
  ....
  proxy:
    image: 'k8s.gcr.io/kube-proxy:v1.21.6'
    extraArgs:
      metrics-bind-address: '10.80.1.1:10249'

and then apply it with talosctl apply-config -n 10.80.1.1 -f controlplane1.yaml
When I watch the pod, I see that the corresponding kube-proxy is restarted (but not recreated), and if I describe it, I see that its command line arguments did not change :

  containers:
  - name: kube-proxy
    command:
    - /usr/local/bin/kube-proxy
    - --cluster-cidr=10.244.0.0/16
    - --conntrack-max-per-core=0
    - --hostname-override=$(NODE_NAME)
    - --kubeconfig=/etc/kubernetes/kubeconfig
    - --proxy-mode=iptables

I tried to delete the pod; it was recreated but still with the same command line.
In fact the daemonset kube-proxy is not updated.

When I changed the cluster.scheduler.extraArgs and cluster.controllerManager.extraArgs the corresponding pods were successfully recreated with the new command.

Environment

• Talos version: 0.14.1
• Kubernetes version: 1.21.6
• Platform: amd64

[smira]

Talos doesn't update the manifests automatically for safety reasons. You need to run talosctl upgrade-k8s -n <controlplanenodeIP> --to v1.23.5 to make it re-apply the manifests.

[smira]

https://www.talos.dev/v1.0/kubernetes-guides/upgrading-kubernetes/

[fredgate]

Yes but I don't want to upgrade kubernetes version. Just add a parameter to the kube-proxy command line.

Can I run the upgrade-k8s with the actual version ?
talosctl upgrade-k8s -n <controlplanenodeIP> --to v1.21.6

[frezbo]

| Can I run the upgrade-k8s with the actual version ?

yes, you can do talosctl upgrade-k8s --to=<existing version> This will pick up the k8s related changes in the manifest and apply them, other component versiosn will remain the same. You could see the diff with talosctl upgrade-k8s --to=<existing version> --dry-run

[smira]

I just noticed you're on 0.14, but it already supports this flow: https://www.talos.dev/v0.14/guides/upgrading-kubernetes/

[mrwulf]

I think this is an actual (and ongoing!) issue- I'm running v1.2.2 with kubernetes 1.23.4 and am seeing the same problem. I've just run through talos apply-config, talos upgrade, and talos upgrade-k8s again and the kube-proxy daemonset is still missing all of the extraArgs.

MachineConfig:

cluster:
  proxy:
    image: k8s.gcr.io/kube-proxy:v1.24.4
    extraArgs:
      feature-gates: MixedProtocolLBService=true,EphemeralContainers=True
      metrics-bind-address: 0.0.0.0:10249

Daemonset:

      - command:
        - /usr/local/bin/kube-proxy
        - --cluster-cidr=10.244.0.0/16
        - --conntrack-max-per-core=0
        - --hostname-override=$(NODE_NAME)
        - --kubeconfig=/etc/kubernetes/kubeconfig
        - --proxy-mode=iptables

[smira]

First of all, you can do kubectl edit ds -n kube-system kube-proxy and patch it yourself, you don't have to make Talos manage this for you if you don't want it.

Second, make sure you have update machine config on all control plane nodes for extraArgs, then talosctl upgrade-k8s should pick it up.

[mrwulf]

Ah! Thanks @smira - I see from a clarifying comment you made on #5959 that talos will never update the kube-proxy manifest! I haven't fully groked WHY kube-proxy isn't updated - especially since the manifests.kubernetes.talos.dev resource has the update, but I'll dig in more

[chamburr]

@smira It seems that running talosctl upgrade-k8s does not update extraArgs for me. Looking at the code, I suspect the command would only update the image. Could this be a bug?

talos/pkg/cluster/kubernetes/daemonset.go

Lines 104 to 109 in 49e9f80

	switch ds {
	case kubeProxy:
	daemonset.Spec.Template.Spec.Containers[0].Image = fmt.Sprintf("%s:v%s", constants.KubernetesProxyImage, options.ToVersion)
	default:
	return fmt.Errorf("failed to build new image spec")
	}

[smira]

I'm sorry, why would extraArgs need to be updated?

[chamburr]

Same reason as in the description of this issue, to change the metrics bind address

[smira]

This might be confusing, we'll get a fix for it. You can always update daemonset directly with kubectl edit ds.