Convert controlplane node to worker

[bcspragu]

Hello there,

I have a cluster of three nodes, all are type controlplane and all have allowSchedulingOnControlPlanes set to true. This ended up causing some of the expected problems, like pods fighting for resources with the control plane infra, so I want to split the cluster into worker node sand controlplane nodes. The main problem is that the nodes are all part of a Rook cluster and spread data across them.

My plan looked like this:

1. Spin up a new control plane node
2. One-by-one, migrate the existing control plane nodes to be worker nodes, by apply-config-ing a worker config onto the node

• Rook is configured for data replication to 2 nodes, and error correction codes on a third, so theoretically, the new worker node should come up and fix any data problems, and then I can move onto the next node.

The Problem: I apply-config'd the worker config to an existing controlplane node, and after reboot, it still appears to be a controlplane node. It has node-role.kubernetes.io/control-plane: "" and—oh wait actually nevermind, it looks like kube-apiserver isn't running on the node, so I think it might have worked! I guess my new question is: how do I change the role from control-plane to worker, it doesn't appear to be as simple as changing the metadata.annotations.

[smira]

Worker nodes can't control special node labels, so you need to remove a node label yourself manually using kubectl.

[smira]

But in general such conversion is not supported the way you described, as the node will not clean up etcd members, etc.

A better way might be to do talosctl reset --reboot for a single node, wait for it to reboot in maintenance mode, apply-config, wait for Rook to recover and become healthy, continue with the next node.

[bcspragu]

Ah, that explains why my etcd cluster went unhealthy and I've since broken ~everything! Good to know! If I ever get this cluster functioning again, I'll try doing it that way, thanks!

[smira]

you can try removing "dead" members with talosctl etcd remove-member

[bcspragu]

I'll definitely try that when I attempt the migration again, thanks! For now, I was able to get the cluster functioning by reverting my changes (e.g. making the nodes control planes again) and then following your etcd snapshot/restore disaster docs. That brought back the cluster + all configuration (e.g. all K8s resources).

Now I'm just trying to recover my Rook cluster, which didn't survive the talosctl ... reset --graceful=false --reboot --system-labels-to-wipe=EPHEMERAL, but I think (hope?) the data is still intact