How do I convert a control plane node into a worker node without resetting it?

[Aex12]

Hi!

My current setup looks like this:

• Node A (Control Plane with allowSchedulingOnControlPlanes: true)
• Node B (Control Plane without allowSchedulingOnControlPlanes)
• Node C (Control Plane without allowSchedulingOnControlPlanes)

Node A have some deployments using local-path-provisioner PVCs, that I'll like to keep when doing the conversion.

Now, I'll like to add a Node D, make it a ControlPlane without allowSchedule, and convert Node A into a worker node while keeping its /var partition intact (to avoid losing data from local-path-provisioner). Later on, I'm planing on adding more worker nodes, but that shouldn't be problematic.

I've searched previous discussions, and found this: #7187
It doesnt exactly applies to my setup as I'm not running Rook Ceph, which should be able to recover data from the other working nodes while doing the conversion, but still provided some useful data to me.

This is the plan I've in mind, but I'll like to confirm with the community it's okay and it wont cause data loss or cluster failure:

• Snapshot the etcd database, just in case something goes wrong.
• Generate a worker.yaml configuration for Node A using secrets.yaml, and apply it so that it stops scheduling control plane pods on it.
• Update DNS records to remove Node A from the control plane domain (I use a domain to distribute load between all control planes, using multiple A records)
• Run kubectl label node <Node A name> node-role.kubernetes.io/control-plane- to remove the control plane label from it.
• Run talosctl etcd remove-member <Node A ID> to remove Node A from etcd cluster.
• Setup the new control plane node and apply its controlplane.yaml configuration
• Patch all control planes machineconfig so that cluster.apiServer.certSANs is updated to remove Node A from the list and add Node D to it
• Update DNS records to add Node D to the external control plane domain

Is this plan ok or am I missing something?

[smira]

In general, we don't document/test procedure like this which doesn't involve full machine reset.

I would probably scale up before scaling down, so going 3->4->3 vs. 3->2->3 (for etcd quorum reasons).

Patch all control planes machineconfig so that cluster.apiServer.certSANs is updated to remove Node A from the list and add Node D to it

This is never needed, it should just contain the DNS name in your case, but probably doesn't even need that if the controlplane endpoint is configured correctly to point to the domain name.

[Aex12]

In general, we don't document/test procedure like this which doesn't involve full machine reset.

I'm aware this procedure is not officially supported by the Talos team, I'm just trying to make sure it's still viable and my steps made sense.

I would probably scale up before scaling down, so going 3->4->3 vs. 3->2->3 (for etcd quorum reasons).

My reasoning for scaling down first, then scaling up is that both a 3 nodes, and a 2 nodes etcd cluster needs 2 machines up for quorum to be achieved. A 4 node cluster would need 3 nodes to be up, so there is a lowest chance of quorum being lost if scaling down first, since there are less nodes that could fail (2 vs 3).

This is never needed, it should just contain the DNS name in your case, but probably doesn't even need that if the controlplane endpoint is configured correctly to point to the domain name.

For some reason, I was previously getting TLS errors when individual nodes weren't part of the certSANs. Maybe I got my config wrong at some point and a node didnt have the shared dns record as the endpoint. I'll try that and see if everything works alright.

[smira]

My reasoning for scaling down first, then scaling up is that both a 3 nodes, and a 2 nodes etcd cluster needs 2 machines up for quorum to be achieved. A 4 node cluster would need 3 nodes to be up, so there is a lowest chance of quorum being lost if scaling down first, since there are less nodes that could fail (2 vs 3).

I don't think I can understand this. When you scale down 3->2, your quorum is 2, so any additional failure would be catastrophic. If you scale up 3->4, you can still lose a single machine and be fine.