Automatic replacement of MC NotReady nodes

[whites11]

Similarly to what we do on WCs, we should have an automatic replacement feature of MC nodes when they get not ready

[T-Kukawka]

Ideal approach would be to have an operator that recycles nodes regardless of the provider. This could be used for CAPI then as well. Operator would have to be HA as the node it is running on can be affected.

[whites11]

@giantswarm/team-hydra is there any feature like this implemented in the CAPI world?

[alex-dabija]

CAPI does have the concept of health checks for machines. Unfortunately, it only works with machine deployments and not with machine pools because a machine resource CR (or machine set CR) needs to be present.

[T-Kukawka]

Waiting for CAPA to stabilize implementation of machinepools/machinedeployments

[whites11]

did some progress in giantnetes-terraform 14.12.0.
We run a script in the masters that take the node down if services are down

[nprokopic]

In CAPI all MCs will also be WCs at the same time (with all CAPI CRs and managed my CAPI), so it will be possible to use MachineHealthChecks (however well/bad they work).

Currently it should be possible to use MachineHealthChecks for control plane nodes and workers created with machine deployments (CAPG and CAPZ, and I think all onprem providers).

When it comes to MachinePool support (currently used in CAPA), there is an open PR to implement MachinePoolMachine, which should then make it possible to implement MachineHealthChecks for MachinePool as well.

[fiunchinho]

CAPA is currently using it for the control planes nodes.

[fiunchinho]

Nowadays I think this is a duplicate of https://github.com/giantswarm/giantswarm/issues/28006. I think I'd close this one @T-Kukawka

[T-Kukawka]

agreed, closing