Statefulset updates with forbidden updates should re-create Statefulset

[applike-ss]

Sometimes we have to make adjustments to our configuration for druid which can include updating fields on a statefulset that is not allowed to be updated (e.g. labels).

When updating the pod labels, currently this results in the operator changing the service label selector before the statefulset has been rolled out (either because it fails and the operator just proceeds or because the order in which the service resource is updated is not correct?).

This behaviour is not ideal, so i would suggest to let the druid operator re-create statefulsets when they can't be updated to ensure that resources are in the desired state before proceeding. Ideally that would be done with the cascade=orphan option to keep the pods that are running and exchange them after the statefulset was re-created.

I have seen this behaviour in other operators, but maybe there was a reason it was not implemented here?

[AdheipSingh]

I have seen this behaviour in other operators, but maybe there was a reason it was not implemented here?

can you list few operators.

Also the reason of not adding this was to respect the k8s API error's for statefulset. The only edge case which druid operator implements is with deletion of sts using non cascading deletion is for scaling storage vertically in statefulsets.

If you can give more usecases of what exactly is the edge case to implement this. Operator cannot/will not delete sts to force in desired state, unless it needs to.

When updating the pod labels, currently this results in the operator changing the service label selector before the statefulset has been rolled out (either because it fails and the operator just proceeds or because the order in which the service resource is updated is not correct?).

can you post a config, this might be a bug, though havent seen this issue.

[applike-ss]

can you list few operators.

i actually do not remember, sorry.

can you post a config, this might be a bug, though havent seen this issue.

that should be easy to test:

• create a base config where you have one pod label for a component (sts, not deployment)
• apply it
• check the endpoints are all set
• add another label
• apply the updated config
• sts update will fail, service will be updated to select the new label(s)
• endpoints will be empty until sts got deleted and recreated (as well as pods)

[AdheipSingh]

@applike-ss yes, the steps you mentioned are not allowed by the statefulset. It allows only podSpec, rollingUpgrade and podmanagempolicy to be changed. I dont see any operator issue here.
This is what k8s statefulset controller implements. Not sure why druid operator should override this behavior and re-create.

[applike-ss]

In our case we had to adjust the labels for the pods, so that our scaling provider would not scale down an instance where druid is running.
That is because we don't want our historicals to restart as often as they did.