Make sure rollout restarts are performed for stateful sets #281
Conversation
…e annotation is set.
Codecov Report
@@ Coverage Diff @@
## master #281 +/- ##
=======================================
Coverage 75.33% 75.33%
=======================================
Files 32 32
Lines 4103 4103
=======================================
Hits 3091 3091
Misses 724 724
Partials 288 288 Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Changes look good to me overall! The only question I have is if we want to make the rollouts configurable via a cluster's spec? Specifically, would it be worth requiring that the user enables rollouts via a field in the spec so that the current behavior remains the default?
pkg/controller/controller.go
Outdated
| if sts.Spec.Template.Annotations == nil { | ||
| sts.Spec.Template.Annotations = map[string]string{} | ||
| } | ||
| sts.Spec.Template.Annotations[annotations.Rollout] = time.Now().Format(rolloutTimeFormat) |
There was a problem hiding this comment.
Nit: can you use the k8s-native time helpers? That should handle formatting for you.
There was a problem hiding this comment.
Will do. I assume k8s annotation values don't have similar restrictions as label values?
* master: Backwards compatibility when using the original update annoation with an OnDelete update strategy (#284) Add support for parallel node updates within a statefulset (#283) Support namespace ExtendedOptions in cluster spec (#282) [controller] Support multi instance placement add (#275) [gomod] Update M3DB dependency (#277) [cmd] Fix instrument package name (#280) # Conflicts: # pkg/k8sops/annotations/annotations.go
|
What's the status on this? Could it be merged? Maybe rollouts through cluster's spec could be implemented as a separate PR later? |
| } | ||
|
|
||
| now, _ := metav1.Now().MarshalQueryParameter() | ||
| sts.Spec.Template.Annotations[annotations.Rollout] = now |
There was a problem hiding this comment.
I wonder if we should only do this if the user opts in to this behavior by setting a corresponding field in the cluster spec? I just worry it might come as a surprise to someone that was expecting the old behavior of the operator not performing an update if the statefulsets hadn't changed.
There was a problem hiding this comment.
You still need to update sts with update annotation for rollout to happen, I am assuming that if someone already added this annotation they probably expect that sts should be updated. Otherwise what's the point for adding an update annotation in the first place?
There was a problem hiding this comment.
The main time I can think of that having the operator skip statefulsets that don't need to be updated has come in handy is when an update to a cluster has been interrupted partway through. For example, we would set the annotation on all statefulsets, but then if we ran into an issue we could remove the annotation from all statefulsets to ensure the operator doesn't perform any more updates while we debug any issue that we encountered. Once we've decided to continue with the update we could once again set the annotation on all statefulsets and leave it up to the operator to determine which statefulsets still need to be updated and which don't, skipping any nodes which have already been updated. It's admittedly not a very common use case but I worry that if anyone has become used to it then they may be surprised if the behavior changes here. But I'm also not convinced that we necessarily need to keep supporting this behavior. @schallert what do you think?
It would be really useful if operator would be able to initiate rolling restart for stateful set when update annotation is set even when there are no changes in cluster spec. Since this would be done only when update annotation is set, there shouldn't be any unexpected outcomes.
Currently, rollout updates won't be initiated because in order for them to be performed, spec pod's template needs to change.
This PR ensures that spec pod's template is always updated when update annotation is set.
Thanks for contributing to the M3DB Operator! We'd love to accept your contribution, but we require that most issues
outside of trivial changes such as typo corrections have an issue associated with the pull request. So please open an
issue first!