Reconfigurator: Decommission cockroach nodes that belong to expunged cockroach omicron zones #5903
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andrewjstone
approved these changes
Jun 17, 2024
| .await | ||
| } | ||
|
|
||
| async fn invoke_cli_with_format_csv<'a, F, I, T>( |
| @@ -203,17 +229,146 @@ impl NodeStatus { | |||
| } | |||
| } | |||
|
|
|||
| // The cockroach CLI and `crdb_internal.gossip_liveness` table use a string for | |||
There was a problem hiding this comment.
Nice way to deal with this!
| @@ -435,4 +653,82 @@ mod tests { | |||
| db.cleanup().await.unwrap(); | |||
| logctx.cleanup_successful(); | |||
| } | |||
|
|
|||
| // Ensure that if `cockroach node decommission` changes in a future CRDB | |||
There was a problem hiding this comment.
I was going to suggest something like this, and here it is! Thanks for adding this.
| @@ -914,8 +914,8 @@ impl Nexus { | |||
| *mid | |||
| } | |||
|
|
|||
| pub(crate) async fn resolver(&self) -> internal_dns::resolver::Resolver { | |||
There was a problem hiding this comment.
I definitely came across this recently as well. Good fix.
| @@ -85,29 +100,235 @@ pub(crate) async fn deploy_zones( | |||
| } | |||
| } | |||
|
|
|||
| /// Idempontently perform any cleanup actions necessary for expunged zones. | |||
| /// | |||
| /// # Panics | |||
There was a problem hiding this comment.
Presumably we could instead log an error and return rather than taking out Nexus. I'm a bit concerned about taking out nexus, because it limits our debugging ability via omdb and the API. This is a pretty serious error, and hard to do accidentally though. It also leaves things in a state where invariants may be violated, which is almost certainly why a panic is the right thing to do.
Presumably in the future we will have an E2E test that actually runs the reconfigurator and will panic if the caller in execution/src/lib.rs gets changed. However, even with that test we cannot guarantee the absence of new callers doing the wrong thing.
Thinking about whether it's better to panic or log is definitely a false dichotomy also, and only exists because we don't have other options. It would be pretty great if we could instead trigger an alert and prevent the reconfigurator from running without an override rather than killing nexus, which will end up in all Nexuses eventually panicking because of the timer based activation of the executor.
There was a problem hiding this comment.
I think we can also reasonably do nothing here; just make this function filter out any non-expunged zones. It's a little less obvious, but much easier to use safely - call it with whatever zones you want, and it cleans up any expunged ones. Changed in 2d0c388
|
Testing this in a4x2 looks good. I used Confirming that the add sled / add crdb worked, we see that new sixth CRDB in DNS: and as part of the cockroach cluster: I then hyperstopped g2, and cockroach realized the node was no longer available / live: I then expunged the sled and generated a new blueprint which expunged all its zones: After setting this as the new target blueprint, the sixth CRDB was removed from the cluster: If I ask for the status of I also confirmed we had a DNS version bump that goes back to just the original 5 CRDB zones: It took about 6 minutes for cockroach to transition the node from |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a "decommission a node" endpoint to the
cockroach-adminserver, and a step to blueprint execution to clean up any expunged zones (which for now, only does anything for cockroach).Before merging, I need to run this through a4x2 and try expunging a sled running cockroach to ensure this is all working end-to-end, but I think it's close enough to be ready for review.