Helix Node/Instance Swap

[zpinto]

Helix Node/Instance Swap

Implementation for N -> N + 1 -> N instance swap for all replicas on the instance

1. Bootstrap new replicas on the SWAP_IN node: New replicas are created for all the partitions hosted on the node to be replaced. These new replicas are then assigned and bootstrapped on the SWAP_IN-node.
2. Remove replicas from the SWAP_OUT node: After successfully bootstrapping all the new replicas, the SWAP_OUT node will have its replicas dropped. Waiting until all replicas bootstrap on the SWAP_IN node will ensure we maintain all partitions desired replication factor.
3. Do not populate SWAP_IN replicas in routing tables until SWAP is completed. (Avoid spectator serving traffic to SWAP_IN node)

Requirements

• Swapping of a node requires “transaction”. When bootstrapping new replicas on the SWAP_IN host, replica count of all partitions should be N + 1 (and should be in topState or secondTopState) before SWAP_OUT node can begin dropping replicas (which will return replica count to N)

Assumptions

When a node is being swapped, the following criteria must be met for the SWAP_IN node:

1. The cluster must have TOPOLOGY with a FAULT_ZONE_TYPE and EndNodeType:

• EndNodeType is the last key in the TOPOLOGY. The corresponding value is considered to be the logicalId. This is necessary as logicalId will need to be used for deterministic placement when the SWAP_IN node is introduced as an assignable node.

2. DOMAIN of SWAP_IN and SWAP_OUT must have identical fault zone and logicalId.
3. This will be further explained in the topology section
4. If WAGED is enabled in the cluster, the INSTANCE_CAPACITY_MAP should match exactly.

Without these assumptions, it is not possible to swap a node as this would not be considered the best possible assignment for the cluster. If these assumptions cannot be met, this functionality should not be utilized. Instead the old node should use the Node Evacuation flow. A new instance can be added in the ENABLED state either before or after to replenish the capacity of removing the old node.

Execution

• HelixAdmin.setInstanceOperation and addInstance sanity checks for SWAP_IN & SWAP_OUT
• Create HelixAdmin.canCompleteSwap and completeSwapIfPossible(returns if swap finished)
• Refactor BaseControllerDataProvider only allow AssignableNodes (either SWAP_OUT or SWAP_IN) and update all places with either getAssignableInstances or getAllInstances.
• Refactor BestPossibleStateCalcStage to add SWAP_IN node with correct states to stateMaps containing SWAP_OUT node following partitionAssignment and partitionState calculation.
• Refactor WAGED to do assignment based on logicalId instead of instanceName.
• Refactor routing provider to not include replicas from SWAP_IN nodes in routing tables

[zpinto]

This issue has now been closed with the merging of Application Cluster Manager branch. #2714