Change stoppable to perform min active check sequentially in mz

[GrantPSpencer]

Issues

• My PR addresses the following Helix issues and references them in the PR description:

#2885
Helix stoppable does not account for multiple replicas in the same fault zone. This can occur during instance operations such as swap but also as a result of the issue linked above.

Description

• Here are some details about my PR, including screenshots of any UI changes:

Currently, the batch stoppable check parallelizes the performHelixOwnInstanceCheck, which includes the min active replica check. This parallelization works on the assumption that a single partition will not have more than 2 replicas hosted in 1 fault zone. This assumption is not fully correct as n+1 transitions can result in 2 replicas in the same fault zone, as well instance operations such as SWAP.

The parallelization of the other helix health checks can be kept to not lose out on that optimization. The min active check can then be done sequentially for each instance.

There is a potential for optimizing the min active replica check. Currently it is done per instance and reads the externalView and idealStates for each resource from ZK. It could be optimized to read the EVs and IS's once and use those for all instances the check is being ran on. However, I did not want to complicate this first draft of the PR with that refactoring.

Tests

• The following tests are written for this issue:

TestInstancesAccessor.java#testMultipleReplicasInSameMZ

• The following is the result of the "mvn test" command on the appropriate module:

$ mvn test -o -Dtest=TestPerInstanceAccessor,TestInstancesAccessor,TestMaintenanceManagementService -pl=helix-rest

[INFO] Results:
[INFO] 
[INFO] Tests run: 45, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:31 min
[INFO] Finished at: 2024-08-15T20:43:09-07:00
[INFO] ------------------------------------------------------------------------

Changes that Break Backward Compatibility (Optional)

• My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:

Not a significant incompatibility, but the STOPPABLE_CHECK_LIST no longer contains the min_active_replica check as it is done separately.

Commits

• My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Code Quality

• My diff has been formatted using helix-style.xml
  (helix-style-intellij.xml if IntelliJ IDE is used)

[junkaixue]

We should be consistent with behavior. What if user does not specific MZs?

[GrantPSpencer]

We should be consistent with behavior. What if user does not specific MZs?

If there is no topology set up for the cluster, I believe batch stoppable will actually return an empty.
My current assumption is that this is due to the single instance stoppable check calling getZoneBasedInstances which seems will return an empty list when no topology set.
For the batch instances check, it will return empty because it relies on _orderOfZones not being empty.

the per instance stoppable check should work normally in this case.

[junkaixue]

We should be consistent with behavior. What if user does not specific MZs?

If there is no topology set up for the cluster, I believe batch stoppable will actually return an empty. My current assumption is that this is due to the single instance stoppable check calling getZoneBasedInstances which seems will return an empty list when no topology set. For the batch instances check, it will return empty because it relies on _orderOfZones not being empty.

the per instance stoppable check should work normally in this case.

If this PR just make the fix without adding the non-zone based support, let's create ticket and making a TODO item in the code to make sure that support will happen later.

[MarkGaox]

Same as above, by placing the min active replica check before submitting the asynchronous checks, there is no longer a need to check the future and get the value.

[zpinto]

LGTM

[GrantPSpencer]

Pull request approved by: @MarkGaox @zpinto
Commit message: Batch stoppable check now performs min_active check sequentially in zone

[GrantPSpencer]

We should be consistent with behavior. What if user does not specific MZs?

If there is no topology set up for the cluster, I believe batch stoppable will actually return an empty. My current assumption is that this is due to the single instance stoppable check calling getZoneBasedInstances which seems will return an empty list when no topology set. For the batch instances check, it will return empty because it relies on _orderOfZones not being empty.
the per instance stoppable check should work normally in this case.

If this PR just make the fix without adding the non-zone based support, let's create ticket and making a TODO item in the code to make sure that support will happen later.

Created issue to track this
#2893

[junkaixue]

Yeah. Please add it to TODO comment in the code as well.

[GrantPSpencer]

Yeah. Please add it to TODO comment in the code as well.

Added comment in InstancesAccessor class

      // TODO: Add support for clusters that do not have topology set up.
      // Issue #2893: https://github.com/apache/helix/issues/2893