Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix partitionAssignment API failing due to NPE when no resource config #2653

Merged
merged 1 commit into from
Oct 18, 2023
Merged

Fix partitionAssignment API failing due to NPE when no resource config #2653

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions ...ava/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizerAccessor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -359,8 +359,10 @@ private void computeWagedAssignmentResult(List<IdealState> wagedResourceIdealSta
ConfigAccessor cfgAccessor = getConfigAccessor();
List<ResourceConfig> wagedResourceConfigs = new ArrayList<>();
for (IdealState idealState : wagedResourceIdealState) {
wagedResourceConfigs
.add(cfgAccessor.getResourceConfig(clusterId, idealState.getResourceName()));
ResourceConfig resourceConfig = cfgAccessor.getResourceConfig(clusterId, idealState.getResourceName());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we not check if the resource is WAGED enabled?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check for WAGED resources happens in the function (computeOptimalAssignmentForResources) that calls this function (computeWagedAssignmentResult). My understanding is there is an assumption that the only resources passed to it are waged resources.

The calling function is computeOptimalAssignmentForResources() in ResourceAssignmentOptimizerAccessor line 254

      // Compute all Waged resources in a batch later.
      if (idealState.getRebalancerClassName() != null && idealState.getRebalancerClassName()
          .equals(WagedRebalancer.class.getName())) {
        wagedResourceIdealState.add(idealState);
        continue;
      }

and then:

    if (!wagedResourceIdealState.isEmpty()) {
      computeWagedAssignmentResult(wagedResourceIdealState, inputFields, clusterState, clusterId,
          result);
    }

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aha, so in some way, user has configured resource as WAGED but hasn't provided WAGED resource config? isn't this user error? we can prevent null pointer exception but shouldn't user know that the config is wrong too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically waged rebalance "works" for clusters without resource configs. This is actually the current set up of our super clusters. Using waged without any resource configs or relevant instance capacity configs.

I'm not 100% on this part, but I believe if there are no resource configs then the score calculated for each node will be 0 and tiebreak will go to the node without any resources assigned to it. I don't think there's a guarantee of evenness if there's no resource and instance capacity configs, but it will guarantee that each node will have at least 1 replica assigned to it (given # replicas > # nodes)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The NPE only occurs for the partitionAssignment API, but the actual controller rebalance algorithm works fine in the same scenario

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that means, we fill out the value using some default values in Waged workflow but not in this workflow.
Please look at: WagedValidationUtil::validateAndGetPartitionCapacity.

But your fix should be good too.

if (resourceConfig != null) {
wagedResourceConfigs.add(resourceConfig);
}
}

Map<String, ResourceAssignment> wagedAssignmentResult;
Expand Down
Loading