[Nemo-472] Implement Runtime Components of Hierarchical Aggregation

[Kangji]

JIRA: NEMO-472: Fix and Implement Hierarchical Aggregation

Major changes:
Continuing from #318

• Implemented new stage execution property, TaskIndexToExecutorIDProperty, that stores task allocation information.
  • Information will be stored in runtime, when TaskDispatcher dispatches a task to any executor.
  • It will be used when upstream tasks of partial shuffle communication channel decide where to emit data.
• Implemented data transfer logic on partial shuffle communication channel.
  • Data from each executor must be transferred to nearby executors only.
• Implemented constraint when scheduling intermediate combine stage(the stage consists of intermediate accumulator vertex), so that the tasks are evenly distributed across all node groups. Each group represents physically close nodes.

Minor changes to note:

• None

Tests for the changes:

• Tested on my Mac and ubuntu machine

Other comments:

• Also see [NEMO-472] Implement Intermediate Combine #318, which implemented optimization pass that inserts intermediate accumulator vertex.
• [TODO] Currently the implemented scheduling constraint simply distributes tasks in the intermediate combine stage evenly to each executor groups. It is expected to be updated, related to WAN aware scheduling policy.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information