Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Implement an Accurate Simulator based on Functional model #322

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

Lemarais
Copy link
Contributor

@Lemarais Lemarais commented Aug 30, 2021

JIRA: NEMO-478: Implement an Accurate Simulator based on Functional model

Major changes:

  • Added run_beam_simulator.sh and Created SimulatorLauncherto run simulator.

    • In addition to the existing arguments, pass hardware specification including network information through -node_json argument
  • Created NetworkSimulator and ContainerManageSimulator to manage node and network specification.

    • NetworkSimulator manages network bandwidth and latency between nodes.
    • ContainerManageSimulator manages nodes cpu resources.
  • Created PlanSimulator and StreamingPlanSimulator to Emulate on the job level.

    • It manages the size of transferred data between tasks while triggering simulation for ExecutorSimulator
      *StreamingPlanSimulator has an internal timestamp and iterate cycles until the size of transferred data between tasks reaches equilibrium.
  • Created ExecutorSimulator and TaskHarness to Simulate execution of task.

    • TaskHarness has DataFetchSimulator and WriterSimulator.
      • DataFetchSimulator manages the number of tuples that are read from upstream tasks.
      • WriterSimulator manages the number of tuples to transfer to downstream tasks.

Minor changes to note:

Tests for the changes:

  • Tested on ubuntu machine

Other comments:

  • This is the PR for the GSoC2021 project.

Closes #GITHUB_PR_NUMBER

@Lemarais Lemarais changed the title [WIP] AccurateSimulator [WIP] Implement an Accurate Simulator based on Functional model Aug 30, 2021
@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@wonook
Copy link
Member

wonook commented Sep 5, 2021

Hi @Lemarais , can you share what tasks are left for the completion of this PR?

@Lemarais
Copy link
Contributor Author

Lemarais commented Sep 5, 2021

I tried to merge the metric-related PR First. Because until then, I cannot clear up the metric related part of simulator. Then It will be the latest version under the situation that there is no further research. Of course, I think that a review is necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants