Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add reduce_scatter t3k perf to pipeline #14950

Merged
merged 2 commits into from
Nov 14, 2024
Merged

Conversation

Aswinmcw
Copy link
Contributor

Ticket

#14406

Adds reduce_scatter t3k perf to pipeline

https://github.com/tenstorrent/tt-metal/actions/runs/11795073011/job/32854208345

Checklist

  • Post commit CI passes
  • Blackhole Post commit (if applicable)
  • Model regression CI testing passes (if applicable)
  • Device performance regression CI testing passes (if applicable)
  • New/Existing tests provide coverage for changes

@Aswinmcw Aswinmcw marked this pull request as ready for review November 13, 2024 15:34
@@ -22,7 +22,8 @@ jobs:
{ name: "t3k LLM llama3 model perf tests", model: "llama3", model-type: "LLM", arch: wormhole_b0, cmd: run_t3000_llama3_tests, timeout: 60, owner_id: U03PUAKE719}, # Miguel Tairum
{ name: "t3k LLM falcon40b model perf tests", model: "falcon40b", model-type: "LLM", arch: wormhole_b0, cmd: run_t3000_falcon40b_tests, timeout: 75, owner_id: U053W15B6JF}, # Djordje Ivanovic
{ name: "t3k CNN resnet50 model perf tests", model: "resnet50", model-type: "CNN", arch: wormhole_b0, cmd: run_t3000_resnet50_tests, timeout: 75, owner_id: U013121KDH9}, # Austin Ho
{ name: "t3k CCL all_gather perf tests", arch: wormhole_b0, cmd: run_t3000_ccl_all_gather_perf_tests, timeout: 75, tracy: true, owner_id: ULMEPM2MA}, # Sean Nijjar
{ name: "t3k CCL all_gather perf tests", model: "all_gather", arch: wormhole_b0, cmd: run_t3000_ccl_all_gather_perf_tests, timeout: 75, tracy: true, owner_id: ULMEPM2MA}, # Sean Nijjar
{ name: "t3k CCL reduce_scatter perf tests", model: "reduce_scatter", arch: wormhole_b0, cmd: run_t3000_ccl_reduce_scatter_perf_tests, timeout: 75, tracy: true, owner_id: ULMEPM2MA}, # Sean Nijjar
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since these are the same owners and both all_gather and reduce_scatter tests take < 3mins.
Would it be a good idea to combine these into the same job?

The overhead to setup the same job is ~50s, and the time to run each individual CCL job is ~1mins

@Aswinmcw Aswinmcw force-pushed the Aswinmcw/t3k_reduce_scatter_ci branch 2 times, most recently from e803b9c to 39d58a9 Compare November 14, 2024 06:10
@Aswinmcw Aswinmcw force-pushed the Aswinmcw/t3k_reduce_scatter_ci branch from 39d58a9 to 96d2b13 Compare November 14, 2024 07:05
@Aswinmcw Aswinmcw merged commit 4dc379c into main Nov 14, 2024
29 of 37 checks passed
@Aswinmcw Aswinmcw deleted the Aswinmcw/t3k_reduce_scatter_ci branch November 14, 2024 10:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants