Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Collect CUDA/CPU profiling info into result sheets. #5921

Merged
merged 2 commits into from
Nov 28, 2023
Merged

Collect CUDA/CPU profiling info into result sheets. #5921

merged 2 commits into from
Nov 28, 2023

Conversation

golechwierowicz
Copy link
Collaborator

@golechwierowicz golechwierowicz commented Nov 23, 2023 •
edited
Loading

This PR:

  1. Adds CUDA/CPU collection capabilties to the script.
  2. Modifies result_analyzer.py to analyze newly collected results.
  3. Moves CUDA synchronize/XLA device synchronize into the profiler.
  4. Fixes list typing for Python 3.8+.

Tested with command:

python3 xla/benchmarks/experiment_runner.py --dynamo=openxla --xla=PJRT --test=train --filter=basic_gnn_gcn$ --suite-name=torchbench --accelerator=cuda --progress-bar --output-dirname=/tmp/output --repeat=2 --print-subprocess --no-resume --profile-cuda-cpu-collect --profile-cuda python3 xla/benchmarks/result_analyzer.py --output-dir=/tmp/output

Collect CUDA/CPU profiling info into result sheets.
80ca0fc 
This PR:
0. Adds CUDA/CPU collection capabilties to the script.
1. Modifies result_analyzer.py to analyze newly collected results.
2. Moves CUDA synchronize/XLA device synchronize into the profiler.
3. Fixes list typing for Python 3.8+.

Tested with command:
python3 xla/benchmarks/experiment_runner.py --dynamo=openxla --xla=PJRT --test=train --filter=basic_gnn_gcn$ --suite-name=torchbench --accelerator=cuda --progress-bar --output-dirname=/tmp/output --repeat=2 --print-subprocess --no-resume --profile-cuda-cpu-collect --profile-cuda
python3 xla/benchmarks/result_analyzer.py --output-dir=/tmp/output
frgossen
frgossen reviewed Nov 24, 2023
View reviewed changes
Copy link
Collaborator

@frgossen frgossen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, ty!

benchmarks/experiment_runner.py Outdated Show resolved Hide resolved
frgossen
frgossen approved these changes Nov 24, 2023
View reviewed changes
Copy link
Collaborator

@frgossen frgossen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, one comment.

@golechwierowicz golechwierowicz force-pushed the olechwierowicz/add_profiling_data branch from 53b80a7 to dde4ea4 Compare November 27, 2023 15:51
@golechwierowicz golechwierowicz force-pushed the olechwierowicz/add_profiling_data branch from dde4ea4 to ac84ed1 Compare November 27, 2023 15:52
vanbasten23
vanbasten23 reviewed Nov 28, 2023
View reviewed changes
benchmarks/experiment_runner.py
)
return

kernel_dump = prof.profiler.total_average()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious where is this total_average() defined?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/pytorch/pytorch/blob/00412e6dfacdec5a8508c841b3ea0846388dc872/torch/autograd/profiler_util.py#L339-L350

@golechwierowicz golechwierowicz merged commit c03afb1 into master Nov 28, 2023
18 checks passed
@golechwierowicz golechwierowicz deleted the olechwierowicz/add_profiling_data branch November 28, 2023 08:05
@zpcore zpcore mentioned this pull request Nov 28, 2023
Closed
ManfeiBai pushed a commit to ManfeiBai/PyTorchXLA that referenced this pull request Dec 1, 2023
@golechwierowicz @ManfeiBai
Collect CUDA/CPU profiling info into result sheets. (pytorch#5921)
c36f0fd 
* Collect CUDA/CPU profiling info into result sheets.

This PR:
0. Adds CUDA/CPU collection capabilties to the script.
1. Modifies result_analyzer.py to analyze newly collected results.
2. Moves CUDA synchronize/XLA device synchronize into the profiler.
3. Fixes list typing for Python 3.8+.

Tested with command:
python3 xla/benchmarks/experiment_runner.py --dynamo=openxla --xla=PJRT --test=train --filter=basic_gnn_gcn$ --suite-name=torchbench --accelerator=cuda --progress-bar --output-dirname=/tmp/output --repeat=2 --print-subprocess --no-resume --profile-cuda-cpu-collect --profile-cuda
python3 xla/benchmarks/result_analyzer.py --output-dir=/tmp/output

* Lint, and add _s suffix to metrics

---------

Co-authored-by: root <[email protected]>
ManfeiBai pushed a commit to ManfeiBai/PyTorchXLA that referenced this pull request Dec 1, 2023
@golechwierowicz @ManfeiBai
Collect CUDA/CPU profiling info into result sheets. (pytorch#5921)
b100a95 
* Collect CUDA/CPU profiling info into result sheets.

This PR:
0. Adds CUDA/CPU collection capabilties to the script.
1. Modifies result_analyzer.py to analyze newly collected results.
2. Moves CUDA synchronize/XLA device synchronize into the profiler.
3. Fixes list typing for Python 3.8+.

Tested with command:
python3 xla/benchmarks/experiment_runner.py --dynamo=openxla --xla=PJRT --test=train --filter=basic_gnn_gcn$ --suite-name=torchbench --accelerator=cuda --progress-bar --output-dirname=/tmp/output --repeat=2 --print-subprocess --no-resume --profile-cuda-cpu-collect --profile-cuda
python3 xla/benchmarks/result_analyzer.py --output-dir=/tmp/output

* Lint, and add _s suffix to metrics

---------

Co-authored-by: root <[email protected]>
@miladm
Copy link
Collaborator

miladm commented Dec 4, 2023

Thanks.

cc @zpcore to take advantage of this feature in future benchmarking automation work.

@miladm miladm requested a review from zpcore December 4, 2023 16:47
@miladm miladm added the xla:gpu label Dec 4, 2023
chunnienc pushed a commit to chunnienc/xla that referenced this pull request Dec 14, 2023
@golechwierowicz @chunnienc
Collect CUDA/CPU profiling info into result sheets. (pytorch#5921)
e37c3da 
* Collect CUDA/CPU profiling info into result sheets.

This PR:
0. Adds CUDA/CPU collection capabilties to the script.
1. Modifies result_analyzer.py to analyze newly collected results.
2. Moves CUDA synchronize/XLA device synchronize into the profiler.
3. Fixes list typing for Python 3.8+.

Tested with command:
python3 xla/benchmarks/experiment_runner.py --dynamo=openxla --xla=PJRT --test=train --filter=basic_gnn_gcn$ --suite-name=torchbench --accelerator=cuda --progress-bar --output-dirname=/tmp/output --repeat=2 --print-subprocess --no-resume --profile-cuda-cpu-collect --profile-cuda
python3 xla/benchmarks/result_analyzer.py --output-dir=/tmp/output

* Lint, and add _s suffix to metrics

---------

Co-authored-by: root <[email protected]>
golechwierowicz added a commit that referenced this pull request Jan 12, 2024
@golechwierowicz
Collect CUDA/CPU profiling info into result sheets. (#5921)
0bb588e 
* Collect CUDA/CPU profiling info into result sheets.

This PR:
0. Adds CUDA/CPU collection capabilties to the script.
1. Modifies result_analyzer.py to analyze newly collected results.
2. Moves CUDA synchronize/XLA device synchronize into the profiler.
3. Fixes list typing for Python 3.8+.

Tested with command:
python3 xla/benchmarks/experiment_runner.py --dynamo=openxla --xla=PJRT --test=train --filter=basic_gnn_gcn$ --suite-name=torchbench --accelerator=cuda --progress-bar --output-dirname=/tmp/output --repeat=2 --print-subprocess --no-resume --profile-cuda-cpu-collect --profile-cuda
python3 xla/benchmarks/result_analyzer.py --output-dir=/tmp/output

* Lint, and add _s suffix to metrics

---------

Co-authored-by: root <[email protected]>
bhavya01 pushed a commit that referenced this pull request Apr 22, 2024
@golechwierowicz @bhavya01
Collect CUDA/CPU profiling info into result sheets. (#5921)
bb34f16 
* Collect CUDA/CPU profiling info into result sheets.

This PR:
0. Adds CUDA/CPU collection capabilties to the script.
1. Modifies result_analyzer.py to analyze newly collected results.
2. Moves CUDA synchronize/XLA device synchronize into the profiler.
3. Fixes list typing for Python 3.8+.

Tested with command:
python3 xla/benchmarks/experiment_runner.py --dynamo=openxla --xla=PJRT --test=train --filter=basic_gnn_gcn$ --suite-name=torchbench --accelerator=cuda --progress-bar --output-dirname=/tmp/output --repeat=2 --print-subprocess --no-resume --profile-cuda-cpu-collect --profile-cuda
python3 xla/benchmarks/result_analyzer.py --output-dir=/tmp/output

* Lint, and add _s suffix to metrics

---------

Co-authored-by: root <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vanbasten23 vanbasten23 vanbasten23 left review comments

@frgossen frgossen frgossen approved these changes

@zpcore zpcore Awaiting requested review from zpcore

Assignees
No one assigned
Labels
xla:gpu
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@golechwierowicz @miladm @vanbasten23 @frgossen