Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Truncate python stack when outputting frame that cause the graph executation #5933

Merged
merged 3 commits into from
Nov 29, 2023
Merged

Truncate python stack when outputting frame that cause the graph executation #5933

merged 3 commits into from
Nov 29, 2023

Conversation

JackCaoG
Copy link
Collaborator

@JackCaoG JackCaoG commented Nov 29, 2023 •
edited
Loading

motivation of this pr is simple, currently frame analysis looks like

Execution Analysis: ================================================================================
Execution Analysis: Execution Cause
Execution Analysis:   user mark_step
Execution Analysis: Python Frame Triggered Execution: 
Execution Analysis:   mark_step (/src/pytorch/xla/torch_xla/core/xla_model.py:826)
Execution Analysis:   broadcast_master_param (/src/pytorch/xla/torch_xla/core/xla_model.py:1216)
Execution Analysis:   train_imagenet (/src/pytorch/xla/test/test_train_mp_imagenet.py:261)
Execution Analysis:   _mp_fn (/src/pytorch/xla/test/test_train_mp_imagenet.py:365)
Execution Analysis:   __call__ (/src/pytorch/xla/torch_xla/_internal/pjrt.py:176)
Execution Analysis:   _thread_fn (/src/pytorch/xla/torch_xla/_internal/pjrt.py:70)
Execution Analysis:   run (/usr/local/lib/python3.8/concurrent/futures/thread.py:57)
Execution Analysis:   _worker (/usr/local/lib/python3.8/concurrent/futures/thread.py:80)
Execution Analysis:   run (/usr/local/lib/python3.8/threading.py:870)
Execution Analysis:   _bootstrap_inner (/usr/local/lib/python3.8/threading.py:932)
Execution Analysis:   _bootstrap (/usr/local/lib/python3.8/threading.py:890)
Execution Analysis: --------------------------------------------------------------------------------
Execution Analysis: ================================================================================

but the stack below _mp_fn are not very useful so I want to truncate them, after this pr it becomes.

Execution Analysis: ================================================================================
Execution Analysis: Execution Cause
Execution Analysis:   user mark_step
Execution Analysis: Python Frame Triggered Execution: 
Execution Analysis:   mark_step (/workspaces/dk2/pytorch/xla/torch_xla/core/xla_model.py:826)
Execution Analysis:   broadcast_master_param (/workspaces/dk2/pytorch/xla/torch_xla/core/xla_model.py:1216)
Execution Analysis:   train_imagenet (/workspaces/dk2/pytorch/xla/test/test_train_mp_imagenet.py:261)
Execution Analysis:   _mp_fn (/workspaces/dk2/pytorch/xla/test/test_train_mp_imagenet.py:365)
Execution Analysis:   ..........
Execution Analysis: --------------------------------------------------------------------------------
Execution Analysis: ================================================================================

@JackCaoG JackCaoG changed the title [WIP]Truncate python stack when outputting frame that cause the graph exec… Truncate python stack when outputting frame that cause the graph executation Nov 29, 2023
@JackCaoG JackCaoG added the usability Bugs/features related to improving the usability of PyTorch/XLA label Nov 29, 2023
will-cromar
will-cromar approved these changes Nov 29, 2023
View reviewed changes
Copy link
Collaborator

@will-cromar will-cromar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your original TODO mentioned configurability. Would it make sense to have a flag to print the entire stack?

@JackCaoG
Copy link
Collaborator Author

yea.. I am thinking about something less fine grain like verbose level for pt xla debug, when it is high enough it will print all sorts of things.

@JackCaoG JackCaoG merged commit a5e0738 into master Nov 29, 2023
17 checks passed
ManfeiBai pushed a commit to ManfeiBai/PyTorchXLA that referenced this pull request Dec 1, 2023
@JackCaoG @ManfeiBai
Truncate python stack when outputting frame that cause the graph exec…
169a7fd 
…utation (pytorch#5933)

* Truncate python stack when outputting frame that cause the graph execution

* add mp tests

* move tests to a new dir

---------

Co-authored-by: root <[email protected]>
ManfeiBai pushed a commit to ManfeiBai/PyTorchXLA that referenced this pull request Dec 1, 2023
@JackCaoG @ManfeiBai
Truncate python stack when outputting frame that cause the graph exec…
79555a4 
…utation (pytorch#5933)

* Truncate python stack when outputting frame that cause the graph execution

* add mp tests

* move tests to a new dir

---------

Co-authored-by: root <[email protected]>
chunnienc pushed a commit to chunnienc/xla that referenced this pull request Dec 14, 2023
@JackCaoG @chunnienc
Truncate python stack when outputting frame that cause the graph exec…
6e5cb6c 
…utation (pytorch#5933)

* Truncate python stack when outputting frame that cause the graph execution

* add mp tests

* move tests to a new dir

---------

Co-authored-by: root <[email protected]>
golechwierowicz pushed a commit that referenced this pull request Jan 12, 2024
@JackCaoG @golechwierowicz
Truncate python stack when outputting frame that cause the graph exec…
9d72a5a 
…utation (#5933)

* Truncate python stack when outputting frame that cause the graph execution

* add mp tests

* move tests to a new dir

---------

Co-authored-by: root <[email protected]>
bhavya01 pushed a commit that referenced this pull request Apr 22, 2024
@JackCaoG @bhavya01
Truncate python stack when outputting frame that cause the graph exec…
86b0290 
…utation (#5933)

* Truncate python stack when outputting frame that cause the graph execution

* add mp tests

* move tests to a new dir

---------

Co-authored-by: root <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@will-cromar will-cromar will-cromar approved these changes

@yeounoh yeounoh Awaiting requested review from yeounoh

Assignees
No one assigned
Labels
usability Bugs/features related to improving the usability of PyTorch/XLA
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@JackCaoG @will-cromar