Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to checkpoint/restore NCCL state between multiple nvidia GPUs ? #29

Open
botieking98 opened this issue Sep 23, 2023 · 0 comments
Open

How to checkpoint/restore NCCL state between multiple nvidia GPUs ? #29

botieking98 opened this issue Sep 23, 2023 · 0 comments
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@botieking98
Copy link

botieking98 commented Sep 23, 2023 •
edited
Loading

Assume we want to migrate an AI task from one node to another node, for NCCL, the only way that I know is to re-establish the NCCL communition ring. So can anybody know how to checkpoint and restore the NCCL communitions between multiple ranks?
@RWTH-ACS RWTH-ACS deleted a comment from Cattacker Dec 30, 2023
@n-eiling n-eiling added enhancement New feature or request help wanted Extra attention is needed labels Dec 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@n-eiling @botieking98