Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Publishes rosetta CUDA 12.1 and containers for pax and t5x AND containers for nightlies based on jax-pinned images for CUDA 12.1 and 12.2 #286

Merged
merged 24 commits into from
Oct 14, 2023
Merged

Publishes rosetta CUDA 12.1 and containers for pax and t5x AND containers for nightlies based on jax-pinned images for CUDA 12.1 and 12.2 #286

merged 24 commits into from
Oct 14, 2023

Conversation

terrykong
Copy link
Contributor

No description provided.

@terrykong terrykong requested a review from yhtang October 6, 2023 03:46
@terrykong terrykong changed the title Publishes rosetta 12.1 containers for pax and t5x Publishes rosetta CUDA 12.1 containers for pax and t5x Oct 6, 2023
@yhtang yhtang requested a review from nouiz October 6, 2023 18:37
yhtang
yhtang reviewed Oct 6, 2023
View reviewed changes
Copy link
Collaborator

@yhtang yhtang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this build arm64 images once #284 is merged?
We probably don't want to build arm64+CUDA12.1 images since they do not work on Grace Hopper.

@terrykong
Copy link
Contributor Author

Oops. You are right. I need to rebase this on #284 and only add the amd64 platform. Will do that now

@terrykong
Copy link
Contributor Author

terrykong commented Oct 6, 2023 •
edited
Loading

Done. Launched this run to test: https://github.com/NVIDIA/JAX-Toolbox/actions/runs/6436638266

(edit: build failed, but not due to this change)

@terrykong
Copy link
Contributor Author

Re-running CI after rebasing on top of the multiarch PR to make sure nothing broke: https://github.com/NVIDIA/JAX-Toolbox/actions/runs/6465287847

@terrykong
Copy link
Contributor Author

Ahh, even that build failed. I forgot I haven't cherry-picked @yhtang 's fix for NCCL in 12.1 images

@yhtang
Copy link
Collaborator

yhtang commented Oct 10, 2023

Rerunning agaaain after rebasing with the latest NCCL fix.

https://github.com/NVIDIA/JAX-Toolbox/actions/runs/6466124323

@terrykong terrykong changed the title Publishes rosetta CUDA 12.1 containers for pax and t5x Publishes rosetta CUDA 12.1 and containers for pax and t5x AND containers for nightlies based on jax-pinned images Oct 10, 2023
@terrykong terrykong changed the title Publishes rosetta CUDA 12.1 and containers for pax and t5x AND containers for nightlies based on jax-pinned images Publishes rosetta CUDA 12.1 and containers for pax and t5x AND containers for nightlies based on jax-pinned images for CUDA 12.1 and 12.2 Oct 10, 2023
@terrykong
Copy link
Contributor Author

terrykong commented Oct 11, 2023 •
edited
Loading

Launched sandbox CI:

(edit: re-ran due to t5x build failure; waiting on google-research/t5x#1416, but configured workflow to default to this branch for now)
(edit2: re-ran due to rosetta-t5x build failure. Building from a fork was problematic since PRs weren't present. Switched to mirror branch patches instead of upstream PR patches)
(edit3: re-ran b/c TE change removes API so added functionality to pin TE)

@yhtang
Copy link
Collaborator

yhtang commented Oct 13, 2023 •
edited
Loading

Rerun on top-of-tree for JAX/XLA after openxla/xla#6305 is merged.

https://github.com/NVIDIA/JAX-Toolbox/actions/runs/6512759676: build succeeded.

@yhtang yhtang merged commit c2b721c into main Oct 14, 2023
6 checks passed
@yhtang yhtang deleted the rosetta-cuda-121 branch October 14, 2023 05:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yhtang yhtang yhtang left review comments

@nouiz nouiz Awaiting requested review from nouiz

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@terrykong @yhtang