Revert "Bump CUDA version to 12.2.0-devel-ubuntu22.04" #176
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reverts commit d0e6e84.
yhtang
reviewed
Aug 24, 2023
|
@nouiz mentioned something about compat drivers. I see that forward compat drivers are installed: docker run --rm ghcr.io/nvidia/t5x:nightly-2023-08-20 dpkg -l | grep cuda-compat
ii cuda-compat-12-2 535.54.03-1 amd64 CUDA Compatibility Platformso that checks that box. I get an issue when we add the nvidia container runtime: # ok
docker run -it --rm ghcr.io/nvidia/t5x:nightly-2023-08-20 bash
# errors
docker run --gpus=all -it --rm ghcr.io/nvidia/t5x:nightly-2023-08-20 bash
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.2, please update your driver to a newer version, or use an earlier cuda container: unknown.So maybe the nvidia container plugin is where we start looking? |
yhtang
added a commit
that referenced
this pull request
Aug 25, 2023
…2.1.1 (#182) This PR addresses #176 and #159 in a single shot. The default CUDA version will only be encoded in Dockerfile.base. A default value of latest will be passed between various workflows and eventually being properly resolved into the BASE_IMAGE build arg by the _build_base.yaml workflow. Closes #176 Closes #159
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reverts #161
Proposing the reversion of the 12.2 bump. The images produced from the base cuda 12.2.0 image are not functioning and all tests on the CI have been failing for a few days now. I have also seen this fail on my workstation and on selene. It might be better to revert this until we know what's going on. I have attached an image to the issue below to help debug in parallel.
Related: #165