Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Properly install NCCL for CUDA 12.2/12.1 #293

Merged
merged 8 commits into from
Oct 10, 2023
Merged

Properly install NCCL for CUDA 12.2/12.1 #293

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
7cb526b
add nccl installer script that automatically selects a version compat…
yhtang Oct 9, 2023
6a4d5a4
add missing file
yhtang Oct 9, 2023
b4c37d3
minor fix to dockerfile
yhtang Oct 9, 2023
dd092c7
fix file permission
yhtang Oct 9, 2023
aa30e4b
do not reinstall NCCL if one already exists
yhtang Oct 9, 2023
aebca02
do not reinstall NCCL if one already exists
yhtang Oct 9, 2023
6882265
env var
yhtang Oct 10, 2023
4a85459
reset sandbox
yhtang Oct 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .github/container/Dockerfile.base
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,21 @@ ARG BASE_IMAGE=nvidia/cuda:12.2.0-devel-ubuntu22.04
FROM ${BASE_IMAGE}

###############################################################################
## Install Python
## Install Python and essential tools
###############################################################################

RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive \
TZ=America/Los_Angeles \
apt-get install -y \
build-essential \
checkinstall \
clang \
cmake \
curl \
git \
git \
lld \
vim \
python-is-python3 \
python3-pip \
wget \
Expand All @@ -34,10 +36,8 @@ RUN install-cudnn.sh
## Install NCCL
###############################################################################

RUN apt-get update && \
apt-get install -y libnccl-dev && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
ADD install-nccl.sh /usr/local/bin
RUN install-nccl.sh

###############################################################################
## RoCE and InfiniteBand support
Expand Down
35 changes: 35 additions & 0 deletions .github/container/install-nccl.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#!/bin/bash

set -ex -o pipefail

export DEBIAN_FRONTEND=noninteractive
export TZ=America/Los_Angeles

# If NCCL is already installed, don't reinstall it. Print a message and exit
if dpkg -s libnccl2 libnccl-dev &> /dev/null; then
echo "NCCL is already installed. Skipping installation."
exit 0
fi

apt-get update

# Extract CUDA version from `nvcc --version` output line
# Input: "Cuda compilation tools, release X.Y, VX.Y.Z"
# Output: X.Y
cuda_version=$(nvcc --version | sed -n 's/^.*release \([0-9]*\.[0-9]*\).*$/\1/p')

# Find latest NCCL version compatible with existing CUDA by matching
# ${cuda_version} in the package version string
libnccl2_version=$(apt-cache show libnccl-dev | sed -n "s/^Version: \(.*+cuda${cuda_version}\)$/\1/p" | head -n 1)
libnccl_dev_version=$(apt-cache show libnccl-dev | sed -n "s/^Version: \(.*+cuda${cuda_version}\)$/\1/p" | head -n 1)
if [[ -z "${libnccl2_version}" || -z "${libnccl_dev_version}" ]]; then
echo "Could not find compatible NCCL version for CUDA ${cuda_version}"
exit 1
fi

apt-get install -y \
libnccl2=${libnccl2_version} \
libnccl-dev=${libnccl_dev_version}

apt-get clean
rm -rf /var/lib/apt/lists/*
Loading