Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use hosted CI runners for building docker images #1915

Merged
merged 19 commits into from
Aug 31, 2023
Merged

Use hosted CI runners for building docker images #1915

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
ba45568
New technique
muellerzr Aug 31, 2023
b3ab86a
needs
muellerzr Aug 31, 2023
f6fa31f
explicit all
muellerzr Aug 31, 2023
da08806
Volume prune not going
muellerzr Aug 31, 2023
8205d98
Skip volume
muellerzr Aug 31, 2023
08bd505
versions
muellerzr Aug 31, 2023
e5c5ef2
Avoid checkout perhaps?
muellerzr Aug 31, 2023
2347f54
Working dir
muellerzr Aug 31, 2023
53c2bdc
Don't include dot-slash?
muellerzr Aug 31, 2023
e6bd329
Accelerate prefix?
muellerzr Aug 31, 2023
9993a03
Working directory?
muellerzr Aug 31, 2023
f9f8436
Context?
muellerzr Aug 31, 2023
0ae63b2
other workingdir
muellerzr Aug 31, 2023
8d261e3
Faster iteration
muellerzr Aug 31, 2023
3718675
Right tag
muellerzr Aug 31, 2023
60bd51c
Full
muellerzr Aug 31, 2023
80e6ba7
Release
muellerzr Aug 31, 2023
aaf4857
GPU
muellerzr Aug 31, 2023
533d4f8
Merge branch 'main' into runner-for-docker
muellerzr Aug 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions .github/workflows/build-docker-images-release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,40 +21,40 @@ jobs:

version-cpu:
name: "Latest Accelerate CPU [version]"
runs-on: ubuntu-latest
runs-on: [self-hosted, docker-gpu, multi-gpu]
ydshieh marked this conversation as resolved.
Show resolved Hide resolved
needs: get-version
steps:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
- name: Check out code
uses: actions/checkout@v2
uses: docker/setup-buildx-action@v2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Never heard of Docker buildx

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's the CLI extension for docker build-kit: https://github.com/docker/buildx

- name: Login to DockerHub
uses: docker/login-action@v1
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PASSWORD }}

- name: Build and Push CPU
run: |
docker build -t huggingface/accelerate-cpu -f ./docker/accelerate-cpu/Dockerfile .
docker push huggingface/accelerate-cpu:${{needs.get-version.outputs.version}}
uses: docker/build-push-action@v4
with:
file: docker/accelerate-cpu/Dockerfile
push: true
tags: huggingface/accelerate-cpu:${{needs.get-version.outputs.version}}

version-cuda:
name: "Latest Accelerate GPU [version]"
runs-on: ubuntu-latest
runs-on: [self-hosted, docker-gpu, multi-gpu]
ydshieh marked this conversation as resolved.
Show resolved Hide resolved
needs: get-version
steps:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
- name: Check out code
uses: actions/checkout@v2
uses: docker/setup-buildx-action@v2
- name: Login to DockerHub
uses: docker/login-action@v1
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PASSWORD }}

- name: Build and Push GPU
run: |
docker build -t huggingface/accelerate-gpu -f ./docker/accelerate-gpu/Dockerfile .
docker push huggingface/accelerate-gpu:${{needs.get-version.outputs.version}}
uses: docker/build-push-action@v4
with:
file: docker/accelerate-gpu/Dockerfile
push: true
tags: huggingface/accelerate-gpu:${{needs.get-version.outputs.version}}
43 changes: 27 additions & 16 deletions .github/workflows/build_docker_images.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,39 +11,50 @@ concurrency:
cancel-in-progress: false

jobs:
clean-storage:
name: "Clean docker image storage"
runs-on: [self-hosted, docker-gpu, multi-gpu]
steps:
- name: Clean storage
run: |
docker image prune --all -f --filter "until=48h"
docker system prune --all -f --filter "until=48h"

latest-cpu:
name: "Latest Accelerate CPU [dev]"
runs-on: ubuntu-latest
runs-on: [self-hosted, docker-gpu, multi-gpu]
needs: clean-storage
steps:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
- name: Check out code
uses: actions/checkout@v2
uses: docker/setup-buildx-action@v2
- name: Login to DockerHub
uses: docker/login-action@v1
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PASSWORD }}
- name: Build and Push CPU
run: |
docker build -t huggingface/accelerate-cpu -f ./docker/accelerate-cpu/Dockerfile .
docker push huggingface/accelerate-cpu:latest
uses: docker/build-push-action@v4
with:
file: docker/accelerate-cpu/Dockerfile
push: true
tags: huggingface/accelerate-cpu

latest-cuda:
name: "Latest Accelerate GPU [dev]"
runs-on: ubuntu-latest
runs-on: [self-hosted, docker-gpu, multi-gpu]
needs: clean-storage
steps:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
- name: Check out code
uses: actions/checkout@v2
uses: docker/setup-buildx-action@v2
- name: Login to DockerHub
uses: docker/login-action@v1
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PASSWORD }}

- name: Build and Push GPU
run: |
docker build -t huggingface/accelerate-gpu -f ./docker/accelerate-gpu/Dockerfile .
docker push huggingface/accelerate-gpu:latest
uses: docker/build-push-action@v4
with:
file: docker/accelerate-gpu/Dockerfile
push: true
tags: huggingface/accelerate-gpu
Loading