Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(docker): provide modular images for openadkit planning simulator visualizer #4673

Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
0dd842c
add modular containers
oguzkaganozt Apr 30, 2024
990bf3e
.
oguzkaganozt Apr 30, 2024
ba15545
style(pre-commit): autofix
pre-commit-ci[bot] Apr 30, 2024
242ace0
refine dockerfiles
oguzkaganozt May 3, 2024
68a82a1
Simplify run.sh
oguzkaganozt May 16, 2024
b4ba9c2
Simply workspace argument
oguzkaganozt May 16, 2024
4aaaff9
Simplify directories
oguzkaganozt May 16, 2024
4df70c2
switch from autoware-openadkit to openadkit for simplification
oguzkaganozt May 17, 2024
39b3cd0
update documentation
oguzkaganozt May 17, 2024
efb80e5
add base metadata into autoware
oguzkaganozt May 17, 2024
ff54c31
add base to build.sh
oguzkaganozt May 17, 2024
450ce3e
fix docker tags
oguzkaganozt May 20, 2024
471e60e
update readme
oguzkaganozt May 20, 2024
fba279e
style(pre-commit): autofix
pre-commit-ci[bot] May 20, 2024
96b7b9b
.
oguzkaganozt May 20, 2024
0c4571a
.
oguzkaganozt May 20, 2024
1788b7f
.
oguzkaganozt May 20, 2024
1a9c784
add base tar
oguzkaganozt May 20, 2024
f0bd4fb
fix typo
oguzkaganozt May 20, 2024
4b34a4a
ommit building all images in all matrix threads
oguzkaganozt May 21, 2024
31f1a70
.
oguzkaganozt May 21, 2024
39fb791
fix typo
oguzkaganozt May 21, 2024
803df77
update readme
oguzkaganozt May 21, 2024
cc841e2
fix ansible style-check
oguzkaganozt May 22, 2024
ef5d1a6
fix stylecheck
oguzkaganozt May 22, 2024
1df3098
.
oguzkaganozt May 22, 2024
99514cf
remove targets from openadki build.sh
oguzkaganozt May 22, 2024
fc237f8
style(pre-commit): autofix
pre-commit-ci[bot] May 22, 2024
6608bfe
fix https://github.com/autowarefoundation/autoware/issues/4765
oguzkaganozt May 23, 2024
eb8b55a
style(pre-commit): autofix
pre-commit-ci[bot] May 23, 2024
e0d0a31
Update from reviewers
oguzkaganozt May 24, 2024
24d84f1
remove upload artifacts because of the disk space issue
oguzkaganozt May 24, 2024
43824c5
style(pre-commit): autofix
pre-commit-ci[bot] May 24, 2024
cf43ae9
.
oguzkaganozt May 27, 2024
1705246
update yamllint settings
oguzkaganozt May 27, 2024
c32b00f
Remove data_path as it is included in the image again
oguzkaganozt May 27, 2024
29d1a47
download artifacts inside the dockerfile again
oguzkaganozt May 27, 2024
83dad77
.
oguzkaganozt May 27, 2024
6293ea2
fix playbook naming
oguzkaganozt May 27, 2024
9ddc399
add caching
oguzkaganozt May 27, 2024
e6d49b5
remove base from build.sh
oguzkaganozt May 27, 2024
d0a6d20
remove extra dash'-' from docker image tagging
oguzkaganozt May 28, 2024
573fbde
Merge branch 'main' into 4538-provide-modular-images-for-openadkit-pl…
oguzkaganozt May 28, 2024
95f1bb5
add clone repositories into build.sh
oguzkaganozt May 28, 2024
0b28c0e
style(pre-commit): autofix
pre-commit-ci[bot] May 28, 2024
ef2683a
.
oguzkaganozt May 28, 2024
deb0d40
Merge branch 'main' into 4538-provide-modular-images-for-openadkit-pl…
oguzkaganozt May 29, 2024
04477cf
style(pre-commit): autofix
pre-commit-ci[bot] May 29, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
197 changes: 197 additions & 0 deletions .github/actions/docker-build-and-push-openadkit/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,197 @@
name: docker-build-and-push-openadkit
description: ""

inputs:
bake-target:
description: ""
required: true
bake-images:
description: ""
required: false
build-args:
description: ""
required: false
tag-prefix:
description: ""
required: false
tag-suffix:
description: ""
required: false
tag-arch:
description: ""
required: false
allow-push:
description: ""
default: "true"
required: false

runs:
using: composite
steps:
- name: Setup Docker Buildx
uses: docker/setup-buildx-action@v2

- name: Set Docker tags
id: set-docker-tags
run: |
tags=()
if [ "${{ github.event_name }}" == "push" ] && [ "${{ github.ref_type }}" == "tag" ]; then
tags+=("$(echo "${{ github.ref }}" | sed -E 's/.*([vV][0-9]+\.[0-9]+\.[0-9]+).*/\1/')")
fi

tags+=("{{date 'YYYYMMDD'}}")
tags+=("latest")
tags+=("latest-${{ inputs.tag-prefix }}")

# Output multiline strings: https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#multiline-strings
EOF=$(dd if=/dev/urandom bs=15 count=1 status=none | base64)
echo "tags<<$EOF" >> $GITHUB_OUTPUT
echo "$(printf "%s\n" "${tags[@]}")" >> $GITHUB_OUTPUT
echo "$EOF" >> $GITHUB_OUTPUT
shell: bash

# openadkit modular images metadata
- name: Docker meta for openadkit:planning-control
id: meta-planning-control
uses: docker/metadata-action@v4
with:
images: ghcr.io/${{ github.repository_owner }}/${{ inputs.bake-target }}
tags: ${{ steps.set-docker-tags.outputs.tags }}
bake-target: docker-metadata-action-planning-control
flavor: |
latest=false
suffix=-planning-control${{ inputs.tag-arch }}

- name: Docker meta for openadkit:visualizer
id: meta-visualizer
uses: docker/metadata-action@v4
with:
images: ghcr.io/${{ github.repository_owner }}/${{ inputs.bake-target }}
tags: ${{ steps.set-docker-tags.outputs.tags }}
bake-target: docker-metadata-action-visualizer
flavor: |
latest=false
suffix=-visualizer${{ inputs.tag-arch }}

- name: Docker meta for openadkit:simulator
id: meta-simulator
uses: docker/metadata-action@v4
with:
images: ghcr.io/${{ github.repository_owner }}/${{ inputs.bake-target }}
tags: ${{ steps.set-docker-tags.outputs.tags }}
bake-target: docker-metadata-action-simulator
flavor: |
latest=false
suffix=-simulator${{ inputs.tag-arch }}

- name: Login to GitHub Container Registry
if: ${{ github.event_name != 'pull_request' }}
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ github.token }}

- name: Build and Push to GitHub Container Registry
if: ${{ github.event_name == 'push' || github.event_name == 'schedule' || ( github.event_name == 'workflow_dispatch' && github.event.inputs.artifacts-destination == 'registry') }}
uses: docker/bake-action@v3
with:
push: ${{ inputs.allow-push == 'true' }}
files: |
docker/${{ inputs.bake-target }}/docker-bake.hcl
${{ steps.meta-planning-control.outputs.bake-file }}
${{ steps.meta-visualizer.outputs.bake-file }}
${{ steps.meta-simulator.outputs.bake-file }}
provenance: false
targets: |
${{ inputs.bake-images }}
set: |
${{ inputs.build-args }}

- name: Build only
if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.artifacts-destination == 'tarball' }}
uses: docker/bake-action@v3
with:
push: false
files: |
docker/${{ inputs.bake-target }}/docker-bake.hcl
${{ steps.meta-planning-control.outputs.bake-file }}
${{ steps.meta-visualizer.outputs.bake-file }}
${{ steps.meta-simulator.outputs.bake-file }}
provenance: false
targets: |
${{ inputs.bake-images }}
set: |
${{ inputs.build-args }}

- name: Output artifact URLs
if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.artifacts-destination == 'tarball' }}
id: warn-no-artifacts
run: |
echo 'No artifacts uploaded because of disk space issue'
shell: bash

# TODO:Enable after solving the issue with the disk space
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are using the version control tool. We can revert the commit. That's why removing them is simple.

# - name: Build and Save Artifacts
# if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.artifacts-destination == 'tarball' }}
# uses: docker/bake-action@v3
# with:
# push: false
# files: |
# docker/${{ inputs.bake-target }}/docker-bake.hcl
# ${{ steps.meta-planning-control.outputs.bake-file }}
# ${{ steps.meta-visualizer.outputs.bake-file }}
# ${{ steps.meta-simulator.outputs.bake-file }}
# provenance: false
# targets: |
# ${{ inputs.bake-images }}
# set: |
# ${{ inputs.build-args }}
# planning-control.output=type=docker,dest=/tmp/planning-control.tar
# visualizer.output=type=docker,dest=/tmp/visualizer.tar
# simulator.output=type=docker,dest=/tmp/simulator.tar

# - name: Upload Artifact - planning-control
# if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.artifacts-destination == 'tarball' }}
# id: artifact-upload-step-planning-control
# uses: actions/upload-artifact@v4
# with:
# name: planning-control-image${{ inputs.tag-suffix }}
# path: /tmp/planning-control.tar
# retention-days: 7
# compression-level: 6
# overwrite: true
# if-no-files-found: warn

# - name: Upload Artifact - visualizer
# if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.artifacts-destination == 'tarball' }}
# id: artifact-upload-step-visualizer
# uses: actions/upload-artifact@v4
# with:
# name: visualizer-image${{ inputs.tag-suffix }}
# path: /tmp/visualizer.tar
# retention-days: 7
# compression-level: 6
# overwrite: true
# if-no-files-found: warn

# - name: Upload Artifact - simulator
# if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.artifacts-destination == 'tarball' }}
# id: artifact-upload-step-simulator
# uses: actions/upload-artifact@v4
# with:
# name: simulator-image${{ inputs.tag-suffix }}
# path: /tmp/simulator.tar
# retention-days: 7
# compression-level: 6
# overwrite: true
# if-no-files-found: warn

# - name: Output artifact URLs
# if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.artifacts-destination == 'tarball' }}
# id: output-artifact-urls
# run: |
# echo 'planning-control URL ${{ steps.artifact-upload-step-planning-control.outputs.artifact-url }}'
# echo 'visualizer URL ${{ steps.artifact-upload-step-visualizer.outputs.artifact-url }}'
# echo 'simulator URL ${{ steps.artifact-upload-step-simulator.outputs.artifact-url }}'
# shell: bash
38 changes: 27 additions & 11 deletions .github/actions/docker-build-and-push/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ inputs:
bake-target:
description: ""
required: true
bake-images:
description: ""
required: false
build-args:
description: ""
required: false
Expand All @@ -14,6 +17,9 @@ inputs:
tag-suffix:
description: ""
required: false
tag-arch:
description: ""
required: false
allow-push:
description: ""
default: "true"
Expand All @@ -25,12 +31,6 @@ runs:
- name: Setup Docker Buildx
uses: docker/setup-buildx-action@v2

- name: Install jq
run: |
sudo apt-get -y update
sudo apt-get -y install jq
shell: bash

- name: Set Docker tags
id: set-docker-tags
run: |
Expand All @@ -50,6 +50,17 @@ runs:
echo "$EOF" >> $GITHUB_OUTPUT
shell: bash

- name: Docker meta for base
id: meta-base
uses: docker/metadata-action@v4
with:
images: ghcr.io/${{ github.repository_owner }}/${{ inputs.bake-target }}
tags: ${{ steps.set-docker-tags.outputs.tags }}
bake-target: docker-metadata-action-base
flavor: |
latest=false
suffix=-base${{ inputs.tag-arch }}

oguzkaganozt marked this conversation as resolved.
Show resolved Hide resolved
- name: Docker meta for prebuilt
id: meta-prebuilt
uses: docker/metadata-action@v4
Expand All @@ -59,7 +70,7 @@ runs:
bake-target: docker-metadata-action-prebuilt
flavor: |
latest=false
suffix=-prebuilt${{ inputs.tag-suffix }}
suffix=-prebuilt${{ inputs.tag-suffix }}${{ inputs.tag-arch }}

- name: Docker meta for devel
id: meta-devel
Expand All @@ -70,10 +81,9 @@ runs:
bake-target: docker-metadata-action-devel
flavor: |
latest=false
suffix=-devel${{ inputs.tag-suffix }}
suffix=-devel${{ inputs.tag-suffix }}${{ inputs.tag-arch }}

- name: Docker meta for runtime
if: ${{ github.event_name == 'workflow_dispatch' }} || ${{ (github.event_name == 'push' && github.ref_type == 'tag') }}
id: meta-runtime
uses: docker/metadata-action@v4
with:
Expand All @@ -82,7 +92,7 @@ runs:
bake-target: docker-metadata-action-runtime
flavor: |
latest=${{ github.event_name == 'push' && github.ref_type == 'tag' }}
suffix=-runtime${{ inputs.tag-suffix }}
suffix=-runtime${{ inputs.tag-suffix }}${{ inputs.tag-arch }}

- name: Login to GitHub Container Registry
if: ${{ github.event_name != 'pull_request' }}
Expand All @@ -99,23 +109,29 @@ runs:
push: ${{ inputs.allow-push == 'true' }}
files: |
docker/docker-bake.hcl
${{ steps.meta-base.outputs.bake-file }}
${{ steps.meta-prebuilt.outputs.bake-file }}
${{ steps.meta-devel.outputs.bake-file }}
${{ steps.meta-runtime.outputs.bake-file }}
provenance: false
targets: |
${{ inputs.bake-images }}
set: |
${{ inputs.build-args }}

- name: Build only
- name: Build Only
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please focus on the minimum change. If you want to refactor please make another PR.

Suggested change
- name: Build Only
- name: Build only

if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.artifacts-destination == 'tarball' }}
uses: docker/bake-action@v3
with:
push: false
files: |
docker/docker-bake.hcl
${{ steps.meta-base.outputs.bake-file }}
${{ steps.meta-prebuilt.outputs.bake-file }}
${{ steps.meta-devel.outputs.bake-file }}
${{ steps.meta-runtime.outputs.bake-file }}
provenance: false
targets: |
${{ inputs.bake-images }}
set: |
${{ inputs.build-args }}
25 changes: 24 additions & 1 deletion .github/workflows/docker-build-and-push-main-self-hosted.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,12 @@ jobs:
lib_dir: aarch64
setup-args: --no-nvidia
additional-tag-suffix: ""
bake-images: default
- name: cuda
base_image_env: base_image
lib_dir: aarch64
additional-tag-suffix: -cuda
bake-images: default
steps:
# https://github.com/actions/checkout/issues/211
- name: Change permission of workspace
Expand Down Expand Up @@ -85,10 +87,31 @@ jobs:
*.args.BASE_IMAGE=${{ needs.load-env.outputs[format('{0}', matrix.base_image_env)] }}
*.args.SETUP_ARGS=${{ matrix.setup-args }}
*.args.LIB_DIR=${{ matrix.lib_dir }}
*.args.ARCH=arm64
*.cache-from=type=registry,ref=ghcr.io/${{ github.repository }}:buildcache-${{ matrix.name }}
*.cache-to=type=registry,ref=ghcr.io/${{ github.repository }}:buildcache-${{ matrix.name }},mode=max
tag-suffix: ${{ matrix.additional-tag-suffix }}-arm64
tag-suffix: ${{ matrix.additional-tag-suffix }}
tag-arch: -arm64
tag-prefix: ${{ needs.load-env.outputs.rosdistro }}
bake-images: ${{ matrix.bake-images }}
allow-push: true

- name: Build 'Open AD Kit'
if: ${{ matrix.name == 'no-cuda' }}
uses: ./.github/actions/docker-build-and-push-openadkit
with:
bake-target: openadkit
build-args: |
*.platform=linux/arm64
*.args.ROS_DISTRO=${{ needs.load-env.outputs.rosdistro }}
*.args.BASE_IMAGE=${{ needs.load-env.outputs[format('{0}', matrix.base_image_env)] }}
*.args.SETUP_ARGS=${{ matrix.setup-args }}
*.args.LIB_DIR=${{ matrix.lib_dir }}
*.args.ARCH=arm64
tag-suffix: ${{ matrix.additional-tag-suffix }}
tag-arch: -arm64
tag-prefix: ${{ needs.load-env.outputs.rosdistro }}
bake-images: ${{ matrix.bake-images }}
allow-push: true

- name: Show disk space
Expand Down
Loading
Loading