Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: run build docker image action on self hosted runners #2534

Merged
merged 16 commits into from
Sep 14, 2023
Merged

chore: run build docker image action on self hosted runners #2534

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
dfaa3c1
Run build docker image action on self hosted runners
lokey Aug 30, 2023
4f1f8e5
Remove cache to test
lokey Aug 30, 2023
5d6ffd5
test commit
lokey Aug 30, 2023
0113a59
Test
lokey Aug 30, 2023
41d95fe
Build only amd64 image
lokey Aug 30, 2023
819c5ab
build multi step multi arch image
lokey Aug 30, 2023
325163c
test
lokey Aug 30, 2023
6000283
Remove rudderlabs docker hub repo for testing
lokey Aug 30, 2023
316d609
Update the manifest command to use imagetools
lokey Sep 3, 2023
fb63550
Login to dockerhub
lokey Sep 3, 2023
f31c2a0
Remove unnecessary step
lokey Sep 3, 2023
77fbb1f
Merge branch 'develop' into IDP-9-run-actions-on-self-hosted-transfor…
koladilip Sep 12, 2023
8f092e5
chore: uncommented test runs
krishna2020 Sep 13, 2023
8451d8c
Merge branch 'develop' into IDP-9-run-actions-on-self-hosted-transfor…
krishna2020 Sep 13, 2023
787bfcc
Merge branch 'develop' into IDP-9-run-actions-on-self-hosted-transfor…
krishna2020 Sep 13, 2023
ab1c118
Merge branch 'develop' into IDP-9-run-actions-on-self-hosted-transfor…
krishna2020 Sep 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/build-pr-artifacts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ jobs:
uses: ./.github/workflows/build-push-docker-image.yml
with:
build_tag: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name }}
push_tags: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name }},rudderlabs/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name }}
push_tags: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name }}
img_tag: ${{ needs.generate-tag-names.outputs.tag_name }}
dockerfile: Dockerfile
load_target: development
Expand All @@ -59,7 +59,7 @@ jobs:
uses: ./.github/workflows/build-push-docker-image.yml
with:
build_tag: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name_ut }}
push_tags: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name_ut }},rudderlabs/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name_ut }}
push_tags: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name_ut }}
img_tag: ${{ needs.generate-tag-names.outputs.tag_name_ut }}
dockerfile: Dockerfile-ut-func
load_target: development
Expand Down
90 changes: 79 additions & 11 deletions .github/workflows/build-push-docker-image.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,9 @@ env:
DOCKERHUB_USERNAME: rudderlabs

jobs:
build-transformer-image:
name: Build Transformer Docker Image
runs-on: ubuntu-latest
build-transformer-image-arm64:
name: Build Transformer Docker Image ARM64
runs-on: [self-hosted, Linux, ARM64]
steps:
- name: Checkout
uses: actions/[email protected]
Expand All @@ -55,12 +55,12 @@ jobs:
target: ${{ inputs.load_target }}
load: true
tags: ${{ inputs.build_tag }}
cache-from: type=gha
cache-to: type=gha,mode=max
# cache-from: type=gha
# cache-to: type=gha,mode=max

- name: Run Tests
run: |
docker run ${{ inputs.build_tag }} npm run test:js:ci
docker run ${{ inputs.build_tag }} npm run test:js:ci
docker run ${{ inputs.build_tag }} npm run test:ts:ci

- name: Build and Push Multi-platform Images
Expand All @@ -70,12 +70,80 @@ jobs:
file: ${{ inputs.dockerfile }}
target: ${{ inputs.push_target }}
push: true
tags: ${{ inputs.push_tags }}
tags: ${{ inputs.push_tags }}-arm64
platforms: |
linux/amd64
linux/arm64
build-args: |
version=${{ inputs.img_tag }}
version=${{ inputs.img_tag }}-arm64
GIT_COMMIT_SHA=${{ github.sha }}
cache-from: type=gha
cache-to: type=gha,mode=max
# cache-from: type=gha
# cache-to: type=gha,mode=max

build-transformer-image-amd64:
name: Build Transformer Docker Image AMD64
runs-on: [self-hosted, Linux, X64]
steps:
- name: Checkout
uses: actions/[email protected]
with:
fetch-depth: 1

- name: Setup Docker Buildx
uses: docker/[email protected]

- name: Login to DockerHub
uses: docker/[email protected]
with:
username: ${{ env.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PROD_TOKEN }}

- name: Build Docker Image
uses: docker/[email protected]
with:
context: .
file: ${{ inputs.dockerfile }}
target: ${{ inputs.load_target }}
load: true
tags: ${{ inputs.build_tag }}
# cache-from: type=gha
# cache-to: type=gha,mode=max

- name: Run Tests
run: |
docker run ${{ inputs.build_tag }} npm run test:js:ci
docker run ${{ inputs.build_tag }} npm run test:ts:ci

- name: Build and Push Multi-platform Images
uses: docker/[email protected]
with:
context: .
file: ${{ inputs.dockerfile }}
target: ${{ inputs.push_target }}
push: true
tags: ${{ inputs.push_tags }}-amd64
platforms: |
linux/amd64
build-args: |
version=${{ inputs.img_tag }}-amd64
GIT_COMMIT_SHA=${{ github.sha }}
# cache-from: type=gha
# cache-to: type=gha,mode=max

create-manifest:
name: Create multi-arch manifest
runs-on: ubuntu-latest
needs: [build-transformer-image-amd64, build-transformer-image-arm64]

steps:
- name: Set up Docker Buildx
uses: docker/[email protected]

- name: Login to DockerHub
uses: docker/[email protected]
with:
username: ${{ env.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PROD_TOKEN }}

- name: Create multi-arch manifest
run: |
docker buildx imagetools create -t ${{ inputs.push_tags }} ${{ inputs.push_tags }}-amd64 ${{ inputs.push_tags }}-arm64
4 changes: 2 additions & 2 deletions .github/workflows/prepare-for-dev-deploy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ jobs:
uses: ./.github/workflows/build-push-docker-image.yml
with:
build_tag: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name }}
push_tags: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name }},rudderlabs/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name }}
push_tags: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name }}
img_tag: ${{ needs.generate-tag-names.outputs.tag_name }}
dockerfile: Dockerfile
load_target: development
Expand All @@ -67,7 +67,7 @@ jobs:
uses: ./.github/workflows/build-push-docker-image.yml
with:
build_tag: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name_ut }}
push_tags: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name_ut }},rudderlabs/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name_ut }}
push_tags: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name_ut }}
img_tag: ${{ needs.generate-tag-names.outputs.tag_name_ut }}
dockerfile: Dockerfile-ut-func
load_target: development
Expand Down