Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create CI for building word-count applications #19

Closed
wants to merge 25 commits into from
Closed

Create CI for building word-count applications #19

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
40a80d4
Test build and test stages
irux Sep 19, 2023
003348c
Test change of directory
irux Sep 19, 2023
b087abe
Rollback
irux Sep 19, 2023
743b209
Change gitignore
irux Sep 19, 2023
39028bd
Add wrapper again
irux Sep 19, 2023
279cc44
Test images
irux Sep 19, 2023
9cfd202
Change for env variable
irux Sep 19, 2023
5206c21
Fix replacement
irux Sep 19, 2023
04ff945
Test publish
irux Sep 19, 2023
3b1152a
Fix verification
irux Sep 19, 2023
ee27b9c
Fix load
irux Sep 19, 2023
3ade98f
Add run id for the image tag
irux Sep 19, 2023
014e89d
Add release
irux Sep 19, 2023
b8f55da
Remove upload and download of images
irux Sep 20, 2023
15e851e
Rename stage
irux Sep 20, 2023
b0ee4b8
Rename stage
irux Sep 20, 2023
142e87a
Change to jib command line only
irux Sep 25, 2023
6aeb0e7
Fix end if
irux Sep 25, 2023
b324bc5
Add action for the jib
irux Oct 2, 2023
87c1e73
Move to actions
irux Oct 2, 2023
d8a817b
Delete if for always there build
irux Oct 2, 2023
2ea08b4
Test metadata docker
irux Oct 9, 2023
2543294
Test without images param
irux Oct 9, 2023
de985e9
Disable other jobs
irux Oct 9, 2023
c190681
Comment temporary
irux Oct 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
157 changes: 157 additions & 0 deletions .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
name: Build, Test and Publish

on:
push:
tags: [ '**' ]
branches: [ '**' ]

env:
DOCKER_REGISTRY: bakdata
SENTENCE_PRODUCER_IMAGE_NAME: kpops-demo-sentence-producer
WORD_COUNT_APPLICATION_IMAGE_NAME: kpops-demo-word-count-app
WORKING_DIRECTORY_APP: "./word-count"

jobs:
build:
name: Build
runs-on: ubuntu-22.04

steps:
- name: Build
uses: bakdata/ci-templates/actions/[email protected]
with:
java-distribution: "microsoft"
java-version: "17"
gradle-version: "wrapper"
working-directory: ${{env.WORKING_DIRECTORY_APP}}

test:
name: Test
runs-on: ubuntu-22.04
needs: build

steps:
- name: Test
uses: bakdata/ci-templates/actions/[email protected]
with:
java-distribution: "microsoft"
java-version: "17"
gradle-version: "wrapper"
working-directory: ${{env.WORKING_DIRECTORY_APP}}

build-jib:
name: Build tarball images
runs-on: ubuntu-22.04
needs: test

steps:
- name: Check out repository
uses: bakdata/ci-templates/actions/[email protected]

- name: Set up Gradle
uses: bakdata/ci-templates/actions/[email protected]
with:
java-distribution: "microsoft"
java-version: "17"
gradle-version: "wrapper"

- name: Build Docker images
run: |
./gradlew jibBuildTar \
--info --stacktrace \
--image=$SENTENCE_PRODUCER_IMAGE_NAME:${{ github.run_id }} \
-Djib.outputPaths.tar=build/$SENTENCE_PRODUCER_IMAGE_NAME.tar -Djib.container.mainClass=com.bakdata.kpops.examples.SentenceProducer

./gradlew jibBuildTar \
--info --stacktrace \
--image=$WORD_COUNT_APPLICATION_IMAGE_NAME:${{ github.run_id }} \
-Djib.outputPaths.tar=build/$WORD_COUNT_APPLICATION_IMAGE_NAME.tar -Djib.container.mainClass=com.bakdata.kpops.examples.WordCountApplication
shell: bash
working-directory: ${{env.WORKING_DIRECTORY_APP}}

- name: Upload ${{env.SENTENCE_PRODUCER_IMAGE_NAME}} image artifact
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please avoid using artifacts as it is slow. Just have another step next that directly pushes the images

uses: actions/upload-artifact@v3
with:
name: ${{env.SENTENCE_PRODUCER_IMAGE_NAME}}
path: ${{env.WORKING_DIRECTORY_APP}}/build/${{env.SENTENCE_PRODUCER_IMAGE_NAME}}.tar
retention-days: 1

- name: Upload ${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}} image artifact
uses: actions/upload-artifact@v3
with:
name: ${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}
path: ${{env.WORKING_DIRECTORY_APP}}/build/${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}.tar
retention-days: 1


publish-jib-image:
name: Publish tarball images
runs-on: ubuntu-22.04
needs: build-jib

steps:
- name: Download Docker ${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}} tar artifact
uses: actions/download-artifact@v3
with:
name: ${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}
path: ${{env.WORKING_DIRECTORY_APP}}/build

- name: Download Docker ${{env.SENTENCE_PRODUCER_IMAGE_NAME}} tar artifact
uses: actions/download-artifact@v3
with:
name: ${{env.SENTENCE_PRODUCER_IMAGE_NAME}}
path: ${{env.WORKING_DIRECTORY_APP}}/build

- name: Login to DockerHub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Publish Docker image
run: |
if [[ $(ls -1 build/*.tar 2>/dev/null | wc -l) != 2 ]]; then
>&2 echo "Error: images tar files are needed in the downloaded artifact. You can upload them before using this action: https://github.com/actions/upload-artifact."
exit 1
fi
docker load --input build/${{env.SENTENCE_PRODUCER_IMAGE_NAME}}.tar
docker load --input build/${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}.tar
if [[ "$GITHUB_REF" =~ ^refs/tags/.* ]]; then
# SENTENCE_PRODUCER
docker tag ${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:${{ github.run_id }} ${{ env.DOCKER_REGISTRY }}/${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:latest
docker tag ${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:${{ github.run_id }} ${{ env.DOCKER_REGISTRY }}/${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:${GITHUB_REF/refs\/tags\//}
docker push ${{ env.DOCKER_REGISTRY }}/${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:latest
docker push ${{ env.DOCKER_REGISTRY }}/${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:${GITHUB_REF/refs\/tags\//}

# WORD_COUNT_APP
docker tag ${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:${{ github.run_id }} ${{ env.DOCKER_REGISTRY }}/${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:latest
docker tag ${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:${{ github.run_id }} ${{ env.DOCKER_REGISTRY }}/${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:${GITHUB_REF/refs\/tags\//}
docker push ${{ env.DOCKER_REGISTRY }}/${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:latest
docker push ${{ env.DOCKER_REGISTRY }}/${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:${GITHUB_REF/refs\/tags\//}
else
# SENTENCE_PRODUCER
docker tag ${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:${{ github.run_id }} ${{ env.DOCKER_REGISTRY }}/${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:${{ github.run_id }}
docker push ${{ env.DOCKER_REGISTRY }}/${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:${{ github.run_id }}

# WORD_COUNT_APP
docker tag ${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:${{ github.run_id }} ${{ env.DOCKER_REGISTRY }}/${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:${{ github.run_id }}
docker push ${{ env.DOCKER_REGISTRY }}/${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:${{ github.run_id }}
fi
shell: bash
working-directory: ${{env.WORKING_DIRECTORY_APP}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not just

# deploy producer app
gradle jib -Djib.to.image=bakdata/kpops-demo-sentence-producer -Djib.container.mainClass=com.bakdata.kpops.examples.SentenceProducer


# deploy streams-app
gradle jib -Djib.to.image=bakdata/kpops-demo-word-count-app-Djib.container.mainClass=com.bakdata.kpops.examples.WordCountApplication

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It builds and pushes the image, and IMO it is completely sufficient for this simple example

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then you cannot control the tags

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not?

# deploy producer app
gradle jib -Djib.to.image=bakdata/kpops-demo-sentence-producer:tag1 -Djib.container.mainClass=com.bakdata.kpops.examples.SentenceProducer


# deploy streams-app
gradle jib -Djib.to.image=bakdata/kpops-demo-word-count-app:tag2 -Djib.container.mainClass=com.bakdata.kpops.examples.WordCountApplication

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to set multiple tags. latest and version for tag builds e.g.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I don't understand why we can't use the flag -Djib.to.tags=a,b,c

gradle jib -Djib.to.image=bakdata/kpops-demo-word-count-app -Djib.container.mainClass=com.bakdata.kpops.examples.WordCountApplication
-Djib.to.tags=latest,version-tag,bla

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That should work

Copy link
Contributor

@raminqaf raminqaf Sep 22, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@irux Then please remove all the docker commands and directly use the jib command above to build, tag, and push.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

like this @raminqaf


release:
name: Create GitHub release
if: startsWith(github.ref, 'refs/tags/')
runs-on: ubuntu-22.04
needs: publish-jib-image

steps:
- name: Release on Github
uses: bakdata/ci-templates/actions/[email protected]
with:
github-username: ${{ secrets.USERNAME }}
github-token: ${{ secrets.GH_TOKEN }}
java-distribution: "microsoft"
java-version: "17"
working-directory: ${{env.WORKING_DIRECTORY_APP}}
70 changes: 70 additions & 0 deletions .github/workflows/release.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
name: Release

env:
WORKING_DIRECTORY_APP: "./word-count"

on:
workflow_dispatch:
inputs:
release-type:
type: choice
required: true
default: patch
options:
- patch
- minor
- major

jobs:
build:
name: Build
runs-on: ubuntu-22.04

steps:
- name: Build
uses: bakdata/ci-templates/actions/[email protected]
with:
java-distribution: "microsoft"
java-version: "17"
gradle-version: "wrapper"
working-directory: ${{env.WORKING_DIRECTORY_APP}}

test:
name: Test
runs-on: ubuntu-22.04
needs: build

steps:
- name: Test
uses: bakdata/ci-templates/actions/[email protected]
with:
java-distribution: "microsoft"
java-version: "17"
gradle-version: "wrapper"
working-directory: ${{env.WORKING_DIRECTORY_APP}}

release:
name: Release
runs-on: ubuntu-22.04
needs: test

steps:
- name: Check out repository
uses: bakdata/ci-templates/actions/[email protected]
with:
ref: ${{ github.event.repository.default_branch }}
persist-credentials: false # required for pushing to protected branch later
fetch-depth: 0 # required to build the changelog


- name: Release on Github
id: release
uses: bakdata/ci-templates/actions/[email protected]
with:
release-type: ${{ inputs.release-type }}
github-email: ${{ secrets.GH_EMAIL }}
github-username: ${{ secrets.GH_USERNAME }}
github-token: ${{ secrets.GH_TOKEN }}
java-distribution: "microsoft"
java-version: "17"
working-directory: ${{ inputs.working-directory }}
37 changes: 7 additions & 30 deletions word-count/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,30 +1,7 @@
### Java
# Compiled class file
*.class

# Log file
*.log

# BlueJ files
*.ctxt

# Mobile Tools for Java (J2ME)
.mtj.tmp/

# Package Files #
*.jar
*.war
*.nar
*.ear
*.zip
*.tar.gz
*.rar
build
bin
.idea
.vscode
.DS_Store

*.bin
*.lock
.gradle
.*
!.github
!.gitignore
!.gitmodules
word-count/build/
word-count/out/
word-count/bin/
Binary file added word-count/gradle/wrapper/gradle-wrapper.jar
View file
Open in desktop
Binary file not shown.