Create CI for building word-count applications #19

irux · 2023-09-19T08:41:33Z

closes #16

philipp94831 · 2023-09-20T08:12:26Z

.github/workflows/ci.yaml

+          shell: bash
+          working-directory: ${{env.WORKING_DIRECTORY_APP}}
+
+        - name: Upload ${{env.SENTENCE_PRODUCER_IMAGE_NAME}} image artifact


Please avoid using artifacts as it is slow. Just have another step next that directly pushes the images

raminqaf · 2023-09-20T11:15:28Z

.github/workflows/ci.yaml

+  publish-jib-image:
+    name: Publish tarball images
+    runs-on: ubuntu-22.04
+    needs: build-jib
+
+    steps:
+      - name: Download Docker ${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}} tar artifact
+        uses: actions/download-artifact@v3
+        with:
+          name: ${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}
+          path: ${{env.WORKING_DIRECTORY_APP}}/build
+
+      - name: Download Docker ${{env.SENTENCE_PRODUCER_IMAGE_NAME}} tar artifact
+        uses: actions/download-artifact@v3
+        with:
+          name: ${{env.SENTENCE_PRODUCER_IMAGE_NAME}}
+          path: ${{env.WORKING_DIRECTORY_APP}}/build
+
+      - name: Login to DockerHub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Publish Docker image
+        run: |
+          if [[ $(ls -1 build/*.tar 2>/dev/null | wc -l) != 2 ]]; then
+            >&2 echo "Error: images tar files are needed in the downloaded artifact. You can upload them before using this action: https://github.com/actions/upload-artifact."
+            exit 1
+          fi
+          docker load --input build/${{env.SENTENCE_PRODUCER_IMAGE_NAME}}.tar
+          docker load --input build/${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}.tar
+          if [[ "$GITHUB_REF" =~ ^refs/tags/.* ]]; then
+            # SENTENCE_PRODUCER
+            docker tag ${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:${{ github.run_id }} ${{ env.DOCKER_REGISTRY }}/${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:latest
+            docker tag ${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:${{ github.run_id }} ${{ env.DOCKER_REGISTRY }}/${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:${GITHUB_REF/refs\/tags\//}
+            docker push ${{ env.DOCKER_REGISTRY }}/${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:latest
+            docker push ${{ env.DOCKER_REGISTRY }}/${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:${GITHUB_REF/refs\/tags\//}
+
+            # WORD_COUNT_APP
+            docker tag ${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:${{ github.run_id }} ${{ env.DOCKER_REGISTRY }}/${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:latest
+            docker tag ${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:${{ github.run_id }} ${{ env.DOCKER_REGISTRY }}/${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:${GITHUB_REF/refs\/tags\//}
+            docker push ${{ env.DOCKER_REGISTRY }}/${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:latest
+            docker push ${{ env.DOCKER_REGISTRY }}/${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:${GITHUB_REF/refs\/tags\//}
+          else
+            # SENTENCE_PRODUCER
+            docker tag ${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:${{ github.run_id }} ${{ env.DOCKER_REGISTRY }}/${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:${{ github.run_id }}
+            docker push ${{ env.DOCKER_REGISTRY }}/${{env.SENTENCE_PRODUCER_IMAGE_NAME}}:${{ github.run_id }}
+
+            # WORD_COUNT_APP
+            docker tag ${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:${{ github.run_id }} ${{ env.DOCKER_REGISTRY }}/${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:${{ github.run_id }}
+            docker push ${{ env.DOCKER_REGISTRY }}/${{env.WORD_COUNT_APPLICATION_IMAGE_NAME}}:${{ github.run_id }}
+          fi
+        shell: bash
+        working-directory: ${{env.WORKING_DIRECTORY_APP}}


why not just

# deploy producer app gradle jib -Djib.to.image=bakdata/kpops-demo-sentence-producer -Djib.container.mainClass=com.bakdata.kpops.examples.SentenceProducer # deploy streams-app gradle jib -Djib.to.image=bakdata/kpops-demo-word-count-app-Djib.container.mainClass=com.bakdata.kpops.examples.WordCountApplication

It builds and pushes the image, and IMO it is completely sufficient for this simple example

Then you cannot control the tags

Why not?

# deploy producer app gradle jib -Djib.to.image=bakdata/kpops-demo-sentence-producer:tag1 -Djib.container.mainClass=com.bakdata.kpops.examples.SentenceProducer # deploy streams-app gradle jib -Djib.to.image=bakdata/kpops-demo-word-count-app:tag2 -Djib.container.mainClass=com.bakdata.kpops.examples.WordCountApplication

We want to set multiple tags. latest and version for tag builds e.g.

But I don't understand why we can't use the flag -Djib.to.tags=a,b,c

gradle jib -Djib.to.image=bakdata/kpops-demo-word-count-app -Djib.container.mainClass=com.bakdata.kpops.examples.WordCountApplication -Djib.to.tags=latest,version-tag,bla

That should work

@irux Then please remove all the docker commands and directly use the jib command above to build, tag, and push.

like this @raminqaf

raminqaf · 2023-09-25T08:48:01Z

.github/workflows/ci.yaml

+              ./gradlew jib \
+                --info --stacktrace \
+                --image=$DOCKER_REGISTRY/$SENTENCE_PRODUCER_IMAGE_NAME \
+                -Djib.container.mainClass=com.bakdata.kpops.examples.SentenceProducer \
+                -Djib.to.tags=latest,${GITHUB_REF/refs\/tags\//}
+
+              ./gradlew jib \
+                --info --stacktrace \
+                --image=$DOCKER_REGISTRY/$WORD_COUNT_APPLICATION_IMAGE_NAME \
+                -Djib.container.mainClass=com.bakdata.kpops.examples.WordCountApplication \
+                -Djib.to.tags=latest,${GITHUB_REF/refs\/tags\//}
+            else
+              ./gradlew jib \
+                --info --stacktrace \
+                --image=$DOCKER_REGISTRY/$SENTENCE_PRODUCER_IMAGE_NAME \
+                 -Djib.container.mainClass=com.bakdata.kpops.examples.SentenceProducer \
+                 -Djib.to.tags=${{ github.run_id }}
+
+              ./gradlew jib \
+                --info --stacktrace \
+                --image=$DOCKER_REGISTRY/$WORD_COUNT_APPLICATION_IMAGE_NAME \
+                -Djib.container.mainClass=com.bakdata.kpops.examples.WordCountApplication \
+                -Djib.to.tags=${{ github.run_id }}
+            fi


You can move these to its own reusable action

So it takes the image, main class, and the tags as the input parameter

raminqaf · 2023-10-04T08:16:36Z

.github/workflows/jib-uploader/action.yaml

+  working-directory:
+    description: "working directory to run the commands"
+    required: true


It can be optional. With the default of ./

raminqaf · 2023-10-04T08:17:33Z

.github/workflows/jib-uploader/action.yaml

@@ -0,0 +1,29 @@
+name: "Build and upload Jib "


WDYT?

Suggested change

name: "Build and upload Jib "

name: "Jib: build and push image"

raminqaf · 2023-10-04T08:18:39Z

.github/workflows/jib-uploader/action.yaml

@@ -0,0 +1,29 @@
+name: "Build and upload Jib "
+description: "Build the Jib image and upload it to the image repository"


Suggested change

description: "Build the Jib image and upload it to the image repository"

description: "Build the Jib image and push it to the image repository."

raminqaf · 2023-10-04T08:19:33Z

.github/workflows/jib-uploader/action.yaml

A PR in https://github.com/bakdata/ci-templates would be appreciated! ❤️

Do you think it makes sense to add this as an action ?

yes, definitely! we would also need it for https://github.com/bakdata/pipeline-word-count :)

raminqaf · 2023-10-09T08:39:10Z

.github/actions/jib-uploader/action.yaml

Could you please change the name of this action to jib only? The -uploader is a bit misleading. I thought it would use the upload action of GitHub to upload the tarball.

because we don't want to use the artifacts uploader anymore (#19 (comment)) this action already push to the docker repository

raminqaf · 2024-12-12T12:49:17Z

Closing due to inactivity and solved in #29

irux added 13 commits September 19, 2023 10:40

Test build and test stages

40a80d4

Test change of directory

003348c

Rollback

b087abe

Change gitignore

743b209

Add wrapper again

39028bd

Test images

279cc44

Change for env variable

9cfd202

Fix replacement

5206c21

Test publish

04ff945

Fix verification

3b1152a

Fix load

ee27b9c

Add run id for the image tag

3ade98f

Add release

014e89d

philipp94831 reviewed Sep 20, 2023

View reviewed changes

disrupted changed the title ~~Add CI to the word-count pipeline~~ Create CI for building word-count applications Sep 20, 2023

raminqaf requested changes Sep 20, 2023

View reviewed changes

irux added 5 commits September 20, 2023 13:33

Remove upload and download of images

b8f55da

Rename stage

15e851e

Rename stage

b0ee4b8

Change to jib command line only

142e87a

Fix end if

6aeb0e7

raminqaf reviewed Sep 25, 2023

View reviewed changes

irux added 3 commits October 2, 2023 14:55

Add action for the jib

b324bc5

Move to actions

87c1e73

Delete if for always there build

d8a817b

raminqaf reviewed Oct 4, 2023

View reviewed changes

irux added 4 commits October 9, 2023 09:52

Test metadata docker

2ea08b4

Test without images param

2543294

Disable other jobs

de985e9

Comment temporary

c190681

raminqaf reviewed Oct 9, 2023

View reviewed changes

disrupted assigned irux Feb 28, 2024

disrupted mentioned this pull request Jun 5, 2024

Create CI for publishing Jib container images to Docker Hub bakdata/pipeline-atm-fraud#2

Open

disrupted mentioned this pull request Dec 11, 2024

Update word-count to streams bootstrap 3 #28

Merged

philipp94831 mentioned this pull request Dec 11, 2024

Add CI for build and publish docker Images #29

Merged

raminqaf closed this Dec 12, 2024

disrupted deleted the feature/ci branch December 12, 2024 12:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create CI for building word-count applications #19

Create CI for building word-count applications #19

irux commented Sep 19, 2023

philipp94831 Sep 20, 2023

raminqaf Sep 20, 2023

raminqaf Sep 20, 2023

philipp94831 Sep 20, 2023

raminqaf Sep 20, 2023

philipp94831 Sep 20, 2023

raminqaf Sep 22, 2023

philipp94831 Sep 22, 2023

raminqaf Sep 22, 2023 •

edited

Loading

irux Sep 25, 2023

raminqaf Sep 25, 2023 •

edited

Loading

raminqaf Sep 25, 2023

raminqaf Oct 4, 2023

raminqaf Oct 4, 2023 •

edited

Loading

raminqaf Oct 4, 2023 •

edited

Loading

raminqaf Oct 4, 2023

irux Oct 4, 2023

disrupted Mar 27, 2024

raminqaf Oct 9, 2023

irux Oct 9, 2023

raminqaf commented Dec 12, 2024

	name: "Build and upload Jib "
	name: "Jib: build and push image"

		@@ -0,0 +1,29 @@
		name: "Build and upload Jib "
		description: "Build the Jib image and upload it to the image repository"

	description: "Build the Jib image and upload it to the image repository"
	description: "Build the Jib image and push it to the image repository."

Create CI for building word-count applications #19

Create CI for building word-count applications #19

Conversation

irux commented Sep 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raminqaf Sep 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raminqaf Sep 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raminqaf Oct 4, 2023 • edited Loading

Choose a reason for hiding this comment

raminqaf Oct 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raminqaf commented Dec 12, 2024

raminqaf Sep 22, 2023 •

edited

Loading

raminqaf Sep 25, 2023 •

edited

Loading

raminqaf Oct 4, 2023 •

edited

Loading

raminqaf Oct 4, 2023 •

edited

Loading