Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(bqstream): event ordering #2558

Closed
wants to merge 20 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/build-pr-artifacts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ jobs:
uses: ./.github/workflows/build-push-docker-image.yml
with:
build_tag: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name }}
push_tags: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name }}
push_tags: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name }},rudderlabs/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name }}
img_tag: ${{ needs.generate-tag-names.outputs.tag_name }}
dockerfile: Dockerfile
load_target: development
Expand All @@ -59,7 +59,7 @@ jobs:
uses: ./.github/workflows/build-push-docker-image.yml
with:
build_tag: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name_ut }}
push_tags: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name_ut }}
push_tags: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name_ut }},rudderlabs/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name_ut }}
img_tag: ${{ needs.generate-tag-names.outputs.tag_name_ut }}
dockerfile: Dockerfile-ut-func
load_target: development
Expand Down
90 changes: 11 additions & 79 deletions .github/workflows/build-push-docker-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,59 +29,9 @@ env:
DOCKERHUB_USERNAME: rudderlabs

jobs:
build-transformer-image-arm64:
name: Build Transformer Docker Image ARM64
runs-on: [self-hosted, Linux, ARM64]
steps:
- name: Checkout
uses: actions/[email protected]
with:
fetch-depth: 1

- name: Setup Docker Buildx
uses: docker/[email protected]

- name: Login to DockerHub
uses: docker/[email protected]
with:
username: ${{ env.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PROD_TOKEN }}

- name: Build Docker Image
uses: docker/[email protected]
with:
context: .
file: ${{ inputs.dockerfile }}
target: ${{ inputs.load_target }}
load: true
tags: ${{ inputs.build_tag }}
# cache-from: type=gha
# cache-to: type=gha,mode=max

- name: Run Tests
run: |
docker run ${{ inputs.build_tag }} npm run test:js:ci
docker run ${{ inputs.build_tag }} npm run test:ts:ci

- name: Build and Push Multi-platform Images
uses: docker/[email protected]
with:
context: .
file: ${{ inputs.dockerfile }}
target: ${{ inputs.push_target }}
push: true
tags: ${{ inputs.push_tags }}-arm64
platforms: |
linux/arm64
build-args: |
version=${{ inputs.img_tag }}-arm64
GIT_COMMIT_SHA=${{ github.sha }}
# cache-from: type=gha
# cache-to: type=gha,mode=max

build-transformer-image-amd64:
name: Build Transformer Docker Image AMD64
runs-on: [self-hosted, Linux, X64]
build-transformer-image:
name: Build Transformer Docker Image
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/[email protected]
Expand All @@ -105,12 +55,12 @@ jobs:
target: ${{ inputs.load_target }}
load: true
tags: ${{ inputs.build_tag }}
# cache-from: type=gha
# cache-to: type=gha,mode=max
cache-from: type=gha
cache-to: type=gha,mode=max

- name: Run Tests
run: |
docker run ${{ inputs.build_tag }} npm run test:js:ci
docker run ${{ inputs.build_tag }} npm run test:js:ci
docker run ${{ inputs.build_tag }} npm run test:ts:ci

- name: Build and Push Multi-platform Images
Expand All @@ -120,30 +70,12 @@ jobs:
file: ${{ inputs.dockerfile }}
target: ${{ inputs.push_target }}
push: true
tags: ${{ inputs.push_tags }}-amd64
tags: ${{ inputs.push_tags }}
platforms: |
linux/amd64
linux/arm64
build-args: |
version=${{ inputs.img_tag }}-amd64
version=${{ inputs.img_tag }}
GIT_COMMIT_SHA=${{ github.sha }}
# cache-from: type=gha
# cache-to: type=gha,mode=max

create-manifest:
name: Create multi-arch manifest
runs-on: ubuntu-latest
needs: [build-transformer-image-amd64, build-transformer-image-arm64]

steps:
- name: Set up Docker Buildx
uses: docker/[email protected]

- name: Login to DockerHub
uses: docker/[email protected]
with:
username: ${{ env.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PROD_TOKEN }}

- name: Create multi-arch manifest
run: |
docker buildx imagetools create -t ${{ inputs.push_tags }} ${{ inputs.push_tags }}-amd64 ${{ inputs.push_tags }}-arm64
cache-from: type=gha
cache-to: type=gha,mode=max
4 changes: 2 additions & 2 deletions .github/workflows/prepare-for-dev-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ jobs:
uses: ./.github/workflows/build-push-docker-image.yml
with:
build_tag: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name }}
push_tags: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name }}
push_tags: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name }},rudderlabs/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name }}
img_tag: ${{ needs.generate-tag-names.outputs.tag_name }}
dockerfile: Dockerfile
load_target: development
Expand All @@ -67,7 +67,7 @@ jobs:
uses: ./.github/workflows/build-push-docker-image.yml
with:
build_tag: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name_ut }}
push_tags: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name_ut }}
push_tags: rudderstack/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name_ut }},rudderlabs/develop-rudder-transformer:${{ needs.generate-tag-names.outputs.tag_name_ut }}
img_tag: ${{ needs.generate-tag-names.outputs.tag_name_ut }}
dockerfile: Dockerfile-ut-func
load_target: development
Expand Down
23 changes: 0 additions & 23 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,6 @@

All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.

### [1.41.1](https://github.com/rudderlabs/rudder-transformer/compare/v1.41.0...v1.41.1) (2023-09-14)


### Bug Fixes

* **redis:** add destination_id in the key ([55629d6](https://github.com/rudderlabs/rudder-transformer/commit/55629d6d871d275de601a11ec223e0c283641d7b))

## [1.41.0](https://github.com/rudderlabs/rudder-transformer/compare/v1.40.2...v1.41.0) (2023-09-11)


### Features

* add support for updating profile properties through track call ([#2581](https://github.com/rudderlabs/rudder-transformer/issues/2581)) ([f0f20d6](https://github.com/rudderlabs/rudder-transformer/commit/f0f20d654ec5ee8eb078ce7f5610a4666d73fd8c))
* **INT-580:** messageId to event_id mapping support ([#2570](https://github.com/rudderlabs/rudder-transformer/issues/2570)) ([b38843b](https://github.com/rudderlabs/rudder-transformer/commit/b38843bce9bc02d73dceedc6f751f402251fd23a))
* **tiktok_ads:** messageId to event_id mapping support ([72f87bf](https://github.com/rudderlabs/rudder-transformer/commit/72f87bfa381ed7a5b74fb5907f932b78d0257ab9))


### Bug Fixes

* **bugsnag:** alerts ([266514b](https://github.com/rudderlabs/rudder-transformer/commit/266514bd56c150d6c88c1db0733a1da0a4367c02))
* **bugsnag:** alerts ([#2580](https://github.com/rudderlabs/rudder-transformer/issues/2580)) ([9e9eeac](https://github.com/rudderlabs/rudder-transformer/commit/9e9eeacdf79cf8175f87f302242542060f668db8))
* json paths for non tracks events for warehouse ([#2571](https://github.com/rudderlabs/rudder-transformer/issues/2571)) ([e455368](https://github.com/rudderlabs/rudder-transformer/commit/e45536805cf9545b73f4d5bf1be5fad1565ab075))

### [1.40.2](https://github.com/rudderlabs/rudder-transformer/compare/v1.40.1...v1.40.2) (2023-09-06)


Expand Down
Loading