Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup hatch #779

Closed
wants to merge 20 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
a1cf815
draft commit
mikealfare Apr 20, 2024
2ab9ff5
merge internal pypi release into full release pipeline
mikealfare Apr 20, 2024
c2da70c
draft commit
mikealfare Apr 20, 2024
c30dd66
merge internal pypi release into full release pipeline
mikealfare Apr 20, 2024
2944b5e
commit changes to break into separate prs
mikealfare Apr 21, 2024
5a7ccab
Merge remote-tracking branch 'origin/setup-hatch' into setup-hatch
mikealfare Apr 22, 2024
b4580e6
clean up rebase artifacts
mikealfare Apr 22, 2024
30ac1ba
clean up rebase artifacts
mikealfare Apr 22, 2024
f511e4c
clean up rebase artifacts
mikealfare Apr 22, 2024
8ad4412
rename daily checks
mikealfare Apr 22, 2024
07197e3
Merge branch 'refs/heads/workflow-updates/reformatting' into setup-hatch
mikealfare Apr 22, 2024
154a89d
rename scheduled workflows and variables
mikealfare Apr 22, 2024
b7d2a7d
Merge branch 'refs/heads/workflow-updates/reformatting' into setup-hatch
mikealfare Apr 22, 2024
d4600b2
typo
mikealfare Apr 22, 2024
e9d7bd7
Merge branch 'refs/heads/workflow-updates/reformatting' into setup-hatch
mikealfare Apr 22, 2024
e929fd9
move schedule back into workflow
mikealfare Apr 22, 2024
2dd1a2e
Merge branch 'refs/heads/workflow-updates/reformatting' into setup-hatch
mikealfare Apr 22, 2024
fb2b0ba
merge aws iam user configuration into the integration tests workflow
mikealfare Apr 22, 2024
5b094f1
add ci workflows back at the adapter level
mikealfare Apr 23, 2024
8d27c17
updates refleching changes in actions repo
mikealfare Apr 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 0 additions & 35 deletions .bumpversion.cfg

This file was deleted.

38 changes: 38 additions & 0 deletions .github/actions/update-dev-branches/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: "Update dev branches"
description: "Update dev branches for testing cross-repo changes"

inputs:
dbt-adapters-branch:
description: "The branch/tag of `dbt-adapters` to use"
default: ""
dbt-common-branch:
description: "The branch/tag of `dbt-common` to use"
default: ""
dbt-core-branch:
description: "The branch/tag of `dbt-core` to use"
default: ""

runs:
using: composite
steps:
- name: "[DEBUG] Inputs"
shell: bash
run: |
echo dbt-adapters : ${{ inputs.dbt-adapters-branch }}
echo dbt-common : ${{ inputs.dbt-common-branch }}
echo dbt-core : ${{ inputs.dbt-core-branch }}

- name: "Update `dbt-adapters` branch"
if: ${{ inputs.dbt-adapters-branch }}
shell: bash
run: ./.github/actions/update-dev-branches/update_dev_branch.sh "dbt-adapters" ${{ inputs.dbt-adapters-branch }}

- name: "Update `dbt-common` branch"
if: ${{ inputs.dbt-common-branch }}
shell: bash
run: ./.github/actions/update-dev-branches/update_dev_branch.sh "dbt-common" ${{ inputs.dbt-common-branch }}

- name: "Update `dbt-core` branch"
if: ${{ inputs.dbt-core-branch }}
shell: bash
run: ./.github/actions/update-dev-branches/update_dev_branch.sh "dbt-core" ${{ inputs.dbt-core-branch }}
15 changes: 15 additions & 0 deletions .github/actions/update-dev-branches/update_dev_branch.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
#!/bin/bash -e
set -e

package=$1
branch=$2

file="pyproject.toml"
sed_pattern="s|${package}.git@main*|${package}.git@${branch}|g"

# mac ships with a different version of sed that requires a delimiter arg
if [[ "$OSTYPE" == darwin* ]]; then
sed -i "" "$sed_pattern" $file
else
sed -i "$sed_pattern" $file
fi
87 changes: 0 additions & 87 deletions .github/scripts/integration-test-matrix.js

This file was deleted.

20 changes: 0 additions & 20 deletions .github/scripts/update_dbt_core_branch.sh

This file was deleted.

44 changes: 44 additions & 0 deletions .github/workflows/build-artifacts.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
name: "Build artifacts"

on:
workflow_call:
inputs:
ref:
description: "The branch/tag/commit to run integration tests on"
type: string
required: true
archive-name:
description: "Name of the archive for build artifacts, used during a release"
type: string
default: ""

permissions: read-all

concurrency:
group: "${{ github.workflow }}-${{ github.event_name }}-${{ inputs.ref }}"
cancel-in-progress: true

jobs:
build-artifacts:
name: "Build artifacts"
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ${{ fromJSON(vars.DBT_PYTHON_VERSIONS) }}
steps:
- name: "Check out ${{ github.repository }}@${{ inputs.ref }}"
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref }}
persist-credentials: false

- name: "Setup environment"
uses: dbt-labs/actions/hatch/environment/create@add-hatch-actions
with:
python-version: ${{ matrix.python-version }}

- name: "Build artifacts"
uses: dbt-labs/actions/hatch/artifacts/create.yml@add-hatch-actions
with:
archive-name: ${{ inputs.archive-name }}
45 changes: 45 additions & 0 deletions .github/workflows/ci-test-suite.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
name: "CI test suite"

on:
workflow_call:
inputs:
ref:
description: "The branch/tag/commit to run integration tests on"
type: string
required: true
archive-name:
description: "Name of the archive for build artifacts, used during a release"
type: string
default: ""

permissions: read-all

concurrency:
group: "${{ github.workflow }}-${{ github.event_name }}-${{ inputs.ref }}"
cancel-in-progress: true

jobs:
code-quality:
name: "Run code quality"
uses: dbt-labs/actions/.github/workflows/code-quality.yml@add-hatch-actions
with:
ref: ${{ inputs.ref }}

unit-tests:
name: "Run unit tests"
uses: dbt-labs/actions/.github/workflows/unit-tests.yml@add-hatch-actions
with:
ref: ${{ inputs.ref }}

integration-tests:
name: "Run integration tests"
uses: dbt-labs/actions/.github/workflows/integration-tests.yml@add-hatch-actions
with:
ref: ${{ inputs.ref }}

build-artifacts:
name: "Verify build artifacts"
uses: dbt-labs/actions/.github/workflows/build-artifacts.yml@add-hatch-actions
with:
ref: ${{ inputs.ref }}
archive-name: ${{ inputs.archive-name }}
52 changes: 52 additions & 0 deletions .github/workflows/code-quality.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# **what?**
# Run code quality checks, e.g. `black`, `flake8`, `mypy`, via `pre-commit`
#
# **why?**
# Ensure code quality meets dbt Labs standards
#
# **when?**
# - runs on all PRs into protected branches
# - runs indirectly during releases and release tests
# - can be manually run
name: "Code quality"

on:
workflow_call:
inputs:
ref:
description: "The branch/tag/commit to run code quality on"
type: string
required: true

permissions: read-all

concurrency:
group: "${{ github.workflow }}-${{ github.event_name }}-${{ inputs.ref }}"
cancel-in-progress: true

jobs:
code-quality:
name: "Code quality"
runs-on: ubuntu-latest
steps:
- name: "Check out ${{ github.repository }}@${{ inputs.ref }}"
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref }}
persist-credentials: false

- name: "Setup environment"
uses: dbt-labs/actions/hatch/environment/create@add-hatch-actions
with:
python-version: ${{ vars.DBT_PYTHON_VERSION }}

- name: "Run code quality"
shell: bash
run: hatch run code-quality

- name: "[INFO] Run code quality"
shell: bash
run: echo "::notice title=$TITLE::$MESSAGE"
env:
TITLE: "Run code quality"
MESSAGE: "Code quality checks completed successfully!"
40 changes: 40 additions & 0 deletions .github/workflows/daily-checks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# **what?**
# - run scheduled CI on protected branches
# - post a failure message in #dev-adapter-alerts for failures
#
# **why?**
# - ensure release branches are always shippable
# - raise awareness of dependencies shifting beneath us
#
# **when?**
# - daily at 5:00 AM UTC
# - manually
name: "Daily checks"

on:
schedule:
- cron: 0 5 * * *
workflow_dispatch:

permissions: read-all

jobs:
ci-test-suite:
name: "CI test suite"
uses: ./.github/workflows/ci-test-suite.yml
with:
ref: "main"

post-failure:
runs-on: ubuntu-latest
needs: test
if: ${{ failure() }}
steps:
- name: Posting scheduled run failures
uses: ravsamhq/notify-slack-action@v2
if: ${{ github.event_name == 'schedule' }}
with:
notification_title: 'Redshift nightly integration test failed'
status: ${{ job.status }}
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_DEV_ADAPTER_ALERTS }}
Loading
Loading