Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify hatch setup #126

Merged
merged 10 commits into from
Jul 16, 2024
2 changes: 1 addition & 1 deletion .github/workflows/integration-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ jobs:
python-version: ${{ matrix.python-version }}

- name: Run integration tests
run: hatch run integration-tests:all
run: hatch run integration-tests
env:
POSTGRES_TEST_HOST: localhost
POSTGRES_TEST_PORT: 5432
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/release_prep_hatch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
# 1. Bump the version if it has not been bumped
# 2. Generate the changelog (via changie) if there is no markdown file for this version
name: "Release prep"
run-name: "Release prep: Generate changelog and bump ${{ inputs.package }} to ${{ inputs.version }} for release to ${{ inputs.deploy-to }}"
run-name: "Release prep: Generate changelog and bump to ${{ inputs.version }} for release to ${{ inputs.deploy-to }}"
on:
workflow_call:
inputs:
Expand Down Expand Up @@ -342,7 +342,7 @@ jobs:
uses: dbt-labs/dbt-adapters/.github/actions/setup-hatch@main

- name: "Run unit tests"
run: hatch run unit-tests:all
run: hatch run unit-tests

integration-tests:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -387,7 +387,7 @@ jobs:
uses: dbt-labs/dbt-adapters/.github/actions/setup-hatch@main

- name: "Run integration tests"
run: hatch run integration-tests:all
run: hatch run integration-tests
env:
POSTGRES_TEST_HOST: localhost
POSTGRES_TEST_PORT: 5432
Expand Down Expand Up @@ -464,6 +464,6 @@ jobs:
run: echo "sha=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT

# if this is a real release and a release branch was created, delete it
- name: "Delete release branch: ${{ needs.branch.outputs.name }}"
- name: "Delete release branch: ${{ needs.release-branch.outputs.name }}"
if: ${{ inputs.deploy-to == 'prod' && inputs.is-nightly-release == 'false' && needs.release-branch.outputs.name != '' }}
run: git push origin -d ${{ needs.branch.outputs.name }}
run: git push origin -d ${{ needs.release-branch.outputs.name }}
2 changes: 1 addition & 1 deletion .github/workflows/unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,5 @@ jobs:
python-version: ${{ matrix.python-version }}

- name: Run unit tests
run: hatch run unit-tests:all
run: hatch run unit-tests
shell: bash
79 changes: 64 additions & 15 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,26 +66,42 @@ Rather than forking `dbt-labs/dbt-postgres`, use `dbt-labs/dbt-postgres` directl

### Installation

1. Ensure the latest version of `pip` is installed:
1. Ensure the latest versions of `pip` and `hatch` are installed:
```shell
pip install --upgrade pip
pip install --user --upgrade pip hatch
```
2. Configure and activate a virtual environment using `virtualenv` as described in
[Setting up an environment](https://github.com/dbt-labs/dbt-core/blob/HEAD/CONTRIBUTING.md#setting-up-an-environment)
3. Install `dbt-postgres` and development dependencies in the virtual environment
2. This step is optional, but it's recommended. Configure `hatch` to create its virtual environments in the project. Add this block to your `hatch` `config.toml` file:
```toml
# MacOS: ~/Library/Application Support/hatch/config.toml
[dirs.env]
virtual = ".hatch"
```
This makes `hatch` create all virtual environments in the project root inside of the directory `/.hatch`, similar to `/.tox` for `tox`.
It also makes it easier to add this environment as a runner in common IDEs like VSCode and PyCharm.
3. Create a `hatch` environment with all of the development dependencies and activate it:
```shell
hatch run setup
hatch shell
```
4. Run any commands within the virtual environment by prefixing the command with `hatch run`:
```shell
pip install -e .[dev]
hatch run <command>
```

When `dbt-postgres` is installed this way, any changes made to the `dbt-postgres` source code
will be reflected in the virtual environment immediately.


## Testing

`dbt-postgres` contains [unit](https://github.com/dbt-labs/dbt-postgres/tree/main/tests/unit)
and [functional](https://github.com/dbt-labs/dbt-postgres/tree/main/tests/functional) tests.
`dbt-postgres` contains [code quality checks](https://github.com/dbt-labs/dbt-postgres/tree/main/.pre-commit-config.yaml), [unit tests](https://github.com/dbt-labs/dbt-postgres/tree/main/tests/unit),
and [functional tests](https://github.com/dbt-labs/dbt-postgres/tree/main/tests/functional).

### Code quality

Code quality checks can run with a single command:
```shell
hatch run code-quality
```

### Unit tests

Expand All @@ -94,10 +110,14 @@ Unit tests can be run locally without setting up a database connection:
```shell
# Note: replace $strings with valid names

# run all unit tests
hatch run unit-test

# run all unit tests in a module
python -m pytest tests/unit/$test_file_name.py
hatch run unit-tests tests/unit/$test_file_name.py

# run a specific unit test
python -m pytest tests/unit/$test_file_name.py::$test_class_name::$test_method_name
hatch run unit-tests tests/unit/$test_file_name.py::$test_class_name::$test_method_name
```

### Functional tests
Expand All @@ -120,16 +140,45 @@ Functional tests can be run locally with a valid database connection configured
```shell
# Note: replace $strings with valid names

# run all functional tests
hatch run integration-tests

# run all functional tests in a directory
python -m pytest tests/functional/$test_directory
hatch run integration-tests tests/functional/$test_directory

# run all functional tests in a module
python -m pytest tests/functional/$test_dir_and_filename.py
hatch run integration-tests tests/functional/$test_directory/$test_filename.py

# run all functional tests in a class
python -m pytest tests/functional/$test_dir_and_filename.py::$test_class_name
hatch run integration-tests tests/functional/$test_directory/$test_filename.py::$test_class_name

# run a specific functional test
python -m pytest tests/functional/$test_dir_and_filename.py::$test_class_name::$test__method_name
hatch run integration-tests tests/functional/$test_directory/$test_filename.py::$test_class_name::$test__method_name
```

### Testing against a development branch

Some changes require a change in `dbt-common` and/or `dbt-adapters`.
In that case, the dependency on `dbt-common` and/or `dbt-adapters` must be updated to point to the development branch. For example:

```toml
[tool.hatch.envs.default]
dependencies = [
"dbt-common @ git+https://github.com/dbt-labs/dbt-common.git@my-dev-branch",
"dbt-adapters @ git+https://github.com/dbt-labs/dbt-adapters.git@my-dev-branch",
"dbt-tests-adapter @ git+https://github.com/dbt-labs/dbt-adapters.git@my-dev-branch#subdirectory=dbt-tests-adapter",
...,
]
```

This will install `dbt-common`/`dbt-adapters`/`dbt-tests-adapter` as snapshots. In other words, if `my-dev-branch` is updated on GitHub, those updates will not be reflected locally.
In order to pick up those updates, the `hatch` environment(s) will need to be rebuilt:

```shell
exit
hatch env prune
hatch shell
```

## Documentation

Expand Down
42 changes: 14 additions & 28 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@ dependencies = [
"dbt-common>=0.1.0a1,<2.0",
"agate>=1.0,<2.0",
]

[project.urls]
Homepage = "https://github.com/dbt-labs/dbt-postgres"
Documentation = "https://docs.getdbt.com"
Expand All @@ -56,44 +55,31 @@ path = "dbt/adapters/postgres/__version__.py"
dependencies = [
"dbt-adapters @ git+https://github.com/dbt-labs/dbt-adapters.git",
"dbt-common @ git+https://github.com/dbt-labs/dbt-common.git",
"dbt-tests-adapter @ git+https://github.com/dbt-labs/dbt-adapters.git#subdirectory=dbt-tests-adapter",
"dbt-core @ git+https://github.com/dbt-labs/dbt-core.git#subdirectory=core",
'pre-commit==3.7.0;python_version>="3.9"',
'pre-commit==3.5.0;python_version=="3.8"',
]
[tool.hatch.envs.default.scripts]
dev = "pre-commit install"
code-quality = "pre-commit run --all-files"
docker-dev = [
"echo Does not support integration testing, only development and unit testing. See issue https://github.com/dbt-labs/dbt-postgres/issues/99",
"docker build -f docker/dev.Dockerfile -t dbt-postgres-dev .",
"docker run --rm -it --name dbt-postgres-dev -v $(pwd):/opt/code dbt-postgres-dev",
]
docker-prod = "docker build -f docker/Dockerfile -t dbt-postgres ."

[tool.hatch.envs.unit-tests]
dependencies = [
"dbt-adapters @ git+https://github.com/dbt-labs/dbt-adapters.git",
"dbt-common @ git+https://github.com/dbt-labs/dbt-common.git",
"dbt-core @ git+https://github.com/dbt-labs/dbt-core.git#subdirectory=core",
"freezegun",
"pytest",
"pytest-dotenv",
"pytest-mock",
"pytest-xdist",
]
[tool.hatch.envs.unit-tests.scripts]
all = "python -m pytest {args:tests/unit}"

[tool.hatch.envs.integration-tests]
template = "unit-tests"
extra-dependencies = [
"dbt-tests-adapter @ git+https://github.com/dbt-labs/dbt-adapters.git#subdirectory=dbt-tests-adapter",
]
[tool.hatch.envs.integration-tests.env-vars]
[tool.hatch.envs.default.env-vars]
DBT_TEST_USER_1 = "dbt_test_user_1"
DBT_TEST_USER_2 = "dbt_test_user_2"
DBT_TEST_USER_3 = "dbt_test_user_3"
[tool.hatch.envs.integration-tests.scripts]
all = "python -m pytest {args:tests/functional}"
[tool.hatch.envs.default.scripts]
setup = "pre-commit install"
code-quality = "pre-commit run --all-files"
unit-tests = "python -m pytest {args:tests/unit}"
integration-tests = "python -m pytest {args:tests/functional}"
docker-dev = [
"echo Does not support integration testing, only development and unit testing. See issue https://github.com/dbt-labs/dbt-postgres/issues/99",
"docker build -f docker/dev.Dockerfile -t dbt-postgres-dev .",
"docker run --rm -it --name dbt-postgres-dev -v $(pwd):/opt/code dbt-postgres-dev",
]
docker-prod = "docker build -f docker/Dockerfile -t dbt-postgres ."

[tool.hatch.envs.build]
detached = true
Expand Down
Loading