From 341eefa624c984f0da91de668344c522a394f32b Mon Sep 17 00:00:00 2001 From: Matthieu Caneill Date: Fri, 7 Jun 2024 10:01:04 +0200 Subject: [PATCH] Create contributors guide (#39) While there, fix some inconsistencies in the documentation website, and update the readme. --- CHANGELOG.md | 2 + README.md | 104 ++++++----------------- docs/CONTRIBUTING.md | 141 ++++++++++++++++++++++++++++++++ docs/configuration.md | 6 +- docs/create_rules.md | 8 +- docs/index.md | 4 +- docs/reference/rules/generic.md | 3 - mkdocs.yml | 1 + 8 files changed, 175 insertions(+), 94 deletions(-) create mode 100644 docs/CONTRIBUTING.md delete mode 100644 docs/reference/rules/generic.md diff --git a/CHANGELOG.md b/CHANGELOG.md index cd5dfad..b5105b2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,8 @@ and this project adheres to ## [Unreleased] +- Create contributors guide for the documentation website. + ## [0.1.1] - 2023-06-03 ### Added diff --git a/README.md b/README.md index 4542046..d197543 100644 --- a/README.md +++ b/README.md @@ -7,92 +7,34 @@ [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/dbt-score.svg)](https://pypi.org/project/dbt-score) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://makeapullrequest.com) -Linter for dbt model metadata. +## What is `dbt-score`? -## Development +`dbt-score` is a linter for dbt model metadata. -### Prerequisites +[dbt](https://getdbt.com/) (Data Build Tool) is a great framework for creating, +building, organizing, testing and documenting _data models_, i.e. data sets +living in a database or a data warehouse. Through a declarative approach, it +allows data practitioners to build data with a methodology inspired by software +development practices. -You'll need the following prerequisites: +This leads to data models being bundled with a lot of metadata, such as +documentation, data tests, access control information, column types and +constraints, 3rd party integrations... Not to mention any other metadata that +organizations need, fully supported through the `meta` parameter. -- Any Python version starting from 3.11 -- [pre-commit](https://pre-commit.com/) -- [PDM](https://pdm-project.org/2.12/) +At scale, with hundreds or thousands of data models, all this metadata can +become confusing, disparate, and inconsistent. It's hard to enforce good +practices and maintain them in continuous integration systems. This is where +`dbt-score` plays its role: by allowing data teams to programatically define and +enforce metadata rules, in an easy and scalable manner. -Configure development environment running these commands from the project's -root: +## Documentation -```shell -pre-commit install -pdm install --group :all -``` +Everything you need (and more) can be found in +[`dbt-score` documentation website](https://dbt-score.picnic.tech/). -The pdm command will install all project's dependency groups, including all the -dependencies needed for development purposes. +## Contributing -### Lint - -`dbt_score` uses: - -- [ruff](https://docs.astral.sh/ruff/) for fast linting and formatting. -- [mypy](https://mypy.readthedocs.io/en/stable/) for type checking. -- [pre-commit-hooks](https://github.com/pre-commit/pre-commit-hooks). -- [prettier-hooks](https://github.com/pre-commit/mirrors-prettier). - -Cheatsheet: - -```shell -pdm run ruff check . -pdm run ruff check --fix -pdm run mypy . -pdm run tox -e lint -``` - -### Test - -`dbt_score` uses: - -- [pytest](https://docs.pytest.org/) as a main test framework. -- [coverage](https://coverage.readthedocs.io/en/latest/index.html) for test - coverage. -- [tox](https://tox.wiki/en/latest/) for testing against multiple Python - versions. - -Cheatsheet: - -```shell -pdm run tox -e py -pdm run pytest -pdm run coverage run -m pytest -``` - -### Docs - -`dbt_score` uses: - -- [mkdocs](https://www.mkdocs.org/) for docs generation. -- [mkdocstrings](https://mkdocstrings.github.io/) for automatic docs from - sources. - -Cheatsheet: - -```shell -pdm run mkdocs build -pdm run mkdocs serve -``` - -### Pre-commit - -Cheatsheet: - -Execute hooks manually: - -```shell -pre-commit run --all-files -``` - -Create a commit bypassing hooks: - -```shell -git commit --no-verify -``` +Would you like to contribute to `dbt-score`? That's great news! Please follow +[the guide on the documentation website](https://dbt-score.picnic.tech/contributors_guide). +🚀 diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md new file mode 100644 index 0000000..7cea68f --- /dev/null +++ b/docs/CONTRIBUTING.md @@ -0,0 +1,141 @@ +# Contributor's guide + +`dbt-score` is free software, your contributions are welcome! 🚀 + +## Reporting bugs + +If you encountered a bug, check the +[issue tracker on Github](https://github.com/PicnicSupermarket/dbt-score/issues) +to see if it is already known. If not, feel free to +[open a new issue](https://github.com/PicnicSupermarket/dbt-score/issues/new). +Share all relevant details, especially how to reproduce the problem. + +We'd love to hear from you and help make `dbt-score` as stable as we can. + +## Developing new features + +### Adding rules + +The linting rules bundled with `dbt-score` aim to be as generic as possible, and +apply to a large majority of dbt projects. + +This is why they are not very opinionated: for example, we believe documenting +data models is important, and hope it's not a controversial opinion ;) Of +course, it's always possible to disable any rule. + +If you think a new rule should be created in `dbt-score`, feel free to +[open an issue](https://github.com/PicnicSupermarket/dbt-score/issues/new) to +discuss it first - this might save you some work in case maintainers don't see a +fit. + +If your rule idea is not generic and applies to your own project and logic, +`dbt-score` has been designed to fully [support custom rules](create_rules.md). +Create as many as you need for your purposes! + +### Fixing bugs + +We love bug squashing! You can open a pull request to fix any bug you encounter +in `dbt-score`. If the changes are large enough, refer to the next section +first - discussing a solution in a Github issue is always a good idea to avoid +unnecessary work and orchestrate efforts. + +### Adding or changing core features + +Before implementing or changing a new feature, we kindly ask you to +[open a Github issue](https://github.com/PicnicSupermarket/dbt-score/issues/new/) +to get the maintainers' opinion on that feature. It might have been already +considered, discussed, or already in the works. + +We aim to maintain a high code coverage in `dbt-score`'s unit tests, so new +features should be properly tested for happy and unhappy paths. + +If the feature has direct impact on users, it should also be reflected in the +documentation website. + +## Development environment + +### Prerequisites + +You'll need the following: + +- Any Python version starting from 3.11 +- [pre-commit](https://pre-commit.com/) (recommended) +- [PDM](https://pdm-project.org/2.12/) + +After cloning the repository with git, configure your development environment by +running these commands from the project's root: + +```shell +pre-commit install +pdm install --group :all +``` + +The pdm command will install all project's dependency groups, including all the +dependencies needed for development purposes. + +### Lint + +`dbt-score` uses: + +- [ruff](https://docs.astral.sh/ruff/) for fast linting and formatting. +- [mypy](https://mypy.readthedocs.io/en/stable/) for type checking. +- [pre-commit-hooks](https://github.com/pre-commit/pre-commit-hooks). +- [prettier-hooks](https://github.com/pre-commit/mirrors-prettier). + +Cheatsheet: + +```shell +pdm run ruff check . +pdm run ruff check --fix +pdm run mypy . +pdm run tox -e lint +``` + +### Test + +`dbt-score` uses: + +- [pytest](https://docs.pytest.org/) as a main test framework. +- [coverage](https://coverage.readthedocs.io/en/latest/index.html) for test + coverage. +- [tox](https://tox.wiki/en/latest/) for testing against multiple Python + versions. + +Cheatsheet: + +```shell +pdm run tox -e py +pdm run pytest +pdm run coverage run -m pytest +``` + +### Docs + +`dbt-score` uses: + +- [mkdocs](https://www.mkdocs.org/) for docs generation. +- [mkdocstrings](https://mkdocstrings.github.io/) for automatic docs from + sources. + +Cheatsheet: + +```shell +pdm run mkdocs build +pdm run mkdocs serve +``` + +### Pre-commit + +Cheatsheet: + +Execute hooks manually: + +```shell +pre-commit run --all-files +``` + +Create a commit bypassing hooks: + +```shell +git commit --no-verify +``` diff --git a/docs/configuration.md b/docs/configuration.md index e7fb6f6..6f8f746 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -66,7 +66,7 @@ All badges except `wip` can be configured with the following option: score required for a model to be rewarded with a certain badge. The default values can be found in the -[BadgeConfig](/reference/config/#dbt_score.config.BadgeConfig). +[BadgeConfig](reference/config.md#dbt_score.config.BadgeConfig). #### Rule configuration @@ -81,10 +81,10 @@ Every rule can be configured with the following option: of 4. Some rules have additional configuration options, e.g. -[sql_has_reasonable_number_of_lines](/rules/generic/#sql_has_reasonable_number_of_lines). +[sql_has_reasonable_number_of_lines](rules/generic.md#sql_has_reasonable_number_of_lines). Depending on the rule, the options will have different names, types and default values. In the case of the -[sql_has_reasonable_number_of_lines](/rules/generic/#sql_has_reasonable_number_of_lines), +[sql_has_reasonable_number_of_lines](rules/generic.md#sql_has_reasonable_number_of_lines), the `max_lines` option can be configured. ## Command line diff --git a/docs/create_rules.md b/docs/create_rules.md index 7e2016b..054c458 100644 --- a/docs/create_rules.md +++ b/docs/create_rules.md @@ -2,8 +2,9 @@ In order to lint and score models, `dbt-score` uses a set of rules that are applied to each model. A rule can pass or fail when it is run. Based on the -severity of the rule models are scored. `dbt-score` has a set of rules enabled -by default, which can be found [here](reference/rules/generic.md). +severity of the rule, models are scored with the weighted average of the rules +results. Note that `dbt-score` comes bundled with a +[set of default rules](rules/generic.md). On top of the generic rules, it's possible to add your own rules. Two ways exist to create a new rule: @@ -19,7 +20,6 @@ The `@rule` decorator can be used to easily create a new rule: ```python from dbt_score import Model, rule, RuleViolation - @rule def has_description(model: Model) -> RuleViolation | None: """A model should have a description.""" @@ -36,7 +36,6 @@ The severity of a rule can be set using the `severity` argument: ```python from dbt_score import rule, Severity - @rule(severity=Severity.HIGH) ``` @@ -48,7 +47,6 @@ class: ```python from dbt_score import Model, Rule, RuleViolation - class HasDescription(Rule): description = "A model should have a description." diff --git a/docs/index.md b/docs/index.md index 043a536..0793a49 100644 --- a/docs/index.md +++ b/docs/index.md @@ -10,8 +10,8 @@ encourage) good practices. ## Example -```shell -$ dbt-score lint +``` +> dbt-score lint 🥇 customers (score: 10.0) OK dbt_score.rules.generic.has_description OK dbt_score.rules.generic.has_owner: Model lacks an owner. diff --git a/docs/reference/rules/generic.md b/docs/reference/rules/generic.md deleted file mode 100644 index 53f7f7a..0000000 --- a/docs/reference/rules/generic.md +++ /dev/null @@ -1,3 +0,0 @@ -# Generic - -::: dbt_score.rules.generic diff --git a/mkdocs.yml b/mkdocs.yml index 54f4928..de053b6 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -38,4 +38,5 @@ nav: - reference/formatters/index.md - reference/formatters/human_readable_formatter.md - reference/formatters/manifest_formatter.md + - Contributor's guide: CONTRIBUTING.md - Changelog: https://github.com/PicnicSupermarket/dbt-score/blob/master/CHANGELOG.md