Welcome to your Starter Project! This repository is designed to help users kickstart their journey with dbt (data build tool) and Airflow. Whether you're new to data modeling or an experienced data engineer, this repo will assist you in setting up dbt for your environment and tailoring it to your specific data needs.
Datacoves is an enterprise dbt DataOps platform which helps organizations overcome their data delivery challenges quickly using dbt and Airflow, implementing best practices from the start without the need for multiple vendors or costly consultants.
To make use of this repo on your dbt journey, follow these steps:
- Configure your CICD:
- Edit your gitlab-ci,yml or .github/workflows files based on your Data Warehouse and dbt project location. This will involve commenting and uncommenting lines of code in the files.
- For Gitlab users: Generate your Personal Access token.
- Head to user > preferences > Access Tokens
- Name the Token
GITLAB_PUSH_TOKEN - Select the expiration data
- Select api, read_repository, write_repository
- Copy the token since it will not be viewable once you navigate from that screen
- Configure the
GITLAB_PUSH_TOKENvariable in your workflow environment. Settings > CICD > Variables. Be sure to select Masked for sensitive values.
- Configure your workflow environment variables in Gitlab or Github. See .gitlab-ci,yml or .github/workflows. Be sure to select Masked for sensitive values in gitlab or set secrets in github.
-
Configure dbt:
- Configure dbt for your environment by editing the
profiles.ymlfile in theautomate/dbt/directory. Ensure you provide accurate connection details for your Data Warehouse.
- Configure dbt for your environment by editing the
-
Customize Your Project:
- Define your data models in the
modelsdirectory using SQL files. Organize your models according to your data warehouse schema and naming conventions.
- Define your data models in the
-
Run dbt:
- Execute your dbt transformations using the
dbt debugcommand within the repository directory. - Execute your dbt transformations using the
dbt runcommand within the repository directory. This will compile your SQL models and execute them against your warehouse.
- Execute your dbt transformations using the
Happy modeling! 🚀