Skip to content

Commit

Permalink
Improve configuration documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
Kayrnt committed Nov 29, 2024
1 parent 5f2d7b8 commit 9efd0d4
Showing 1 changed file with 12 additions and 19 deletions.
31 changes: 12 additions & 19 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,6 @@ slug: /configuration

# Configuration

### Configure the package

Settings have default values that can be overriden using:

- dbt project variables (and therefore also by CLI variable override)
Expand All @@ -16,9 +14,9 @@ Please note that the default region is `us` and there's no way, at the time of w

To know which region is related to a job, in the BQ UI, use the `Job history` (bottom panel), take a job and look at `Location` field when clicking on a job. You can also access the region of a dataset/table by opening the details panel of it and check the `Data location` field.

#### Modes
## Modes

##### Region mode (default)
### Region mode (default)

In this mode, the package will monitor all the GCP projects in the region specified in the `dbt_project.yml` file.

Expand All @@ -33,17 +31,18 @@ vars:
- Execution project needs to be the same as the storage project else you'll need to use the second mode.
- If you have multiple GCP Projects in the same region, you should use the "project mode" (with `input_gcp_projects` setting to specify them) as else you will run into errors such as: `Within a standard SQL view, references to tables/views require explicit project IDs unless the entity is created in the same project that is issuing the query, but these references are not project-qualified: "region-us.INFORMATION_SCHEMA.JOBS"`.

##### Project mode
### Project mode

To enable the "project mode", you'll need to define explicitly one mandatory setting to set in the `dbt_project.yml` file:

```yml
vars:
# dbt bigquery monitoring vars
bq_region: 'us'
input_gcp_projects: [ 'my-gcp-project', 'my-gcp-project-2' ]
```

##### GCP Billing export
### GCP Billing export

GCP Billing export is a feature that allows you to export your billing data to BigQuery. It allows the package to track the real cost of your queries and storage overtime.

Expand All @@ -60,23 +59,20 @@ vars:
gcp_billing_export_table: 'my_table'
```

#### Customizing the package configuration
### Customizing the package configuration

<details>
<summary>
Settings details
</summary>

Following settings are defined with following template: `dbt_project_variable` (__Environment variable__) : description (default if any).

#### Optional settings
### Optional settings

##### Environment
#### Environment

- `input_gcp_projects` (__DBT_BQ_MONITORING_GCP_PROJECTS__) : list of GCP projects to monitor (default: `[]`)
- `bq_region` (__DBT_BQ_MONITORING_REGION__) : region where the monitored projects are located (default: `us`)

##### Pricing
#### Pricing

- `use_flat_pricing` (__DBT_BQ_MONITORING_USE_FLAT_PRICING__) : whether to use flat pricing or not (default: `true`)
- `per_billed_tb_price` (__DBT_BQ_MONITORING_PER_BILLED_TB_PRICE__) : price in US dollars per billed TB of data processed (default: `6,25`)
Expand All @@ -89,24 +85,21 @@ Following settings are defined with following template: `dbt_project_variable` (
- `bi_engine_gb_hourly_price` (__DBT_BQ_MONITORING_BI_ENGINE_GB_HOURLY_PRICE__): hourly price in US dollars per BI engine GB of memory (default: `0.0416`)
- `free_storage_gb_per_month` (__DBT_BQ_MONITORING_FREE_STORAGE_GB_PER_MONTH__) : free storage GB per month (default: `10`)

###### Package
#### Package

- `lookback_window_days` (__DBT_BQ_MONITORING_LOOKBACK_WINDOW_DAYS__) : number of days to look back for monitoring (default: `7`)
- `lookback_incremental_billing_window_days` (__DBT_BQ_MONITORING_LOOKBACK_INCREMENTAL_BILLING_WINDOW_DAYS__) : number of days to look back for monitoring (default: `3`)
- `output_limit_size` (__DBT_BQ_MONITORING_OUTPUT_LIMIT_SIZE__) : limit size to use for the models (default: `1000`)
- `output_partition_expiration_days` (__DBT_BQ_MONITORING_OUTPUT_LIMIT_SIZE__) : default table expiration in days for incremental models (default: `365` days)
- `use_copy_partitions` (__DBT_BQ_MONITORING_USE_COPY_PARTITIONS__) : whether to use copy partitions or not (default: `true`)

###### GCP Billing export
#### GCP Billing export
- `enable_gcp_billing_export` (__DBT_BQ_MONITORING_ENABLE_GCP_BILLING_EXPORT__) : toggle to enable GCP billing export monitoring (default: `false`)
- `gcp_billing_export_storage_project` (__DBT_BQ_MONITORING_GCP_BILLING_EXPORT_STORAGE_PROJECT__) : the GCP project where billing export data is stored (default: `'placeholder'` if `enable_gcp_billing_export` is `true`; otherwise `None`)
- `gcp_billing_export_dataset` (__DBT_BQ_MONITORING_GCP_BILLING_EXPORT_DATASET__) : the dataset for GCP billing export data (default: `'placeholder'` if `enable_gcp_billing_export` is `true`; otherwise `None`)
- `gcp_billing_export_table` (__DBT_BQ_MONITORING_GCP_BILLING_EXPORT_TABLE__) : the table for GCP billing export data (default: `'placeholder'` if `enable_gcp_billing_export` is `true`; otherwise `None`)


</details>

#### Add metadata to queries (Recommended but optional)
### Add metadata to queries (Recommended but optional)

To enhance your query metadata with dbt model information, the package provides a dedicated macro that leverage "dbt query comments" (the header set at the top of each query)
To configure the query comments, add the following config to `dbt_project.yml`.
Expand Down

0 comments on commit 9efd0d4

Please sign in to comment.