diff --git a/docs/configuration.md b/docs/configuration.md index 24275f1..9028d25 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -5,8 +5,6 @@ slug: /configuration # Configuration -### Configure the package - Settings have default values that can be overriden using: - dbt project variables (and therefore also by CLI variable override) @@ -16,9 +14,9 @@ Please note that the default region is `us` and there's no way, at the time of w To know which region is related to a job, in the BQ UI, use the `Job history` (bottom panel), take a job and look at `Location` field when clicking on a job. You can also access the region of a dataset/table by opening the details panel of it and check the `Data location` field. -#### Modes +## Modes -##### Region mode (default) +### Region mode (default) In this mode, the package will monitor all the GCP projects in the region specified in the `dbt_project.yml` file. @@ -33,17 +31,18 @@ vars: - Execution project needs to be the same as the storage project else you'll need to use the second mode. - If you have multiple GCP Projects in the same region, you should use the "project mode" (with `input_gcp_projects` setting to specify them) as else you will run into errors such as: `Within a standard SQL view, references to tables/views require explicit project IDs unless the entity is created in the same project that is issuing the query, but these references are not project-qualified: "region-us.INFORMATION_SCHEMA.JOBS"`. -##### Project mode +### Project mode To enable the "project mode", you'll need to define explicitly one mandatory setting to set in the `dbt_project.yml` file: ```yml vars: # dbt bigquery monitoring vars + bq_region: 'us' input_gcp_projects: [ 'my-gcp-project', 'my-gcp-project-2' ] ``` -##### GCP Billing export +### GCP Billing export GCP Billing export is a feature that allows you to export your billing data to BigQuery. It allows the package to track the real cost of your queries and storage overtime. @@ -60,23 +59,20 @@ vars: gcp_billing_export_table: 'my_table' ``` -#### Customizing the package configuration +### Customizing the package configuration -
- Settings details - Following settings are defined with following template: `dbt_project_variable` (__Environment variable__) : description (default if any). -#### Optional settings +### Optional settings -##### Environment +#### Environment - `input_gcp_projects` (__DBT_BQ_MONITORING_GCP_PROJECTS__) : list of GCP projects to monitor (default: `[]`) - `bq_region` (__DBT_BQ_MONITORING_REGION__) : region where the monitored projects are located (default: `us`) -##### Pricing +#### Pricing - `use_flat_pricing` (__DBT_BQ_MONITORING_USE_FLAT_PRICING__) : whether to use flat pricing or not (default: `true`) - `per_billed_tb_price` (__DBT_BQ_MONITORING_PER_BILLED_TB_PRICE__) : price in US dollars per billed TB of data processed (default: `6,25`) @@ -89,7 +85,7 @@ Following settings are defined with following template: `dbt_project_variable` ( - `bi_engine_gb_hourly_price` (__DBT_BQ_MONITORING_BI_ENGINE_GB_HOURLY_PRICE__): hourly price in US dollars per BI engine GB of memory (default: `0.0416`) - `free_storage_gb_per_month` (__DBT_BQ_MONITORING_FREE_STORAGE_GB_PER_MONTH__) : free storage GB per month (default: `10`) -###### Package +#### Package - `lookback_window_days` (__DBT_BQ_MONITORING_LOOKBACK_WINDOW_DAYS__) : number of days to look back for monitoring (default: `7`) - `lookback_incremental_billing_window_days` (__DBT_BQ_MONITORING_LOOKBACK_INCREMENTAL_BILLING_WINDOW_DAYS__) : number of days to look back for monitoring (default: `3`) @@ -97,16 +93,13 @@ Following settings are defined with following template: `dbt_project_variable` ( - `output_partition_expiration_days` (__DBT_BQ_MONITORING_OUTPUT_LIMIT_SIZE__) : default table expiration in days for incremental models (default: `365` days) - `use_copy_partitions` (__DBT_BQ_MONITORING_USE_COPY_PARTITIONS__) : whether to use copy partitions or not (default: `true`) -###### GCP Billing export +#### GCP Billing export - `enable_gcp_billing_export` (__DBT_BQ_MONITORING_ENABLE_GCP_BILLING_EXPORT__) : toggle to enable GCP billing export monitoring (default: `false`) - `gcp_billing_export_storage_project` (__DBT_BQ_MONITORING_GCP_BILLING_EXPORT_STORAGE_PROJECT__) : the GCP project where billing export data is stored (default: `'placeholder'` if `enable_gcp_billing_export` is `true`; otherwise `None`) - `gcp_billing_export_dataset` (__DBT_BQ_MONITORING_GCP_BILLING_EXPORT_DATASET__) : the dataset for GCP billing export data (default: `'placeholder'` if `enable_gcp_billing_export` is `true`; otherwise `None`) - `gcp_billing_export_table` (__DBT_BQ_MONITORING_GCP_BILLING_EXPORT_TABLE__) : the table for GCP billing export data (default: `'placeholder'` if `enable_gcp_billing_export` is `true`; otherwise `None`) - -
- -#### Add metadata to queries (Recommended but optional) +### Add metadata to queries (Recommended but optional) To enhance your query metadata with dbt model information, the package provides a dedicated macro that leverage "dbt query comments" (the header set at the top of each query) To configure the query comments, add the following config to `dbt_project.yml`.