diff --git a/website/docs/docs/deploy/advanced-ci.md b/website/docs/docs/deploy/advanced-ci.md index 624620d6623..923fa9cdd53 100644 --- a/website/docs/docs/deploy/advanced-ci.md +++ b/website/docs/docs/deploy/advanced-ci.md @@ -32,12 +32,12 @@ dbt reports the comparison differences in: ### Considerations -You can set a [event_time](/reference/resource-configs/event-time) for a model, seed, snapshot, or source to represent the actual timestamp of the event, rather than something like loading date. This configuration is important for certain features, such as [Incremental microbatch](/docs/build/incremental-microbatch) and compare changes in CI/CD workflows, where it ensures the same time-slice of data is accurately compared between your CI and production environments. +You can set the [`event_time`](/reference/resource-configs/event-time) config for a model, seed, snapshot, or source to represent the actual timestamp of the event, rather than something like loading date. This configuration is important for certain features, such as [Incremental microbatch](/docs/build/incremental-microbatch) and compare changes in CI/CD workflows, where it ensures the same time-slice of data is accurately compared between your CI and production environments. When configured, `event_time` enables compare changes to: -- Compare data in CI vs. production for overlapping times only, reducing false discrepancies. -- Handle scenarios where CI contains fresher data than production by using only the overlapping timeframe, which avoids incorrect row-count changes. +- Compare data in CI versus production for overlapping times only, reducing false discrepancies. +- Handle scenarios where CI has "fresher" data than production by using only the overlapping timeframe, allowing you to avoid incorrect row-count changes. - Account for subset data builds in CI without flagging filtered-out rows as "deleted" when compared with production. - Coming soon, you'll be able to add a flag to the `dbt compare` command to select the specific time slice to compare. diff --git a/website/docs/reference/resource-configs/event-time.md b/website/docs/reference/resource-configs/event-time.md index 3d7827835e1..4a1bb8930c3 100644 --- a/website/docs/reference/resource-configs/event-time.md +++ b/website/docs/reference/resource-configs/event-time.md @@ -139,9 +139,9 @@ sources: ## Definition -Set the `event_time` to the name of the field that represents the timestamp of the event, as opposed to a date like data loading date. You can configure `event_time` for a [model](/docs/build/models), [seed](/docs/build/seeds), or [source](/docs/build/sources) in your `dbt_project.yml` file, property YAML file, or config block. +Set the `event_time` to the name of the field that represents the timestamp of the event, as opposed to a date like data loading date. You can configure `event_time` for a [model](/docs/build/models), [seed](/docs/build/seeds), [snapshot](/docs/build/snapshots), or [source](/docs/build/sources) in your `dbt_project.yml` file, property YAML file, or config block. -`event_time` is required for [Incremental mircrobatch](/docs/build/incremental-microbatch) and [Advanced CI compare changes](/docs/deploy/advanced-ci) in CI/CD workflows, where it ensures the same time-slice of data is correctly compared between your CI and production environments. +`event_time` is required for [Incremental mircrobatch](/docs/build/incremental-microbatch) and [Advanced CI compare changes](/docs/deploy/advanced-ci#considerations) in CI/CD workflows, where it ensures the same time-slice of data is correctly compared between your CI and production environments. When you configure `event_time`, it enables compare changes to: diff --git a/website/docs/reference/seed-configs.md b/website/docs/reference/seed-configs.md index 5d5c39071d6..a18f1fc28f7 100644 --- a/website/docs/reference/seed-configs.md +++ b/website/docs/reference/seed-configs.md @@ -79,6 +79,8 @@ seeds: + + ```yaml seeds: [](/reference/resource-configs/resource-path): @@ -95,7 +97,28 @@ seeds: [+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {} ``` + + + + +```yaml +seeds: + [](/reference/resource-configs/resource-path): + [+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false + [+](/reference/resource-configs/plus-prefix)[tags](/reference/resource-configs/tags): | [] + [+](/reference/resource-configs/plus-prefix)[pre-hook](/reference/resource-configs/pre-hook-post-hook): | [] + [+](/reference/resource-configs/plus-prefix)[post-hook](/reference/resource-configs/pre-hook-post-hook): | [] + [+](/reference/resource-configs/plus-prefix)[database](/reference/resource-configs/database): + [+](/reference/resource-configs/plus-prefix)[schema](/reference/resource-properties/schema): + [+](/reference/resource-configs/plus-prefix)[alias](/reference/resource-configs/alias): + [+](/reference/resource-configs/plus-prefix)[persist_docs](/reference/resource-configs/persist_docs): + [+](/reference/resource-configs/plus-prefix)[full_refresh](/reference/resource-configs/full_refresh): + [+](/reference/resource-configs/plus-prefix)[meta](/reference/resource-configs/meta): {} + [+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {} + [+](/reference/resource-configs/plus-prefix)[event_time](/reference/resource-configs/event-time): my_time_field +``` + @@ -105,6 +128,8 @@ seeds: + + ```yaml version: 2 @@ -122,13 +147,36 @@ seeds: [full_refresh](/reference/resource-configs/full_refresh): [meta](/reference/resource-configs/meta): {} [grants](/reference/resource-configs/grants): {} + [event_time](/reference/resource-configs/event-time): my_time_field + +``` + + + + +```yaml +version: 2 +seeds: + - name: [] + config: + [enabled](/reference/resource-configs/enabled): true | false + [tags](/reference/resource-configs/tags): | [] + [pre_hook](/reference/resource-configs/pre-hook-post-hook): | [] + [post_hook](/reference/resource-configs/pre-hook-post-hook): | [] + [database](/reference/resource-configs/database): + [schema](/reference/resource-properties/schema): + [alias](/reference/resource-configs/alias): + [persist_docs](/reference/resource-configs/persist_docs): + [full_refresh](/reference/resource-configs/full_refresh): + [meta](/reference/resource-configs/meta): {} + [grants](/reference/resource-configs/grants): {} ``` + - ## Configuring seeds diff --git a/website/docs/reference/snapshot-configs.md b/website/docs/reference/snapshot-configs.md index 8543f4facab..504d7c8a606 100644 --- a/website/docs/reference/snapshot-configs.md +++ b/website/docs/reference/snapshot-configs.md @@ -168,6 +168,24 @@ Configurations can be applied to snapshots using [YAML syntax](/docs/build/snaps + + +```yaml +snapshots: + [](/reference/resource-configs/resource-path): + [+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false + [+](/reference/resource-configs/plus-prefix)[tags](/reference/resource-configs/tags): | [] + [+](/reference/resource-configs/plus-prefix)[alias](/reference/resource-configs/alias): + [+](/reference/resource-configs/plus-prefix)[pre-hook](/reference/resource-configs/pre-hook-post-hook): | [] + [+](/reference/resource-configs/plus-prefix)[post-hook](/reference/resource-configs/pre-hook-post-hook): | [] + [+](/reference/resource-configs/plus-prefix)[persist_docs](/reference/resource-configs/persist_docs): {} + [+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {} + [+](/reference/resource-configs/plus-prefix)[event_time](/reference/resource-configs/event-time): my_time_field +``` + + + + ```yaml snapshots: [](/reference/resource-configs/resource-path): @@ -179,6 +197,7 @@ snapshots: [+](/reference/resource-configs/plus-prefix)[persist_docs](/reference/resource-configs/persist_docs): {} [+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {} ``` + @@ -225,6 +244,7 @@ snapshots: [post-hook](/reference/resource-configs/pre-hook-post-hook): | [] [persist_docs](/reference/resource-configs/persist_docs): {} [grants](/reference/resource-configs/grants): {} + [event_time](/reference/resource-configs/event-time): my_time_field ``` @@ -292,7 +312,6 @@ The following examples demonstrate how to configure snapshots using the `dbt_pro ```yml - snapshots: +unique_key: id ``` @@ -307,7 +326,6 @@ The following examples demonstrate how to configure snapshots using the `dbt_pro ```yml - snapshots: jaffle_shop: +unique_key: id diff --git a/website/docs/reference/source-configs.md b/website/docs/reference/source-configs.md index 64dda8bffde..2aa4cd9ed9b 100644 --- a/website/docs/reference/source-configs.md +++ b/website/docs/reference/source-configs.md @@ -8,7 +8,17 @@ import ConfigGeneral from '/snippets/_config-description-general.md'; ## Available configurations -Sources only support one configuration, [`enabled`](/reference/resource-configs/enabled). + + +Sources supports [`enabled`](/reference/resource-configs/enabled) and [`meta`](/reference/resource-configs/meta). + + + + + +Sources configurations support [`enabled`](/reference/resource-configs/enabled), [`event_time`](/reference/resource-configs/event-time), and [`meta`](/reference/resource-configs/meta) + + ### General configurations @@ -27,12 +37,29 @@ Sources only support one configuration, [`enabled`](/reference/resource-configs/ + + ```yaml sources: [](/reference/resource-configs/resource-path): [+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false + [+](/reference/resource-configs/plus-prefix)[event_time](/reference/resource-configs/event-time): my_time_field + [+](/reference/resource-configs/plus-prefix)[meta](/reference/resource-configs/meta): + key: value ``` + + + + +```yaml +sources: + [](/reference/resource-configs/resource-path): + [+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false + [+](/reference/resource-configs/plus-prefix)[meta](/reference/resource-configs/meta): + key: value +``` + @@ -43,6 +70,8 @@ sources: + + ```yaml version: 2 @@ -50,12 +79,37 @@ sources: - name: [] [config](/reference/resource-properties/config): [enabled](/reference/resource-configs/enabled): true | false + [event_time](/reference/resource-configs/event-time): my_time_field + [meta](/reference/resource-configs/meta): {} + tables: - name: [] [config](/reference/resource-properties/config): [enabled](/reference/resource-configs/enabled): true | false + [event_time](/reference/resource-configs/event-time): my_time_field + [meta](/reference/resource-configs/meta): {} ``` + + + + +```yaml +version: 2 + +sources: + - name: [] + [config](/reference/resource-properties/config): + [enabled](/reference/resource-configs/enabled): true | false + [meta](/reference/resource-configs/meta): {} + tables: + - name: [] + [config](/reference/resource-properties/config): + [enabled](/reference/resource-configs/enabled): true | false + [meta](/reference/resource-configs/meta): {} + +``` + @@ -74,6 +128,8 @@ You can disable sources imported from a package to prevent them from rendering i + + ```yaml sources: your_project_name: @@ -81,11 +137,34 @@ You can disable sources imported from a package to prevent them from rendering i source_name: source_table_name: +enabled: false + +event_time: my_time_field ``` + + + + + ```yaml + sources: + your_project_name: + subdirectory_name: + source_name: + source_table_name: + +enabled: false + ``` + ### Examples + +The following examples show how to configure sources in your dbt project. + +— [Disable all sources imported from a package](#disable-all-sources-imported-from-a-package)
+— [Conditionally enable a single source](#conditionally-enable-a-single-source)
+— [Disable a single source from a package](#disable-a-single-source-from-a-package)
+— [Configure a source with an `event_time`](#configure-a-source-with-an-event_time)
+— [Configure meta to a source](#configure-meta-to-a-source)
+ #### Disable all sources imported from a package To apply a configuration to all sources included from a [package](/docs/build/packages), state your configuration under the [project name](/reference/project-configs/name.md) in the @@ -172,6 +251,53 @@ sources:
+#### Configure a source with an `event_time` + + + +Configuring an [`event_time`](/reference/resource-configs/event-time) for a source is only available in dbt Cloud Versionless or dbt Core versions 1.9 and later. + + + + + +To configure a source with an `event_time`, specify the `event_time` field in the source configuration. This field is used to represent the actual timestamp of the event, rather than something like a loading date. + +For example, if you had a source table called `clickstream` in the `events` source, you can use the timestamp for each event in the `event_timestamp` column as follows: + + + +```yaml +sources: + events: + clickstream: + +event_time: event_timestamp +``` + + +In this example, the `event_time` is set to `event_timestamp`, which has the exact time each clickstream event happened. +Not only is this required for the [incremental microbatching strategy]((/docs/build/incremental-microbatch)), but when you compare data across [CI and production](/docs/deploy/advanced-ci#considerations) environments, dbt will use `event_timestamp` to filter and match data by this event-based timeframe, ensuring that only overlapping timeframes are compared. + + + +#### Configure meta to a source + +Use the `meta` field to assign metadata information to sources. This is useful for tracking additional context, documentation, logging, and more. + +For example, you can add `meta` information to a `clickstream` source to include information about the data source system: + + + +```yaml +sources: + events: + clickstream: + +meta: + source_system: "Google analytics" + data_owner: "marketing_team" +``` + + ## Example source configuration The following is a valid source configuration for a project with: * `name: jaffle_shop`