Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add event _time to sources/snapshots/models/seeds #6384

Merged
merged 17 commits into from
Nov 19, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions website/docs/docs/deploy/advanced-ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,12 @@ dbt reports the comparison differences in:
<Lightbox src="/img/docs/dbt-cloud/example-ci-compare-changes-tab.png" width="85%" title="Example of the Compare tab" />

### Considerations
You can set a [event_time](/reference/resource-configs/event-time) for a model, seed, snapshot, or source to represent the actual timestamp of the event, rather than something like loading date. This configuration is important for certain features, such as [Incremental microbatch](/docs/build/incremental-microbatch) and compare changes in CI/CD workflows, where it ensures the same time-slice of data is accurately compared between your CI and production environments.
You can set the [`event_time`](/reference/resource-configs/event-time) config for a model, seed, snapshot, or source to represent the actual timestamp of the event, rather than something like loading date. This configuration is important for certain features, such as [Incremental microbatch](/docs/build/incremental-microbatch) and compare changes in CI/CD workflows, where it ensures the same time-slice of data is accurately compared between your CI and production environments.

When configured, `event_time` enables compare changes to:

- Compare data in CI versus production for overlapping times only, reducing false discrepancies.
- Handle scenarios where CI contains fresher data than production by using only the overlapping timeframe, which avoids incorrect row-count changes.
- Handle scenarios where CI has "fresher" data than production by using only the overlapping timeframe, allowing you to avoid incorrect row-count changes.
- Account for subset data builds in CI without flagging filtered-out rows as "deleted" when compared with production.
- Coming soon, you'll be able to add a flag to the command list allowing you to select the specific time slice to compare.

Expand Down
78 changes: 76 additions & 2 deletions website/docs/reference/model-configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,8 @@ models:

<File name='dbt_project.yml'>

<VersionBlock lastVersion="1.8">

```yaml
models:
[<resource-path>](/reference/resource-configs/resource-path):
Expand All @@ -121,7 +123,29 @@ models:
[+](/reference/resource-configs/plus-prefix)[contract](/reference/resource-configs/contract): {<dictionary>}

```
</VersionBlock>

<VersionBlock firstVersion="1.9">

```yaml
models:
[<resource-path>](/reference/resource-configs/resource-path):
[+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false
[+](/reference/resource-configs/plus-prefix)[tags](/reference/resource-configs/tags): <string> | [<string>]
[+](/reference/resource-configs/plus-prefix)[pre-hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[+](/reference/resource-configs/plus-prefix)[post-hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[+](/reference/resource-configs/plus-prefix)[database](/reference/resource-configs/database): <string>
[+](/reference/resource-configs/plus-prefix)[schema](/reference/resource-properties/schema): <string>
[+](/reference/resource-configs/plus-prefix)[alias](/reference/resource-configs/alias): <string>
[+](/reference/resource-configs/plus-prefix)[persist_docs](/reference/resource-configs/persist_docs): <dict>
[+](/reference/resource-configs/plus-prefix)[full_refresh](/reference/resource-configs/full_refresh): <boolean>
[+](/reference/resource-configs/plus-prefix)[meta](/reference/resource-configs/meta): {<dictionary>}
[+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {<dictionary>}
[+](/reference/resource-configs/plus-prefix)[contract](/reference/resource-configs/contract): {<dictionary>}
[+](/reference/resource-configs/plus-prefix)[event_time](/reference/resource-configs/event-time): my_time_field

```
</VersionBlock>
</File>

</TabItem>
Expand All @@ -131,6 +155,8 @@ models:

<File name='models/properties.yml'>

<VersionBlock lastVersion="1.8">

```yaml
version: 2

Expand All @@ -150,17 +176,63 @@ models:
[grants](/reference/resource-configs/grants): {<dictionary>}
[contract](/reference/resource-configs/contract): {<dictionary>}
```
</VersionBlock>

</File>
<VersionBlock firstVersion="1.9">

</TabItem>
```yaml
version: 2

models:
- name: [<model-name>]
config:
[enabled](/reference/resource-configs/enabled): true | false
[tags](/reference/resource-configs/tags): <string> | [<string>]
[pre_hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[post_hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[database](/reference/resource-configs/database): <string>
[schema](/reference/resource-properties/schema): <string>
[alias](/reference/resource-configs/alias): <string>
[persist_docs](/reference/resource-configs/persist_docs): <dict>
[full_refresh](/reference/resource-configs/full_refresh): <boolean>
[meta](/reference/resource-configs/meta): {<dictionary>}
[grants](/reference/resource-configs/grants): {<dictionary>}
[contract](/reference/resource-configs/contract): {<dictionary>}
[event_time](/reference/resource-configs/event-time): my_time_field
```

</VersionBlock>
</File>

</TabItem>

<TabItem value="config">

<File name='models/<model_name>.sql'>

<VersionBlock lastVersion="1.8">

```jinja

{{ config(
[enabled](/reference/resource-configs/enabled)=true | false,
[tags](/reference/resource-configs/tags)="<string>" | ["<string>"],
[pre_hook](/reference/resource-configs/pre-hook-post-hook)="<sql-statement>" | ["<sql-statement>"],
[post_hook](/reference/resource-configs/pre-hook-post-hook)="<sql-statement>" | ["<sql-statement>"],
[database](/reference/resource-configs/database)="<string>",
[schema](/reference/resource-properties/schema)="<string>",
[alias](/reference/resource-configs/alias)="<string>",
[persist_docs](/reference/resource-configs/persist_docs)={<dict>},
[meta](/reference/resource-configs/meta)={<dict>},
[grants](/reference/resource-configs/grants)={<dict>},
[contract](/reference/resource-configs/contract)={<dictionary>}
) }}

```
</VersionBlock>

<VersionBlock firstVersion="1.9">

```jinja

{{ config(
Expand All @@ -175,9 +247,11 @@ models:
[meta](/reference/resource-configs/meta)={<dict>},
[grants](/reference/resource-configs/grants)={<dict>},
[contract](/reference/resource-configs/contract)={<dictionary>}
[event_time](/reference/resource-configs/event-time): my_time_field
) }}

```
</VersionBlock>

</File>

Expand Down
2 changes: 1 addition & 1 deletion website/docs/reference/resource-configs/event-time.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ sources:

## Definition

Set the `event_time` to the name of the field that represents the timestamp of the event, as opposed to a date like data loading date. You can configure `event_time` for a [model](/docs/build/models), [seed](/docs/build/seeds), or [source](/docs/build/sources) in your `dbt_project.yml` file, property YAML file, or config block.
Set the `event_time` to the name of the field that represents the timestamp of the event, as opposed to a date like data loading date. You can configure `event_time` for a [model](/docs/build/models), [seed](/docs/build/seeds), [snapshot](/docs/build/snapshots), or [source](/docs/build/sources) in your `dbt_project.yml` file, property YAML file, or config block.

`event_time` is required for [Incremental mircrobatch](/docs/build/incremental-microbatch) and [Advanced CI's compare changes](/docs/deploy/advanced-ci) in CI/CD workflows, where it ensures the same time-slice of data is correctly compared between your CI and production environments.

Expand Down
50 changes: 49 additions & 1 deletion website/docs/reference/seed-configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@ seeds:

<File name='dbt_project.yml'>

<VersionBlock lastVersion="1.8">

```yaml
seeds:
[<resource-path>](/reference/resource-configs/resource-path):
Expand All @@ -95,7 +97,28 @@ seeds:
[+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {<dictionary>}

```
</VersionBlock>

<VersionBlock firstVersion="1.9">

```yaml
seeds:
[<resource-path>](/reference/resource-configs/resource-path):
[+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false
[+](/reference/resource-configs/plus-prefix)[tags](/reference/resource-configs/tags): <string> | [<string>]
[+](/reference/resource-configs/plus-prefix)[pre-hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[+](/reference/resource-configs/plus-prefix)[post-hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[+](/reference/resource-configs/plus-prefix)[database](/reference/resource-configs/database): <string>
[+](/reference/resource-configs/plus-prefix)[schema](/reference/resource-properties/schema): <string>
[+](/reference/resource-configs/plus-prefix)[alias](/reference/resource-configs/alias): <string>
[+](/reference/resource-configs/plus-prefix)[persist_docs](/reference/resource-configs/persist_docs): <dict>
[+](/reference/resource-configs/plus-prefix)[full_refresh](/reference/resource-configs/full_refresh): <boolean>
[+](/reference/resource-configs/plus-prefix)[meta](/reference/resource-configs/meta): {<dictionary>}
[+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {<dictionary>}
[+](/reference/resource-configs/plus-prefix)[event_time](/reference/resource-configs/event-time): my_time_field

```
</VersionBlock>
</File>

</TabItem>
Expand All @@ -105,6 +128,8 @@ seeds:

<File name='seeds/properties.yml'>

<VersionBlock firstVersion="1.9">

```yaml
version: 2

Expand All @@ -122,13 +147,36 @@ seeds:
[full_refresh](/reference/resource-configs/full_refresh): <boolean>
[meta](/reference/resource-configs/meta): {<dictionary>}
[grants](/reference/resource-configs/grants): {<dictionary>}
[event_time](/reference/resource-configs/event-time): my_time_field

```
</VersionBlock>

<VersionBlock lastVersion="1.8">

```yaml
version: 2

seeds:
- name: [<seed-name>]
config:
[enabled](/reference/resource-configs/enabled): true | false
[tags](/reference/resource-configs/tags): <string> | [<string>]
[pre_hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[post_hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[database](/reference/resource-configs/database): <string>
[schema](/reference/resource-properties/schema): <string>
[alias](/reference/resource-configs/alias): <string>
[persist_docs](/reference/resource-configs/persist_docs): <dict>
[full_refresh](/reference/resource-configs/full_refresh): <boolean>
[meta](/reference/resource-configs/meta): {<dictionary>}
[grants](/reference/resource-configs/grants): {<dictionary>}
```

</VersionBlock>
</File>

</TabItem>

</Tabs>

## Configuring seeds
Expand Down
22 changes: 20 additions & 2 deletions website/docs/reference/snapshot-configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,24 @@ Configurations can be applied to snapshots using [YAML syntax](/docs/build/snaps

<File name='dbt_project.yml'>

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

grace mentioned event_time is supported for anythign that is ref-able. since snapshot is, i added event_time to it

<VersionBlock firstVersion="1.9">

```yaml
snapshots:
[<resource-path>](/reference/resource-configs/resource-path):
[+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false
[+](/reference/resource-configs/plus-prefix)[tags](/reference/resource-configs/tags): <string> | [<string>]
[+](/reference/resource-configs/plus-prefix)[alias](/reference/resource-configs/alias): <string>
[+](/reference/resource-configs/plus-prefix)[pre-hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[+](/reference/resource-configs/plus-prefix)[post-hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[+](/reference/resource-configs/plus-prefix)[persist_docs](/reference/resource-configs/persist_docs): {<dict>}
[+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {<dict>}
[+](/reference/resource-configs/plus-prefix)[event_time](/reference/resource-configs/event-time): my_time_field
```
</VersionBlock>

<VersionBlock lastVersion="1.8">

```yaml
snapshots:
[<resource-path>](/reference/resource-configs/resource-path):
Expand All @@ -179,6 +197,7 @@ snapshots:
[+](/reference/resource-configs/plus-prefix)[persist_docs](/reference/resource-configs/persist_docs): {<dict>}
[+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {<dict>}
```
</VersionBlock>
</File>

</TabItem>
Expand Down Expand Up @@ -225,6 +244,7 @@ snapshots:
[post-hook](/reference/resource-configs/pre-hook-post-hook): <sql-statement> | [<sql-statement>]
[persist_docs](/reference/resource-configs/persist_docs): {<dict>}
[grants](/reference/resource-configs/grants): {<dictionary>}
[event_time](/reference/resource-configs/event-time): my_time_field
```

</File>
Expand Down Expand Up @@ -292,7 +312,6 @@ The following examples demonstrate how to configure snapshots using the `dbt_pro
<File name='dbt_project.yml'>

```yml

snapshots:
+unique_key: id
```
Expand All @@ -307,7 +326,6 @@ The following examples demonstrate how to configure snapshots using the `dbt_pro
<File name='dbt_project.yml'>

```yml

snapshots:
jaffle_shop:
+unique_key: id
Expand Down
Loading
Loading