From a2d5c4cec9cba83d4f29d9b57a44ab5ea7bb7854 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Thu, 17 Oct 2024 13:05:00 +0100 Subject: [PATCH 01/24] add page --- website/docs/docs/build/snapshots.md | 2 + .../resource-configs/dbt_valid_to_current.md | 99 +++++++++++++++++++ website/sidebars.js | 1 + 3 files changed, 102 insertions(+) create mode 100644 website/docs/reference/resource-configs/dbt_valid_to_current.md diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index f5321aa626a..e6e5eafeff5 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -70,6 +70,7 @@ snapshots: [updated_at](/reference/resource-configs/updated_at): column_name [invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes): true | false [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): dictionary + [dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current): string ``` @@ -88,6 +89,7 @@ The following table outlines the configurations available for snapshots: | [updated_at](/reference/resource-configs/updated_at) | If using the `timestamp` strategy, the timestamp column to compare | Only if using the `timestamp` strategy | updated_at | | [invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) | Find hard deleted records in source and set `dbt_valid_to` to current time if the record no longer exists | No | True | | [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names) | Customize the names of the snapshot meta fields | No | dictionary | +| [dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current) | Set a custom future date for `dbt_valid_to` in new snapshot columns. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. | No | string | - In versions prior to v1.9, the `target_schema` (required) and `target_database` (optional) configurations defined a single schema or database to build a snapshot across users and environment. This created problems when testing or developing a snapshot, as there was no clear separation between development and production environments. In v1.9, `target_schema` became optional, allowing snapshots to be environment-aware. By default, without `target_schema` or `target_database` defined, snapshots now use the `generate_schema_name` or `generate_database_name` macros to determine where to build. Developers can still set a custom location with [`schema`](/reference/resource-configs/schema) and [`database`](/reference/resource-configs/database) configs, consistent with other resource types. - A number of other configurations are also supported (for example, `tags` and `post-hook`). For the complete list, refer to [Snapshot configurations](/reference/snapshot-configs). diff --git a/website/docs/reference/resource-configs/dbt_valid_to_current.md b/website/docs/reference/resource-configs/dbt_valid_to_current.md new file mode 100644 index 00000000000..7f0bc5a2855 --- /dev/null +++ b/website/docs/reference/resource-configs/dbt_valid_to_current.md @@ -0,0 +1,99 @@ +--- +resource_types: [snapshots] +description: "Snapshot dbt_valid_to_current custom date" +datatype: "{}" +default_value: {"dbt_valid_from": "dbt_valid_from", "dbt_valid_to": "dbt_valid_to", "dbt_scd_id": "dbt_scd_id", "dbt_updated_at": "dbt_updated_at"} +id: "dbt_valid_to_current" +--- + +Available in 1.9 or with [Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) dbt Cloud. + + + +```yaml: +snapshots: + my_project: + +dbt_valid_to_current: "to_date('9999-12-31')" + +``` + + + + + +```sql +{{ + config( + unique_key='id', + strategy='timestamp', + updated_at='updated_at', + dbt_valid_to_current='to_date('9999-12-31')' + ) +}} + + +``` + + + + + +```yml +snapshots: + [](/reference/resource-configs/resource-path): + +dbt_valid_to_current: "to_date('9999-12-31')" +``` + + + +## Description + +Use the `dbt_valid_to_current` config to set a custom future date for `dbt_valid_to` in new snapshot columns. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. + +This simplifies makes it easier to assign date and range-based filtering with a concrete end date. + +## Default + +By default, `dbt_valid_to` is set to `NULL` for current (most recent) records in your snapshot table. This indicates that these records are still valid and have no defined end date. + +If you prefer to use a specific value instead of `NULL` for `dbt_valid_to` in current and future records (such as a date set in the far future, '9999-12-31'), you can use the `dbt_valid_to_current` configuration option. + +The value assigned to `dbt_valid_to_current` should be a string representing a valid date or timestamp, depending on your database's requirements. + +:::info +For existing records — To avoid any unintentional data modification, dbt will **not** automatically adjust the current value in the existing `dbt_valid_to` column. Existing current records will still have `dbt_valid_to` set to `NULL`. + +For new records — Any new records inserted after applying the `dbt_valid_to_current` configuration will have `dbt_valid_to` set to the specified value (for example, '9999-12-31'), instead of `NULL`. + +This means your snapshot table will have current records with `dbt_valid_to` values of both `NULL` (from existing data) and the new specified value (from new data). If you'd rather have consistent `dbt_valid_to` values for current records, you can either manually update existing records in your snapshot table where `dbt_valid_to` is `NULL` to match your `dbt_valid_to_current` value or rebuild your snapshot table. +::: + +## Example + + + +```yaml +snapshots: + - name: my_snapshot + config: + strategy: timestamp + updated_at: updated_at + dbt_valid_to_current: "to_date('9999-12-31')" + columns: + - name: dbt_valid_from + description: The timestamp when the record became valid. + - name: dbt_valid_to + description: > + The timestamp when the record ceased to be valid. For current records, + this is either `NULL` or the value specified in `dbt_valid_to_current` + (e.g., `'9999-12-31'`). +``` + + + +The resulting snapshot table contains the configured dbt_valid_to column value: + +| id | dbt_scd_id | dbt_updated_at | dbt_valid_from | dbt_valid_to | +| -- | -------------------- | -------------------- | -------------------- | -------------------- | +| 1 | 60a1f1dbdf899a4dd... | 2024-10-02 ... | 2024-10-02 ... | 9999-12-31 ... | +| 2 | b1885d098f8bcff51... | 2024-10-02 ... | 2024-10-02 ... | 9999-12-31 ... | diff --git a/website/sidebars.js b/website/sidebars.js index d31f6d03f57..13535a29df3 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -976,6 +976,7 @@ const sidebarSettings = { "reference/resource-configs/updated_at", "reference/resource-configs/invalidate_hard_deletes", "reference/resource-configs/snapshot_meta_column_names", + "reference/resource-configs/dbt_valid_to_current", ], }, { From 3d54fee75f7b16c9b0bbf849e9f4fcc478b267dd Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Thu, 17 Oct 2024 16:01:15 +0100 Subject: [PATCH 02/24] add custom future date --- website/docs/docs/build/snapshots.md | 37 +++++++++++-------- .../docs/docs/dbt-versions/release-notes.md | 2 +- .../resource-configs/dbt_valid_to_current.md | 19 ++++------ 3 files changed, 31 insertions(+), 27 deletions(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index e6e5eafeff5..d4256cfcbf0 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -36,12 +36,6 @@ This order is now in the "shipped" state, but we've lost the information about w ## Configuring snapshots -:::info Previewing or compiling snapshots in IDE not supported - -It is not possible to "preview data" or "compile sql" for snapshots in dbt Cloud. Instead, [run the `dbt snapshot` command](#how-snapshots-work) in the IDE. - -::: - - To configure snapshots in versions 1.8 and earlier, refer to [Configure snapshots in versions 1.8 and earlier](#configure-snapshots-in-versions-18-and-earlier). These versions use an older syntax where snapshots are defined within a snapshot block in a `.sql` file, typically located in your `snapshots` directory. @@ -71,7 +65,6 @@ snapshots: [invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes): true | false [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): dictionary [dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current): string - ``` @@ -88,14 +81,14 @@ The following table outlines the configurations available for snapshots: | [check_cols](/reference/resource-configs/check_cols) | If using the `check` strategy, then the columns to check | Only if using the `check` strategy | ["status"] | | [updated_at](/reference/resource-configs/updated_at) | If using the `timestamp` strategy, the timestamp column to compare | Only if using the `timestamp` strategy | updated_at | | [invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) | Find hard deleted records in source and set `dbt_valid_to` to current time if the record no longer exists | No | True | -| [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names) | Customize the names of the snapshot meta fields | No | dictionary | | [dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current) | Set a custom future date for `dbt_valid_to` in new snapshot columns. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. | No | string | +| [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names) | Customize the names of the snapshot meta fields | No | dictionary | + - In versions prior to v1.9, the `target_schema` (required) and `target_database` (optional) configurations defined a single schema or database to build a snapshot across users and environment. This created problems when testing or developing a snapshot, as there was no clear separation between development and production environments. In v1.9, `target_schema` became optional, allowing snapshots to be environment-aware. By default, without `target_schema` or `target_database` defined, snapshots now use the `generate_schema_name` or `generate_database_name` macros to determine where to build. Developers can still set a custom location with [`schema`](/reference/resource-configs/schema) and [`database`](/reference/resource-configs/database) configs, consistent with other resource types. - A number of other configurations are also supported (for example, `tags` and `post-hook`). For the complete list, refer to [Snapshot configurations](/reference/snapshot-configs). - You can configure snapshots from both the `dbt_project.yml` file and a `config` block. For more information, refer to the [configuration docs](/reference/snapshot-configs). - ### Add a snapshot to your project To add a snapshot to your project follow these steps. For users on versions 1.8 and earlier, refer to [Configure snapshots in versions 1.8 and earlier](#configure-snapshots-in-versions-18-and-earlier). @@ -114,6 +107,7 @@ To add a snapshot to your project follow these steps. For users on versions 1.8 unique_key: id strategy: timestamp updated_at: updated_at + dbt_valid_to_current: "to_date('9999-12-31')" # Specifies that current records should have `dbt_valid_to` set to `'9999-12-31'` instead of `NULL`. ``` @@ -174,6 +168,15 @@ This strategy handles column additions and deletions better than the `check` str + + + +By default, `dbt_valid_to` is `NULL` for current records. However, if you set the [`dbt_valid_to_current` configuration](/reference/resource-configs/dbt_valid_to_current) (available in Versionless and 1.9 and higher), `dbt_valid_to` will be set to your specified value (such as `9999-12-31`) for current records. + +This simplifies your SQL queries by avoiding `NULL` checks and allowing for straightforward date range filtering. + + + The unique key is used by dbt to match rows up, so it's extremely important to make sure this key is actually unique! If you're snapshotting a source, I'd recommend adding a uniqueness test to your source ([example](https://github.com/dbt-labs/jaffle_shop/blob/8e7c853c858018180bef1756ec93e193d9958c5b/models/staging/schema.yml#L26)). @@ -206,12 +209,14 @@ Snapshots can't be rebuilt. Because of this, it's a good idea to put snapshots i ### How snapshots work When you run the [`dbt snapshot` command](/reference/commands/snapshot): -* **On the first run:** dbt will create the initial snapshot table — this will be the result set of your `select` statement, with additional columns including `dbt_valid_from` and `dbt_valid_to`. All records will have a `dbt_valid_to = null`. +* **On the first run:** dbt will create the initial snapshot table — this will be the result set of your `select` statement, with additional columns including `dbt_valid_from` and `dbt_valid_to`. All records will have a `dbt_valid_to = null` or the value specified in [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current) if configured. * **On subsequent runs:** dbt will check which records have changed or if any new records have been created: - - The `dbt_valid_to` column will be updated for any existing records that have changed - - The updated record and any new records will be inserted into the snapshot table. These records will now have `dbt_valid_to = null` + - The `dbt_valid_to` column will be updated for any existing records that have changed. + - The updated record and any new records will be inserted into the snapshot table. These records will now have `dbt_valid_to = null` or the value configured in `dbt_valid_to_current`. -Note, these column names can be customized to your team or organizational conventions using the [snapshot_meta_column_names](#snapshot-meta-fields) config. +#### Note +- These column names can be customized to your team or organizational conventions using the [snapshot_meta_column_names](#snapshot-meta-fields) config. +- If you have set the `dbt_valid_to_current` configuration option, then instead of `NULL`, the `dbt_valid_to` field in future records will be set to your specified value (such as `9999-12-31`). Snapshots can be referenced in downstream models the same way as referencing models — by using the [ref](/reference/dbt-jinja-functions/ref) function. @@ -419,12 +424,14 @@ Basically – keep your query as simple as possible! Some reasonable exceptions Snapshot tables will be created as a clone of your source dataset, plus some additional meta-fields*. -Starting in 1.9 or with [dbt Cloud Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless), these column names can be customized to your team or organizational conventions via the [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) config. +Starting in 1.9 or with [dbt Cloud Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless): +- These column names can be customized to your team or organizational conventions using the [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) config. +- Use the [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current) config to set a custom future date for `dbt_valid_to` in new snapshot columns. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. | Field | Meaning | Usage | | -------------- | ------- | ----- | | dbt_valid_from | The timestamp when this snapshot row was first inserted | This column can be used to order the different "versions" of a record. | -| dbt_valid_to | The timestamp when this row became invalidated. | The most recent snapshot record will have `dbt_valid_to` set to `null`. | +| dbt_valid_to | The timestamp when this row became invalidated.
For current records, this is `NULL` by default or the value specified in `dbt_valid_to_current`. | The most recent snapshot record will have `dbt_valid_to` set to `NULL` or the specified value. | | dbt_scd_id | A unique key generated for each snapshotted record. | This is used internally by dbt | | dbt_updated_at | The updated_at timestamp of the source record when this snapshot row was inserted. | This is used internally by dbt | diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index 662fd0f381a..71865974b6d 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -44,7 +44,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo - +- **New**: Use the [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current) config to set a custom future date for `dbt_valid_to` in new snapshot columns. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. - **New**: The [dbt Semantic Layer Python software development kit](/docs/dbt-cloud-apis/sl-python) is now [generally available](/docs/dbt-versions/product-lifecycles). It provides users with easy access to the dbt Semantic Layer with Python and enables developers to interact with the dbt Semantic Layer APIs to query metrics/dimensions in downstream tools. - **Enhancement**: You can now add a description to a singular data test in dbt Cloud Versionless. Use the [`description` property](/reference/resource-properties/description) to document [singular data tests](/docs/build/data-tests#singular-data-tests). You can also use [docs block](/docs/build/documentation#using-docs-blocks) to capture your test description. The enhancement will be included in upcoming dbt Core 1.9 release. - **New**: Introducing the [microbatch incremental model strategy](/docs/build/incremental-microbatch) (beta), available in dbt Cloud Versionless and will soon be supported in dbt Core 1.9. The microbatch strategy allows for efficient, batch-based processing of large time-series datasets for improved performance and resiliency, especially when you're working with data that changes over time (like new records being added daily). To enable this feature in dbt Cloud, set the `DBT_EXPERIMENTAL_MICROBATCH` environment variable to `true` in your project. diff --git a/website/docs/reference/resource-configs/dbt_valid_to_current.md b/website/docs/reference/resource-configs/dbt_valid_to_current.md index 7f0bc5a2855..d2af1847470 100644 --- a/website/docs/reference/resource-configs/dbt_valid_to_current.md +++ b/website/docs/reference/resource-configs/dbt_valid_to_current.md @@ -10,7 +10,7 @@ Available in 1.9 or with [Versionless](/docs/dbt-versions/upgrade-dbt-version-in -```yaml: +```yaml snapshots: my_project: +dbt_valid_to_current: "to_date('9999-12-31')" @@ -30,8 +30,6 @@ snapshots: dbt_valid_to_current='to_date('9999-12-31')' ) }} - - ``` @@ -50,23 +48,22 @@ snapshots: Use the `dbt_valid_to_current` config to set a custom future date for `dbt_valid_to` in new snapshot columns. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. -This simplifies makes it easier to assign date and range-based filtering with a concrete end date. +This simplifies makes it easier to assign date, work in a join, and range-based filtering with an end date. ## Default -By default, `dbt_valid_to` is set to `NULL` for current (most recent) records in your snapshot table. This indicates that these records are still valid and have no defined end date. +By default, `dbt_valid_to` is set to `NULL` for current (most recent) records in your snapshot table. This means that these records are still valid and have no defined end date. -If you prefer to use a specific value instead of `NULL` for `dbt_valid_to` in current and future records (such as a date set in the far future, '9999-12-31'), you can use the `dbt_valid_to_current` configuration option. +If you prefer to use a specific value instead of `NULL` for `dbt_valid_to` in current and future records, you can use the `dbt_valid_to_current` configuration option. For example, setting a date in the far future, `9999-12-31`. The value assigned to `dbt_valid_to_current` should be a string representing a valid date or timestamp, depending on your database's requirements. -:::info -For existing records — To avoid any unintentional data modification, dbt will **not** automatically adjust the current value in the existing `dbt_valid_to` column. Existing current records will still have `dbt_valid_to` set to `NULL`. +### Managing records +- For existing records — To avoid any unintentional data modification, dbt will _not_ automatically adjust the current value in the existing `dbt_valid_to` column. Existing current records will still have `dbt_valid_to` set to `NULL`. -For new records — Any new records inserted after applying the `dbt_valid_to_current` configuration will have `dbt_valid_to` set to the specified value (for example, '9999-12-31'), instead of `NULL`. +- For new records — Any new records inserted after applying the `dbt_valid_to_current` configuration will have `dbt_valid_to` set to the specified value (for example, '9999-12-31'), instead of `NULL`. This means your snapshot table will have current records with `dbt_valid_to` values of both `NULL` (from existing data) and the new specified value (from new data). If you'd rather have consistent `dbt_valid_to` values for current records, you can either manually update existing records in your snapshot table where `dbt_valid_to` is `NULL` to match your `dbt_valid_to_current` value or rebuild your snapshot table. -::: ## Example @@ -86,7 +83,7 @@ snapshots: description: > The timestamp when the record ceased to be valid. For current records, this is either `NULL` or the value specified in `dbt_valid_to_current` - (e.g., `'9999-12-31'`). + (like `'9999-12-31'`). ``` From 44b6e143fc7d7bfd048b26e62f1f5c196478f051 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 4 Nov 2024 12:20:16 +0000 Subject: [PATCH 03/24] Update website/docs/reference/resource-configs/dbt_valid_to_current.md --- website/docs/reference/resource-configs/dbt_valid_to_current.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/dbt_valid_to_current.md b/website/docs/reference/resource-configs/dbt_valid_to_current.md index d2af1847470..40ca392920c 100644 --- a/website/docs/reference/resource-configs/dbt_valid_to_current.md +++ b/website/docs/reference/resource-configs/dbt_valid_to_current.md @@ -56,7 +56,7 @@ By default, `dbt_valid_to` is set to `NULL` for current (most recent) records in If you prefer to use a specific value instead of `NULL` for `dbt_valid_to` in current and future records, you can use the `dbt_valid_to_current` configuration option. For example, setting a date in the far future, `9999-12-31`. -The value assigned to `dbt_valid_to_current` should be a string representing a valid date or timestamp, depending on your database's requirements. +The value assigned to `dbt_valid_to_current` should be a string representing a valid date or timestamp, depending on your database's requirements. Use expressions that work within the data platform. ### Managing records - For existing records — To avoid any unintentional data modification, dbt will _not_ automatically adjust the current value in the existing `dbt_valid_to` column. Existing current records will still have `dbt_valid_to` set to `NULL`. From 33483f0348ed19a25e316e0cf8667916ac5a41b0 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 13 Nov 2024 12:04:19 +0000 Subject: [PATCH 04/24] fold in doug's feedback --- website/docs/docs/build/snapshots.md | 4 ++-- .../resource-configs/dbt_valid_to_current.md | 14 ++++++++++---- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index 7412643eb4c..584704dc728 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -209,10 +209,10 @@ Snapshots can't be rebuilt. Because of this, it's a good idea to put snapshots i ### How snapshots work When you run the [`dbt snapshot` command](/reference/commands/snapshot): -* **On the first run:** dbt will create the initial snapshot table — this will be the result set of your `select` statement, with additional columns including `dbt_valid_from` and `dbt_valid_to`. All records will have a `dbt_valid_to = null` or the value specified in [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current) if configured. +* **On the first run:** dbt will create the initial snapshot table — this will be the result set of your `select` statement, with additional columns including `dbt_valid_from` and `dbt_valid_to`. All records will have a `dbt_valid_to = null` or the value specified in [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current) (available in Versionless and 1.9 and higher) if configured. * **On subsequent runs:** dbt will check which records have changed or if any new records have been created: - The `dbt_valid_to` column will be updated for any existing records that have changed. - - The updated record and any new records will be inserted into the snapshot table. These records will now have `dbt_valid_to = null` or the value configured in `dbt_valid_to_current`. + - The updated record and any new records will be inserted into the snapshot table. These records will now have `dbt_valid_to = null` or the value configured in `dbt_valid_to_current` (available in Versionless and 1.9 and higher). #### Note - These column names can be customized to your team or organizational conventions using the [snapshot_meta_column_names](#snapshot-meta-fields) config. diff --git a/website/docs/reference/resource-configs/dbt_valid_to_current.md b/website/docs/reference/resource-configs/dbt_valid_to_current.md index 40ca392920c..aa3a3dbe1a8 100644 --- a/website/docs/reference/resource-configs/dbt_valid_to_current.md +++ b/website/docs/reference/resource-configs/dbt_valid_to_current.md @@ -2,7 +2,7 @@ resource_types: [snapshots] description: "Snapshot dbt_valid_to_current custom date" datatype: "{}" -default_value: {"dbt_valid_from": "dbt_valid_from", "dbt_valid_to": "dbt_valid_to", "dbt_scd_id": "dbt_scd_id", "dbt_updated_at": "dbt_updated_at"} +default_value: {NULL} id: "dbt_valid_to_current" --- @@ -48,7 +48,13 @@ snapshots: Use the `dbt_valid_to_current` config to set a custom future date for `dbt_valid_to` in new snapshot columns. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. -This simplifies makes it easier to assign date, work in a join, and range-based filtering with an end date. +This approach makes it easier to assign a custom date date, work in a join, or perform range-based filtering that require an end date. + +## Usage + +- **Date expressions** — Provide a hardcoded date expression compatible with your data platform, such as to_date`('9999-12-31')`. Note that syntax may vary by warehouse (for example, to_date('YYYY-MM-DD') or date(YYYY, MM, DD)). +- **Jinja limitation** — `dbt_valid_to_current` only accepts static SQL expressions. Jinja expressions (like `{{ var('my_future_date') }}`) are not supported. +- **Deferral and `state:modified`** — Changes to `dbt_valid_to_current` are compatible with deferral and `--select state:modified`. When this configuration changes, it'll appear in `state:modified` selections, raising a warning to manually make the necessary snapshot updates. ## Default @@ -59,9 +65,9 @@ If you prefer to use a specific value instead of `NULL` for `dbt_valid_to` in cu The value assigned to `dbt_valid_to_current` should be a string representing a valid date or timestamp, depending on your database's requirements. Use expressions that work within the data platform. ### Managing records -- For existing records — To avoid any unintentional data modification, dbt will _not_ automatically adjust the current value in the existing `dbt_valid_to` column. Existing current records will still have `dbt_valid_to` set to `NULL`. +- **For existing records** — To avoid any unintentional data modification, dbt will _not_ automatically adjust the current value in the existing `dbt_valid_to` column. Existing current records will still have `dbt_valid_to` set to `NULL`. -- For new records — Any new records inserted after applying the `dbt_valid_to_current` configuration will have `dbt_valid_to` set to the specified value (for example, '9999-12-31'), instead of `NULL`. +- **For new records** — Any new records inserted after applying the `dbt_valid_to_current` configuration will have `dbt_valid_to` set to the specified value (for example, '9999-12-31'), instead of `NULL`. This means your snapshot table will have current records with `dbt_valid_to` values of both `NULL` (from existing data) and the new specified value (from new data). If you'd rather have consistent `dbt_valid_to` values for current records, you can either manually update existing records in your snapshot table where `dbt_valid_to` is `NULL` to match your `dbt_valid_to_current` value or rebuild your snapshot table. From 9be457a0111fc27347b68be966d04c2155b75f32 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Wed, 13 Nov 2024 12:08:00 +0000 Subject: [PATCH 05/24] Update release-notes.md --- website/docs/docs/dbt-versions/release-notes.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index d3ed9967309..ee8890a1e25 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -19,6 +19,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo \* The official release date for this new format of release notes is May 15th, 2024. Historical release notes for prior dates may not reflect all available features released earlier this year or their tenancy availability. ## November 2024 +- **New**: Use the [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current) config to set a custom future date for `dbt_valid_to` in new snapshot columns. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. - **Fix**: This update improves [dbt Semantic Layer Tableau integration](/docs/cloud-integrations/semantic-layer/tableau) making query parsing more reliable. Some key fixes include: - Error messages for unsupported joins between saved queries and ALL tables. - Improved handling of queries when multiple tables are selected in a data source. @@ -51,8 +52,6 @@ Release notes are grouped by month for both multi-tenant and virtual private clo - -- **New**: Use the [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current) config to set a custom future date for `dbt_valid_to` in new snapshot columns. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. - **Behavior change:** [Multi-factor authentication](/docs/cloud/manage-access/mfa) is now enforced on all users who log in with username and password credentials. - **Enhancement**: The dbt Semantic Layer JDBC now allows users to paginate `semantic_layer.metrics()` and `semantic_layer.dimensions()` for metrics and dimensions using `page_size` and `page_number` parameters. Refer to [Paginate metadata calls](/docs/dbt-cloud-apis/sl-jdbc#querying-the-api-for-metric-metadata) for more information. - **Enhancement**: The dbt Semantic Layer JDBC now allows you to filter your metrics to include only those that contain a specific substring, using the `search` parameter. If no substring is provided, the query returns all metrics. Refer to [Fetch metrics by substring search](/docs/dbt-cloud-apis/sl-jdbc#querying-the-api-for-metric-metadata) for more information. From 40eac92e5ba1c519fb68c02ff3cd8e924f29a9e1 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Wed, 13 Nov 2024 12:09:20 +0000 Subject: [PATCH 06/24] Update website/docs/reference/resource-configs/dbt_valid_to_current.md --- website/docs/reference/resource-configs/dbt_valid_to_current.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/dbt_valid_to_current.md b/website/docs/reference/resource-configs/dbt_valid_to_current.md index aa3a3dbe1a8..eedbc2a6ddd 100644 --- a/website/docs/reference/resource-configs/dbt_valid_to_current.md +++ b/website/docs/reference/resource-configs/dbt_valid_to_current.md @@ -50,7 +50,7 @@ Use the `dbt_valid_to_current` config to set a custom future date for `dbt_valid This approach makes it easier to assign a custom date date, work in a join, or perform range-based filtering that require an end date. -## Usage +### Considerations - **Date expressions** — Provide a hardcoded date expression compatible with your data platform, such as to_date`('9999-12-31')`. Note that syntax may vary by warehouse (for example, to_date('YYYY-MM-DD') or date(YYYY, MM, DD)). - **Jinja limitation** — `dbt_valid_to_current` only accepts static SQL expressions. Jinja expressions (like `{{ var('my_future_date') }}`) are not supported. From bf40b218a182f1e0f8012b3ed88d2e71fa0aaa83 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Wed, 13 Nov 2024 12:10:00 +0000 Subject: [PATCH 07/24] Update dbt_valid_to_current.md --- .../docs/reference/resource-configs/dbt_valid_to_current.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/dbt_valid_to_current.md b/website/docs/reference/resource-configs/dbt_valid_to_current.md index eedbc2a6ddd..28e7b19dd41 100644 --- a/website/docs/reference/resource-configs/dbt_valid_to_current.md +++ b/website/docs/reference/resource-configs/dbt_valid_to_current.md @@ -52,8 +52,10 @@ This approach makes it easier to assign a custom date date, work in a join, or p ### Considerations -- **Date expressions** — Provide a hardcoded date expression compatible with your data platform, such as to_date`('9999-12-31')`. Note that syntax may vary by warehouse (for example, to_date('YYYY-MM-DD') or date(YYYY, MM, DD)). +- **Date expressions** — Provide a hardcoded date expression compatible with your data platform, such as to_date`('9999-12-31')`. Note that syntax may vary by warehouse (for example, `to_date('YYYY-MM-DD'`) or `date(YYYY, MM, DD)`). + - **Jinja limitation** — `dbt_valid_to_current` only accepts static SQL expressions. Jinja expressions (like `{{ var('my_future_date') }}`) are not supported. + - **Deferral and `state:modified`** — Changes to `dbt_valid_to_current` are compatible with deferral and `--select state:modified`. When this configuration changes, it'll appear in `state:modified` selections, raising a warning to manually make the necessary snapshot updates. ## Default From 15e71c904edfb5c710a4db23f928a04d20fef17c Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 15 Nov 2024 09:24:40 +0000 Subject: [PATCH 08/24] Update website/docs/reference/resource-configs/dbt_valid_to_current.md --- website/docs/reference/resource-configs/dbt_valid_to_current.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/dbt_valid_to_current.md b/website/docs/reference/resource-configs/dbt_valid_to_current.md index 28e7b19dd41..69a7da9d44d 100644 --- a/website/docs/reference/resource-configs/dbt_valid_to_current.md +++ b/website/docs/reference/resource-configs/dbt_valid_to_current.md @@ -48,7 +48,7 @@ snapshots: Use the `dbt_valid_to_current` config to set a custom future date for `dbt_valid_to` in new snapshot columns. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. -This approach makes it easier to assign a custom date date, work in a join, or perform range-based filtering that require an end date. +This approach makes it easier to assign a custom date, work in a join, or perform range-based filtering that requires an end date. ### Considerations From 704879f23a547a56ecff78aa995a39ced2a6a6ad Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 15 Nov 2024 09:52:33 +0000 Subject: [PATCH 09/24] Update website/docs/docs/build/snapshots.md --- website/docs/docs/build/snapshots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index a6fd3a13c1c..5637d2aa248 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -81,7 +81,7 @@ The following table outlines the configurations available for snapshots: | [check_cols](/reference/resource-configs/check_cols) | If using the `check` strategy, then the columns to check | Only if using the `check` strategy | ["status"] | | [updated_at](/reference/resource-configs/updated_at) | If using the `timestamp` strategy, the timestamp column to compare | Only if using the `timestamp` strategy | updated_at | | [invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) | Find hard deleted records in source and set `dbt_valid_to` to current time if the record no longer exists | No | True | -| [dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current) | Set a custom future date for `dbt_valid_to` in new snapshot columns. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. | No | string | +| [dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current) | Set a custom indicator for the value of `dbt_valid_to` in current snapshot records. By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` in the snapshot table.| No | string | | [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names) | Customize the names of the snapshot meta fields | No | dictionary | From 3fb54357c07a04645d2438985a353fac858267bd Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 15 Nov 2024 10:37:13 +0000 Subject: [PATCH 10/24] Update website/docs/docs/build/snapshots.md --- website/docs/docs/build/snapshots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index 5637d2aa248..062cadfb168 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -81,7 +81,7 @@ The following table outlines the configurations available for snapshots: | [check_cols](/reference/resource-configs/check_cols) | If using the `check` strategy, then the columns to check | Only if using the `check` strategy | ["status"] | | [updated_at](/reference/resource-configs/updated_at) | If using the `timestamp` strategy, the timestamp column to compare | Only if using the `timestamp` strategy | updated_at | | [invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) | Find hard deleted records in source and set `dbt_valid_to` to current time if the record no longer exists | No | True | -| [dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current) | Set a custom indicator for the value of `dbt_valid_to` in current snapshot records. By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` in the snapshot table.| No | string | +| [dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current) | Set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` in the snapshot table.| No | string | | [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names) | Customize the names of the snapshot meta fields | No | dictionary | From aa55081bb8e217107b6470f860a1b77f262aa399 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Mon, 18 Nov 2024 16:17:10 -0500 Subject: [PATCH 11/24] grace's feedback --- website/docs/docs/build/snapshots.md | 6 +++--- .../docs/docs/dbt-versions/release-notes.md | 2 +- .../resource-configs/dbt_valid_to_current.md | 18 +++++++++++++++--- 3 files changed, 19 insertions(+), 7 deletions(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index 062cadfb168..760fd635223 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -216,8 +216,8 @@ When you run the [`dbt snapshot` command](/reference/commands/snapshot): #### Note - These column names can be customized to your team or organizational conventions using the [snapshot_meta_column_names](#snapshot-meta-fields) config. -- If you have set the `dbt_valid_to_current` configuration option, then instead of `NULL`, the `dbt_valid_to` field in future records will be set to your specified value (such as `9999-12-31`). - +- Use the `dbt_valid_to_current` config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date such as `9999-12-31`). By default, this value is `NULL`. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. + Snapshots can be referenced in downstream models the same way as referencing models — by using the [ref](/reference/dbt-jinja-functions/ref) function. ## Detecting row changes @@ -403,7 +403,7 @@ Snapshot tables will be created as a clone of your sourc Starting in 1.9 or with [dbt Cloud Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless): - These column names can be customized to your team or organizational conventions using the [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) config. -- Use the [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current) config to set a custom future date for `dbt_valid_to` in new snapshot columns. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. +- Use the `dbt_valid_to_current` config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date such as `9999-12-31`). By default, this value is `NULL`. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. | Field | Meaning | Usage | | -------------- | ------- | ----- | diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index ee8890a1e25..a49537caaac 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -19,7 +19,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo \* The official release date for this new format of release notes is May 15th, 2024. Historical release notes for prior dates may not reflect all available features released earlier this year or their tenancy availability. ## November 2024 -- **New**: Use the [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current) config to set a custom future date for `dbt_valid_to` in new snapshot columns. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. +- **New**: Use the [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current) config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. This feature is available in dbt Cloud Versionless and dbt Core v1.9. - **Fix**: This update improves [dbt Semantic Layer Tableau integration](/docs/cloud-integrations/semantic-layer/tableau) making query parsing more reliable. Some key fixes include: - Error messages for unsupported joins between saved queries and ALL tables. - Improved handling of queries when multiple tables are selected in a data source. diff --git a/website/docs/reference/resource-configs/dbt_valid_to_current.md b/website/docs/reference/resource-configs/dbt_valid_to_current.md index 69a7da9d44d..6a925d1cdf6 100644 --- a/website/docs/reference/resource-configs/dbt_valid_to_current.md +++ b/website/docs/reference/resource-configs/dbt_valid_to_current.md @@ -46,10 +46,18 @@ snapshots: ## Description -Use the `dbt_valid_to_current` config to set a custom future date for `dbt_valid_to` in new snapshot columns. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. +Use the `dbt_valid_to_current` config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. This approach makes it easier to assign a custom date, work in a join, or perform range-based filtering that requires an end date. +:::warning + +To avoid any unintentional data modification, dbt will _not_ automatically adjust the current value in the existing `dbt_valid_to` column. Existing current records will still have `dbt_valid_to` set to `NULL`. + +Any new records inserted _after_ applying the `dbt_valid_to_current` configuration will have `dbt_valid_to` set to the specified value (like '9999-12-31'), instead of the default `NULL` value. + +::: + ### Considerations - **Date expressions** — Provide a hardcoded date expression compatible with your data platform, such as to_date`('9999-12-31')`. Note that syntax may vary by warehouse (for example, `to_date('YYYY-MM-DD'`) or `date(YYYY, MM, DD)`). @@ -66,12 +74,16 @@ If you prefer to use a specific value instead of `NULL` for `dbt_valid_to` in cu The value assigned to `dbt_valid_to_current` should be a string representing a valid date or timestamp, depending on your database's requirements. Use expressions that work within the data platform. -### Managing records + +## Impact on snapshot records + +When you set `dbt_valid_to_current`, it affects how dbt manages the `dbt_valid_to` column in your snapshot table: + - **For existing records** — To avoid any unintentional data modification, dbt will _not_ automatically adjust the current value in the existing `dbt_valid_to` column. Existing current records will still have `dbt_valid_to` set to `NULL`. - **For new records** — Any new records inserted after applying the `dbt_valid_to_current` configuration will have `dbt_valid_to` set to the specified value (for example, '9999-12-31'), instead of `NULL`. -This means your snapshot table will have current records with `dbt_valid_to` values of both `NULL` (from existing data) and the new specified value (from new data). If you'd rather have consistent `dbt_valid_to` values for current records, you can either manually update existing records in your snapshot table where `dbt_valid_to` is `NULL` to match your `dbt_valid_to_current` value or rebuild your snapshot table. +This means your snapshot table will have current records with `dbt_valid_to` values of both `NULL` (from existing data) and the new specified value (from new data). If you'd rather have consistent `dbt_valid_to` values for current records, you can manually update existing records in your snapshot table where `dbt_valid_to` is `NULL` to match your `dbt_valid_to_current` value. ## Example From 9696244095f89d94583d6532ec813ff2286ffa63 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 19 Nov 2024 11:44:20 -0500 Subject: [PATCH 12/24] Update website/docs/docs/dbt-versions/release-notes.md Co-authored-by: Grace Goheen <53586774+graciegoheen@users.noreply.github.com> --- website/docs/docs/dbt-versions/release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index a49537caaac..4540f67b825 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -19,7 +19,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo \* The official release date for this new format of release notes is May 15th, 2024. Historical release notes for prior dates may not reflect all available features released earlier this year or their tenancy availability. ## November 2024 -- **New**: Use the [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current) config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. This feature is available in dbt Cloud Versionless and dbt Core v1.9. +- **New**: Use the [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current) config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. This feature is available in dbt Cloud Versionless and dbt Core v1.9. - **Fix**: This update improves [dbt Semantic Layer Tableau integration](/docs/cloud-integrations/semantic-layer/tableau) making query parsing more reliable. Some key fixes include: - Error messages for unsupported joins between saved queries and ALL tables. - Improved handling of queries when multiple tables are selected in a data source. From 7bd7c146cd2cdada54b8c693b4f4c475728ecdc2 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 19 Nov 2024 11:44:28 -0500 Subject: [PATCH 13/24] Update website/docs/docs/build/snapshots.md Co-authored-by: Grace Goheen <53586774+graciegoheen@users.noreply.github.com> --- website/docs/docs/build/snapshots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index 760fd635223..78b3d4148eb 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -403,7 +403,7 @@ Snapshot tables will be created as a clone of your sourc Starting in 1.9 or with [dbt Cloud Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless): - These column names can be customized to your team or organizational conventions using the [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) config. -- Use the `dbt_valid_to_current` config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date such as `9999-12-31`). By default, this value is `NULL`. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. +- Use the `dbt_valid_to_current` config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date such as `9999-12-31`). By default, this value is `NULL`. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. | Field | Meaning | Usage | | -------------- | ------- | ----- | From f1db813ef12808d9e810020ba0512d36cbbe3d07 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 19 Nov 2024 11:44:34 -0500 Subject: [PATCH 14/24] Update website/docs/reference/resource-configs/dbt_valid_to_current.md Co-authored-by: Grace Goheen <53586774+graciegoheen@users.noreply.github.com> --- website/docs/reference/resource-configs/dbt_valid_to_current.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/dbt_valid_to_current.md b/website/docs/reference/resource-configs/dbt_valid_to_current.md index 6a925d1cdf6..b2469d0be36 100644 --- a/website/docs/reference/resource-configs/dbt_valid_to_current.md +++ b/website/docs/reference/resource-configs/dbt_valid_to_current.md @@ -46,7 +46,7 @@ snapshots: ## Description -Use the `dbt_valid_to_current` config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. +Use the `dbt_valid_to_current` config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. This approach makes it easier to assign a custom date, work in a join, or perform range-based filtering that requires an end date. From 4f0455457bd60d8b2d3e29837e330065eb8cedda Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 19 Nov 2024 11:44:43 -0500 Subject: [PATCH 15/24] Update website/docs/docs/build/snapshots.md Co-authored-by: Grace Goheen <53586774+graciegoheen@users.noreply.github.com> --- website/docs/docs/build/snapshots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index 78b3d4148eb..c0119231d99 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -216,7 +216,7 @@ When you run the [`dbt snapshot` command](/reference/commands/snapshot): #### Note - These column names can be customized to your team or organizational conventions using the [snapshot_meta_column_names](#snapshot-meta-fields) config. -- Use the `dbt_valid_to_current` config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date such as `9999-12-31`). By default, this value is `NULL`. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. +- Use the `dbt_valid_to_current` config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date such as `9999-12-31`). By default, this value is `NULL`. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. Snapshots can be referenced in downstream models the same way as referencing models — by using the [ref](/reference/dbt-jinja-functions/ref) function. From 1adb008b2d72c8d5dc99665cb0e502041d50bedc Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 19 Nov 2024 11:44:50 -0500 Subject: [PATCH 16/24] Update website/docs/docs/build/snapshots.md Co-authored-by: Grace Goheen <53586774+graciegoheen@users.noreply.github.com> --- website/docs/docs/build/snapshots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index c0119231d99..accfa049b19 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -81,7 +81,7 @@ The following table outlines the configurations available for snapshots: | [check_cols](/reference/resource-configs/check_cols) | If using the `check` strategy, then the columns to check | Only if using the `check` strategy | ["status"] | | [updated_at](/reference/resource-configs/updated_at) | If using the `timestamp` strategy, the timestamp column to compare | Only if using the `timestamp` strategy | updated_at | | [invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) | Find hard deleted records in source and set `dbt_valid_to` to current time if the record no longer exists | No | True | -| [dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current) | Set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` in the snapshot table.| No | string | +| [dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current) | Set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table.| No | string | | [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names) | Customize the names of the snapshot meta fields | No | dictionary | From 4be87ac1ec9d3e610f12d715bd11f233c11d9a01 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Tue, 19 Nov 2024 14:34:09 -0500 Subject: [PATCH 17/24] grace feedback --- website/docs/docs/build/snapshots.md | 2 +- website/docs/reference/resource-configs/dbt_valid_to_current.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index accfa049b19..1ff2298d8f7 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -173,7 +173,7 @@ This strategy handles column additions and deletions better than the `check` str By default, `dbt_valid_to` is `NULL` for current records. However, if you set the [`dbt_valid_to_current` configuration](/reference/resource-configs/dbt_valid_to_current) (available in Versionless and 1.9 and higher), `dbt_valid_to` will be set to your specified value (such as `9999-12-31`) for current records. -This simplifies your SQL queries by avoiding `NULL` checks and allowing for straightforward date range filtering. +This allows for straightforward date range filtering. diff --git a/website/docs/reference/resource-configs/dbt_valid_to_current.md b/website/docs/reference/resource-configs/dbt_valid_to_current.md index b2469d0be36..2aba7365511 100644 --- a/website/docs/reference/resource-configs/dbt_valid_to_current.md +++ b/website/docs/reference/resource-configs/dbt_valid_to_current.md @@ -1,6 +1,6 @@ --- resource_types: [snapshots] -description: "Snapshot dbt_valid_to_current custom date" +description: "Use the `dbt_valid_to_current` config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records" datatype: "{}" default_value: {NULL} id: "dbt_valid_to_current" From 92104b69c46eca17c5f8269c8555c1fcc3f88029 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 19 Nov 2024 14:54:21 -0500 Subject: [PATCH 18/24] Update website/docs/reference/resource-configs/dbt_valid_to_current.md --- website/docs/reference/resource-configs/dbt_valid_to_current.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/dbt_valid_to_current.md b/website/docs/reference/resource-configs/dbt_valid_to_current.md index 2aba7365511..b6a99d8425c 100644 --- a/website/docs/reference/resource-configs/dbt_valid_to_current.md +++ b/website/docs/reference/resource-configs/dbt_valid_to_current.md @@ -83,7 +83,7 @@ When you set `dbt_valid_to_current`, it affects how dbt manages the `dbt_valid_t - **For new records** — Any new records inserted after applying the `dbt_valid_to_current` configuration will have `dbt_valid_to` set to the specified value (for example, '9999-12-31'), instead of `NULL`. -This means your snapshot table will have current records with `dbt_valid_to` values of both `NULL` (from existing data) and the new specified value (from new data). If you'd rather have consistent `dbt_valid_to` values for current records, you can manually update existing records in your snapshot table where `dbt_valid_to` is `NULL` to match your `dbt_valid_to_current` value. +This means your snapshot table will have current records with `dbt_valid_to` values of both `NULL` (from existing data) and the new specified value (from new data). If you'd rather have consistent `dbt_valid_to` values for current records, you can manually update existing records in your snapshot table (where `dbt_valid_to` is `NULL`) to match your `dbt_valid_to_current` value. ## Example From 7ecd040c03649a10d13713f34a7f9e4eb5ce867f Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 19 Nov 2024 14:55:31 -0500 Subject: [PATCH 19/24] Update website/docs/docs/dbt-versions/release-notes.md --- website/docs/docs/dbt-versions/release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index e2134bc657b..2f6eb753f41 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -19,7 +19,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo \* The official release date for this new format of release notes is May 15th, 2024. Historical release notes for prior dates may not reflect all available features released earlier this year or their tenancy availability. ## November 2024 -- **New**: Use the [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current) config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. This feature is available in dbt Cloud Versionless and dbt Core v1.9. +- **New**: Use the [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current) config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. This feature is available in dbt Cloud Versionless and dbt Core v1.9 and later. - **New**: Use the `event_time` configuration to specify "at what time did the row occur." This configuration is required for [Incremental microbatch](/docs/build/incremental-microbatch) and can be added to ensure you're comparing overlapping times in [Advanced CI's compare changes](/docs/deploy/advanced-ci). Available in dbt Cloud Versionless and dbt Core v1.9 and higher. Refer to [event_time](/reference/resource-configs/event-time) for more information. - **Fix**: This update improves [dbt Semantic Layer Tableau integration](/docs/cloud-integrations/semantic-layer/tableau) making query parsing more reliable. Some key fixes include: - Error messages for unsupported joins between saved queries and ALL tables. From 44d88a7f7bae1426078c8fa0070e1466a75f70ed Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 19 Nov 2024 14:55:54 -0500 Subject: [PATCH 20/24] Update website/docs/reference/resource-configs/dbt_valid_to_current.md --- website/docs/reference/resource-configs/dbt_valid_to_current.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/dbt_valid_to_current.md b/website/docs/reference/resource-configs/dbt_valid_to_current.md index b6a99d8425c..7c0e33aa5d7 100644 --- a/website/docs/reference/resource-configs/dbt_valid_to_current.md +++ b/website/docs/reference/resource-configs/dbt_valid_to_current.md @@ -6,7 +6,7 @@ default_value: {NULL} id: "dbt_valid_to_current" --- -Available in 1.9 or with [Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) dbt Cloud. +Available from dbt v1.9 or with [Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) dbt Cloud. From ca7350dff0826d22520848a670e2fd30ae3b52fc Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 19 Nov 2024 15:18:15 -0500 Subject: [PATCH 21/24] Update release-notes.md swap links in release note --- website/docs/docs/dbt-versions/release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index 2f6eb753f41..7946d90297a 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -20,7 +20,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo ## November 2024 - **New**: Use the [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current) config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. This feature is available in dbt Cloud Versionless and dbt Core v1.9 and later. -- **New**: Use the `event_time` configuration to specify "at what time did the row occur." This configuration is required for [Incremental microbatch](/docs/build/incremental-microbatch) and can be added to ensure you're comparing overlapping times in [Advanced CI's compare changes](/docs/deploy/advanced-ci). Available in dbt Cloud Versionless and dbt Core v1.9 and higher. Refer to [event_time](/reference/resource-configs/event-time) for more information. +- **New**: Use the [`event_time`](/reference/resource-configs/dbt_valid_to_current) configuration to specify "at what time did the row occur." This configuration is required for [Incremental microbatch](/docs/build/incremental-microbatch) and can be added to ensure you're comparing overlapping times in [Advanced CI's compare changes](/docs/deploy/advanced-ci). Available in dbt Cloud Versionless and dbt Core v1.9 and higher. - **Fix**: This update improves [dbt Semantic Layer Tableau integration](/docs/cloud-integrations/semantic-layer/tableau) making query parsing more reliable. Some key fixes include: - Error messages for unsupported joins between saved queries and ALL tables. - Improved handling of queries when multiple tables are selected in a data source. From 75e8f4bc063b242a416d099ed1388afc991ad3f1 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 19 Nov 2024 15:33:06 -0500 Subject: [PATCH 22/24] Delete .github/ISSUE_TEMPLATE/internal-orch-team.yml this pr deletes the internal orch team github issue yaml template as it's no longer needed. also raised by @joellabes in slack thread https://dbt-labs.slack.com/archives/C02NCQ9483C/p1729894304530849 --- .github/ISSUE_TEMPLATE/internal-orch-team.yml | 49 ------------------- 1 file changed, 49 deletions(-) delete mode 100644 .github/ISSUE_TEMPLATE/internal-orch-team.yml diff --git a/.github/ISSUE_TEMPLATE/internal-orch-team.yml b/.github/ISSUE_TEMPLATE/internal-orch-team.yml deleted file mode 100644 index 8c4d61df10c..00000000000 --- a/.github/ISSUE_TEMPLATE/internal-orch-team.yml +++ /dev/null @@ -1,49 +0,0 @@ -name: Orchestration team - Request changes to docs -description: File a docs update request that is not already tracked in Orch team's Release Plans (Notion database). -labels: ["content","internal-orch-team"] -body: - - type: markdown - attributes: - value: | - * You can ask questions or submit ideas for the dbt docs in [Issues](https://github.com/dbt-labs/docs-internal/issues/new/choose) - * Before you file an issue read the [Contributing guide](https://github.com/dbt-labs/docs-internal#contributing). - * Check to make sure someone hasn't already opened a similar [issue](https://github.com/dbt-labs/docs-internal/issues). - - - type: checkboxes - id: contributions - attributes: - label: Contributions - description: Please read the contribution docs before opening an issue or pull request. - options: - - label: I have read the contribution docs, and understand what's expected of me. - - - type: textarea - attributes: - label: Link to the page on docs.getdbt.com requiring updates - description: Please link to the page or pages you'd like to see improved. - validations: - required: true - - - type: textarea - attributes: - label: What part(s) of the page would you like to see updated? - description: | - - Give as much detail as you can to help us understand the change you want to see. - - Why should the docs be changed? What use cases does it support? - - What is the expected outcome? - validations: - required: true - - - type: textarea - attributes: - label: Reviewers/Stakeholders/SMEs - description: List the reviewers, stakeholders, and subject matter experts (SMEs) to collaborate with for the docs update. - validations: - required: true - - - type: textarea - attributes: - label: Related Jira tickets - description: Add any other context or screenshots about the feature request here. - validations: - required: false From 1ad6e3657b6e72f1d6355f31edebad7a9b8537e3 Mon Sep 17 00:00:00 2001 From: Eric Baumann <24817191+ericbaumann@users.noreply.github.com> Date: Tue, 19 Nov 2024 16:17:48 -0500 Subject: [PATCH 23/24] Correct details about DBT_CLOUD_ENVIRONMENT_TYPE --- website/docs/docs/build/environment-variables.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/environment-variables.md b/website/docs/docs/build/environment-variables.md index e9381644713..c24090db536 100644 --- a/website/docs/docs/build/environment-variables.md +++ b/website/docs/docs/build/environment-variables.md @@ -118,7 +118,7 @@ The following environment variables are set automatically: - `DBT_ENV` — This key is reserved for the dbt Cloud application and will always resolve to 'prod'. For deployment runs only. - `DBT_CLOUD_ENVIRONMENT_NAME` — The name of the dbt Cloud environment in which `dbt` is running. -- `DBT_CLOUD_ENVIRONMENT_TYPE` — The type of dbt Cloud environment in which `dbt` is running. The valid values are `development` or `deployment`. +- `DBT_CLOUD_ENVIRONMENT_TYPE` — The type of dbt Cloud environment in which `dbt` is running. The valid values are `dev`, `staging`, or `prod`. This value can remain unset, so be sure to access it with a default (e.g. `{{env_var('DBT_CLOUD_ENVIRONMENT_TYPE', '')}}`). #### Run details From 736634807d4e0410d3b0097154d9814af8a2cb59 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 19 Nov 2024 16:21:33 -0500 Subject: [PATCH 24/24] Update website/docs/docs/build/environment-variables.md --- website/docs/docs/build/environment-variables.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/environment-variables.md b/website/docs/docs/build/environment-variables.md index c24090db536..99129cea8c9 100644 --- a/website/docs/docs/build/environment-variables.md +++ b/website/docs/docs/build/environment-variables.md @@ -118,7 +118,7 @@ The following environment variables are set automatically: - `DBT_ENV` — This key is reserved for the dbt Cloud application and will always resolve to 'prod'. For deployment runs only. - `DBT_CLOUD_ENVIRONMENT_NAME` — The name of the dbt Cloud environment in which `dbt` is running. -- `DBT_CLOUD_ENVIRONMENT_TYPE` — The type of dbt Cloud environment in which `dbt` is running. The valid values are `dev`, `staging`, or `prod`. This value can remain unset, so be sure to access it with a default (e.g. `{{env_var('DBT_CLOUD_ENVIRONMENT_TYPE', '')}}`). +- `DBT_CLOUD_ENVIRONMENT_TYPE` — The type of dbt Cloud environment in which `dbt` is running. The valid values are `dev`, `staging`, or `prod`. It can be unset, so use a default like `{{env_var('DBT_CLOUD_ENVIRONMENT_TYPE', '')}}`. #### Run details