Skip to content

Commit

Permalink
Merge branch 'current' into mwong-add-cookbook
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Sep 6, 2024
2 parents a40aba0 + 10b4d99 commit fa46df7
Show file tree
Hide file tree
Showing 38 changed files with 281 additions and 141 deletions.
2 changes: 2 additions & 0 deletions website/dbt-versions.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@
* @property {string} EOLDate "End of Life" date which is used to show the EOL banner
* @property {boolean} isPrerelease Boolean used for showing the prerelease banner
* @property {string} customDisplay Allows setting a custom display name for the current version
*
* customDisplay for dbt Cloud should be a version ahead of latest dbt Core release (GA or beta).
*/
exports.versions = [
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import AvailIntegrations from '/snippets/_sl-partner-links.md';
### Custom integration

- [Exports](/docs/use-dbt-semantic-layer/exports) enable custom integration with additional tools that don't natively connect with the dbt Semantic Layer, such as PowerBI.
- Develop custom integrations using different languages and tools, supported through JDBC, ADBC, and GraphQL APIs. For more info, check out [our examples on GitHub](https://github.com/dbt-labs/example-semantic-layer-clients/).
- [Consume metrics](/docs/use-dbt-semantic-layer/consume-metrics) and develop custom integrations using different languages and tools, supported through [JDBC](/docs/dbt-cloud-apis/sl-jdbc), ADBC, and [GraphQL](/docs/dbt-cloud-apis/sl-graphql) APIs, and [Python SDK library](/docs/dbt-cloud-apis/sl-python). For more info, check out [our examples on GitHub](https://github.com/dbt-labs/example-semantic-layer-clients/).
- Connect to any tool that supports SQL queries. These tools must meet one of the two criteria:
- Offers a generic JDBC driver option (such as DataGrip) or
- Is compatible Arrow Flight SQL JDBC driver version 12.0.0 or higher.
Expand Down
34 changes: 20 additions & 14 deletions website/docs/docs/collaborate/data-tile.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,20 @@ image: /img/docs/collaborate/dbt-explorer/data-tile-pass.jpg
# Embed data health tile in dashboards <Lifecycle status='beta' />

With data health tiles, stakeholders will get an at-a-glance confirmation on whether the data they’re looking at is stale or degraded. This trust signal allows teams to immediately go back into Explorer to see more details and investigate issues.

:::info Available in beta
Data health tile is currently available in open beta.
:::

The data health tile:

- Distills trust signals for data consumers.
- Deep links you into dbt Explorer where you can further dive into upstream data issues.
- Provides richer information and makes it easier to debug.
- Revamps the existing, [job-based tiles](#job-based-data-health).

Data health tiles rely on [exposures](/docs/build/exposures) to surface trust signals in your dashboards. When you configure exposures in your dbt project, you are explicitly defining how specific outputs—like dashboards or reports—depend on your data models.

<Lightbox src="/img/docs/collaborate/dbt-explorer/data-tiles.png" width="60%" title="Embed data health tiles in your dashboards to distill trust signals for data consumers." />

## Prerequisites
Expand All @@ -34,43 +38,45 @@ First, be sure to enable [source freshness](/docs/deploy/source-freshness) in

1. Navigate to dbt Explorer by clicking on the **Explore** link in the navigation.
2. In the main **Overview** page, go to the left navigation.
3. Under the **Resources** tab, click on **Exposures** to view the exposures list.
3. Under the **Resources** tab, click on **Exposures** to view the [exposures](/docs/build/exposures) list.
4. Select a dashboard exposure and go to the **General** tab to view the data health information.
5. In this tab, you’ll see:
- Data health status: Data freshness passed, Data quality passed, Data may be stale, Data quality degraded
- Name of the exposure.
5. In this tab, you’ll see:
- Name of the exposure.
- Data health status: Data freshness passed, Data quality passed, Data may be stale, Data quality degraded.
- Resource type (model, source, and so on).
- Dashboard status: Failure, Pass, Stale.
- You can also see the last check completed, the last check time, and the last check duration.
6. You can also click the **Open Dashboard** button on the upper right to immediately view this in your analytics tool.
6. You can click the **Open Dashboard** button on the upper right to immediately view this in your analytics tool.

<Lightbox src="/img/docs/collaborate/dbt-explorer/data-tile-exposures.jpg" width="95%" title="View an exposure in dbt Explorer." />

## Embed in your dashboard

Once you’ve navigated to the auto-exposure in dbt Explorer, you’ll need to set up your dashboard status tile and [service token](/docs/dbt-cloud-apis/service-tokens):
Once you’ve navigated to the auto-exposure in dbt Explorer, you’ll need to set up your data health tile and [service token](/docs/dbt-cloud-apis/service-tokens). You can embed data health tile to any analytics tool that supports URL or iFrame embedding.

Follow these steps to set up your data health tile:

1. Go to **Account settings** in dbt Cloud.
2. Select **API tokens** in the left sidebar and then **Service tokens**.
3. Click on **Create service token** and give it a name.
4. Select the [**Metadata Only** permission](/docs/dbt-cloud-apis/service-tokens). This token will be used to embed the exposure tile in your dashboard in the later steps.
4. Select the [**Metadata Only**](/docs/dbt-cloud-apis/service-tokens) permission. This token will be used to embed the tile in your dashboard in the later steps.
<Lightbox src="/img/docs/collaborate/dbt-explorer/data-tile-setup.jpg" width="95%" title="Set up your dashboard status tile and service token to embed a data health tile" />

5. Copy the **Metadata Only token** and save it in a secure location. You'll need it token in the next steps.
5. Copy the **Metadata Only** token and save it in a secure location. You'll need it token in the next steps.
6. Navigate back to dbt Explorer and select an exposure.
7. Below the **Data health** section, expand on the toggle for instructions on how to embed the exposure tile (if you're an account admin with develop permissions).
8. In the expanded toggle, you'll see a text field where you can paste your **Metadata Only token**.
<Lightbox src="/img/docs/collaborate/dbt-explorer/data-tile-example.jpg" width="85%" title="Expand the toggle to embded data health tile into your dashboard." />

9. Once you’ve pasted your token, you can select either **URL** or **iFrame** depending on which you need to install into your dashboard.
9. Once you’ve pasted your token, you can select either **URL** or **iFrame** depending on which you need to add to your dashboard.

If your analytics tool supports iFrames, you can embed the dashboard tile within it.

### Embed data health tile in Tableau
To embed the data health tile in Tableau, follow these steps:
#### Tableau example
Here’s an example with Tableau, where you can embed the iFrame in a web page object:

1. Ensure you've copied the embed iFrame content in dbt Explorer.
2. For the revamped environment-based exposure tile you can insert these fields into the following iFrame, and then embed them with your dashboard. This is the iFrame that is available from the **Exposure details** page in dbt Explorer.
- Ensure you've copied the embed iFrame snippet from the dbt Explorer **Data health** section.
- **For the revamped environment-based exposure tile** &mdash; Insert the following fields into the following iFrame. Then embed them with your dashboard. This is the iFrame available from the **Exposure details** page in dbt Explorer.

`<iframe src='https://metadata.YOUR_ACCESS_URL/exposure-tile?uniqueId=<exposure_unique_id>&environmentType=production&environmentId=<environment_id>&token=<metadata_token>' />`

Expand All @@ -82,7 +88,7 @@ To embed the data health tile in Tableau, follow these steps:
<Lightbox src="/img/docs/collaborate/dbt-explorer/data-tile-stale.jpg" width="60%" title="Example of stale of degraded Data health tile in your dashboard." />
</DocCarousel>

3. For the job-based exposure tile you can insert these three fields into the following iFrame, and then embed them with your dashboard. The next section will have more details on the job-based exposure tile.
- **For job-based exposure tile** &mdash; Insert the following fields into the following iFrame. Then embed them with your dashboard. The next [section](#job-based-data-health) will have more details on the job-based exposure tile.

`<iframe src='https://metadata.YOUR_ACCESS_URL/exposure-tile?name=<exposure_name>&environment_id=<environment_id>&token=<metadata_token>' />`

Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/collaborate/explore-projects.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Navigate the dbt Explorer overview page to access your project's resources and m
- **Lineage graph** &mdash; Explore your project's or account's [lineage graph](#project-lineage) to visualize the relationships between resources.
- **Latest updates** &mdash; View the latest changes or issues related to your project's resources, including the most recent job runs, changed properties, lineage, and issues.
- **Marts and public models** &mdash; View the [marts](/best-practices/how-we-structure/1-guide-overview#guide-structure-overview) and [public models](/docs/collaborate/govern/model-access#access-modifiers) in your project.
- **Model query history** &mdash; Use [model query history](/docs/collaborate/model-query-history) to track the history of queries on your models for deeper insights.
- **Model query history** &mdash; Use [model query history](/docs/collaborate/model-query-history) to track consumption queries on your models for deeper insights.
- **Auto-exposures** &mdash; [Set up and view auto-exposures](/docs/collaborate/auto-exposures) to automatically expose relevant data models from Tableau to enhance visibility.

<Lightbox src="/img/docs/collaborate/dbt-explorer/explorer-main-page.gif" width="100%" title="Access dbt Explorer from dbt Cloud by clicking Explore in the navigation."/>
Expand Down
40 changes: 24 additions & 16 deletions website/docs/docs/collaborate/model-query-history.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,18 @@ image: /img/docs/collaborate/dbt-explorer/model-query-queried-models.jpg

# About model query history <Lifecycle status='beta' />

The model query history tile allows you to:
Model query history allows you to:

- View the query count for a model based on the data warehouse's query logs.
- View the count of consumption queries for a model based on the data warehouse's query logs.
- Provides data teams insight, so they can focus their time and infrastructure spend on the worthwhile used data products.
- Enable analysts to find the most popular models used by other people.

:::info Available in beta
Model query history is powered by a single query of the query log table in your data warehouse aggregated on a daily basis. It filters down to `select` statements only to gauge model consumption and excludes dbt model build and test executions.
Model query history is powered by a single consumption query of the query log table in your data warehouse aggregated on a daily basis.

:::info What is a consumption query?
Consumption query is a metric of queries in your dbt project that has used the model in a given time. It filters down to `select` statements only to gauge model consumption and excludes dbt model build and test executions.

So for example, if `model_super_santi` was queried 10 times in the past week, it would count as having 10 consumption queries for that particular time period.
:::

## Prerequisites
Expand Down Expand Up @@ -72,31 +76,35 @@ During beta, the dbt Labs team will manually enable query history for your dbt C

## View query history in Explorer

To enhance your discovery, you can view your model query history in various locations within dbt Explorer. For details on how to access model query history in each location, expand the following toggles:
To enhance your discovery, you can view your model query history in various locations within dbt Explorer:
- [View from Performance charts](#view-from-performance-charts)
* [View from Project lineage](#view-from-project-lineage)
- [View from Model list](#view-from-model-list)

### View from Performance charts

1. Navigate to dbt Explorer by clicking on the **Explore** link in the navigation.
2. In the main **Overview** page, under **Project** click **Performance** and scroll down to view the most queried models
2. In the main **Overview** page, click on **Performance** under the **Project details** section. Scroll down to view the **Most consumed models**.
3. Use the dropdown menu on the right to select the desired time period, with options available for up to the past 3 months.

<Lightbox src="/img/docs/collaborate/dbt-explorer/model-query-queried-models.jpg" width="70%" title="View most queried models in 'Performance' page in dbt Explorer." />
<Lightbox src="/img/docs/collaborate/dbt-explorer/most-consumed-models.jpg" width="85%" title="View most consumed models on the 'Performance' page in dbt Explorer." />

4. In the model performance tab, open the **Usage** chart to see queries over time for that model.
<Lightbox src="/img/docs/collaborate/dbt-explorer/model-query-usage-queries.jpg" width="70%" title="View queries over time for a given model." />
4. Click on a model for more details and go to the **Performance** tab.
5. On the **Performance** tab, scroll down to the **Model performance** section.
6. Select the **Consumption queries** tab to view the consumption queries over a given time for that model.
<Lightbox src="/img/docs/collaborate/model-consumption-queries.jpg" width="85%" title="View consumption queries over time for a given model." />

### View from Project lineage

1. To view your model in your project lineage, go to the main **Overview page** and click on **Project lineage.**
2. In the lower left of your lineage, click on **Lenses** and select **Usage queries**.
<Lightbox src="/img/docs/collaborate/dbt-explorer/model-query-lenses.jpg" width="85%" title="View model usage query in your lineage using the 'Lenses' feature." />
2. In the lower left of your lineage, click on **Lenses** and select **Consumption queries**.
<Lightbox src="/img/docs/collaborate/dbt-explorer/model-consumption-lenses.jpg" width="85%" title="View model consumption queries in your lineage using the 'Lenses' feature." />

3. Your lineage should display a small red box above each model, indicating the usage query number for each model. The query number for each model represents the query history over the last 30 days.
3. Your lineage should display a small red box above each model, indicating the consumption query number. The number for each model represents the model consumption over the last 30 days.

### View from Model list

1. To view your model in your project lineage, go to the main **Overview page**.
1. To view a list of models, go to the main **Overview page**.
2. In the left navigation, go to the **Resources** tab and click on **Models** to view the models list.
3. You can view the usage query count for the models and sort by most or least queried. The query number for each model represents the query history over the last 30 days.
<Lightbox src="/img/docs/collaborate/dbt-explorer/model-query-list.jpg" width="85%" title="View models query history in the 'Models' list page under the 'Usage' column." />

3. You can view the consumption query count for the models and sort by most or least consumed. The consumption query number for each model represents the consumption over the last 30 days.
<Lightbox src="/img/docs/collaborate/dbt-explorer/model-consumption-list.jpg" width="85%" title="View models consumption in the 'Models' list page under the 'Consumption' column." />
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ id: "postgres-setup"
meta:
maintained_by: dbt Labs
authors: 'core dbt maintainers'
github_repo: 'dbt-labs/dbt-core'
github_repo: 'dbt-labs/dbt-postgres'
pypi_package: 'dbt-postgres'
min_core_version: 'v0.4.0'
cloud_support: Supported
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,13 +98,13 @@ The ability for installed packages to override built-in materializations without

### Managing changes to legacy behaviors

dbt Core v1.8 has introduced flags for [managing changes to legacy behaviors](/reference/global-configs/legacy-behaviors). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `True` / `False` values, respectively, for `flags` in `dbt_project.yml`.
dbt Core v1.8 has introduced flags for [managing changes to legacy behaviors](/reference/global-configs/behavior-changes). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `True` / `False` values, respectively, for `flags` in `dbt_project.yml`.

You can read more about each of these behavior changes in the following links:

- (Mature, enabled by default) [Require explicit package overrides for builtin materializations](/reference/global-configs/legacy-behaviors#require_explicit_package_overrides_for_builtin_materializations)
- (Introduced, disabled by default) [Require resource names without spaces](https://docs.getdbt.com/reference/global-configs/legacy-behaviors#require_resource_names_without_spaces)
- (Introduced, disabled by default) [Run project hooks (`on-run-*`) in the `dbt source freshness` command](/reference/global-configs/legacy-behaviors#source_freshness_run_project_hooks)
- (Mature, enabled by default) [Require explicit package overrides for builtin materializations](/reference/global-configs/behavior-changes#require_explicit_package_overrides_for_builtin_materializations)
- (Introduced, disabled by default) [Require resource names without spaces](/reference/global-configs/behavior-changes#require_resource_names_without_spaces)
- (Introduced, disabled by default) [Run project hooks (`on-run-*`) in the `dbt source freshness` command](/reference/global-configs/behavior-changes#source_freshness_run_project_hooks)

## Quick hits

Expand Down
Loading

0 comments on commit fa46df7

Please sign in to comment.