Skip to content

Commit

Permalink
Merge branch 'current' into ly-docs-idle-timeout
Browse files Browse the repository at this point in the history
  • Loading branch information
nghi-ly authored Sep 25, 2024
2 parents 1070a3c + f17cf48 commit faf4c43
Show file tree
Hide file tree
Showing 12 changed files with 43 additions and 36 deletions.
21 changes: 6 additions & 15 deletions website/docs/docs/collaborate/data-tile.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,8 @@ description: "Embed data health tiles in your dashboards to distill trust signal
image: /img/docs/collaborate/dbt-explorer/data-tile-pass.jpg
---

# Embed data health tile in dashboards <Lifecycle status='beta' />

With data health tiles, stakeholders will get an at-a-glance confirmation on whether the data they’re looking at is stale or degraded. This trust signal allows teams to immediately go back into Explorer to see more details and investigate issues.

:::info Available in beta
Data health tile is currently available in open beta.
:::

The data health tile:

- Distills trust signals for data consumers.
Expand Down Expand Up @@ -122,17 +116,14 @@ Follow these steps to embed the data health tile in Tableau:
<Lightbox src="/img/docs/collaborate/dbt-explorer/tableau-example.png" width="80%" title="Embed data health tile iFrame in Tableau"/>

1. Create a dashboard in Tableau and connect to your database to pull in the data.
2. Ensure you've copied the iFrame snippet available in dbt Explorer's **Data health** section, under the **Embed data health into your dashboard** toggle.
3. Embed the snippet in your dashboard.

`<iframe src='https://metadata.YOUR_ACCESS_URL/exposure-tile?uniqueId=<exposure_unique_id>&environmentType=production&environmentId=<environment_id>&token=<metadata_token>' />`
2. Ensure you've copied the URL or iFrame snippet available in dbt Explorer's **Data health** section, under the **Embed data health into your dashboard** toggle.
3. Insert a **Web Page** object.
4. Insert the URL and click **Ok**.

`https://metadata.cloud.getdbt.com/exposure-tile?uniqueId=exposure.snowflake_tpcds_sales_spoke.customer360_test&environmentType=production&environmentId=220370&token=<YOUR_METADATA_TOKEN>`

*Note, replace the placeholders with your actual values.*

- **For job-based exposure tile** &mdash; Insert the following fields into the following iFrame. Then embed them with your dashboard. The next [section](#job-based-data-health) will have more details on the job-based exposure tile.
- `<iframe src='https://metadata.YOUR_ACCESS_URL/exposure-tile?name=<exposure_name>&environment_id=<environment_id>&token=<metadata_token>' />`

*Note, replace the placeholders with your actual values.*
5. You should now see the data health tile embedded in your Tableau dashboard.

</TabItem>
</Tabs>
Expand Down
45 changes: 24 additions & 21 deletions website/docs/docs/collaborate/model-query-history.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,15 @@ description: "Import and auto-generate exposures from dashboards and understand
image: /img/docs/collaborate/dbt-explorer/model-query-queried-models.jpg
---

# About model query history <Lifecycle status='beta' />
# About model query history <Lifecycle status='preview' />

Model query history allows you to:

- View the count of consumption queries for a model based on the data warehouse's query logs.
- View the count of consumption queries for a model based on the data warehouse's query logs.
- Provides data teams insight, so they can focus their time and infrastructure spend on the worthwhile used data products.
- Enable analysts to find the most popular models used by other people.

Model query history is powered by a single consumption query of the query log table in your data warehouse aggregated on a daily basis.
Model query history is powered by a single consumption query of the query log table in your data warehouse aggregated on a daily basis. It currently supports Snowflake and BigQuery only, with additional platforms coming soon.

:::info What is a consumption query?
Consumption query is a metric of queries in your dbt project that has used the model in a given time. It filters down to `select` statements only to gauge model consumption and excludes dbt model build and test executions.
Expand All @@ -28,7 +28,23 @@ To access the features, you should meet the following:
1. You have a dbt Cloud account on the [Enterprise plan](https://www.getdbt.com/pricing/).
2. You have set up a [production](https://docs.getdbt.com/docs/deploy/deploy-environments#set-as-production-environment) deployment environment for each project you want to explore, with at least one successful job run.
3. You have [admin permissions](/docs/cloud/manage-access/enterprise-permissions) in dbt Cloud to edit project settings or production environment settings.
4. Use Snowflake or BigQuery as your data warehouse and can enable query history permissions or work with an admin to do so (Support for additional data platforms coming soon).
4. Use Snowflake or BigQuery as your data warehouse and can enable query history permissions or work with an admin to do so. Support for additional data platforms coming soon.

## Enable query history in dbt Cloud

To enable model query history in dbt Cloud, follow these steps:

1. Navigate to **Deploy** and then **Environments**.
2. Select the environment marked **PROD** and click **Settings**.
3. Click **Edit** and scroll to the **Query History** section to enable the query history toggle. When it’s green and to the right, it's enabled.
4. Click the **Test Permissions** button to validate the deployment credentials permissions are sufficient to support query history.

<DocCarousel slidesPerView={1}>

<Lightbox src="/img/docs/collaborate/dbt-explorer/enable-query-history.jpg" width="95%" title="Enable query history in your environment settings." />
<Lightbox src="/img/docs/collaborate/dbt-explorer/enable-query-history-success.jpg" width="95%" title="Example of permissions verified result after clicking Test Permissions." />

</DocCarousel>

## Credential permissions

Expand Down Expand Up @@ -61,25 +77,12 @@ The model query history feature uses the credentials in your production environm

</Expandable>

## Enable query history in dbt Cloud

:::info
During beta, the dbt Labs team will manually enable query history for your dbt Cloud projects. Later on, you’ll be able to do it yourself.

:::
<!--
1. Navigate to **Deploy** and then **Environments**.
2. Select the environment marked **PROD** and click **Settings**.
3. Enable the checkbox for query history in **General Settings**.
4. Click the **Test** button to validate the deployment credentials permissions are sufficient to support query history.
-->

## View query history in Explorer

To enhance your discovery, you can view your model query history in various locations within dbt Explorer:
- [View from Performance charts](#view-from-performance-charts)
* [View from Project lineage](#view-from-project-lineage)
- [View from Model list](#view-from-model-list)
- [View from Performance charts](#view-from-performance-charts)
* [View from Project lineage](#view-from-project-lineage)
- [View from Model list](#view-from-model-list)

### View from Performance charts

Expand All @@ -92,7 +95,7 @@ To enhance your discovery, you can view your model query history in various loca
4. Click on a model for more details and go to the **Performance** tab.
5. On the **Performance** tab, scroll down to the **Model performance** section.
6. Select the **Consumption queries** tab to view the consumption queries over a given time for that model.
<Lightbox src="/img/docs/collaborate/model-consumption-queries.jpg" width="85%" title="View consumption queries over time for a given model." />
<Lightbox src="/img/docs/collaborate/model-consumption-queries.jpg" width="90%" title="View consumption queries over time for a given model." />

### View from Project lineage

Expand Down
2 changes: 2 additions & 0 deletions website/docs/docs/dbt-versions/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ Release notes are grouped by month for both multi-tenant and virtual private clo

## September 2024

- **New**: [Data health tile](/docs/collaborate/data-tile) is now generally available in dbt Explorer. Data health tiles provide a quick at-a-glance view of your data quality, highlighting potential issues in your data. You can embed these tiles in your dashboards to quickly identify and address data quality issues in your dbt project.
- **New**: dbt Explorer's Model query history feature is now in Preview for dbt Cloud Enterprise customers. Model query history allows you to view the count of consumption queries for a model based on the data warehouse's query logs. This feature provides data teams insight, so they can focus their time and infrastructure spend on the worthwhile used data products. To learn more, refer to [Model query history](/docs/collaborate/model-query-history).
- **Enhancement**: You can now use [Extended Attributes](/docs/dbt-cloud-environments#extended-attributes) and [Environment Variables](/docs/build/environment-variables) when connecting to the Semantic Layer. If you set a value directly in the Semantic Layer Credentials, it will have a higher priority than Extended Attributes. When using environment variables, the default value for the environment will be used. If you're using exports, job environment variable overrides aren't supported yet, but they will be soon.
- **New:** There are two new [environment variable defaults](/docs/build/environment-variables#dbt-cloud-context) &mdash; `DBT_CLOUD_ENVIRONMENT_NAME` and `DBT_CLOUD_ENVIRONMENT_TYPE`.
- **New:** The [Amazon Athena warehouse connection](/docs/cloud/connect-data-platform/connect-amazon-athena) is available as a public preview for dbt Cloud accounts that have upgraded to [`versionless`](/docs/dbt-versions/versionless-cloud).
Expand Down
2 changes: 2 additions & 0 deletions website/docs/docs/deploy/ci-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ dbt Labs recommends that you create your CI job in a dedicated dbt Cloud [deploy
- You have a dbt Cloud account.
- CI features:
- For both the [concurrent CI checks](/docs/deploy/continuous-integration#concurrent-ci-checks) and [smart cancellation of stale builds](/docs/deploy/continuous-integration#smart-cancellation) features, your dbt Cloud account must be on the [Team or Enterprise plan](https://www.getdbt.com/pricing/).
- The [SQL linting](/docs/deploy/continuous-integration#sql-linting) feature is currently available in beta to a limited group of users and is gradually being rolled out. If you're in the beta, the **Linting** option is available for use.
- [Advanced CI](/docs/deploy/advanced-ci) features:<Lifecycle status="preview" />
- For the [compare changes](/docs/deploy/advanced-ci#compare-changes) feature, your dbt Cloud account must have access to Advanced CI. Please ask your [dbt Cloud administrator to enable](/docs/cloud/account-settings#account-access-to-advanced-ci-features) this for you.
- Set up a [connection with your Git provider](/docs/cloud/git/git-configuration-in-dbt-cloud). This integration lets dbt Cloud run jobs on your behalf for job triggering.
Expand All @@ -35,6 +36,7 @@ To make CI job creation easier, many options on the **CI job** page are set to d

1. Options in the **Execution settings** section:
- **Commands** &mdash; By default, it includes the `dbt build --select state:modified+` command. This informs dbt Cloud to build only new or changed models and their downstream dependents. Importantly, state comparison can only happen when there is a deferred environment selected to compare state to. Click **Add command** to add more [commands](/docs/deploy/job-commands) that you want to be invoked when this job runs.
- **Linting**<Lifecycle status="beta" /> &mdash; Enable this option for dbt to [lint the SQL files](/docs/deploy/continuous-integration#sql-linting) in your project as the first step in `dbt run`. If this check runs into an error, dbt can either **Fail job run** or **Continue running job**.
- **Run compare changes**<Lifecycle status="preview" /> &mdash; Enable this option to compare the last applied state of the production environment (if one exists) with the latest changes from the pull request, and identify what those differences are. To enable record-level comparison and primary key analysis, you must add a [primary key constraint](/reference/resource-properties/constraints) or [uniqueness test](/reference/resource-properties/data-tests#unique). Otherwise, you'll receive a "Primary key missing" error message in dbt Cloud.

To review the comparison report, navigate to the [Compare tab](/docs/deploy/run-visibility#compare-tab) in the job run's details. A summary of the report is also available from the pull request in your Git provider (see the [CI report example](#example-ci-report)).
Expand Down
9 changes: 9 additions & 0 deletions website/docs/docs/deploy/continuous-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ The [dbt Cloud scheduler](/docs/deploy/job-scheduler) executes CI jobs different
- **Concurrent CI checks** &mdash; CI runs triggered by the same dbt Cloud CI job execute concurrently (in parallel), when appropriate.
- **Smart cancellation of stale builds** &mdash; Automatically cancels stale, in-flight CI runs when there are new commits to the PR.
- **Run slot treatment** &mdash; CI runs don't consume a run slot.
- **SQL linting**<Lifecycle status="beta" /> &mdash; When enabled, automatically lints all SQL files in your project as a run step before your CI job builds.

### Concurrent CI checks

Expand All @@ -54,3 +55,11 @@ When you push a new commit to a PR, dbt Cloud enqueues a new CI run for the late
### Run slot treatment <Lifecycle status="team,enterprise" />

CI runs don't consume run slots. This guarantees a CI check will never block a production run.

### SQL linting <Lifecycle status="beta" />

When enabled for your CI job, dbt invokes [SQLFluff](https://sqlfluff.com/) which is a modular and configurable SQL linter that warns you of complex functions, syntax, formatting, and compilation errors. By default, it lints all the SQL files in your project.

If the linter runs into errors, you can specify whether dbt should fail the job or continue running it. When failing jobs, it helps reduce compute costs by avoiding builds for pull requests that don't meet your SQL code quality CI check.

To override the default linting behavior, create an `.sqlfluff` config file in your project and add your linting rules to it. dbt Cloud will use the rules defined in the config file when linting. For details about linting rules, refer to [Custom Usage](https://docs.sqlfluff.com/en/stable/gettingstarted.html#custom-usage) in the SQLFluff documentation.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit faf4c43

Please sign in to comment.