Skip to content

Commit

Permalink
Merge branch 'current' into explorer-workshop-guide
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Sep 26, 2024
2 parents 510fce4 + 22f8ab4 commit d0ba5ed
Show file tree
Hide file tree
Showing 42 changed files with 252 additions and 183 deletions.
23 changes: 20 additions & 3 deletions website/docs/docs/build/environment-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,10 +139,14 @@ _The following variables are currently only available for GitHub, GitLab, and Az

Environment variables can be used in many ways, and they give you the power and flexibility to do what you want to do more easily in dbt Cloud.

#### Clone private packages
<Expandable alt_header="Clone private packages">

Now that you can set secrets as environment variables, you can pass git tokens into your package HTTPS URLs to allow for on-the-fly cloning of private repositories. Read more about enabling [private package cloning](/docs/build/packages#private-packages).

#### Dynamically set your warehouse in your Snowflake connection
</Expandable>

<Expandable alt_header="Dynamically set your warehouse in your Snowflake connection">

Environment variables make it possible to dynamically change the Snowflake virtual warehouse size depending on the job. Instead of calling the warehouse name directly in your project connection, you can reference an environment variable which will get set to a specific virtual warehouse at runtime.

For example, suppose you'd like to run a full-refresh job in an XL warehouse, but your incremental job only needs to run in a medium-sized warehouse. Both jobs are configured in the same dbt Cloud environment. In your connection configuration, you can use an environment variable to set the warehouse name to `{{env_var('DBT_WAREHOUSE')}}`. Then in the job settings, you can set a different value for the `DBT_WAREHOUSE` environment variable depending on the job's workload.
Expand All @@ -163,7 +167,10 @@ However, there are some limitations when using env vars with Snowflake OAuth Con
Something to note, if you supply an environment variable in the account/host field, Snowflake OAuth Connection will **fail** to connect. This happens because the field doesn't pass through Jinja rendering, so dbt Cloud simply passes the literal `env_var` code into a URL string like `{{ env_var("DBT_ACCOUNT_HOST_NAME") }}.snowflakecomputing.com`, which is an invalid hostname. Use [extended attributes](/docs/deploy/deploy-environments#deployment-credentials) instead.
:::

#### Audit your run metadata
</Expandable>

<Expandable alt_header="Audit your run metadata">

Here's another motivating example that uses the dbt Cloud run ID, which is set automatically at each run. This additional data field can be used for auditing and debugging:

```sql
Expand All @@ -189,3 +196,13 @@ select *,

from users_aggregated
```

</Expandable>

<Expandable alt_header="Configure Semantic Layer credentials">

import SLEnvVars from '/snippets/_sl-env-vars.md';

<SLEnvVars/>

</Expandable>
1 change: 0 additions & 1 deletion website/docs/docs/cloud/configure-cloud-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,6 @@ To set environment variables in the dbt Cloud CLI for your dbt project:
2. Then select **Profile Settings**, then **Credentials**.
3. Click on your project and scroll to the **Environment Variables** section.
4. Click **Edit** on the lower right and then set the user-level environment variables.
- Note, when setting up the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl), using [environment variables](/docs/build/environment-variables) like `{{env_var('DBT_WAREHOUSE')}}` is not supported. You should use the actual credentials instead.

## Use the dbt Cloud CLI

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ Nice job, you're ready to start developing and building models 🎉!
### Considerations
- To improve your experience using dbt Cloud, we suggest that you turn off ad blockers. This is because some project file names, such as `google_adwords.sql`, might resemble ad traffic and trigger ad blockers.
- To preserve performance, there's a file size limitation for repositories over 6 GB. If you have a repo over 6 GB, please contact [dbt Support](mailto:[email protected]) before running dbt Cloud.
- The IDE's idle session timeout is one hour.
- <Expandable alt_header="About the start up process and work retention">

### Start-up process
Expand Down
253 changes: 161 additions & 92 deletions website/docs/docs/cloud/manage-access/about-access.md

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion website/docs/docs/cloud/manage-access/audit-log.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,6 @@ The audit log supports various events for different objects in dbt Cloud. You wi
| Event Name | Event Type | Description |
| -------------------------- | ---------------------------------------- | ------------------------------------------------------ |
| Auth Provider Changed | auth_provider.Changed | Authentication provider settings changed |
| Credential Login Failed | auth.CredentialsLoginFailed | User login via username and password failed |
| Credential Login Succeeded | auth.CredentialsLoginSucceeded | User successfully logged in with username and password |
| SSO Login Failed | auth.SsoLoginFailed | User login via SSO failed |
| SSO Login Succeeded | auth.SsoLoginSucceeded | User successfully logged in via SSO
Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/cloud/manage-access/sso-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Then, assign all of these (and only these) to the user license. This step will a

## SSO enforcement

* **SSO Enforcement:** If you have SSO turned on in your organization, dbt Cloud will enforce SSO-only logins for all non-admin users. If an Account Admin already has a password, they can continue logging in with a password.
* **SSO Enforcement:** If SSO is turned on in your organization, dbt Cloud will enforce SSO-only logins for all non-admin users. By default, if an Account Admin or Security Admin already has a password, they can continue logging in with a password. To restrict admins from using passwords, turn off **Allow password logins for account administrators** in the **Single sign-on** section of your organization's **Account settings**.
* **SSO Re-Authentication:** dbt Cloud will prompt you to re-authenticate using your SSO provider every 24 hours to ensure high security.

### How should non-admin users log in?
Expand Down
8 changes: 4 additions & 4 deletions website/docs/docs/cloud/migration.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,15 @@ dbt Labs is in the process of rolling out a new cell-based architecture for dbt

We're scheduling migrations by account. When we're ready to migrate your account, you will receive a banner or email communication with your migration date. If you have not received this communication, then you don't need to take action at this time. dbt Labs will share information about your migration with you, with appropriate advance notice, when applicable to your account.

Your account will be automatically migrated on its scheduled date. However, if you use certain features, you must take action before that date to avoid service disruptions.
Your account will be automatically migrated on or after its scheduled date. However, if you use certain features, you must take action before that date to avoid service disruptions.

## Recommended actions

We highly recommended you take these actions:

- Ensure pending user invitations are accepted or note outstanding invitations. Pending user invitations will be voided during the migration and must be resent after it is complete.
- Commit unsaved changes in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud). Unsaved changes will be lost during migration.
- Export and download [audit logs](/docs/cloud/manage-access/audit-log) older than 90 days, as they will be lost during migration. If you lose critical logs older than 90 days during the migration, you will have to work with the dbt Labs Customer Support team to recover.
- Ensure pending user invitations are accepted or note outstanding invitations. Pending user invitations might be voided during the migration. You can resend user invitations after the migration is complete.
- Commit unsaved changes in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud). Unsaved changes might be lost during migration.
- Export and download [audit logs](/docs/cloud/manage-access/audit-log) older than 90 days, as they will be unavailable from dbt Cloud after the migration is complete. Logs older than 90 days while within the data retention period are not deleted, but you will have to work with the dbt Labs Customer Support team to recover.

## Required actions

Expand Down
21 changes: 6 additions & 15 deletions website/docs/docs/collaborate/data-tile.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,8 @@ description: "Embed data health tiles in your dashboards to distill trust signal
image: /img/docs/collaborate/dbt-explorer/data-tile-pass.jpg
---

# Embed data health tile in dashboards <Lifecycle status='beta' />

With data health tiles, stakeholders will get an at-a-glance confirmation on whether the data they’re looking at is stale or degraded. This trust signal allows teams to immediately go back into Explorer to see more details and investigate issues.

:::info Available in beta
Data health tile is currently available in open beta.
:::

The data health tile:

- Distills trust signals for data consumers.
Expand Down Expand Up @@ -122,17 +116,14 @@ Follow these steps to embed the data health tile in Tableau:
<Lightbox src="/img/docs/collaborate/dbt-explorer/tableau-example.png" width="80%" title="Embed data health tile iFrame in Tableau"/>

1. Create a dashboard in Tableau and connect to your database to pull in the data.
2. Ensure you've copied the iFrame snippet available in dbt Explorer's **Data health** section, under the **Embed data health into your dashboard** toggle.
3. Embed the snippet in your dashboard.

`<iframe src='https://metadata.YOUR_ACCESS_URL/exposure-tile?uniqueId=<exposure_unique_id>&environmentType=production&environmentId=<environment_id>&token=<metadata_token>' />`
2. Ensure you've copied the URL or iFrame snippet available in dbt Explorer's **Data health** section, under the **Embed data health into your dashboard** toggle.
3. Insert a **Web Page** object.
4. Insert the URL and click **Ok**.

`https://metadata.cloud.getdbt.com/exposure-tile?uniqueId=exposure.snowflake_tpcds_sales_spoke.customer360_test&environmentType=production&environmentId=220370&token=<YOUR_METADATA_TOKEN>`

*Note, replace the placeholders with your actual values.*

- **For job-based exposure tile** &mdash; Insert the following fields into the following iFrame. Then embed them with your dashboard. The next [section](#job-based-data-health) will have more details on the job-based exposure tile.
- `<iframe src='https://metadata.YOUR_ACCESS_URL/exposure-tile?name=<exposure_name>&environment_id=<environment_id>&token=<metadata_token>' />`

*Note, replace the placeholders with your actual values.*
5. You should now see the data health tile embedded in your Tableau dashboard.

</TabItem>
</Tabs>
Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/collaborate/explore-projects.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Navigate the dbt Explorer overview page to access your project's resources and m
- **Sidebar** &mdash; Use the left sidebar to access model [performance](/docs/collaborate/model-performance), [project recommendations](/docs/collaborate/project-recommendations) in the **Project details** section. Browse your project's [resources, file tree, and database](#browse-with-the-sidebar) in the lower section of the sidebar.
- **Lineage graph** &mdash; Explore your project's or account's [lineage graph](#project-lineage) to visualize the relationships between resources.
- **Latest updates** &mdash; View the latest changes or issues related to your project's resources, including the most recent job runs, changed properties, lineage, and issues.
- **Marts and public models** &mdash; View the [marts](/best-practices/how-we-structure/1-guide-overview#guide-structure-overview) and [public models](/docs/collaborate/govern/model-access#access-modifiers) in your project.
- **Marts and public models** &mdash; View the [marts](/best-practices/how-we-structure/1-guide-overview#guide-structure-overview) and [public models](/docs/collaborate/govern/model-access#access-modifiers) in your project. You can also navigate to all public models in your account through this view.
- **Model query history** &mdash; Use [model query history](/docs/collaborate/model-query-history) to track consumption queries on your models for deeper insights.
- **Auto-exposures** &mdash; [Set up and view auto-exposures](/docs/collaborate/auto-exposures) to automatically expose relevant data models from Tableau to enhance visibility.

Expand Down
45 changes: 24 additions & 21 deletions website/docs/docs/collaborate/model-query-history.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,15 @@ description: "Import and auto-generate exposures from dashboards and understand
image: /img/docs/collaborate/dbt-explorer/model-query-queried-models.jpg
---

# About model query history <Lifecycle status='beta' />
# About model query history <Lifecycle status='preview' />

Model query history allows you to:

- View the count of consumption queries for a model based on the data warehouse's query logs.
- View the count of consumption queries for a model based on the data warehouse's query logs.
- Provides data teams insight, so they can focus their time and infrastructure spend on the worthwhile used data products.
- Enable analysts to find the most popular models used by other people.

Model query history is powered by a single consumption query of the query log table in your data warehouse aggregated on a daily basis.
Model query history is powered by a single consumption query of the query log table in your data warehouse aggregated on a daily basis. It currently supports Snowflake and BigQuery only, with additional platforms coming soon.

:::info What is a consumption query?
Consumption query is a metric of queries in your dbt project that has used the model in a given time. It filters down to `select` statements only to gauge model consumption and excludes dbt model build and test executions.
Expand All @@ -28,7 +28,23 @@ To access the features, you should meet the following:
1. You have a dbt Cloud account on the [Enterprise plan](https://www.getdbt.com/pricing/).
2. You have set up a [production](https://docs.getdbt.com/docs/deploy/deploy-environments#set-as-production-environment) deployment environment for each project you want to explore, with at least one successful job run.
3. You have [admin permissions](/docs/cloud/manage-access/enterprise-permissions) in dbt Cloud to edit project settings or production environment settings.
4. Use Snowflake or BigQuery as your data warehouse and can enable query history permissions or work with an admin to do so (Support for additional data platforms coming soon).
4. Use Snowflake or BigQuery as your data warehouse and can enable query history permissions or work with an admin to do so. Support for additional data platforms coming soon.

## Enable query history in dbt Cloud

To enable model query history in dbt Cloud, follow these steps:

1. Navigate to **Deploy** and then **Environments**.
2. Select the environment marked **PROD** and click **Settings**.
3. Click **Edit** and scroll to the **Query History** section to enable the query history toggle. When it’s green and to the right, it's enabled.
4. Click the **Test Permissions** button to validate the deployment credentials permissions are sufficient to support query history.

<DocCarousel slidesPerView={1}>

<Lightbox src="/img/docs/collaborate/dbt-explorer/enable-query-history.jpg" width="95%" title="Enable query history in your environment settings." />
<Lightbox src="/img/docs/collaborate/dbt-explorer/enable-query-history-success.jpg" width="95%" title="Example of permissions verified result after clicking Test Permissions." />

</DocCarousel>

## Credential permissions

Expand Down Expand Up @@ -61,25 +77,12 @@ The model query history feature uses the credentials in your production environm

</Expandable>

## Enable query history in dbt Cloud

:::info
During beta, the dbt Labs team will manually enable query history for your dbt Cloud projects. Later on, you’ll be able to do it yourself.

:::
<!--
1. Navigate to **Deploy** and then **Environments**.
2. Select the environment marked **PROD** and click **Settings**.
3. Enable the checkbox for query history in **General Settings**.
4. Click the **Test** button to validate the deployment credentials permissions are sufficient to support query history.
-->

## View query history in Explorer

To enhance your discovery, you can view your model query history in various locations within dbt Explorer:
- [View from Performance charts](#view-from-performance-charts)
* [View from Project lineage](#view-from-project-lineage)
- [View from Model list](#view-from-model-list)
- [View from Performance charts](#view-from-performance-charts)
* [View from Project lineage](#view-from-project-lineage)
- [View from Model list](#view-from-model-list)

### View from Performance charts

Expand All @@ -92,7 +95,7 @@ To enhance your discovery, you can view your model query history in various loca
4. Click on a model for more details and go to the **Performance** tab.
5. On the **Performance** tab, scroll down to the **Model performance** section.
6. Select the **Consumption queries** tab to view the consumption queries over a given time for that model.
<Lightbox src="/img/docs/collaborate/model-consumption-queries.jpg" width="85%" title="View consumption queries over time for a given model." />
<Lightbox src="/img/docs/collaborate/model-consumption-queries.jpg" width="90%" title="View consumption queries over time for a given model." />

### View from Project lineage

Expand Down
3 changes: 3 additions & 0 deletions website/docs/docs/dbt-versions/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ Release notes are grouped by month for both multi-tenant and virtual private clo

## September 2024

- **New**: [Data health tile](/docs/collaborate/data-tile) is now generally available in dbt Explorer. Data health tiles provide a quick at-a-glance view of your data quality, highlighting potential issues in your data. You can embed these tiles in your dashboards to quickly identify and address data quality issues in your dbt project.
- **New**: dbt Explorer's Model query history feature is now in Preview for dbt Cloud Enterprise customers. Model query history allows you to view the count of consumption queries for a model based on the data warehouse's query logs. This feature provides data teams insight, so they can focus their time and infrastructure spend on the worthwhile used data products. To learn more, refer to [Model query history](/docs/collaborate/model-query-history).
- **Enhancement**: You can now use [Extended Attributes](/docs/dbt-cloud-environments#extended-attributes) and [Environment Variables](/docs/build/environment-variables) when connecting to the Semantic Layer. If you set a value directly in the Semantic Layer Credentials, it will have a higher priority than Extended Attributes. When using environment variables, the default value for the environment will be used. If you're using exports, job environment variable overrides aren't supported yet, but they will be soon.
- **New:** There are two new [environment variable defaults](/docs/build/environment-variables#dbt-cloud-context) &mdash; `DBT_CLOUD_ENVIRONMENT_NAME` and `DBT_CLOUD_ENVIRONMENT_TYPE`.
- **New:** The [Amazon Athena warehouse connection](/docs/cloud/connect-data-platform/connect-amazon-athena) is available as a public preview for dbt Cloud accounts that have upgraded to [`versionless`](/docs/dbt-versions/versionless-cloud).

Expand Down
Loading

0 comments on commit d0ba5ed

Please sign in to comment.