Skip to content

Commit

Permalink
Merge branch 'current' into patch-1
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Jul 10, 2024
2 parents beb401b + 4197c55 commit 4caf9ba
Show file tree
Hide file tree
Showing 17 changed files with 83 additions and 126 deletions.
12 changes: 6 additions & 6 deletions website/docs/docs/build/cumulative-metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Note that we use the double colon (::) to indicate whether a parameter is nested
| `type_params::cumulative_type_params` | Allows you to add a `window`, `period_agg`, and `grain_to_date` configuration. Nested under `type_params`. | Optional |
| `cumulative_type_params::window` | The accumulation window, such as 1 month, 7 days, 1 year. This can't be used with `grain_to_date`. | Optional |
| `cumulative_type_params::grain_to_date` | Sets the accumulation grain, such as `month`, which will accumulate data for one month and then restart at the beginning of the next. This can't be used with `window`. | Optional |
| `cumulative_type_params::period_agg` | Specifies how to aggregate the cumulative metric when summarizing data to a different granularity. Can be used with grain_to_date. Options are <br /> - `first` (Takes the first value within the period) <br /> - `last` (Takes the last value within the period <br /> - `avg` (Calculates the average value within the period). <br /> <br /> Defaults to `first` if no `window` is specified. | Optional |
| `cumulative_type_params::period_agg` | Specifies how to aggregate the cumulative metric when summarizing data to a different granularity. Can be used with grain_to_date. Options are <br /> - `first` (Takes the first value within the period) <br /> - `last` (Takes the last value within the period <br /> - `average` (Calculates the average value within the period). <br /> <br /> Defaults to `first` if no `window` is specified. | Optional |
| `type_params::measure` | A dictionary describing the measure you will use. | Required |
| `measure::name` | The measure you are referencing. | Optional |
| `measure::fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional |
Expand All @@ -45,7 +45,7 @@ metrics:
label: The value that will be displayed in downstream tools # Required
type_params: # Required
cumulative_type_params:
period_agg: first # Optional. Defaults to first. Accepted values: first|last|avg
period_agg: first # Optional. Defaults to first. Accepted values: first|last|average
window: The accumulation window, such as 1 month, 7 days, 1 year. # Optional. It cannot be used with grain_to_date.
grain_to_date: Sets the accumulation grain, such as month will accumulate data for one month, then restart at the beginning of the next. # Optional. It cannot be used with window.
measure:
Expand Down Expand Up @@ -105,7 +105,7 @@ metrics:
### Granularity options
Use the `period_agg` parameter with `first()`, `last()`, and `avg()` functions to aggregate cumulative metrics over the requested period. This is because granularity options for cumulative metrics are different than the options for other metric types.
Use the `period_agg` parameter with `first()`, `last()`, and `average()` functions to aggregate cumulative metrics over the requested period. This is because granularity options for cumulative metrics are different than the options for other metric types.
- For other metrics, we use the `date_trunc` function to implement granularity.
- However, cumulative metrics are non-additive (values can't be added up), so we can't use the `date_trunc` function to change their time grain granularity.
- By default, we take the first value of the period. You can change this by specifying a different function using the `period_agg` parameter.
Expand All @@ -122,7 +122,7 @@ In the following example, we define a cumulative metric, `cumulative_revenue`, t
type_params:
measure: revenue
cumulative_type_params:
period_agg: first # Optional. Defaults to first. Accepted values: first|end|avg
period_agg: first # Optional. Defaults to first. Accepted values: first|end|average
```
</File>

Expand All @@ -131,7 +131,7 @@ In this example, `period_agg` is set to `first`, which chooses the first value f

<Expandable alt_header="Expand toggle to view how the SQL compiles">

Note the use of the `window` function to select the `first` value. For `last` and `avg`, we would replace the `first_value()` function in the generated SQL with `last_value()` and `average` respectively.
Note the use of the `window` function to select the `first` value. For `last` and `average`, we would replace the `first_value()` function in the generated SQL with `last_value()` and `average` respectively.

```sql
-- re-aggregate metric via the group by
Expand Down Expand Up @@ -328,7 +328,7 @@ metrics:
measure: order_total
cumulative_type_params:
grain_to_date: month # Resets at the beginning of each month
period_agg: first # Optional. Defaults to first. Accepted values: first|last|avg
period_agg: first # Optional. Defaults to first. Accepted values: first|last|average
```
</File>

Expand Down
6 changes: 3 additions & 3 deletions website/docs/docs/build/packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "Packages"
id: "packages"
description: "Discover how dbt packages help modularize code and transform data efficiently. Learn about git packages, hub packages, private packages, and advanced package configurations."
keywords: [dbt packages, dbt package, dbt private package, dbt data transformation, dbt libraries, how to add a package dbt project]
keywords: [dbt package, private package, dbt private package, dbt data transformation, dbt clone, add dbt package]
---


Expand Down Expand Up @@ -190,11 +190,11 @@ packages:
If you're using dbt Cloud, the SSH key method will not work, but you can use the [HTTPS Git Token Method](https://docs.getdbt.com/docs/build/packages#git-token-method).


#### Git Token Method
#### Git token method
This method allows the user to clone via HTTPS by passing in a git token via an environment variable. Be careful of the expiration date of any token you use, as an expired token could cause a scheduled run to fail. Additionally, user tokens can create a challenge if the user ever loses access to a specific repo.


:::info dbt Cloud Usage
:::info dbt Cloud usage
If you are using dbt Cloud, you must adhere to the naming conventions for environment variables. Environment variables in dbt Cloud must be prefixed with either `DBT_` or <VersionBlock lastVersion="1.5">`DBT_ENV_SECRET_`</VersionBlock><VersionBlock firstVersion="1.6">`DBT_ENV_SECRET`</VersionBlock>. Environment variables keys are uppercased and case sensitive. When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must match exactly the variable defined in dbt Cloud's UI.
:::

Expand Down
12 changes: 8 additions & 4 deletions website/docs/docs/cloud/billing.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,13 +89,17 @@ Viewing usage in the product is restricted to specific roles:

For an account-level view of usage, if you have access to the **Billing** and **Usage** pages, you can see an estimate of the usage for the month. In the Billing page of the **Account Settings**, you can see how your account tracks against its usage. You can also see which projects are building the most models.

<Lightbox src="/img/docs/building-a-dbt-project/billing.jpg" width="80%" title="To view account-level estimated usage, go to 'Account settings' and then select 'Billing'."/>
<Lightbox src="/img/docs/building-a-dbt-project/billing-usage-page.jpg" width="80%" title="To view account-level estimated usage, go to 'Account settings' and then select 'Billing'."/>

As a Team and Developer plan user, you can see how the account is tracking against the included models built. As an Enterprise plan user, you can see how much you have drawn down from your annual commit and how much remains.
As a Team and Developer plan user, you can see how the account is tracking against the included models built. As an Enterprise plan user, you can see how much you have drawn down from your annual commit and how much remains.

On each Project Home page, any user with access to that project can see how many models are built each month. From there, additional details on top jobs by models built can be found on each Environment page.
On each **Project Home** page, any user with access to that project can see how many models are built each month. From there, additional details on top jobs by models built can be found on each **Environment** page.

In addition, you can look at the Job Details page's Insights tab to show how many models are being built per month for that particular job and which models are taking the longest to build.
<Lightbox src="/img/docs/building-a-dbt-project/billing-project-page.jpg" width="80%" title="Your Project home page displays how many models are built each month."/>

In addition, you can look at the **Job Details** page's **Insights** tab to show how many models are being built per month for that particular job and which models are taking the longest to build.

<Lightbox src="/img/docs/building-a-dbt-project/billing-job-page.jpg" width="80%" title="View how many models are being built per month for a particular job by going to the 'Insights' tab in the 'Job details' page."/>

Usage information is available to customers on consumption-based plans, and some usage visualizations might not be visible to customers on legacy plans. Any usage data shown in dbt Cloud is only an estimate of your usage, and there could be a delay in showing usage data in the product. Your final usage for the month will be visible on your monthly statements (statements applicable to Team and Enterprise plans).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ dbt Cloud can connect with a variety of data platform providers including:
- [AlloyDB](/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb)
- [Amazon Redshift](/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb)
- [Apache Spark](/docs/cloud/connect-data-platform/connect-apache-spark)
- [Azure Synapse Analytics](/docs/cloud/connect-data-platform/connect-azure-synapse-analytics) <Lifecycle status="preview" />
- [Azure Synapse Analytics](/docs/cloud/connect-data-platform/connect-azure-synapse-analytics)
- [Databricks](/docs/cloud/connect-data-platform/connect-databricks)
- [Google BigQuery](/docs/cloud/connect-data-platform/connect-bigquery)
- [Microsoft Fabric](/docs/cloud/connect-data-platform/connect-microsoft-fabric)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: "Configure Azure Synapse Analytics connection."
sidebar_label: "Connect Azure Synapse Analytics"
---

# Connect Azure Synapse Analytics <Lifecycle status="preview" />
# Connect Azure Synapse Analytics

## Supported authentication methods
The supported authentication methods are:
Expand Down
11 changes: 8 additions & 3 deletions website/docs/docs/cloud/git/setup-azure.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ A Microsoft Entra ID admin needs to add another redirect URI to your Entra ID ap

2. Select the link next to **Redirect URIs**
3. Click **Add URI** and add the URI, replacing `YOUR_ACCESS_URL` with the [appropriate Access URL](/docs/cloud/about-cloud/access-regions-ip-addresses) for your region and plan:
`https://YOUR_ACCESS_URL/complete/microsoft_entra_id_service_user`
`https://YOUR_ACCESS_URL/complete/azure_active_directory_service_user`
4. Click **Save**.

<Lightbox src="/img/docs/dbt-cloud/connecting-azure-devops/redirect-uri.gif" title="Adding the Service User redirect URI"/>
Expand Down Expand Up @@ -374,7 +374,12 @@ A dbt Cloud account admin with access to the service user's Azure DevOps account
Once connected, dbt Cloud displays the email address of the service user so you know which user's permissions are enabling headless actions in deployment environments. To change which account is connected, disconnect the profile in dbt Cloud, sign into the alternative Azure DevOps service account, and re-link the account in dbt Cloud.

:::info Personal Access Tokens (PATs)
dbt Cloud generates temporary access tokens called Full-scoped PATs for service users to access APIs related to their dbt Cloud project. These tokens are only valid for a short period of 5 minutes and become invalid after they are used to make an API call.
dbt Cloud leverages the service user to generate temporary access tokens called [PATs](https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?toc=%2Fazure%2Fdevops%2Fmarketplace-extensibility%2Ftoc.json&view=azure-devops&tabs=Windows).

The Azure DevOps Administrator can limit the creation of full-scoped PATs by enabling a policy that restricts users to a custom-defined set of scopes. By default, this policy is set to **off**, but enabling it will cause project setup to fail with an error. After disabling this policy and successfully setting up your project, if you wish to use finer-scoped permissions, some features such as webhooks for CI may be lost, so we recommend the service user has full-scoped PATs. To exclude the dbt Cloud service user from the global PAT policy, add them to the allow list as part of your security policy.
These tokens are limited in scope, are only valid for 5 minutes, and become invalid after a single API call.

These tokens are limited to the following [scopes](https://learn.microsoft.com/en-us/azure/devops/integrate/get-started/authentication/oauth?view=azure-devops):
- `vso.code_full`: Grants full access to source code and version control metadata (commits, branches, and so on). Also grants the ability to create and manage code repositories, create and manage pull requests and code reviews, and receive notifications about version control events with service hooks. Also includes limited support for Client OM APIs.
- `vso.project`: Grants the ability to read projects and teams.
- `vso.build_execute`: Grants the ability to access build artifacts, including build results, definitions, and requests, and the ability to queue a build, update build properties, and the ability to receive notifications about build events with service hooks.
:::
24 changes: 11 additions & 13 deletions website/docs/docs/collaborate/govern/project-dependencies.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,21 @@ keyword: dbt mesh, project dependencies, ref, cross project ref, project depende

For a long time, dbt has supported code reuse and extension by installing other projects as [packages](/docs/build/packages). When you install another project as a package, you are pulling in its full source code, and adding it to your own. This enables you to call macros and run models defined in that other project.

While this is a great way to reuse code, share utility macros, and establish a starting point for common transformations, it's not a great way to enable collaboration across teams and at scale, especially at larger organizations.
While this is a great way to reuse code, share utility macros, and establish a starting point for common transformations, it's not a great way to enable collaboration across teams and at scale, especially in larger organizations.

This year, dbt Labs is introducing an expanded notion of `dependencies` across multiple dbt projects:
- **Packages** &mdash; Familiar and pre-existing type of dependency. You take this dependency by installing the package's full source code (like a software library).
- **Projects** &mdash; A _new_ way to take a dependency on another project. Using a metadata service that runs behind the scenes, dbt Cloud resolves references on-the-fly to public models defined in other projects. You don't need to parse or run those upstream models yourself. Instead, you treat your dependency on those models as an API that returns a dataset. The maintainer of the public model is responsible for guaranteeing its quality and stability.

## Prerequisites
- Available in [dbt Cloud Enterprise](https://www.getdbt.com/pricing). If you have an Enterprise account, you can unlock these features by designating a [public model](/docs/collaborate/govern/model-access) and adding a [cross-project ref](#how-to-write-cross-project-ref).
- Set your development and deployment [environments](/docs/dbt-cloud-environments) to use [dbt version](/docs/dbt-versions/core) 1.6 or later. You can also opt [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) to always use the latest version of dbt.
- Available in [dbt Cloud Enterprise](https://www.getdbt.com/pricing). If you have an Enterprise account, you can unlock these features by designating a [public model](/docs/collaborate/govern/model-access) and adding a [cross-project ref](#how-to-write-cross-project-ref). <Lifecycle status="enterprise"/>
- Use a supported version of dbt (v1.6, v1.7, or go versionless with "[Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version)") for both the upstream ("producer") project and the downstream ("consumer") project.
- Define models in an upstream ("producer") project that are configured with [`access: public`](/reference/resource-configs/access). You need at least one successful job run after defining their `access`.
- Define a deployment environment in the upstream ("producer") project [that is set to be your Production environment](/docs/deploy/deploy-environments#set-as-production-environment), and ensure it has at least one successful job run in that environment.
- Each project `name` must be unique in your dbt Cloud account. For example, if you have a dbt project (codebase) for the `jaffle_marketing` team, you should not create separate projects for `Jaffle Marketing - Dev` and `Jaffle Marketing - Prod`. That isolation should instead be handled at the environment level.
- We are adding support for environment-level permissions and data warehouse connections; please contact your dbt Labs account team for beta access.
- The `dbt_project.yml` file is case-sensitive, which means the project name must exactly match the name in your `dependencies.yml`. For example, if your project name is `jaffle_marketing`, you should use `jaffle_marketing` (not `JAFFLE_MARKETING`) in all related files.


import UseCaseInfo from '/snippets/_packages_or_dependencies.md';

Expand All @@ -27,14 +33,6 @@ import UseCaseInfo from '/snippets/_packages_or_dependencies.md';
Refer to the [FAQs](#faqs) for more info.


## Prerequisites

In order to add project dependencies and resolve cross-project `ref`, you must:
- Use a supported version of dbt (v1.6, v1.7, or go versionless with "Keep on latest version") for both the upstream ("producer") project and the downstream ("consumer") project.
- Define models in an upstream ("producer") project that are configured with [`access: public`](/reference/resource-configs/access). You need at least one successful job run after defining their `access`.
- Define a deployment environment in the upstream ("producer") project [that is set to be your Production environment](/docs/deploy/deploy-environments#set-as-production-environment), and ensure it has at least one successful job run in that environment.
- Each project `name` must be unique in your dbt Cloud account. For example, if you have a dbt project (codebase) for the `jaffle_marketing` team, you should not create separate projects for `Jaffle Marketing - Dev` and `Jaffle Marketing - Prod`. That isolation should instead be handled at the environment level. To that end, we are working on adding support for environment-level permissions and data warehouse connections; reach out to your dbt Labs account team for beta access in May/June 2024.

## Example

As an example, let's say you work on the Marketing team at the Jaffle Shop. The name of your team's project is `jaffle_marketing`:
Expand All @@ -48,7 +46,7 @@ name: jaffle_marketing
</File>
As part of your modeling of marketing data, you need to take a dependency on two other projects:
- `dbt_utils` as a [package](#packages-use-case): A collection of utility macros that you can use while writing the SQL for your own models. This package is, open-source public, and maintained by dbt Labs.
- `dbt_utils` as a [package](#packages-use-case): A collection of utility macros you can use while writing the SQL for your own models. This package is open-source public and maintained by dbt Labs.
- `jaffle_finance` as a [project use-case](#projects-use-case): Data models about the Jaffle Shop's revenue. This project is private and maintained by your colleagues on the Finance team. You want to select from some of this project's final models, as a starting point for your own work.

<File name="dependencies.yml">
Expand All @@ -59,7 +57,7 @@ packages:
version: 1.1.1
projects:
- name: jaffle_finance # matches the 'name' in their 'dbt_project.yml'
- name: jaffle_finance # case sensitive and matches the 'name' in the 'dbt_project.yml'
```

</File>
Expand Down
Loading

0 comments on commit 4caf9ba

Please sign in to comment.