Skip to content

Commit

Permalink
Merge branch 'current' into asmclean-patch-1
Browse files Browse the repository at this point in the history
  • Loading branch information
matthewshaver authored Sep 24, 2024
2 parents 3ae1fe3 + 4f3d679 commit a7b34a6
Show file tree
Hide file tree
Showing 44 changed files with 289 additions and 202 deletions.
15 changes: 9 additions & 6 deletions website/docs/docs/build/environment-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ While all environment variables are encrypted at rest in dbt Cloud, dbt Cloud ha

dbt Cloud has a number of pre-defined variables built in. Variables are set automatically and cannot be changed.

**dbt Cloud IDE details**
#### dbt Cloud IDE details

The following environment variable is set automatically for the dbt Cloud IDE:

Expand All @@ -109,13 +109,16 @@ The following environment variable is set automatically for the dbt Cloud IDE:

Use case — This is useful in cases where you want to dynamically use the Git branch name as a prefix for a [development schema](/docs/build/custom-schemas) ( `{{ env_var ('DBT_CLOUD_GIT_BRANCH') }}` ).

**dbt Cloud context**
#### dbt Cloud context

The following environment variables are set automatically for deployment runs:
The following environment variables are set automatically:

- `DBT_ENV`: This key is reserved for the dbt Cloud application and will always resolve to 'prod'
- `DBT_ENV`: This key is reserved for the dbt Cloud application and will always resolve to 'prod'. For deployment runs only.
- `DBT_CLOUD_ENVIRONMENT_NAME`: The name of the dbt Cloud environment in which `dbt` is running.
- `DBT_CLOUD_ENVIRONMENT_TYPE`: The type of dbt Cloud environment in which `dbt` is running. The valid values are `development` or `deployment`.

#### Run details

**Run details**
- `DBT_CLOUD_PROJECT_ID`: The ID of the dbt Cloud Project for this run
- `DBT_CLOUD_JOB_ID`: The ID of the dbt Cloud Job for this run
- `DBT_CLOUD_RUN_ID`: The ID of this particular run
Expand All @@ -124,7 +127,7 @@ The following environment variables are set automatically for deployment runs:
- `DBT_CLOUD_ENVIRONMENT_ID`: The ID of the environment for this run
- `DBT_CLOUD_ACCOUNT_ID`: The ID of the dbt Cloud account for this run

**Git details**
#### Git details

_The following variables are currently only available for GitHub, GitLab, and Azure DevOps PR builds triggered via a webhook_

Expand Down
27 changes: 23 additions & 4 deletions website/docs/docs/build/metricflow-time-spine.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,31 @@ sidebar_label: "MetricFlow time spine"
tags: [Metrics, Semantic Layer]
---

It's common in analytics engineering to have a date dimension or "time spine" table as a base table for different types of time-based joins and aggregations. The structure of this table is typically a base column of daily or hourly dates, with additional columns for other time grains, like fiscal quarter, defined based on the base column. You can join other tables to the time spine on the base column to calculate metrics like revenue at a point in time, or to aggregate to a specific time grain.
It's common in analytics engineering to have a date dimension or "time-spine" table as a base table for different types of time-based joins and aggregations. The structure of this table is typically a base column of daily or hourly dates, with additional columns for other time grains, like fiscal quarters, defined based on the base column. You can join other tables to the time spine on the base column to calculate metrics like revenue at a point in time, or to aggregate to a specific time grain.

MetricFlow requires you to define a time spine table as a project level configuration, which then is used for various time-based joins and aggregations, like cumulative metrics. At a minimum, you need to define a time spine table for a daily grain. You can optionally define a time spine table for a different granularity, like hourly.
MetricFlow requires you to define a time-spine table as a model-level configuration in the Semantic Layer for time-based joins and aggregations, such as cumulative metrics. This configuration informs dbt which model should be used for time range joins. It is especially useful for cumulative metrics or calculating time-based offsets. The time-spine model is joined to other tables when calculating certain types of metrics or dimensions. MetricFlow will join the time-spine model in the compiled SQL for the following types of metrics and dimensions:
- [Cumulative metrics](/docs/build/cumulative)
- [Metric offsets](/docs/build/derived#derived-metric-offset)
- [Conversion metrics](/docs/build/conversion)
- [Slowly Changing Dimensions](/docs/build/dimensions#scd-type-ii)
- [Metrics](/docs/build/metrics-overview) with the `join_to_timespine` configuration set to true
To see the generated SQL for the metric and dimension types that use time-spine joins, refer to the respective documentation or add the `compile=True` flag when querying the Semantic Layer to return the compiled SQL.

If you already have a date dimension or time spine table in your dbt project, you can point MetricFlow to this table by updating the `model` configuration to use this table in the Semantic Layer. For example, given the following directory structure, you can create two time spine configurations, `time_spine_hourly` and `time_spine_daily`.
#### Configuring time-spine
- You only need to configure time-spine models that the Semantic Layer should recognize.
- At a minimum, define a time-spine table for a daily grain.
- You can optionally define a time-spine table for a different granularity, like hourly.
- Note that if you don’t have a date or calendar model in your project, you'll need to create one.
- If you're looking to specify the grain of a time dimension so that MetricFlow can transform the underlying column to the required granularity, refer to the [Time granularity documentation](/docs/build/dimensions?dimension=time_gran)

If you already have a date dimension or time-spine table in your dbt project, you can point MetricFlow to this table by updating the `model` configuration to use this table in the Semantic Layer. This is a model-level configuration that tells dbt to use the model for time range joins in the Semantic Layer.

For example, given the following directory structure, you can create two time spine configurations, `time_spine_hourly` and `time_spine_daily`. MetricFlow supports granularities ranging from milliseconds to years. Refer to the [Dimensions page](/docs/build/dimensions?dimension=time_gran#time) (time_granularity tab) to find the full list of supported granularities.

:::tip
Previously, you were required to create a model called `metricflow_time_spine` in your dbt project. This is no longer required. However, you can build your time spine model from this table if you don't have another date dimension table you want to use in your project.
Previously, you had to create a model called `metricflow_time_spine` in your dbt project. Now, if your project already includes a date dimension or time spine table, you can simply configure MetricFlow to use that table by updating the `model` setting in the Semantic Layer.

If you don’t have a date dimension table, you can still create one by using the code snippet below to build your time spine model.

:::

Expand All @@ -38,6 +55,8 @@ models:
```
</File>
For an example project, refer to our [Jaffle shop](https://github.com/dbt-labs/jaffle-sl-template/blob/main/models/marts/_models.yml) example.
Now, break down the configuration above. It's pointing to a model called `time_spine_daily`. It sets the time spine configurations under the `time_spine` key. The `standard_granularity_column` is the lowest grain of the table, in this case, it's hourly. It needs to reference a column defined under the columns key, in this case, `date_hour`. Use the `standard_granularity_column` as the join key for the time spine table when joining tables in MetricFlow. Here, the granularity of the `standard_granularity_column` is set at the column level, in this case, `hour`.


Expand Down
5 changes: 2 additions & 3 deletions website/docs/docs/build/saved-queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,9 +165,8 @@ To enable saved queries at the project level, you can set the `saved-queries` co
```yaml
saved-queries:
my_saved_query:
config:
+cache:
enabled: true
+cache:
enabled: true
```
</File>

Expand Down
40 changes: 24 additions & 16 deletions website/docs/docs/cloud/configure-cloud-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,21 +52,29 @@ Once you install the dbt Cloud CLI, you need to configure it to connect to a dbt

The config file looks like this:

```yaml
version: "1"
context:
active-project: "<project id from the list below>"
active-host: "<active host from the list>"
defer-env-id: "<optional defer environment id>"
projects:
- project-id: "<project-id>"
account-host: "<account-host>"
api-key: "<user-api-key>"

- project-id: "<project-id>"
account-host: "<account-host>"
api-key: "<user-api-key>"
```
```yaml
version: "1"
context:
active-project: "<project id from the list below>"
active-host: "<active host from the list>"
defer-env-id: "<optional defer environment id>"
projects:
- project-name: "<project-name>"
project-id: "<project-id>"
account-name: "<account-name>"
account-id: "<account-id>"
account-host: "<account-host>" # for example, "cloud.getdbt.com"
token-name: "<pat-or-service-token-name>"
token-value: "<pat-or-service-token-value>"

- project-name: "<project-name>"
project-id: "<project-id>"
account-name: "<account-name>"
account-id: "<account-id>"
account-host: "<account-host>" # for example, "cloud.getdbt.com"
token-name: "<pat-or-service-token-name>"
token-value: "<pat-or-service-token-value>"
```
3. After downloading the config file and creating your directory, navigate to a dbt project in your terminal:
Expand Down Expand Up @@ -195,4 +203,4 @@ This command moves the `dbt_cloud.yml` from the `Downloads` folder to the `.dbt`
By default, [all artifacts](/reference/artifacts/dbt-artifacts) are downloaded when you execute dbt commands from the dbt Cloud CLI. To skip these files from being downloaded, add `--download-artifacts=false` to the command you want to run. This can help improve run-time performance but might break workflows that depend on assets like the [manifest](/reference/artifacts/manifest-json).


</Expandable>
</Expandable>
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@ Starting July 2024, connection management has moved from the project level to th

<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-legacy-model.png" width="55%" title="Previous connection model"/>

The following connection management section describes these changes.
Connections created with APIs before this change cannot be accessed with the [latest APIs](https://docs.getdbt.com/dbt-cloud/api-v3#/operations/List%20Account%20Connections). dbt Labs recommends [recreating the connections](https://docs.getdbt.com/dbt-cloud/api-v3#/operations/Create%20Account%20Connection) with the latest APIs.


:::

Expand Down
14 changes: 12 additions & 2 deletions website/docs/docs/cloud/git/setup-azure.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ To use our native integration with Azure DevOps in dbt Cloud, an account admin n
4. [Connect Azure DevOps to your new app](#connect-azure-devops-to-your-new-app).
5. [Add your Entra ID app to dbt Cloud](#add-your-azure-ad-app-to-dbt-cloud).

Once the Microsoft Entra ID app is added to dbt Cloud, an account admin must also [connect a service user](#connecting-a-service-user) via OAuth, which will be used to power headless actions in dbt Cloud such as deployment runs and CI.
Once the Microsoft Entra ID app is added to dbt Cloud, an account admin must also [connect a service user](/docs/cloud/git/setup-azure#connect-a-service-user) via OAuth, which will be used to power headless actions in dbt Cloud such as deployment runs and CI.


Once the Microsoft Entra ID app is added to dbt Cloud and the service user is connected, then dbt Cloud developers can personally authenticate in dbt Cloud from Azure DevOps. For more on this, see [Authenticate with Azure DevOps](/docs/cloud/git/authenticate-azure).
Expand Down Expand Up @@ -89,7 +89,7 @@ An Azure admin will need one of the following permissions in both the Microsoft
- Azure Service Administrator
- Azure Co-administrator

If your Azure DevOps account is connected to Entra ID, then you can proceed to [Connecting a service user](#connecting-a-service-user). However, if you're just getting set up, connect Azure DevOps to the Microsoft Entra ID app you just created:
If your Azure DevOps account is connected to Entra ID, then you can proceed to [Connect a service user](#connect-a-service-user). However, if you're just getting set up, connect Azure DevOps to the Microsoft Entra ID app you just created:

1. From your Azure DevOps account, select **Organization settings** in the bottom left.
2. Navigate to Microsoft Entra ID.
Expand Down Expand Up @@ -373,6 +373,16 @@ A dbt Cloud account admin with access to the service user's Azure DevOps account

Once connected, dbt Cloud displays the email address of the service user so you know which user's permissions are enabling headless actions in deployment environments. To change which account is connected, disconnect the profile in dbt Cloud, sign into the alternative Azure DevOps service account, and re-link the account in dbt Cloud.

### Using Azure AD for SSO with dbt Cloud and Microsoft tools

If you're using Azure AD for SSO with dbt Cloud and Microsoft tools, the SSO flow may sometimes direct your account admin to their personal user account instead of the service user. If this happens, follow these steps to resolve it:

1. Sign in to the service user's Azure DevOps account (ensure they are also connected to dbt Cloud through SSO).
2. When connected to dbt Cloud, sign out of Azure AD through the [Azure portal](https://portal.azure.com/).
3. Disconnect the service user in dbt Cloud, and follow the steps to set it up again.
4. You should then be prompted to enter service user credentials.


:::info Personal Access Tokens (PATs)
dbt Cloud leverages the service user to generate temporary access tokens called [PATs](https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?toc=%2Fazure%2Fdevops%2Fmarketplace-extensibility%2Ftoc.json&view=azure-devops&tabs=Windows).

Expand Down
Loading

0 comments on commit a7b34a6

Please sign in to comment.