Skip to content

Commit

Permalink
Merge branch 'current' into clone-ci
Browse files Browse the repository at this point in the history
  • Loading branch information
matthewshaver authored Nov 30, 2023
2 parents 79eec08 + b5c27c9 commit e0e2562
Show file tree
Hide file tree
Showing 16 changed files with 106 additions and 55 deletions.
5 changes: 1 addition & 4 deletions website/docs/docs/build/cumulative-metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,7 @@ metrics:

## Limitations
Cumulative metrics are currently under active development and have the following limitations:

1. You can only use the [`metric_time` dimension](/docs/build/dimensions#time) to check cumulative metrics. If you don't use `metric_time` in the query, the cumulative metric will return incorrect results because it won't perform the time spine join. This means you cannot reference time dimensions other than the `metric_time` in the query.
2. If you use `metric_time` in your query filter but don't include "start_time" and "end_time," cumulative metrics will left-censor the input data. For example, if you query a cumulative metric with a 7-day window with the filter `{{ TimeDimension('metric_time') }} BETWEEN '2023-08-15' AND '2023-08-30' `, the values for `2023-08-15` to `2023-08-20` return missing or incomplete data. This is because we apply the `metric_time` filter to the aggregation input. To avoid this, you must use `start_time` and `end_time` in the query filter.

- You are required to use [`metric_time` dimension](/docs/build/dimensions#time) when querying cumulative metrics. If you don't use `metric_time` in the query, the cumulative metric will return incorrect results because it won't perform the time spine join. This means you cannot reference time dimensions other than the `metric_time` in the query.

## Cumulative metrics example

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,15 @@ title: "About data platform connections"
id: about-connections
description: "Information about data platform connections"
sidebar_label: "About data platform connections"
pagination_next: "docs/cloud/connect-data-platform/connect-starburst-trino"
pagination_next: "docs/cloud/connect-data-platform/connect-microsoft-fabric"
pagination_prev: null
---
dbt Cloud can connect with a variety of data platform providers including:
- [Amazon Redshift](/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb)
- [Apache Spark](/docs/cloud/connect-data-platform/connect-apache-spark)
- [Databricks](/docs/cloud/connect-data-platform/connect-databricks)
- [Google BigQuery](/docs/cloud/connect-data-platform/connect-bigquery)
- [Microsoft Fabric](/docs/cloud/connect-data-platform/connect-microsoft-fabric)
- [PostgreSQL](/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb)
- [Snowflake](/docs/cloud/connect-data-platform/connect-snowflake)
- [Starburst or Trino](/docs/cloud/connect-data-platform/connect-starburst-trino)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
---
title: "Connect Microsoft Fabric"
description: "Configure Microsoft Fabric connection."
sidebar_label: "Connect Microsoft Fabric"
---

## Supported authentication methods
The supported authentication methods are:
- Azure Active Directory (Azure AD) service principal
- Azure AD password

SQL password (LDAP) is not supported in Microsoft Fabric Synapse Data Warehouse so you must use Azure AD. This means that to use [Microsoft Fabric](https://www.microsoft.com/en-us/microsoft-fabric) in dbt Cloud, you will need at least one Azure AD service principal to connect dbt Cloud to Fabric, ideally one service principal for each user.

### Active Directory service principal
The following are the required fields for setting up a connection with a Microsoft Fabric using Azure AD service principal authentication.

| Field | Description |
| --- | --- |
| **Server** | The service principal's **host** value for the Fabric test endpoint. |
| **Port** | The port to connect to Microsoft Fabric. You can use `1433` (the default), which is the standard SQL server port number. |
| **Database** | The service principal's **database** value for the Fabric test endpoint. |
| **Authentication** | Choose **Service Principal** from the dropdown. |
| **Tenant ID** | The service principal's **Directory (tenant) ID**. |
| **Client ID** | The service principal's **application (client) ID id**. |
| **Client secret** | The service principal's **client secret** (not the **client secret id**). |


### Active Directory password

The following are the required fields for setting up a connection with a Microsoft Fabric using Azure AD password authentication.

| Field | Description |
| --- | --- |
| **Server** | The server hostname to connect to Microsoft Fabric. |
| **Port** | The server port. You can use `1433` (the default), which is the standard SQL server port number. |
| **Database** | The database name. |
| **Authentication** | Choose **Active Directory Password** from the dropdown. |
| **User** | The AD username. |
| **Password** | The AD username's password. |

## Configuration

To learn how to optimize performance with data platform-specific configurations in dbt Cloud, refer to [Microsoft Fabric DWH configurations](/reference/resource-configs/fabric-configs).
37 changes: 0 additions & 37 deletions website/docs/docs/cloud/dbt-cloud-ide/dbt-cloud-ide.md

This file was deleted.

4 changes: 2 additions & 2 deletions website/docs/docs/core/about-core-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ dbt Core is an [open-source](https://github.com/dbt-labs/dbt-core) tool that ena
- [Connecting to a data platform](/docs/core/connect-data-platform/profiles.yml)
- [How to run your dbt projects](/docs/running-a-dbt-project/run-your-dbt-projects)

To learn about developing dbt projects in dbt Cloud or dbt Core, refer to [Develop dbt](/docs/cloud/about-develop-dbt).
- **Note** — dbt Cloud provides a command line interface with the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation). Both the open-sourced dbt Core and the dbt Cloud CLI are command line tools that let you run dbt commands. The key distinction is the dbt Cloud CLI is tailored for dbt Cloud's infrastructure and integrates with all its [features](/docs/cloud/about-cloud/dbt-cloud-features).
To learn about developing dbt projects in dbt Cloud, refer to [Develop with dbt Cloud](/docs/cloud/about-develop-dbt).
- dbt Cloud provides a command line interface with the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation). Both dbt Core and the dbt Cloud CLI are command line tools that let you run dbt commands. The key distinction is the dbt Cloud CLI is tailored for dbt Cloud's infrastructure and integrates with all its [features](/docs/cloud/about-cloud/dbt-cloud-features).

If you need a more detailed first-time setup guide for specific data platforms, read our [quickstart guides](https://docs.getdbt.com/guides).
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ dbt Core can connect with a variety of data platform providers including:
- [Apache Spark](/docs/core/connect-data-platform/spark-setup)
- [Databricks](/docs/core/connect-data-platform/databricks-setup)
- [Google BigQuery](/docs/core/connect-data-platform/bigquery-setup)
- [Microsoft Fabric](/docs/core/connect-data-platform/fabric-setup)
- [PostgreSQL](/docs/core/connect-data-platform/postgres-setup)
- [Snowflake](/docs/core/connect-data-platform/snowflake-setup)
- [Starburst or Trino](/docs/core/connect-data-platform/trino-setup)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ meta:
github_repo: 'Microsoft/dbt-fabric'
pypi_package: 'dbt-fabric'
min_core_version: '1.4.0'
cloud_support: Not Supported
cloud_support: Supported
platform_name: 'Microsoft Fabric'
config_page: '/reference/resource-configs/fabric-configs'
---
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@ date: 2023-11-28

Public Preview is now available in dbt Cloud for Microsoft Fabric!

To learn more, check out the [Quickstart for dbt Cloud and Microsoft Fabric](/guides/microsoft-fabric?step=1). The guide walks you through:
To learn more, refer to [Connect Microsoft Fabric](/docs/cloud/connect-data-platform/connect-microsoft-fabric) and [Microsoft Fabric DWH configurations](/reference/resource-configs/fabric-configs).

Also, check out the [Quickstart for dbt Cloud and Microsoft Fabric](/guides/microsoft-fabric?step=1). The guide walks you through:

- Loading the Jaffle Shop sample data (provided by dbt Labs) into your Microsoft Fabric warehouse.
- Connecting dbt Cloud to Microsoft Fabric.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
title: "New: Support for Git repository caching"
description: "November 2023: dbt Cloud can cache your project's code (as well as other dbt packages) to ensure runs can begin despite an upstream Git provider's outage."
sidebar_label: "New: Support for Git repository caching"
sidebar_position: 07
tags: [Nov-2023]
date: 2023-11-29
---

Now available for dbt Cloud Enterprise plans is a new option to enable Git repository caching for your job runs. When enabled, dbt Cloud caches your dbt project's Git repository and uses the cached copy instead if there's an outage with the Git provider. This feature improves the reliability and stability of your job runs.

To learn more, refer to [Repo caching](/docs/deploy/deploy-environments#git-repository-caching).

<Lightbox src="/img/docs/deploy/example-repo-caching.png" width="85%" title="Example of the Repository caching option" />
17 changes: 10 additions & 7 deletions website/docs/guides/microsoft-fabric-qs.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ recently_updated: true
---
## Introduction

In this quickstart guide, you'll learn how to use dbt Cloud with Microsoft Fabric. It will show you how to:
In this quickstart guide, you'll learn how to use dbt Cloud with [Microsoft Fabric](https://www.microsoft.com/en-us/microsoft-fabric). It will show you how to:

- Load the Jaffle Shop sample data (provided by dbt Labs) into your Microsoft Fabric warehouse.
- Connect dbt Cloud to Microsoft Fabric.
Expand All @@ -27,7 +27,7 @@ A public preview of Microsoft Fabric in dbt Cloud is now available!
### Prerequisites
- You have a [dbt Cloud](https://www.getdbt.com/signup/) account.
- You have started the Microsoft Fabric (Preview) trial. For details, refer to [Microsoft Fabric (Preview) trial](https://learn.microsoft.com/en-us/fabric/get-started/fabric-trial) in the Microsoft docs.
- As a Microsoft admin, you’ve enabled service principal authentication. For details, refer to [Enable service principal authentication](https://learn.microsoft.com/en-us/fabric/admin/metadata-scanning-enable-read-only-apis) in the Microsoft docs. dbt Cloud needs these authentication credentials to connect to Microsoft Fabric.
- As a Microsoft admin, you’ve enabled service principal authentication. You must add the service principal to the Microsoft Fabric workspace with either a Member (recommended) or Admin permission set. For details, refer to [Enable service principal authentication](https://learn.microsoft.com/en-us/fabric/admin/metadata-scanning-enable-read-only-apis) in the Microsoft docs. dbt Cloud needs these authentication credentials to connect to Microsoft Fabric.

### Related content
- [dbt Courses](https://courses.getdbt.com/collections)
Expand All @@ -54,8 +54,8 @@ A public preview of Microsoft Fabric in dbt Cloud is now available!
CREATE TABLE dbo.customers
(
[ID] [int],
[FIRST_NAME] [varchar] (8000),
[LAST_NAME] [varchar] (8000)
\[FIRST_NAME] [varchar](8000),
\[LAST_NAME] [varchar](8000)
);

COPY INTO [dbo].[customers]
Expand All @@ -72,7 +72,7 @@ A public preview of Microsoft Fabric in dbt Cloud is now available!
[USER_ID] [int],
-- [ORDER_DATE] [int],
[ORDER_DATE] [date],
[STATUS] [varchar] (8000)
\[STATUS] [varchar](8000)
);

COPY INTO [dbo].[orders]
Expand All @@ -87,8 +87,8 @@ A public preview of Microsoft Fabric in dbt Cloud is now available!
(
[ID] [int],
[ORDERID] [int],
[PAYMENTMETHOD] [varchar] (8000),
[STATUS] [varchar] (8000),
\[PAYMENTMETHOD] [varchar](8000),
\[STATUS] [varchar](8000),
[AMOUNT] [int],
[CREATED] [date]
);
Expand All @@ -108,6 +108,9 @@ A public preview of Microsoft Fabric in dbt Cloud is now available!
2. Enter a project name and click **Continue**.
3. Choose **Fabric** as your connection and click **Next**.
4. In the **Configure your environment** section, enter the **Settings** for your new project:
- **Server** &mdash; Use the service principal's **host** value for the Fabric test endpoint.
- **Port** &mdash; 1433 (which is the default).
- **Database** &mdash; Use the service principal's **database** value for the Fabric test endpoint.
5. Enter the **Development credentials** for your new project:
- **Authentication** &mdash; Choose **Service Principal** from the dropdown.
- **Tenant ID** &mdash; Use the service principal’s **Directory (tenant) id** as the value.
Expand Down
4 changes: 4 additions & 0 deletions website/docs/reference/resource-configs/databricks-configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,10 @@ insert into table analytics.databricks_incremental

### The `insert_overwrite` strategy

:::caution
This strategy is currently only compatible with All Purpose Clusters, not SQL Warehouses.
:::

This strategy is most effective when specified alongside a `partition_by` clause in your model config. dbt will run an [atomic `insert overwrite` statement](https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-dml-insert-overwrite-table.html) that dynamically replaces all partitions included in your query. Be sure to re-select _all_ of the relevant data for a partition when using this incremental strategy.

If no `partition_by` is specified, then the `insert_overwrite` strategy will atomically replace all contents of the table, overriding all existing data with only the new records. The column schema of the table remains the same, however. This can be desirable in some limited circumstances, since it minimizes downtime while the table contents are overwritten. The operation is comparable to running `truncate` + `insert` on other databases. For atomic replacement of Delta-formatted tables, use the `table` materialization (which runs `create or replace`) instead.
Expand Down
1 change: 1 addition & 0 deletions website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ const sidebarSettings = {
link: { type: "doc", id: "docs/cloud/connect-data-platform/about-connections" },
items: [
"docs/cloud/connect-data-platform/about-connections",
"docs/cloud/connect-data-platform/connect-microsoft-fabric",
"docs/cloud/connect-data-platform/connect-starburst-trino",
"docs/cloud/connect-data-platform/connect-snowflake",
"docs/cloud/connect-data-platform/connect-bigquery",
Expand Down
2 changes: 1 addition & 1 deletion website/snippets/_adapters-verified.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@

<Card
title="Microsoft Fabric"
body="<a href='/guides/microsoft-fabric?step=1'><img src='/img/icons/dbt-bit.svg' width='7%'/>Set up in dbt Cloud </a> <br /><a href='/docs/core/connect-data-platform/fabric-setup'><img src='/img/icons/dbt-bit.svg' width='7%'/>Install with dbt Core </a> <br /><br /> <a href=https://badge.fury.io/py/dbt-fabric><img src=https://badge.fury.io/py/dbt-fabric.svg/></a>"
body="<a href='/docs/cloud/connect-data-platform/connect-microsoft-fabric'><img src='/img/icons/dbt-bit.svg' width='7%'/>Set up in dbt Cloud </a> <br /><a href='/docs/core/connect-data-platform/fabric-setup'><img src='/img/icons/dbt-bit.svg' width='7%'/>Install with dbt Core </a> <br /><br /> <a href=https://badge.fury.io/py/dbt-fabric><img src=https://badge.fury.io/py/dbt-fabric.svg/></a>"
icon="fabric"/>
<Card
title="Azure Synapse"
Expand Down
18 changes: 18 additions & 0 deletions website/snippets/_cloud-environments-info.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,24 @@ Both development and deployment environments have a section called **General Set
- If you select a current version with `(latest)` in the name, your environment will automatically install the latest stable version of the minor version selected.
:::

### Git repository caching

At the start of every job run, dbt Cloud clones the project's Git repository so it has the latest versions of your project's code and runs `dbt deps` to install your dependencies.

For improved reliability and performance on your job runs, you can enable dbt Cloud to keep a cache of the project's Git repository. So, if there's a third-party outage that causes the cloning operation to fail, dbt Cloud will instead use the cached copy of the repo so your jobs can continue running as scheduled.

dbt Cloud caches your project's Git repo after each successful run and retains it for 8 days if there are no repo updates. It caches all packages regardless of installation method and does not fetch code outside of the job runs.

To enable Git repository caching, select **Account settings** from the gear menu and enable the **Repository caching** option.

<Lightbox src="/img/docs/deploy/example-repo-caching.png" width="85%" title="Example of the Repository caching option" />

:::note

This feature is only available on the dbt Cloud Enterprise plan.

:::

### Custom branch behavior

By default, all environments will use the default branch in your repository (usually the `main` branch) when accessing your dbt code. This is overridable within each dbt Cloud Environment using the **Default to a custom branch** option. This setting have will have slightly different behavior depending on the environment type:
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 5 additions & 1 deletion website/vercel.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,15 @@
"cleanUrls": true,
"trailingSlash": false,
"redirects": [
{
"source": "/docs/cloud/dbt-cloud-ide",
"destination": "/docs/cloud/dbt-cloud-ide/develop-in-the-cloud",
"permanent": true
},
{
"source": "/docs/core/installation",
"destination": "/docs/core/installation-overview",
"permanent": true

},
{
"source": "/docs/cloud/about-cloud-develop",
Expand Down

0 comments on commit e0e2562

Please sign in to comment.