Skip to content

Commit

Permalink
Merge branch 'current' into mwong-sl-architecture-page
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Dec 4, 2023
2 parents 522e1ac + 18acc6f commit acb939b
Show file tree
Hide file tree
Showing 30 changed files with 258 additions and 5 deletions.
2 changes: 2 additions & 0 deletions website/docs/docs/cloud/billing.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,8 @@ All included successful models built numbers above reflect our most current pric

As an Enterprise customer, you pay annually via invoice, monthly in arrears for additional usage (if applicable), and may benefit from negotiated usage rates. Please refer to your order form or contract for your specific pricing details, or [contact the account team](https://www.getdbt.com/contact-demo) with any questions.

Enterprise plan billing information is not available in the dbt Cloud UI. Changes are handled through your dbt Labs Solutions Architect or account team manager.

### Legacy plans

Customers who purchased the dbt Cloud Team plan before August 11, 2023, remain on a legacy pricing plan as long as your account is in good standing. The legacy pricing plan is based on seats and includes unlimited models, subject to reasonable use.
Expand Down
82 changes: 82 additions & 0 deletions website/docs/docs/cloud/secure/vcs-privatelink.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
---
title: "Configuring PrivateLink for self-hosted cloud version control systems (VCS)"
id: vcs-privatelink
description: "Setting up a PrivateLink connection between dbt Cloud and an organization’s cloud hosted git server"
sidebar_label: "PrivateLink for VCS"
---

import SetUpPages from '/snippets/_available-tiers-privatelink.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

AWS PrivateLink provides private connectivity from dbt Cloud to your self-hosted cloud version control system (VCS) service by routing requests through your virtual private cloud (VPC). This type of connection does not require you to publicly expose an endpoint to your VCS repositories or for requests to the service to traverse the public internet, ensuring the most secure connection possible. AWS recommends PrivateLink connectivity as part of its [Well-Architected Framework](https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html) and details this particular pattern in the **Shared Services** section of the [AWS PrivateLink whitepaper](https://docs.aws.amazon.com/pdfs/whitepapers/latest/aws-privatelink/aws-privatelink.pdf).

You will learn, at a high level, the resources necessary to implement this solution. Cloud environments and provisioning processes vary greatly, so information from this guide may need to be adapted to fit your requirements.

## PrivateLink connection overview

<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/privatelink-vcs-architecture.png" width="80%" title="High level overview of the dbt Cloud and AWS PrivateLink for VCS architecture" />

### Required resources for creating a connection

Creating an Interface VPC PrivateLink connection requires creating multiple AWS resources in your AWS account(s) and private network containing the self-hosted VCS instance. You are responsible for provisioning and maintaining these resources. Once provisioned, connection information and permissions are shared with dbt Labs to complete the connection, allowing for direct VPC to VPC private connectivity.

This approach is distinct from and does not require you to implement VPC peering between your AWS account(s) and dbt Cloud.

You need these resource to create a PrivateLink connection, which allows the dbt Cloud application to connect to your self-hosted cloud VCS. These resources can be created via the AWS Console, AWS CLI, or Infrastructure-as-Code such as [Terraform](https://registry.terraform.io/providers/hashicorp/aws/latest/docs) or [AWS CloudFormation](https://aws.amazon.com/cloudformation/).

- **Target Group(s)** - A [Target Group](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html) is attached to a [Listener](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-listeners.html) on the NLB and is responsible for routing incoming requests to healthy targets in the group. If connecting to the VCS system over both SSH and HTTPS, two **Target Groups** will need to be created.
- **Target Type (choose most applicable):**
- **Instance/ASG:** Select existing EC2 instance(s) where the VCS system is running, or [an autoscaling group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/attach-load-balancer-asg.html) (ASG) to automatically attach any instances launched from that ASG.
- **Application Load Balancer (ALB):** Select an ALB that already has VCS EC2 instances attached (HTTP/S traffic only).
- **IP Addresses:** Select the IP address(es) of the EC2 instances where the VCS system is installed. Keep in mind that the IP of the EC2 instance can change if the instance is relaunched for any reason.
- **Protocol/Port:** Choose one protocol and port pair per Target Group, for example:
- TG1 - SSH: TCP/22
- TG2 - HTTPS: TCP/443 or TLS if you want to attach a certificate to decrypt TLS connections ([details](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-tls-listener.html)).
- **VPC:** Choose the VPC in which the VPC Endpoint Service and NLB will be created.
- **Health checks:** Targets must register as healthy in order for the NLB to forward requests. Configure a health check that’s appropriate for your service and the protocol of the Target Group ([details](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html)).
- **Register targets:** Register the targets (see above) for the VCS service ([details](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-register-targets.html)). _It's critical to be sure targets are healthy before attempting connection from dbt Cloud._
- **Network Load Balancer (NLB)** - Requires creating a Listener that attaches to the newly created Target Group(s) for port `443` and/or `22`, as applicable.
- **Scheme:** Internal
- **IP address type:** IPv4
- **Network mapping:** Choose the VPC that the VPC Endpoint Service and NLB are being deployed in, and choose subnets from at least two Availability Zones.
- **Listeners:** Create one Listener per Target Group that maps the appropriate incoming port to the corresponding Target Group ([details](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-listeners.html)).
- **Endpoint Service** - The VPC Endpoint Service is what allows for the VPC to VPC connection, routing incoming requests to the configured load balancer.
- **Load balancer type:** Network.
- **Load balancer:** Attach the NLB created in the previous step.
- **Acceptance required (recommended)**: When enabled, requires a new connection request to the VPC Endpoint Service to be accepted by the customer before connectivity is allowed ([details](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests)).

Once these resources have been provisioned, access needs to be granted for the dbt Labs AWS account to create a VPC Endpoint in our VPC. On the newly created VPC Endpoint Service, add a new [Allowed Principal](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#add-remove-permissions) for the appropriate dbt Labs principal:

- **AWS Account ID:** `arn:aws:iam::<account id>:root` (contact your dbt Labs account representative for appropriate account ID).

### Completing the connection

To complete the connection, dbt Labs must now provision a VPC Endpoint to connect to your VPC Endpoint Service. This requires you send the following information:

- VPC Endpoint Service name:

<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/vpc-endpoint-service-name.png" width="80%" title="Location of the VPC Endpoint Service name in the AWS console" />

- **DNS configuration:** If the connection to the VCS service requires a custom domain and/or URL for TLS, a private hosted zone can be configured by the dbt Labs Infrastructure team in the dbt Cloud private network. For example:
- **Private hosted zone:** `examplecorp.com`
- **DNS record:** `github.examplecorp.com`

### Accepting the connection request

When you have been notified that the resources are provisioned within the dbt Cloud environment, you must accept the endpoint connection (unless the VPC Endpoint Service is set to auto-accept connection requests). Requests can be accepted through the AWS console, as seen below, or through the AWS CLI.

<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/accept-request.png" width="80%" title="Accept the connection request" />

Once you accept the endpoint connection request, you can use the PrivateLink endpoint in dbt Cloud.

## Configure in dbt Cloud

Once dbt confirms that the PrivateLink integration is complete, you can use it in a new or existing git configuration.
1. Select **PrivateLink Endpoint** as the connection type, and your configured integrations will appear in the dropdown menu.
2. Select the configured endpoint from the drop down list.
3. Click **Save**.

<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/vcs-setup-new.png" width="80%" title="Configuring a new git integration with PrivateLink" />

<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/vcs-setup-existing.png" width="80%" title="Editing an existing git integration with PrivateLink" />
2 changes: 1 addition & 1 deletion website/docs/docs/community-adapters.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,4 @@ Community adapters are adapter plugins contributed and maintained by members of
| [TiDB](/docs/core/connect-data-platform/tidb-setup) | [Firebolt](/docs/core/connect-data-platform/firebolt-setup) | [MindsDB](/docs/core/connect-data-platform/mindsdb-setup)
| [Vertica](/docs/core/connect-data-platform/vertica-setup) | [AWS Glue](/docs/core/connect-data-platform/glue-setup) | [MySQL](/docs/core/connect-data-platform/mysql-setup) |
| [Upsolver](/docs/core/connect-data-platform/upsolver-setup) | [Databend Cloud](/docs/core/connect-data-platform/databend-setup) | [fal - Python models](/docs/core/connect-data-platform/fal-setup) |

| [TimescaleDB](https://dbt-timescaledb.debruyn.dev/) | | |
2 changes: 2 additions & 0 deletions website/docs/docs/dbt-cloud-apis/sl-jdbc.md
Original file line number Diff line number Diff line change
Expand Up @@ -352,6 +352,8 @@ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'],
## FAQs
<FAQ path="Troubleshooting/sl-alpn-error" />
- **Why do some dimensions use different syntax, like `metric_time` versus `[Dimension('metric_time')`?**<br />
When you select a dimension on its own, such as `metric_time` you can use the shorthand method which doesn't need the “Dimension” syntax. However, when you perform operations on the dimension, such as adding granularity, the object syntax `[Dimension('metric_time')` is required.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ import AvailIntegrations from '/snippets/_sl-partner-links.md';
- <span><a href="https://docs.getdbt.com/docs/dbt-cloud-apis/sl-api-overview" target="_self">{frontMatter.meta.api_name}</a></span> to learn how to integrate and query your metrics in downstream tools.
- [dbt Semantic Layer API query syntax](/docs/dbt-cloud-apis/sl-jdbc#querying-the-api-for-metric-metadata)
- [Hex dbt Semantic Layer cells](https://learn.hex.tech/docs/logic-cell-types/transform-cells/dbt-metrics-cells) to set up SQL cells in Hex.
- [Resolve 'Failed APN'](/faqs/Troubleshooting/sl-alpn-error) error when connecting to the dbt Semantic Layer.

</VersionBlock>

Expand Down
5 changes: 2 additions & 3 deletions website/docs/docs/use-dbt-semantic-layer/gsheets.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,10 +54,9 @@ To use the filter functionality, choose the [dimension](docs/build/dimensions) y
- For categorical dimensiosn, type in the dimension value you want to filter by (no quotes needed) and press enter.
- Continue adding additional filters as needed with AND and OR. If it's a time dimension, choose the operator and select from the calendar.



**Limited Use Policy Disclosure**

The dbt Semantic Layer for Sheet's use and transfer to any other app of information received from Google APIs will adhere to [Google API Services User Data Policy](https://developers.google.com/terms/api-services-user-data-policy), including the Limited Use requirements.


## FAQs
<FAQ path="Troubleshooting/sl-alpn-error" />
4 changes: 3 additions & 1 deletion website/docs/docs/use-dbt-semantic-layer/tableau.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ This integration provides a live connection to the dbt Semantic Layer through Ta

2. Install the [JDBC driver](/docs/dbt-cloud-apis/sl-jdbc) to the folder based on your operating system:
- Windows: `C:\Program Files\Tableau\Drivers`
- Mac: `~/Library/Tableau/Drivers`
- Mac: `~/Library/Tableau/Drivers` or `/Library/JDBC` or `~/Library/JDBC`
- Linux: ` /opt/tableau/tableau_driver/jdbc`
3. Open Tableau Desktop or Tableau Server and find the **dbt Semantic Layer by dbt Labs** connector on the left-hand side. You may need to restart these applications for the connector to be available.
4. Connect with your Host, Environment ID, and Service Token information dbt Cloud provides during [Semantic Layer configuration](/docs/use-dbt-semantic-layer/setup-sl#:~:text=After%20saving%20it%2C%20you%27ll%20be%20provided%20with%20the%20connection%20information%20that%20allows%20you%20to%20connect%20to%20downstream%20tools).
Expand Down Expand Up @@ -80,3 +80,5 @@ The following Tableau features aren't supported at this time, however, the dbt S
- Filtering on a Date Part time dimension for a Cumulative metric type
- Changing your date dimension to use "Week Number"

## FAQs
<FAQ path="Troubleshooting/sl-alpn-error" />
10 changes: 10 additions & 0 deletions website/docs/faqs/API/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# position: 2.5 # float position is supported
label: 'API'
collapsible: true # make the category collapsible
collapsed: true # keep the category collapsed by default
className: red
link:
type: generated-index
title: API FAQs
customProps:
description: Frequently asked questions about dbt APIs
10 changes: 10 additions & 0 deletions website/docs/faqs/Accounts/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# position: 2.5 # float position is supported
label: 'Accounts'
collapsible: true # make the category collapsible
collapsed: true # keep the category collapsed by default
className: red
link:
type: generated-index
title: Account FAQs
customProps:
description: Frequently asked questions about your account in dbt
10 changes: 10 additions & 0 deletions website/docs/faqs/Core/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# position: 2.5 # float position is supported
label: 'dbt Core'
collapsible: true # make the category collapsible
collapsed: true # keep the category collapsed by default
className: red
link:
type: generated-index
title: 'dbt Core FAQs'
customProps:
description: Frequently asked questions about dbt Core
10 changes: 10 additions & 0 deletions website/docs/faqs/Docs/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# position: 2.5 # float position is supported
label: 'dbt Docs'
collapsible: true # make the category collapsible
collapsed: true # keep the category collapsed by default
className: red
link:
type: generated-index
title: dbt Docs FAQs
customProps:
description: Frequently asked questions about dbt Docs
10 changes: 10 additions & 0 deletions website/docs/faqs/Environments/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# position: 2.5 # float position is supported
label: 'Environments'
collapsible: true # make the category collapsible
collapsed: true # keep the category collapsed by default
className: red
link:
type: generated-index
title: 'Environments FAQs'
customProps:
description: Frequently asked questions about Environments in dbt
10 changes: 10 additions & 0 deletions website/docs/faqs/Git/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# position: 2.5 # float position is supported
label: 'Git'
collapsible: true # make the category collapsible
collapsed: true # keep the category collapsed by default
className: red
link:
type: generated-index
title: Git FAQs
customProps:
description: Frequently asked questions about Git and dbt
10 changes: 10 additions & 0 deletions website/docs/faqs/Jinja/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# position: 2.5 # float position is supported
label: 'Jinja'
collapsible: true # make the category collapsible
collapsed: true # keep the category collapsed by default
className: red
link:
type: generated-index
title: Jinja FAQs
customProps:
description: Frequently asked questions about Jinja and dbt
10 changes: 10 additions & 0 deletions website/docs/faqs/Models/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# position: 2.5 # float position is supported
label: 'Models'
collapsible: true # make the category collapsible
collapsed: true # keep the category collapsed by default
className: red
link:
type: generated-index
title: Models FAQs
customProps:
description: Frequently asked questions about Models in dbt
10 changes: 10 additions & 0 deletions website/docs/faqs/Project/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# position: 2.5 # float position is supported
label: 'Projects'
collapsible: true # make the category collapsible
collapsed: true # keep the category collapsed by default
className: red
link:
type: generated-index
title: Project FAQs
customProps:
description: Frequently asked questions about projects in dbt
10 changes: 10 additions & 0 deletions website/docs/faqs/Runs/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# position: 2.5 # float position is supported
label: 'Runs'
collapsible: true # make the category collapsible
collapsed: true # keep the category collapsed by default
className: red
link:
type: generated-index
title: Runs FAQs
customProps:
description: Frequently asked questions about runs in dbt
10 changes: 10 additions & 0 deletions website/docs/faqs/Seeds/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# position: 2.5 # float position is supported
label: 'Seeds'
collapsible: true # make the category collapsible
collapsed: true # keep the category collapsed by default
className: red
link:
type: generated-index
title: Seeds FAQs
customProps:
description: Frequently asked questions about seeds in dbt
10 changes: 10 additions & 0 deletions website/docs/faqs/Snapshots/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# position: 2.5 # float position is supported
label: 'Snapshots'
collapsible: true # make the category collapsible
collapsed: true # keep the category collapsed by default
className: red
link:
type: generated-index
title: Snapshots FAQs
customProps:
description: Frequently asked questions about snapshots in dbt
10 changes: 10 additions & 0 deletions website/docs/faqs/Tests/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# position: 2.5 # float position is supported
label: 'Tests'
collapsible: true # make the category collapsible
collapsed: true # keep the category collapsed by default
className: red
link:
type: generated-index
title: Tests FAQs
customProps:
description: Frequently asked questions about tests in dbt
10 changes: 10 additions & 0 deletions website/docs/faqs/Troubleshooting/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# position: 2.5 # float position is supported
label: 'Troubleshooting'
collapsible: true # make the category collapsible
collapsed: true # keep the category collapsed by default
className: red
link:
type: generated-index
title: Troubleshooting FAQs
customProps:
description: Frequently asked questions about troubleshooting dbt
14 changes: 14 additions & 0 deletions website/docs/faqs/Troubleshooting/sl-alpn-error.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
title: I'm receiving an `Failed ALPN` error when trying to connect to the dbt Semantic Layer.
description: "To resolve the 'Failed ALPN' error in the dbt Semantic Layer, create a SSL interception exception for the dbt Cloud domain."
sidebar_label: 'Use SSL exception to resolve `Failed ALPN` error'
---

If you're receiving a `Failed ALPN` error when trying to connect the dbt Semantic Layer with the various [data integration tools](/docs/use-dbt-semantic-layer/avail-sl-integrations) (such as Tableau, DBeaver, Datagrip, ADBC, or JDBC), it typically happens when connecting from a computer behind a corporate VPN or Proxy (like Zscaler or Check Point).

The root cause is typically the proxy interfering with the TLS handshake as the dbt Semantic Layer uses gRPC/HTTP2 for connectivity. To resolve this:

- If your proxy supports gRPC/HTTP2 but isn't configured to allow ALPN, adjust its settings accordingly to allow ALPN. Or create an exception for the dbt Cloud domain.
- If your proxy does not support gRPC/HTTP2, add an SSL interception exception for the dbt Cloud domain in your proxy settings

This should help in successfully establishing the connection without the Failed ALPN error.
10 changes: 10 additions & 0 deletions website/docs/faqs/Warehouse/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# position: 2.5 # float position is supported
label: 'Warehouse'
collapsible: true # make the category collapsible
collapsed: true # keep the category collapsed by default
className: red
link:
type: generated-index
title: Warehouse FAQs
customProps:
description: Frequently asked questions about warehouses and dbt
1 change: 1 addition & 0 deletions website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@ const sidebarSettings = {
"docs/cloud/secure/databricks-privatelink",
"docs/cloud/secure/redshift-privatelink",
"docs/cloud/secure/postgres-privatelink",
"docs/cloud/secure/vcs-privatelink",
"docs/cloud/secure/ip-restrictions",
],
}, // PrivateLink
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit acb939b

Please sign in to comment.