From 58a64734af7deafe3a981e48f2cef07cbcbda6e7 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 13 Aug 2024 19:46:38 -0400 Subject: [PATCH] Azure private link (#5929) ## What are you changing in this pull request and why? Enhances the Snowflake instruction for Azure Add Databricks instructions ## Checklist - [ ] Review the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] For [docs versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#about-versioning), review how to [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content). - [ ] Add a checklist item for anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." Adding or removing pages (delete if not applicable): - [ ] Add/remove page in `website/sidebars.js` - [ ] Provide a unique filename for new pages - [ ] Add an entry for deleted pages in `website/vercel.json` - [ ] Run link testing locally with `npm run build` to update the links that point to deleted pages --------- Co-authored-by: Ly Nguyen <107218380+nghi-ly@users.noreply.github.com> --- .../cloud/secure/databricks-privatelink.md | 25 +++++- .../cloud/secure/snowflake-privatelink.md | 80 ++++++++++++++++--- 2 files changed, 92 insertions(+), 13 deletions(-) diff --git a/website/docs/docs/cloud/secure/databricks-privatelink.md b/website/docs/docs/cloud/secure/databricks-privatelink.md index c83e9c67c95..f5cd7f1dc61 100644 --- a/website/docs/docs/cloud/secure/databricks-privatelink.md +++ b/website/docs/docs/cloud/secure/databricks-privatelink.md @@ -10,15 +10,15 @@ import SetUpPages from '/snippets/_available-tiers-privatelink.md'; -The following steps will walk you through the setup of a Databricks AWS PrivateLink endpoint in the dbt Cloud multi-tenant environment. +The following steps will walk you through the setup of a Databricks AWS PrivateLink or Azure Private Link endpoint in the dbt Cloud multi-tenant environment. -## Configure PrivateLink +## Configure AWS PrivateLink 1. Locate your [Databricks instance name](https://docs.databricks.com/en/workspace/workspace-details.html#workspace-instance-names-urls-and-ids) - Example: `cust-success.cloud.databricks.com` -2. Add the required information to the template below, and submit your request to [dbt Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support): +2. Add the required information to the following template and submit your AWS PrivateLink request to [dbt Support](https://docs.getdbt.com/docs/dbt-support#dbt-cloud-support): ``` -Subject: New Multi-Tenant PrivateLink Request +Subject: New AWS Multi-Tenant PrivateLink Request - Type: Databricks - Databricks instance name: - Databricks cluster AWS Region (e.g., us-east-1, eu-west-2): @@ -41,6 +41,23 @@ If using an existing Databricks workspace, all workloads running in the workspac ::: +## Configure Azure Private Link + +1. Navigate to your Azure Databricks workspace. + The path format is: `/subscriptions//resourceGroups//providers/Microsoft.Databricks/workspaces/`. +2. From the workspace overview, click **JSON view**. +3. Copy the value in the `resource_id` field. +4. Add the required information to the following template and submit your Azure Private Link request to [dbt Support](https://docs.getdbt.com/docs/dbt-support#dbt-cloud-support): + ``` + Subject: New Azure Multi-Tenant Private Link Request + - Type: Databricks + - Databricks instance name: + - Databricks Azure resource ID: + - dbt Cloud multi-tenant environment: EMEA + ``` +5. Once our Support team confirms the resources are available in the Azure portal, navigate to the Azure Databricks Workspace and browse to **Networking** > **Private Endpoint Connections**. Then, highlight the `dbt` named option and select **Approve**. + + ## Create Connection in dbt Cloud Once you've completed the setup in the Databricks environment, you will be able to configure a private endpoint in dbt Cloud: diff --git a/website/docs/docs/cloud/secure/snowflake-privatelink.md b/website/docs/docs/cloud/secure/snowflake-privatelink.md index 6cffc373d3b..c6775be2444 100644 --- a/website/docs/docs/cloud/secure/snowflake-privatelink.md +++ b/website/docs/docs/cloud/secure/snowflake-privatelink.md @@ -9,7 +9,7 @@ import SetUpPages from '/snippets/_available-tiers-privatelink.md'; -The following steps walk you through the setup of a Snowflake AWS PrivateLink and Azure Private Link endpoint in the dbt Cloud multi-tenant environment. +The following steps walk you through the setup of a Snowflake AWS PrivateLink or Azure Private Link endpoint in a dbt Cloud multi-tenant environment. :::note Snowflake SSO with PrivateLink Users connecting to Snowflake using SSO over a PrivateLink connection from dbt Cloud will also require access to a PrivateLink endpoint from their local workstation. @@ -19,26 +19,37 @@ Users connecting to Snowflake using SSO over a PrivateLink connection from dbt C - [Snowflake SSO with Private Connectivity](https://docs.snowflake.com/en/user-guide/admin-security-fed-auth-overview#label-sso-private-connectivity) ::: -## Configure PrivateLink +## About private connectivity for Snowflake -1. Open a Support case with Snowflake to allow access from the dbt Cloud AWS account -- Snowflake prefers that the account owner opens the Support case directly, rather than dbt Labs acting on their behalf. For more information, refer to [Snowflake's knowledge base article](https://community.snowflake.com/s/article/HowtosetupPrivatelinktoSnowflakefromCloudServiceVendors) +dbt Cloud supports private connectivity for Snowflake using one of the following services: + +- AWS [PrivateLink](#configure-aws-privatelink) +- Azure [Private Link](#configure-azure-private-link) + +## Configure AWS PrivateLink + +To configure Snowflake instances hosted on AWS for [PrivateLink](https://aws.amazon.com/privatelink): + +1. Open a support case with Snowflake to allow access from the dbt Cloud AWS or Entra ID account. +- Snowflake prefers that the account owner opens the support case directly rather than dbt Labs acting on their behalf. For more information, refer to [Snowflake's knowledge base article](https://community.snowflake.com/s/article/HowtosetupPrivatelinktoSnowflakefromCloudServiceVendors). - Provide them with your dbt Cloud account ID along with any other information requested in the article. - - AWS account ID: `346425330055` - _NOTE: This account ID only applies to dbt Cloud Multi-Tenant environments. For Virtual Private/Single-Tenant account IDs please contact [Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support)._ + - **AWS account ID**: `346425330055` — _NOTE: This account ID only applies to AWS dbt Cloud multi-tenant environments. For AWS Virtual Private/Single-Tenant account IDs, please contact [Support](https://docs.getdbt.com/docs/dbt-support#dbt-cloud-support)._ - You will need to have `ACCOUNTADMIN` access to the Snowflake instance to submit a Support request. 2. After Snowflake has granted the requested access, run the Snowflake system function [SYSTEM$GET_PRIVATELINK_CONFIG](https://docs.snowflake.com/en/sql-reference/functions/system_get_privatelink_config.html) and copy the output. -3. Add the required information to the template below, and submit your request to [dbt Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support): +3. Add the required information to the following template and submit your request to [dbt Support](https://docs.getdbt.com/docs/dbt-support#dbt-cloud-support): ``` -Subject: New Multi-Tenant PrivateLink Request +Subject: New Multi-Tenant (Azure or AWS) PrivateLink Request - Type: Snowflake - SYSTEM$GET_PRIVATELINK_CONFIG output: - *Use privatelink-account-url or regionless-privatelink-account-url?: -- dbt Cloud multi-tenant environment (US, EMEA, AU): +- dbt Cloud multi-tenant environment + - AWS: US, EMEA, or AU + - Azure: EMEA only ``` _*By default dbt Cloud will be configured to use `privatelink-account-url` from the provided [SYSTEM$GET_PRIVATELINK_CONFIG](https://docs.snowflake.com/en/sql-reference/functions/system_get_privatelink_config.html) as the PrivateLink endpoint. Upon request, `regionless-privatelink-account-url` can be used instead._ @@ -47,6 +58,32 @@ import PrivateLinkSLA from '/snippets/_PrivateLink-SLA.md'; +## Configure Azure Private Link + +To configure Snowflake instances hosted on Azure for [Private Link](https://learn.microsoft.com/en-us/azure/private-link/private-link-overview): + +1. In your Snowflake account, run the following SQL statements and copy the output: + +```sql + +USE ROLE ACCOUNTADMIN; +SYSTEM$GET_PRIVATELINK_CONFIG; + +``` + + +2. Add the required information to the following template and submit your request to [dbt Support](https://docs.getdbt.com/docs/dbt-support#dbt-cloud-support): + +``` +Subject: New Multi-Tenant (Azure or AWS) PrivateLink Request +- Type: Snowflake +- The output from SYSTEM$GET_PRIVATELINK_CONFIG: + - Include the privatelink-pls-id +- dbt Cloud Azure multi-tenant environment: +``` + +3. dbt Support will provide the `private endpoint resource_id` of our `private_endpoint` and the `CIDR` range for you to complete the [PrivateLink configuration](https://community.snowflake.com/s/article/HowtosetupPrivatelinktoSnowflakefromCloudServiceVendors) by contacting the Snowflake Support team. + ## Create Connection in dbt Cloud Once dbt Cloud support completes the configuration, you can start creating new connections using PrivateLink. @@ -57,6 +94,27 @@ Once dbt Cloud support completes the configuration, you can start creating new c 4. Configure the remaining data platform details. 5. Test your connection and save it. +## Enable the connection in Snowflake + +To complete the setup, follow the remaining steps from the Snowflake setup guides. The instructions vary based on the platform: + +- [Snowflake AWS PrivateLink](https://docs.snowflake.com/en/user-guide/admin-security-privatelink) +- [Snowflake Azure Private Link](https://docs.snowflake.com/en/user-guide/privatelink-azure) + +There are some nuances for each connection and you will need a Snowflake administrator. As the Snowflake administrator, call the `SYSTEM$AUTHORIZE_STAGE_PRIVATELINK_ACCESS` function using the privateEndpointResourceID value as the function argument. This authorizes access to the Snowflake internal stage through the private endpoint. + +```sql + +USE ROLE ACCOUNTADMIN; + +-- AWS PrivateLink +SELECT SYSTEMS$AUTHORIZE_STATE_PRIVATELINK_ACCESS ( `AWS VPC ID` ); + +-- Azure Private Link +SELECT SYSTEMS$AUTHORIZE_STATE_PRIVATELINK_ACCESS ( `AZURE PRIVATE ENDPOINT RESOURCE ID` ); + +``` + ## Configuring Network Policies If your organization uses [Snowflake Network Policies](https://docs.snowflake.com/en/user-guide/network-policies) to restrict access to your Snowflake account, you will need to add a network rule for dbt Cloud. @@ -84,19 +142,23 @@ Open the Snowflake UI and take the following steps: ### Using SQL + For quick and automated setup of network rules via SQL in Snowflake, the following commands allow you to create and configure access rules for dbt Cloud. These SQL examples demonstrate how to add a network rule and update your network policy accordingly. 1. Create a new network rule with the following SQL: ```sql + CREATE NETWORK RULE allow_dbt_cloud_access MODE = INGRESS TYPE = AWSVPCEID VALUE_LIST = (''); -- Replace '' with the actual ID provided + ``` 2. Add the rule to a network policy with the following SQL: ```sql + ALTER NETWORK POLICY ADD ALLOWED_NETWORK_RULE_LIST =('allow_dbt_cloud_access'); -``` +```