Skip to content

Commit

Permalink
add callout to PrivateLink docs about Environment Variables
Browse files Browse the repository at this point in the history
  • Loading branch information
ryancharris committed Jan 3, 2024
1 parent 2e4df56 commit a15a231
Show file tree
Hide file tree
Showing 4 changed files with 26 additions and 12 deletions.
9 changes: 6 additions & 3 deletions website/docs/docs/cloud/secure/databricks-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ pagination_next: null
---

import SetUpPages from '/snippets/_available-tiers-privatelink.md';
import PrivateLinkHostnameWarning from '/snippets/_privatelink-hostname-restriction.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

Expand All @@ -15,7 +16,7 @@ The following steps will walk you through the setup of a Databricks AWS PrivateL
## Configure PrivateLink

1. Locate your [Databricks Workspace ID](https://kb.databricks.com/en_US/administration/find-your-workspace-id#:~:text=When%20viewing%20a%20Databricks%20workspace,make%20up%20the%20workspace%20ID)
2. Add the required information to the template below, and submit your request to [dbt Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support):
2. Add the required information to the template below, and submit your request to [dbt Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support):
```
Subject: New Multi-Tenant PrivateLink Request
- Type: Databricks
Expand All @@ -29,8 +30,10 @@ Subject: New Multi-Tenant PrivateLink Request

Once you've completed the setup in the Databricks environment, you will be able to configure a private endpoint in dbt Cloud:

1. Navigate to **Settings****Create new project** → select **Databricks**.
2. You will see two radio buttons: **Public** and **Private.** Select **Private**.
1. Navigate to **Settings****Create new project** → select **Databricks**.
2. You will see two radio buttons: **Public** and **Private.** Select **Private**.
3. Select the private endpoint from the dropdown (this will automatically populate the hostname/account field).
4. Configure the remaining data platform details.
5. Test your connection and save it.

<PrivateLinkHostnameWarning features={'/snippets/_privatelink-hostname-restriction.md'}/>
11 changes: 7 additions & 4 deletions website/docs/docs/cloud/secure/postgres-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ description: "Configuring PrivateLink for Postgres"
sidebar_label: "PrivateLink for Postgres"
---
import SetUpPages from '/snippets/_available-tiers-privatelink.md';
import PrivateLinkHostnameWarning from '/snippets/_privatelink-hostname-restriction.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

Expand All @@ -18,14 +19,14 @@ Creating an Interface VPC PrivateLink connection requires creating multiple AWS

- **Security Group (AWS hosted only)** &mdash; If you are connecting to an existing Postgres instance, this likely already exists, however, you may need to add or modify Security Group rules to accept traffic from the Network Load Balancer (NLB) created for this Endpoint Service.
- **Target Group** &mdash; The Target Group will be attached to the NLB to tell it where to route requests. There are various target types available for NLB Target Groups, so choose the one appropriate for your Postgres setup.

- Target Type:

- _[Amazon RDS for PostgreSQL](https://aws.amazon.com/rds/postgresql/)_ - **IP**

- Find the IP address of your RDS instance using a command line tool such as `nslookup <endpoint>` or `dig +short <endpoint>` with your RDS DNS endpoint

- _Note_: With RDS Multi-AZ failover capabilities the IP address of your RDS instance can change, at which point your Target Group would need to be updated. See [this AWS blog post](https://aws.amazon.com/blogs/database/access-amazon-rds-across-vpcs-using-aws-privatelink-and-network-load-balancer/) for more details and a possible solution.
- _Note_: With RDS Multi-AZ failover capabilities the IP address of your RDS instance can change, at which point your Target Group would need to be updated. See [this AWS blog post](https://aws.amazon.com/blogs/database/access-amazon-rds-across-vpcs-using-aws-privatelink-and-network-load-balancer/) for more details and a possible solution.

- _On-prem Postgres server_ - **IP**

Expand All @@ -37,7 +38,7 @@ Creating an Interface VPC PrivateLink connection requires creating multiple AWS

- The IP type can also be used, with the understanding that the IP of the EC2 instance can change if the instance is relaunched for any reason

- Target Group protocol: **TCP**
- Target Group protocol: **TCP**

- **Network Load Balancer (NLB)** &mdash; Requires creating a Listener that attaches to the newly created Target Group for port `5432`
- **VPC Endpoint Service** &mdash; Attach to the newly created NLB.
Expand Down Expand Up @@ -73,7 +74,9 @@ dbt Labs will work on your behalf to complete the PrivateLink setup. Please allo
Once dbt Cloud support completes the configuration, you can start creating new connections using PrivateLink.

1. Navigate to **settings****Create new project** → select **PostgreSQL**
2. You will see two radio buttons: **Public** and **Private.** Select **Private**.
2. You will see two radio buttons: **Public** and **Private.** Select **Private**.
3. Select the private endpoint from the dropdown (this will automatically populate the hostname/account field).
4. Configure the remaining data platform details.
5. Test your connection and save it.

<PrivateLinkHostnameWarning features={'/snippets/_privatelink-hostname-restriction.md'}/>
13 changes: 8 additions & 5 deletions website/docs/docs/cloud/secure/redshift-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,18 @@ sidebar_label: "PrivateLink for Redshift"
---

import SetUpPages from '/snippets/_available-tiers-privatelink.md';
import PrivateLinkHostnameWarning from '/snippets/_privatelink-hostname-restriction.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

AWS provides two different ways to create a PrivateLink VPC endpoint for a Redshift cluster that is running in another VPC:
AWS provides two different ways to create a PrivateLink VPC endpoint for a Redshift cluster that is running in another VPC:
- [Redshift-managed PrivateLink Endpoints](https://docs.aws.amazon.com/redshift/latest/mgmt/managing-cluster-cross-vpc.html)
- [Redshift Interface-type PrivateLink Endpoints](https://docs.aws.amazon.com/redshift/latest/mgmt/security-private-link.html)

dbt Cloud supports both types of endpoints, but there are a number of [considerations](https://docs.aws.amazon.com/redshift/latest/mgmt/managing-cluster-cross-vpc.html#managing-cluster-cross-vpc-considerations) to take into account when deciding which endpoint type to use. Redshift-managed provides a far simpler setup with no additional cost, which might make it the preferred option for many, but may not be an option in all environments. Based on these criteria, you will need to determine which is the right type for your system. Follow the instructions from the section below that corresponds to your chosen endpoint type.

:::note Redshift Serverless
While Redshift Serverless does support Redshift-managed type VPC endpoints, this functionality is not currently available across AWS accounts. Due to this limitation, an Interface-type VPC endpoint service must be used for Redshift Serverless cluster PrivateLink connectivity from dbt Cloud.
While Redshift Serverless does support Redshift-managed type VPC endpoints, this functionality is not currently available across AWS accounts. Due to this limitation, an Interface-type VPC endpoint service must be used for Redshift Serverless cluster PrivateLink connectivity from dbt Cloud.
:::

## Configuring Redshift-managed PrivateLink
Expand Down Expand Up @@ -56,7 +57,7 @@ Creating an Interface VPC PrivateLink connection requires creating multiple AWS

- **Security Group** &mdash; If you are connecting to an existing Redshift cluster, this likely already exists, however, you may need to add or modify Security Group rules to accept traffic from the Network Load Balancer (NLB) created for this Endpoint Service.
- **Target Group** &mdash; The Target Group will be attached to the NLB to tell it where to route requests. There are various target types available for NLB Target Groups, but you will use the IP address type.

- Target Type: **IP**

- **Standard Redshift**
Expand All @@ -73,7 +74,7 @@ Creating an Interface VPC PrivateLink connection requires creating multiple AWS

- From a command line run the command `nslookup <endpoint>` using the endpoint found in the previous step and use the associated IP(s) for the Target Group.

- Target Group protocol: **TCP**
- Target Group protocol: **TCP**

- **Network Load Balancer (NLB)** &mdash; Requires creating a Listener that attaches to the newly created Target Group for port `5439`
- **VPC Endpoint Service** &mdash; Attach to the newly created NLB.
Expand Down Expand Up @@ -109,7 +110,9 @@ dbt Labs will work on your behalf to complete the PrivateLink setup. Please allo
Once dbt Cloud support completes the configuration, you can start creating new connections using PrivateLink.

1. Navigate to **settings****Create new project** → select **Redshift**
2. You will see two radio buttons: **Public** and **Private.** Select **Private**.
2. You will see two radio buttons: **Public** and **Private.** Select **Private**.
3. Select the private endpoint from the dropdown (this will automatically populate the hostname/account field).
4. Configure the remaining data platform details.
5. Test your connection and save it.

<PrivateLinkHostnameWarning features={'/snippets/_privatelink-hostname-restriction.md'}/>
5 changes: 5 additions & 0 deletions website/snippets/_privatelink-hostname-restriction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
:::caution Environment variables

The use of [Environment Variables](https://docs.getdbt.com/docs/build/environment-variables) to define the `hostname` of your PrivateLink endpoint is not supported in dbt Cloud. To dynamically change this value in a given dbt Cloud Environment, you can instead use [Extended Attributes](https://docs.getdbt.com/docs/deploy/deploy-environments#extended-attributes).

:::

0 comments on commit a15a231

Please sign in to comment.