From 9dfb96db8598d7a2f4ab864f874837a0f72b936d Mon Sep 17 00:00:00 2001 From: Dave Haworth Date: Thu, 28 Sep 2023 13:21:26 -0500 Subject: [PATCH 01/10] add postgres privatelink page --- .../docs/cloud/secure/about-privatelink.md | 3 +- .../docs/cloud/secure/postgres-privatelink.md | 76 +++++++++++++++++++ .../docs/cloud/secure/redshift-privatelink.md | 2 +- 3 files changed, 79 insertions(+), 2 deletions(-) create mode 100644 website/docs/docs/cloud/secure/postgres-privatelink.md diff --git a/website/docs/docs/cloud/secure/about-privatelink.md b/website/docs/docs/cloud/secure/about-privatelink.md index 77ee8a6af7a..29003f65a21 100644 --- a/website/docs/docs/cloud/secure/about-privatelink.md +++ b/website/docs/docs/cloud/secure/about-privatelink.md @@ -15,6 +15,7 @@ dbt Labs has a worldwide network of regional VPCs. These VPCs are specifically u dbt Cloud supports the following data platforms for use with the PrivateLink feature. Instructions for enabling PrivateLink for the various data platform providers are unique. The following guides will walk you through the necessary steps, including working with [dbt Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support) to complete the connection in the dbt private network and setting up the endpoint in dbt Cloud. -- [Redshift](/docs/cloud/secure/redshift-privatelink) - [Snowflake](/docs/cloud/secure/snowflake-privatelink) - [Databricks](/docs/cloud/secure/databricks-privatelink) +- [Redshift](/docs/cloud/secure/redshift-privatelink) +- [Postgres](/docs/cloud/secure/postgres-privatelink) diff --git a/website/docs/docs/cloud/secure/postgres-privatelink.md b/website/docs/docs/cloud/secure/postgres-privatelink.md new file mode 100644 index 00000000000..b220d9c936a --- /dev/null +++ b/website/docs/docs/cloud/secure/postgres-privatelink.md @@ -0,0 +1,76 @@ +--- +title: "Configure AWS PrivateLink for Postgres" +id: postgres-privatelink +description: "Configuring PrivateLink for Postgres" +sidebar_label: "PrivateLink for Postgres" +--- + +A Postgres database, hosted either in AWS or in a properly connected on-prem data center, can be accessed via a private network connection using AWS Interface-type PrivateLink. The type of Target Group connected to the Network Load Balancer (NLB) may vary based on the location and type of Postgres instance being connected, as is noted in the steps below. + +## Configuring Postgres Interface-type PrivateLink + +### 1. Provision AWS Resources + +Creating an Interface VPC PrivateLink connection requires creating multiple AWS resources in the account containing, or connected to, the Postgres instance: + +- **Security Group (AWS hosted only)** — If you are connecting to an existing Postgres instance, this likely already exists, however, you may need to add or modify Security Group rules to accept traffic from the Network Load Balancer (NLB) created for this Endpoint Service. +- **Target Group** — The Target Group will be attached to the NLB to tell it where to route requests. There are various target types available for NLB Target Groups, so choose the one appropriate for your Postgres setup. + + - Target Type: + + - _[Amazon RDS for PostgreSQL](https://aws.amazon.com/rds/postgresql/)_ - **IP** + + - Find the IP address of your RDS instance using a command line tool such as `nslookup ` or `dig +short ` with your RDS DNS endpoint + + - _Note_: With RDS Multi-AZ failover capabilities the IP address of your RDS instance can change, at which point your Tagrat Group would need to be updated. See [this AWS blog post](https://aws.amazon.com/blogs/database/access-amazon-rds-across-vpcs-using-aws-privatelink-and-network-load-balancer/) for more details and a possible solution. + + - _On-prem Postgres server_ - **IP** + + - Use the IP address of the on-prem Postgres server linked to AWS through AWS Direct Connect or a Site-to-Site VPN connection + + - _Postgres on EC2_ - **Instance/ASG** (or **IP**) + + - If your Postgres instance is hosted on EC2 the _instance_ Target Group type (or ideally [using the instance type to connect to an auto-scaling group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/attach-load-balancer-asg.html)) can be used to attach the instance without needing a static IP address + + - The IP type can also be used, with the understanding that the IP of the EC2 instance can change if the instance is relaunched for any reason + + - Target Group protocol: **TCP** + +- **Network Load Balancer (NLB)** — Requires creating a Listener that attaches to the newly created Target Group for port `5432` +- **VPC Endpoint Service** — Attach to the newly created NLB. + - Acceptance required (optional) — Requires you to [accept our connection request](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests) after dbt creates the endpoint. + +### 2. Grant dbt AWS Account access to the VPC Endpoint Service + +On the provisioned VPC endpoint service, click the **Allow principals** tab. Click **Allow principals** to grant access. Enter the ARN of the root user in the appropriate production AWS account and save your changes. + + - Principal: `arn:aws:iam::346425330055:role/MTPL_Admin` (_or_ `arn:aws:iam::346425330055:root`) + + + +### 3. Obtain VPC Endpoint Service Name + +Once the VPC Endpoint Service is provisioned, you can find the service name in the AWS console by navigating to **VPC** → **Endpoint Services** and selecting the appropriate endpoint service. You can copy the service name field value and include it in your communication to dbt Cloud support. + + + +### 4. Add the required information to the template below, and submit your request to [dbt Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support): +``` +Subject: New Multi-Tenant PrivateLink Request +- Type: Postgres Interface-type +- VPC Endpoint Service Name: +- Postgres server AWS Region (e.g., us-east-1, eu-west-2): +- dbt Cloud multi-tenant environment (US, EMEA, AU): +``` + +dbt Labs will work on your behalf to complete the PrivateLink setup. Please allow 1-2 business days for this process to complete. Support will contact you when the endpoint is available. + +## Create Connection in dbt Cloud + +Once dbt Cloud support completes the configuration, you can start creating new connections using PrivateLink. + +1. Navigate to **settings** → **Create new project** → select **PostgreSQL** +2. You will see two radio buttons: **Public** and **Private.** Select **Private**. +3. Select the private endpoint from the dropdown (this will automatically populate the hostname/account field). +4. Configure the remaining data platform details. +5. Test your connection and save it. diff --git a/website/docs/docs/cloud/secure/redshift-privatelink.md b/website/docs/docs/cloud/secure/redshift-privatelink.md index fc0ceeea334..8f39bf4f2f6 100644 --- a/website/docs/docs/cloud/secure/redshift-privatelink.md +++ b/website/docs/docs/cloud/secure/redshift-privatelink.md @@ -79,7 +79,7 @@ Creating an Interface VPC PrivateLink connection requires creating multiple AWS On the provisioned VPC endpoint service, click the **Allow principals** tab. Click **Allow principals** to grant access. Enter the ARN of the root user in the appropriate production AWS account and save your changes. - - Principal: `arn:aws:iam::346425330055:root` + - Principal: `arn:aws:iam::346425330055:role/MTPL_Admin` (_or_ `arn:aws:iam::346425330055:root`) From 301a4ce621e943a4ce3a2c174c7b23207cd935a6 Mon Sep 17 00:00:00 2001 From: Dave Haworth Date: Thu, 28 Sep 2023 13:28:24 -0500 Subject: [PATCH 02/10] add postgres pl page to sidebars.js --- website/sidebars.js | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/website/sidebars.js b/website/sidebars.js index af9482a8ddf..2f3ebaed4e6 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -134,8 +134,9 @@ const sidebarSettings = { items: [ "docs/cloud/secure/about-privatelink", "docs/cloud/secure/snowflake-privatelink", - "docs/cloud/secure/redshift-privatelink", "docs/cloud/secure/databricks-privatelink", + "docs/cloud/secure/redshift-privatelink", + "docs/cloud/secure/postgres-privatelink", "docs/cloud/secure/ip-restrictions", ], }, // PrivateLink From 9a555aa4424a521755035180df80aadb1587c0f5 Mon Sep 17 00:00:00 2001 From: Dave Haworth Date: Thu, 28 Sep 2023 13:45:56 -0500 Subject: [PATCH 03/10] fix typo --- website/docs/docs/cloud/secure/postgres-privatelink.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/secure/postgres-privatelink.md b/website/docs/docs/cloud/secure/postgres-privatelink.md index b220d9c936a..68b151fbce7 100644 --- a/website/docs/docs/cloud/secure/postgres-privatelink.md +++ b/website/docs/docs/cloud/secure/postgres-privatelink.md @@ -22,7 +22,7 @@ Creating an Interface VPC PrivateLink connection requires creating multiple AWS - Find the IP address of your RDS instance using a command line tool such as `nslookup ` or `dig +short ` with your RDS DNS endpoint - - _Note_: With RDS Multi-AZ failover capabilities the IP address of your RDS instance can change, at which point your Tagrat Group would need to be updated. See [this AWS blog post](https://aws.amazon.com/blogs/database/access-amazon-rds-across-vpcs-using-aws-privatelink-and-network-load-balancer/) for more details and a possible solution. + - _Note_: With RDS Multi-AZ failover capabilities the IP address of your RDS instance can change, at which point your Target Group would need to be updated. See [this AWS blog post](https://aws.amazon.com/blogs/database/access-amazon-rds-across-vpcs-using-aws-privatelink-and-network-load-balancer/) for more details and a possible solution. - _On-prem Postgres server_ - **IP** From 33a4d7a9ffb08caf5c4570eddf7fa0fb17364985 Mon Sep 17 00:00:00 2001 From: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 29 Sep 2023 12:51:42 +0100 Subject: [PATCH 04/10] Update website/docs/docs/cloud/secure/postgres-privatelink.md --- website/docs/docs/cloud/secure/postgres-privatelink.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/secure/postgres-privatelink.md b/website/docs/docs/cloud/secure/postgres-privatelink.md index 68b151fbce7..d823ce1e431 100644 --- a/website/docs/docs/cloud/secure/postgres-privatelink.md +++ b/website/docs/docs/cloud/secure/postgres-privatelink.md @@ -7,7 +7,7 @@ sidebar_label: "PrivateLink for Postgres" A Postgres database, hosted either in AWS or in a properly connected on-prem data center, can be accessed via a private network connection using AWS Interface-type PrivateLink. The type of Target Group connected to the Network Load Balancer (NLB) may vary based on the location and type of Postgres instance being connected, as is noted in the steps below. -## Configuring Postgres Interface-type PrivateLink +## Configuring Postgres interface-type PrivateLink ### 1. Provision AWS Resources From f7ed223dd6ea72cd6591cfd3e1d636996870490e Mon Sep 17 00:00:00 2001 From: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 29 Sep 2023 12:51:56 +0100 Subject: [PATCH 05/10] Update website/docs/docs/cloud/secure/postgres-privatelink.md --- website/docs/docs/cloud/secure/postgres-privatelink.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/secure/postgres-privatelink.md b/website/docs/docs/cloud/secure/postgres-privatelink.md index d823ce1e431..3dd5f0af165 100644 --- a/website/docs/docs/cloud/secure/postgres-privatelink.md +++ b/website/docs/docs/cloud/secure/postgres-privatelink.md @@ -9,7 +9,7 @@ A Postgres database, hosted either in AWS or in a properly connected on-prem dat ## Configuring Postgres interface-type PrivateLink -### 1. Provision AWS Resources +### 1. Provision AWS resources Creating an Interface VPC PrivateLink connection requires creating multiple AWS resources in the account containing, or connected to, the Postgres instance: From 425a7237f91183ffebd86b2d520ec8b4dad048cc Mon Sep 17 00:00:00 2001 From: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 29 Sep 2023 12:52:25 +0100 Subject: [PATCH 06/10] Update website/docs/docs/cloud/secure/postgres-privatelink.md --- website/docs/docs/cloud/secure/postgres-privatelink.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/secure/postgres-privatelink.md b/website/docs/docs/cloud/secure/postgres-privatelink.md index 3dd5f0af165..b6940441800 100644 --- a/website/docs/docs/cloud/secure/postgres-privatelink.md +++ b/website/docs/docs/cloud/secure/postgres-privatelink.md @@ -46,7 +46,7 @@ On the provisioned VPC endpoint service, click the **Allow principals** tab. Cli - Principal: `arn:aws:iam::346425330055:role/MTPL_Admin` (_or_ `arn:aws:iam::346425330055:root`) - + ### 3. Obtain VPC Endpoint Service Name From 4fbf124da07600a7f32911ef5f2c8be0eb824af1 Mon Sep 17 00:00:00 2001 From: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 29 Sep 2023 12:52:42 +0100 Subject: [PATCH 07/10] Update website/docs/docs/cloud/secure/postgres-privatelink.md --- website/docs/docs/cloud/secure/postgres-privatelink.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/secure/postgres-privatelink.md b/website/docs/docs/cloud/secure/postgres-privatelink.md index b6940441800..e888a0c32e7 100644 --- a/website/docs/docs/cloud/secure/postgres-privatelink.md +++ b/website/docs/docs/cloud/secure/postgres-privatelink.md @@ -40,7 +40,7 @@ Creating an Interface VPC PrivateLink connection requires creating multiple AWS - **VPC Endpoint Service** — Attach to the newly created NLB. - Acceptance required (optional) — Requires you to [accept our connection request](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests) after dbt creates the endpoint. -### 2. Grant dbt AWS Account access to the VPC Endpoint Service +### 2. Grant dbt AWS account access to the VPC Endpoint Service On the provisioned VPC endpoint service, click the **Allow principals** tab. Click **Allow principals** to grant access. Enter the ARN of the root user in the appropriate production AWS account and save your changes. From c6d581fe5ed9e6405d2661bda120a76c3814e672 Mon Sep 17 00:00:00 2001 From: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 29 Sep 2023 13:22:45 +0100 Subject: [PATCH 08/10] Update website/docs/docs/cloud/secure/postgres-privatelink.md --- website/docs/docs/cloud/secure/postgres-privatelink.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/secure/postgres-privatelink.md b/website/docs/docs/cloud/secure/postgres-privatelink.md index e888a0c32e7..dcfff74f61d 100644 --- a/website/docs/docs/cloud/secure/postgres-privatelink.md +++ b/website/docs/docs/cloud/secure/postgres-privatelink.md @@ -52,7 +52,7 @@ On the provisioned VPC endpoint service, click the **Allow principals** tab. Cli Once the VPC Endpoint Service is provisioned, you can find the service name in the AWS console by navigating to **VPC** → **Endpoint Services** and selecting the appropriate endpoint service. You can copy the service name field value and include it in your communication to dbt Cloud support. - + ### 4. Add the required information to the template below, and submit your request to [dbt Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support): ``` From d9127bfff093e7a23ae61c6d33e788771aa0542c Mon Sep 17 00:00:00 2001 From: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 29 Sep 2023 13:30:38 +0100 Subject: [PATCH 09/10] Update website/docs/docs/cloud/secure/postgres-privatelink.md --- website/docs/docs/cloud/secure/postgres-privatelink.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/secure/postgres-privatelink.md b/website/docs/docs/cloud/secure/postgres-privatelink.md index dcfff74f61d..455792a0c6f 100644 --- a/website/docs/docs/cloud/secure/postgres-privatelink.md +++ b/website/docs/docs/cloud/secure/postgres-privatelink.md @@ -5,7 +5,7 @@ description: "Configuring PrivateLink for Postgres" sidebar_label: "PrivateLink for Postgres" --- -A Postgres database, hosted either in AWS or in a properly connected on-prem data center, can be accessed via a private network connection using AWS Interface-type PrivateLink. The type of Target Group connected to the Network Load Balancer (NLB) may vary based on the location and type of Postgres instance being connected, as is noted in the steps below. +A Postgres database, hosted either in AWS or in a properly connected on-prem data center, can be accessed through a private network connection using AWS Interface-type PrivateLink. The type of Target Group connected to the Network Load Balancer (NLB) may vary based on the location and type of Postgres instance being connected, as explained in the following steps. ## Configuring Postgres interface-type PrivateLink From e752c37a155dde13325faad8dfe1c30b4d355645 Mon Sep 17 00:00:00 2001 From: Dave Haworth Date: Fri, 29 Sep 2023 10:53:25 -0500 Subject: [PATCH 10/10] remove root principal for privatelink access --- website/docs/docs/cloud/secure/postgres-privatelink.md | 2 +- website/docs/docs/cloud/secure/redshift-privatelink.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/docs/cloud/secure/postgres-privatelink.md b/website/docs/docs/cloud/secure/postgres-privatelink.md index 68b151fbce7..cd2135d0ace 100644 --- a/website/docs/docs/cloud/secure/postgres-privatelink.md +++ b/website/docs/docs/cloud/secure/postgres-privatelink.md @@ -44,7 +44,7 @@ Creating an Interface VPC PrivateLink connection requires creating multiple AWS On the provisioned VPC endpoint service, click the **Allow principals** tab. Click **Allow principals** to grant access. Enter the ARN of the root user in the appropriate production AWS account and save your changes. - - Principal: `arn:aws:iam::346425330055:role/MTPL_Admin` (_or_ `arn:aws:iam::346425330055:root`) + - Principal: `arn:aws:iam::346425330055:role/MTPL_Admin` diff --git a/website/docs/docs/cloud/secure/redshift-privatelink.md b/website/docs/docs/cloud/secure/redshift-privatelink.md index 8f39bf4f2f6..3ed49e7bb34 100644 --- a/website/docs/docs/cloud/secure/redshift-privatelink.md +++ b/website/docs/docs/cloud/secure/redshift-privatelink.md @@ -79,7 +79,7 @@ Creating an Interface VPC PrivateLink connection requires creating multiple AWS On the provisioned VPC endpoint service, click the **Allow principals** tab. Click **Allow principals** to grant access. Enter the ARN of the root user in the appropriate production AWS account and save your changes. - - Principal: `arn:aws:iam::346425330055:role/MTPL_Admin` (_or_ `arn:aws:iam::346425330055:root`) + - Principal: `arn:aws:iam::346425330055:role/MTPL_Admin`