-
Notifications
You must be signed in to change notification settings - Fork 978
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
add troubleshooting snippet and reference it on three pages
- Loading branch information
Showing
5 changed files
with
126 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| [ | ||
| { | ||
| "host": "jd43444.east-us-2.privatelink.snowflakecomputing.com", | ||
| "port": 443, | ||
| "type": "SNOWFLAKE_DEPLOYMENT" | ||
| }, | ||
| { | ||
| "host": "mgb-mgbdev.privatelink.snowflakecomputing.com", | ||
| "port": 443, | ||
| "type": "SNOWFLAKE_DEPLOYMENT_REGIONLESS" | ||
| }, | ||
| { | ||
| "host": "r3iqwjsfcb1stg.blob.core.windows.net", | ||
| "port": 443, | ||
| "type": "STAGE" | ||
| }, | ||
| { | ||
| "host": "sfc-repo.snowflakecomputing.com", | ||
| "port": 443, | ||
| "type": "SNOWSQL_REPO" | ||
| }, | ||
| { | ||
| "host": "client-telemetry.snowflakecomputing.com", | ||
| "port": 443, | ||
| "type": "OUT_OF_BAND_TELEMETRY" | ||
| }, | ||
| { | ||
| "host": "app-mgb-mgbdevenv.privatelink.snowflakecomputing.com", | ||
| "port": 443, | ||
| "type": "SNOWSIGHT_CLIENT_FAILOVER" | ||
| }, | ||
| { | ||
| "host": "mgb-mgbdevenv.privatelink.snowflakecomputing.com", | ||
| "port": 443, | ||
| "type": "CLIENT_FAILOVER" | ||
| }, | ||
| { | ||
| "host": "ocsp.mgb-mgbdevenv.privatelink.snowflakecomputing.com", | ||
| "port": 80, | ||
| "type": "OCSP_CLIENT_FAILOVER" | ||
| }, | ||
| { | ||
| "host": "ocsp.app-mgb-mgbdevenv.privatelink.snowflakecomputing.com", | ||
| "port": 80, | ||
| "type": "OCSP_CLIENT_FAILOVER" | ||
| }, | ||
| { | ||
| "host": "ocsp.jd43444.east-us-2.privatelink.snowflakecomputing.com", | ||
| "port": 80, | ||
| "type": "OCSP_CACHE" | ||
| }, | ||
| { | ||
| "host": "ocsp.mgb-mgbdev.privatelink.snowflakecomputing.com", | ||
| "port": 80, | ||
| "type": "OCSP_CACHE_REGIONLESS" | ||
| }, | ||
| { | ||
| "host": "api-73a252bb.duosecurity.com", | ||
| "port": 443, | ||
| "type": "DUO_SECURITY" | ||
| }, | ||
| { | ||
| "host": "app-mgb-mgbdev.privatelink.snowflakecomputing.com", | ||
| "port": 443, | ||
| "type": "SNOWSIGHT_DEPLOYMENT_REGIONLESS" | ||
| }, | ||
| { | ||
| "host": "app.east-us-2.privatelink.snowflakecomputing.com", | ||
| "port": 443, | ||
| "type": "SNOWSIGHT_DEPLOYMENT" | ||
| } | ||
| ] | ||
|
|
||
| # Current | ||
| { | ||
| "regionless-snowsight-privatelink-url": "app-mgb-mgbdev.privatelink.snowflakecomputing.com", | ||
| "privatelink-account-name": "jd43444.east-us-2.privatelink", | ||
| "snowsight-privatelink-url": "app.east-us-2.privatelink.snowflakecomputing.com", | ||
| "regionless-privatelink-ocsp-url": "ocsp.mgb-mgbdev.privatelink.snowflakecomputing.com", | ||
| "privatelink-account-url": "jd43444.east-us-2.privatelink.snowflakecomputing.com", | ||
| "privatelink-connection-ocsp-urls": "[ocsp.mgb-mgbdevenv.privatelink.snowflakecomputing.com, ocsp.app-mgb-mgbdevenv.privatelink.snowflakecomputing.com]", | ||
| "privatelink-pls-id": "sf-pvlinksvc-azeastus2.cf82bce2-be9d-4dc6-92ee-3de5fb04d191.eastus2.azure.privatelinkservice", | ||
| "regionless-privatelink-account-url": "mgb-mgbdev.privatelink.snowflakecomputing.com", | ||
| "privatelink_ocsp-url": "ocsp.jd43444.east-us-2.privatelink.snowflakecomputing.com", | ||
| "privatelink-connection-urls": "[mgb-mgbdevenv.privatelink.snowflakecomputing.com, app-mgb-mgbdevenv.privatelink.snowflakecomputing.com]" | ||
| } | ||
|
|
||
| # Received from customer 02/2023 | ||
| { | ||
| "regionless-snowsight-privatelink-url": "app-mgb-mgbdev.privatelink.snowflakecomputing.com", | ||
| "privatelink-account-name": "jd43444.east-us-2.privatelink", | ||
| "snowsight-privatelink-url": "app.east-us-2.privatelink.snowflakecomputing.com", | ||
| "privatelink-account-url": "jd43444.east-us-2.privatelink.snowflakecomputing.com", | ||
| "privatelink-pls-id": "sf-pvlinksvc-azeastus2.cf82bce2-be9d-4dc6-92ee-3de5fb04d191.eastus2.azure.privatelinkservice", | ||
| "regionless-privatelink-account-url": "mgb-mgbdev.privatelink.snowflakecomputing.com", | ||
| "privatelink_ocsp-url": "ocsp.jd43444.east-us-2.privatelink.snowflakecomputing.com", | ||
| "privatelink-connection-urls": "[CLIENT_FAILOVER,mgb-mgbdevenv.privatelink.snowflakecomputing.com]" | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| ## Troubleshooting | ||
|
|
||
| If the PrivateLink endpoint has been provisioned and configured in dbt Cloud, but connectivity is still failing, check the following in your networking setup to ensure requests and responses can be successfully routed between dbt Cloud and the backing service. | ||
|
|
||
| ### Configuration | ||
|
|
||
| 1. **NLB Security Group** - The Network Load Balancer (NLB) associated with the VPC Endpoint Service must either not have an associated Security Group, or the Security Group must have a rule that allows requests from the appropriate dbt Cloud _private CIDR(s)_. Note that this is different than the static public IPs listed on the dbt Cloud Connection page. The correct private CIDR(s) can be provided by dbt Support upon request. | ||
| - NOTE: To test if this is the issue, temporarily adding an allow rule of `10.0.0.0/8` should allow connectivity until the rule can be refined to a smaller CIDR | ||
| 2. **NLB Listener and Target Group** - Check that there is a Listener connected to the NLB that matches the port that dbt Cloud is trying to connect to. This Listener must have a configured action to forward to a Target Group with targets that point to your backing service. At least one (but preferrably all) of these targets must be **Healthy**. Unhealthy targets could suggest that the backing service is in fact unhealthy, or that the service is protected by a Security Group that doesn't allow requests from the NLB. | ||
| 3. **Cross-zone Load Balancing** - Check that _Cross-zone load balancing_ is enabled for your NLB (check the **Attributes** tab of the NLB in the AWS console). If this is disabled, and the zones that dbt Cloud is connected to are misaligned with the zones where the service is running, requests may not be able to be routed correctly. Enabling cross-zone load balancing will also make the connection more resilient in the case of a failover in a zone outage scenario. Note that cross-zone connectivity may incur additional data transfer charges, though this should be minimal for requests from dbt Cloud. | ||
| 4. **Routing tables and ACLs** - If all the above check out, it may be possible that requests are not routing correctly within the private network. This could be due to a misconfiguration in the VPCs routing tables or access control lists. Review these settings with your network administrator to ensure that requests can be routed from the VPC Endpoint Service to the backing service, and that the response can return to the VPC Endpoint Service. One way to test this is to create a VPC endpoint in another VPC in your own network to test that connectivity is working independent of dbt's connection. | ||
|
|
||
| ### Monitoring | ||
|
|
||
| To help isolate connection issues over a PrivateLink connection from dbt Cloud there are a few monitoring sources that can be used to verify request activity. To see anything in the monitoring there will first have to be requests sent to the endpoint. Contact dbt Support to understand when connection testing occurred or request new connection attempts. Use these times to correlate with activity in the following monitoring sources. | ||
|
|
||
| - **VPC Endpoint Service Monitoring** - In the AWS Console, navigate to VPC -> Endpoint Services. Select the Endpoint Service being tested and click the **Monitoring** tab. Update the time selection to include the times when test connection attempts were sent. If there is activity in the _New connections_ and _Bytes processed_ graphs then requests have been received by the Endpoint Service, suggesting that the dbt endpoint is routing properly. | ||
| - **NLB Monitoring** - In the AWS Console, navigate to EC2 -> Load Balancers. Select the Network Load Balancer (NLB) being tested and click the **Monitoring** tab. Update the time selection to include the times when test connection attempts were sent. If there is activity in the _New flow count_ and _Processed bytes_ graphs then requests have been received by the NLB from the Endpoint Service, suggesting the NLB Listener, Target Group, and Security Group are correctly configured. | ||
| - **VPC Flow Logs** - VPC Flow Logs can provide a variety of useful information for requests that are being routed through your VPCs, though they can sometimes be difficult to locate and interpret. Flow logs can be written to either S3 or CloudWatch Logs, so determine the availability of these logs for your VPC and query them accordingly. Flow logs record the Elastic Network Interface (ENI) ID, source and destination IP and port, and whether the request was accepted or rejected by the security group and/or network ACL. This can be useful in understanding if a request arrived at a certain network interface and whether that request was accepted, potentially illuminating overly restrictive rules. For more on accessing and interpreting VPC Flow Logs see the related [AWS documentation](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html). |