Skip to content

Commit

Permalink
Merge branch 'current' into updated-billing-page
Browse files Browse the repository at this point in the history
  • Loading branch information
matthewshaver authored Nov 7, 2023
2 parents bf4259e + 507178d commit 6988358
Show file tree
Hide file tree
Showing 20 changed files with 175 additions and 74 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -15,15 +15,15 @@ The following fields are required when creating a Postgres, Redshift, or AlloyDB

**Note**: When you set up a Redshift or Postgres connection in dbt Cloud, SSL-related parameters aren't available as inputs.

<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/postgres-redshift-connection.png" title="Configuring a Redshift connection"/>
<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/postgres-redshift-connection.png" width="70%" title="Configuring a Redshift connection"/>

### Connecting via an SSH Tunnel

To connect to a Postgres, Redshift, or AlloyDB instance via an SSH tunnel, select the **Use SSH Tunnel** option when creating your connection. When configuring the tunnel, you must supply the hostname, username, and port for the [bastion server](#about-the-bastion-server-in-aws).

Once the connection is saved, a public key will be generated and displayed for the Connection. You can copy this public key to the bastion server to authorize dbt Cloud to connect to your database via the bastion server.

<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/postgres-redshift-ssh-tunnel.png" title="A generated public key for a Redshift connection"/>
<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/postgres-redshift-ssh-tunnel.png" width="70%" title="A generated public key for a Redshift connection"/>

#### About the Bastion server in AWS

Expand Down Expand Up @@ -61,6 +61,37 @@ chmod 600 ~/.ssh/authorized_keys

The Bastion server should now be ready for dbt Cloud to use as a tunnel into the Redshift environment.

#### Intermittent connection issues

<details>
<summary>Database Error - could not connect to server: Connection timed out</summary>
You will have the following components when you configure a connection to a database using an SSH tunnel: <br />
- An Elastic Load Balancer (ELB) or Network Load Balancing (NLB) instance.<br />
- A bastion host (or jump server) running the <code>sshd</code> process.<br />
- A Database (such as Redshift cluster)<br /><br />

dbt Cloud establishes an SSH tunnel connection through the ELB or NLB to the <code>sshd</code> process. This is responsible for routing traffic to the database. When dbt initiates a job run, an SSH tunnel is created at the start of the run. If this SSH tunnel fails at any point, the job will also fail.<br />

The most common causes of tunnel failures are:<br />
- The SSH daemon terminates the session due to an idle timeout.<br />
- The ELB or NLB terminates the connection when it's idle.<br />

dbt Cloud sets a value for its SSH tunnel called `ServerAliveInterval` and `ServerAliveCountMax` that polls the connection every 30 seconds and the underlying OS in our run "pods" will terminate the connection if the `sshd` process fails to respond after 300s. This will, in many cases, prevent an idle timeout entirely so long as the customer is not using ELB with a firewall-level idle timeout of less than 30 seconds. However, if the customer is using ELB and is using an Idle Connection Timeout of less than 30s, this will be insufficient to prevent tunnels from being terminated. <br />

Some versions of Linux used on bastion hosts use a version of `sshd` with additional idle timeout settings `ClientAliveCountMax`. This value sets the number of client alive messages that may be sent without `sshd` receiving any messages back from the client. If this threshold is reached while client alive messages are being sent, `sshd` will disconnect the client, terminating the session. The client-alive mechanism is helpful when the client or server needs to know when a connection has become inactive. The default value is 3.<br /><br />

`ClientAliveInterval`:<br />
This value sets a timeout interval in seconds after which if no data has been received from the client, `sshd` will send a message through the encrypted channel to request a response from the client. The default is 0, indicating that these messages will not be sent to the client.<br />

Using default values, tunnels could be terminated prematurely by `sshd`. To solve this problem, the `/etc/ssh/sshd_config` file on the bastion host can be configured with the following values:<br /><br />
- `ClientAliveCountMax` 10<br />
- `ClientAliveInterval` 30<br />
where `ClientAliveCountMax` should be set to a non-zero value and `ClientAliveInterval` should be a value less than the ELB or NLB idle timeout value.<br />

With these settings, unresponsive SSH clients will be disconnected after approximately 300 seconds, helping to prevent tunnel failures.
</details>


## Configuration

To learn how to optimize performance with data platform-specific configurations in dbt Cloud, refer to [Redshift-specific configuration](/reference/resource-configs/redshift-configs).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -464,7 +464,7 @@ my-profile:
role: dev
runtime_config:
properties:
spark.executor.instances: 3
spark.executor.instances: "3"
spark.driver.memory: 1g
```

Expand Down
82 changes: 66 additions & 16 deletions website/docs/docs/deploy/job-notifications.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,83 @@
---
title: "Job notifications"
id: "job-notifications"
description: "Set up notifications in dbt Cloud to receive Email or Slack alerts for job run status."
description: "Set up notifications in dbt Cloud to receive email or Slack alerts about job run status."
---

Set up notifications in dbt Cloud to receive email or Slack alerts when a job run succeeds, fails, or is cancelled.

Setting up notifications in dbt Cloud will allow you to receive alerts via Email or a chosen Slack channel when a job run succeeds, fails, or is cancelled.
## Email notifications

### Email
You can receive email alerts about jobs by configuring the dbt Cloud email notification settings.

These are the following options for setting up email notifications. Refer to [Users and licenses](/docs/cloud/manage-access/seats-and-users) for info on license types eligible for email notifications.
### Prerequisites
- You must be either a _developer user_ or an _account admin_ to configure email notifications in dbt Cloud. For more details, refer to [Users and licenses](/docs/cloud/manage-access/seats-and-users).
- As a developer user, you can set up email notifications for yourself.
- As an account admin, you can set up notifications for yourself and other team members.

- As a **user** &mdash; You can set up email notifications for yourself under your Profile.
- As an **admin** &mdash; You can set up notifications on behalf of your team members.
### Configure email notifications

To set up job notifications, follow these steps:
1. From the gear menu, choose **Notification settings**.
1. By default, dbt Cloud sends notifications to the email address that's in your **User profile** page.

1. Click the gear menu in the top right corner and select **Notification Settings**.
If you're an account admin, you can choose a different email address to receive notifications. Select the **Notification email** dropdown and choose another address from the list. The list includes **Internal Users** with access to the account and **External Emails** that have been added.
- To add an external email address, select the **Notification email** dropdown and choose **Add external email**. After you add the external email, it becomes available for selection in the **Notification email** dropdown list. External emails can be addresses that are outside of your dbt Cloud account and also for third-party integrations like [channels in Microsoft Teams](https://support.microsoft.com/en-us/office/tip-send-email-to-a-channel-2c17dbae-acdf-4209-a761-b463bdaaa4ca) and [PagerDuty email integration](https://support.pagerduty.com/docs/email-integration-guide).

2. Select **Edit** to begin editing the **Email Notifications** settings.
- **As a user:** Choose the Notification type (Succeeds, Fails, or Is Cancelled) for each Job you want to receive notifications for.
<Lightbox src="/img/docs/deploy/example-notification-external-email.png" width="50%" title="Example of the Notification email dropdown"/>

- **As an admin:** Under **Configure notifications for**, use the dropdown to select one or more users you'd like to set notifications for. If you only see your own name, then you might not have admin privileges. <br /><br />
Choose the Notification type (Succeeds, Fails, or Is Cancelled) for each Job you want them to receive notifications for.
1. Select the **Environment** for the jobs you want to receive notifications about from the dropdown.

3. Click **Save**.
1. Click **Edit** to configure the email notification settings. Choose one or more of the run statuses (**Succeeds**, **Fails**, **Is Canceled**) for each job you want to receive notifications about.

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/email-notifications.png" width="75%" title="Configuring Email Notifications"/>
1. When you're done with the settings, click **Save**.

### Slack
As an account admin, you can add more email recipients by choosing another **Notification email** from the dropdown, **Edit** the job notification settings, and **Save** the changes.

To set up alerts on jobs from a different environment, select another **Environment** from the dropdown, **Edit** those job notification settings, and **Save** the changes.

<Snippet path="slack-notifications-config-steps" />
<Lightbox src="/img/docs/deploy/example-email-notification-settings-page.png" width="100%" title="Example of the Email notifications page"/>

### Unsubscribe from email notifications
1. From the gear menu, choose **Notification settings**.
1. On the **Email notifications** page, click **Unsubscribe from all email notifications**.

## Slack notifications

You can receive Slack alerts about jobs by setting up the Slack integration, then configuring the dbt Cloud Slack notification settings.

:::note
Any account admin can edit the Slack notifications but they'll be limited to configuring the channels that their Slack user has access to. If there has been a change in user roles or Slack permissions where you no longer have access to edit a configured Slack channel, please [contact support](mailto:[email protected]) for assistance.
:::

### Prerequisites
- You must be an administrator of the Slack workspace.
- You must be an account admin to configure Slack notifications in dbt Cloud. For more details, refer to [Users and licenses](/docs/cloud/manage-access/seats-and-users).

### Set up the Slack integration

1. From the gear menu, select **Profile settings**. On your **User profile** page, scroll to the **Linked accounts** section.
1. In the **Linked accounts** section, find the Slack application and click **Link**.
<Lightbox src="/img/docs/dbt-cloud/Link-your-Slack-Profile.png" width="75%" title="Link for the Slack app"/>
1. Allow dbt Labs to access the Slack workspace. If you are a member of multiple workspaces, you can select the appropriate workspace from the dropdown menu in the upper right corner.
<Lightbox src="/img/docs/dbt-cloud/Allow-dbt-to-access-slack.png" width="75%" title="Allow dbt access to Slack"/>

### Configure Slack notifications

1. From the gear menu, choose **Notification settings**.
1. Select **Slack notifications** in the left sidebar.
1. Select the **Notification channel** you want to receive the job run notifications from the dropdown.
<Lightbox src="/img/docs/deploy/example-notification-slack-channels.png" width="75%" title="Example of the Notification channel dropdown"/>
1. Select the **Environment** for the jobs you want to receive notifications about from the dropdown.
1. Click **Edit** to configure the Slack notification settings. Choose one or more of the run statuses (**Succeeds**, **Fails**, **Is Canceled**) for each job you want to receive notifications about.
1. When you're done with the settings, click **Save**.

To send alerts to another Slack channel, select another **Notification channel** from the dropdown, **Edit** those job notification settings, and **Save** the changes.

To set up alerts on jobs from a different environment, select another **Environment** from the dropdown, **Edit** those job notification settings, and **Save** the changes.

<Lightbox src="/img/docs/deploy/example-slack-notification-settings-page.png" width="100%" title="Example of the Slack notifications page"/>

### Disable the Slack integration

1. From the gear menu, select **Profile settings**. On your **User profile** page, scroll to the **Linked accounts** section.
1. Find the Slack application in the **Linked accounts** section, click the trash can icon, and click **Unlink**. Channels that you configured will no longer receive Slack notifications. Channels that are configured by other account admins will continue to receive Slack notifications if they still have active Slack integrations. To migrate ownership of a Slack channel notification configuration, have another account admin edit the configuration.
8 changes: 0 additions & 8 deletions website/docs/faqs/Accounts/slack.md

This file was deleted.

11 changes: 0 additions & 11 deletions website/docs/faqs/Project/docs-for-multiple-projects.md

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ slug: zapier-ms-teams
description: Use Zapier and the dbt Cloud API to post to Microsoft Teams
---

This guide will show you how to set up an integration between dbt Cloud jobs and Microsoft Teams using [dbt Cloud Webhooks](/docs/deploy/webhooks) and Zapier, similar to the [native Slack integration](/faqs/accounts/slack).
This guide will show you how to set up an integration between dbt Cloud jobs and Microsoft Teams using [dbt Cloud Webhooks](/docs/deploy/webhooks) and Zapier, similar to the [native Slack integration](/docs/deploy/job-notifications#slack-notifications).

When a dbt Cloud job finishes running, the integration will:

Expand Down
2 changes: 1 addition & 1 deletion website/docs/guides/orchestration/webhooks/zapier-slack.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ slug: zapier-slack
description: Use Zapier and the dbt Cloud API to post error context to Slack
---

This guide will show you how to set up an integration between dbt Cloud jobs and Slack using [dbt Cloud webhooks](/docs/deploy/webhooks) and Zapier. It builds on the native [native Slack integration](/faqs/accounts/slack) by attaching error message details of models and tests in a thread.
This guide will show you how to set up an integration between dbt Cloud jobs and Slack using [dbt Cloud webhooks](/docs/deploy/webhooks) and Zapier. It builds on the native [native Slack integration](/docs/deploy/job-notifications#slack-notifications) by attaching error message details of models and tests in a thread.

Note: Because there is not a webhook for Run Cancelled, you may want to keep the standard Slack integration installed to receive those notifications. You could also use the [alternative integration](#alternate-approach) that augments the native integration without replacing it.

Expand Down
47 changes: 43 additions & 4 deletions website/docs/reference/resource-configs/resource-path.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,28 @@
The `<resource-path>` nomenclature is used in this documentation when documenting how to configure a model, seed, or snapshot, from your `dbt_project.yml` file. It represents the nested dictionary keys that provide the path to either a directory of models, or a single model.
---
title: Resource path
description: "Learn how to use resource paths to configure resource types in dbt."
id: resource-path
sidebar_label: "About resource paths"
---

The `<resource-path>` nomenclature is used in this documentation when documenting how to configure resource types like models, seeds, snapshots, tests, sources, and others, from your `dbt_project.yml` file.

It represents the nested dictionary keys that provide the path to a directory of that resource type, or a single instance of that resource type by name.

```yml
resource_type:
project_name:
directory_name:
subdirectory_name:
instance_of_resource_type (by name):
...
```
## Example
:::info
This example is for models, but the same concepts apply for seeds and snapshots.
The following examples are mostly for models and a source, but the same concepts apply for seeds, snapshots, tests, sources, and other resource types.
:::
### Apply config to all models
To apply a configuration to all models, do not use a `<resource-path>`:

Expand All @@ -18,6 +35,8 @@ models:

</File>

### Apply config to all models in your project

To apply a configuration to all models in _your_ project only, use your [project name](/reference/project-configs/name) as the `<resource-path>`:

<File name='dbt_project.yml'>
Expand All @@ -32,6 +51,8 @@ models:

</File>

### Apply config to all models in a subdirectory

To apply a configuration to all models in a subdirectory of your project, e.g. `staging`, nest the directory under the project name:

<File name='dbt_project.yml'>
Expand All @@ -57,6 +78,8 @@ In the following project, this would apply to models in the `staging/` directory
```

### Apply config to all models in one model

To apply a configuration to one model, nest the full path under the project name. For a model at `/staging/stripe/payments.sql`, this would look like:

<File name='dbt_project.yml'>
Expand Down Expand Up @@ -92,3 +115,19 @@ In the following project, this would only apply to the `payments` model:
   └── payments.sql

```
### Apply config source nested in a subfolder
To disable a source nested in a YAML file in a subfolder, you will need to supply the path to that YAML file, as well as the source name in the `dbt_project.yml` file.<br /><br />
The following example shows how to disable a source nested in a YAML file in a subfolder:
<File name='dbt_project.yml'>
```yaml
sources:
your_project_name:
subdirectory_name:
source_yaml_file_name:
source_name:
+enabled: false # This will apply to sources nested in subfolders.
```
</File>
2 changes: 1 addition & 1 deletion website/docs/reference/resource-properties/constraints.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ models:
contract:
enforced: true
columns:
- name: customer_id
- name: id
data_type: int
constraints:
- type: not_null
Expand Down
Loading

0 comments on commit 6988358

Please sign in to comment.