diff --git a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md index ee5b09e83ef..486aa787936 100644 --- a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md +++ b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md @@ -15,7 +15,7 @@ The following fields are required when creating a Postgres, Redshift, or AlloyDB **Note**: When you set up a Redshift or Postgres connection in dbt Cloud, SSL-related parameters aren't available as inputs. - + ### Connecting via an SSH Tunnel @@ -23,7 +23,7 @@ To connect to a Postgres, Redshift, or AlloyDB instance via an SSH tunnel, selec Once the connection is saved, a public key will be generated and displayed for the Connection. You can copy this public key to the bastion server to authorize dbt Cloud to connect to your database via the bastion server. - + #### About the Bastion server in AWS @@ -61,6 +61,37 @@ chmod 600 ~/.ssh/authorized_keys The Bastion server should now be ready for dbt Cloud to use as a tunnel into the Redshift environment. +#### Intermittent connection issues + +
+ Database Error - could not connect to server: Connection timed out + You will have the following components when you configure a connection to a database using an SSH tunnel:
+ - An Elastic Load Balancer (ELB) or Network Load Balancing (NLB) instance.
+ - A bastion host (or jump server) running the sshd process.
+ - A Database (such as Redshift cluster)

+ +dbt Cloud establishes an SSH tunnel connection through the ELB or NLB to the sshd process. This is responsible for routing traffic to the database. When dbt initiates a job run, an SSH tunnel is created at the start of the run. If this SSH tunnel fails at any point, the job will also fail.
+ + The most common causes of tunnel failures are:
+ - The SSH daemon terminates the session due to an idle timeout.
+ - The ELB or NLB terminates the connection when it's idle.
+ +dbt Cloud sets a value for its SSH tunnel called `ServerAliveInterval` and `ServerAliveCountMax` that polls the connection every 30 seconds and the underlying OS in our run "pods" will terminate the connection if the `sshd` process fails to respond after 300s. This will, in many cases, prevent an idle timeout entirely so long as the customer is not using ELB with a firewall-level idle timeout of less than 30 seconds. However, if the customer is using ELB and is using an Idle Connection Timeout of less than 30s, this will be insufficient to prevent tunnels from being terminated.
+ +Some versions of Linux used on bastion hosts use a version of `sshd` with additional idle timeout settings `ClientAliveCountMax`. This value sets the number of client alive messages that may be sent without `sshd` receiving any messages back from the client. If this threshold is reached while client alive messages are being sent, `sshd` will disconnect the client, terminating the session. The client-alive mechanism is helpful when the client or server needs to know when a connection has become inactive. The default value is 3.

+ +`ClientAliveInterval`:
+This value sets a timeout interval in seconds after which if no data has been received from the client, `sshd` will send a message through the encrypted channel to request a response from the client. The default is 0, indicating that these messages will not be sent to the client.
+ +Using default values, tunnels could be terminated prematurely by `sshd`. To solve this problem, the `/etc/ssh/sshd_config` file on the bastion host can be configured with the following values:

+- `ClientAliveCountMax` 10
+- `ClientAliveInterval` 30
+where `ClientAliveCountMax` should be set to a non-zero value and `ClientAliveInterval` should be a value less than the ELB or NLB idle timeout value.
+ +With these settings, unresponsive SSH clients will be disconnected after approximately 300 seconds, helping to prevent tunnel failures. +
+ + ## Configuration To learn how to optimize performance with data platform-specific configurations in dbt Cloud, refer to [Redshift-specific configuration](/reference/resource-configs/redshift-configs). diff --git a/website/docs/docs/core/connect-data-platform/bigquery-setup.md b/website/docs/docs/core/connect-data-platform/bigquery-setup.md index 4169b782594..a630cbb0bd3 100644 --- a/website/docs/docs/core/connect-data-platform/bigquery-setup.md +++ b/website/docs/docs/core/connect-data-platform/bigquery-setup.md @@ -464,7 +464,7 @@ my-profile: role: dev runtime_config: properties: - spark.executor.instances: 3 + spark.executor.instances: "3" spark.driver.memory: 1g ``` diff --git a/website/docs/docs/deploy/job-notifications.md b/website/docs/docs/deploy/job-notifications.md index 8d242abac78..548e34fc2f3 100644 --- a/website/docs/docs/deploy/job-notifications.md +++ b/website/docs/docs/deploy/job-notifications.md @@ -1,33 +1,83 @@ --- title: "Job notifications" id: "job-notifications" -description: "Set up notifications in dbt Cloud to receive Email or Slack alerts for job run status." +description: "Set up notifications in dbt Cloud to receive email or Slack alerts about job run status." --- +Set up notifications in dbt Cloud to receive email or Slack alerts when a job run succeeds, fails, or is cancelled. -Setting up notifications in dbt Cloud will allow you to receive alerts via Email or a chosen Slack channel when a job run succeeds, fails, or is cancelled. +## Email notifications -### Email +You can receive email alerts about jobs by configuring the dbt Cloud email notification settings. -These are the following options for setting up email notifications. Refer to [Users and licenses](/docs/cloud/manage-access/seats-and-users) for info on license types eligible for email notifications. +### Prerequisites +- You must be either a _developer user_ or an _account admin_ to configure email notifications in dbt Cloud. For more details, refer to [Users and licenses](/docs/cloud/manage-access/seats-and-users). + - As a developer user, you can set up email notifications for yourself. + - As an account admin, you can set up notifications for yourself and other team members. -- As a **user** — You can set up email notifications for yourself under your Profile. -- As an **admin** — You can set up notifications on behalf of your team members. +### Configure email notifications -To set up job notifications, follow these steps: +1. From the gear menu, choose **Notification settings**. +1. By default, dbt Cloud sends notifications to the email address that's in your **User profile** page. -1. Click the gear menu in the top right corner and select **Notification Settings**. + If you're an account admin, you can choose a different email address to receive notifications. Select the **Notification email** dropdown and choose another address from the list. The list includes **Internal Users** with access to the account and **External Emails** that have been added. + - To add an external email address, select the **Notification email** dropdown and choose **Add external email**. After you add the external email, it becomes available for selection in the **Notification email** dropdown list. External emails can be addresses that are outside of your dbt Cloud account and also for third-party integrations like [channels in Microsoft Teams](https://support.microsoft.com/en-us/office/tip-send-email-to-a-channel-2c17dbae-acdf-4209-a761-b463bdaaa4ca) and [PagerDuty email integration](https://support.pagerduty.com/docs/email-integration-guide). -2. Select **Edit** to begin editing the **Email Notifications** settings. - - **As a user:** Choose the Notification type (Succeeds, Fails, or Is Cancelled) for each Job you want to receive notifications for. + - - **As an admin:** Under **Configure notifications for**, use the dropdown to select one or more users you'd like to set notifications for. If you only see your own name, then you might not have admin privileges.

- Choose the Notification type (Succeeds, Fails, or Is Cancelled) for each Job you want them to receive notifications for. +1. Select the **Environment** for the jobs you want to receive notifications about from the dropdown. -3. Click **Save**. +1. Click **Edit** to configure the email notification settings. Choose one or more of the run statuses (**Succeeds**, **Fails**, **Is Canceled**) for each job you want to receive notifications about. - +1. When you're done with the settings, click **Save**. -### Slack + As an account admin, you can add more email recipients by choosing another **Notification email** from the dropdown, **Edit** the job notification settings, and **Save** the changes. + + To set up alerts on jobs from a different environment, select another **Environment** from the dropdown, **Edit** those job notification settings, and **Save** the changes. - + + +### Unsubscribe from email notifications +1. From the gear menu, choose **Notification settings**. +1. On the **Email notifications** page, click **Unsubscribe from all email notifications**. + +## Slack notifications + +You can receive Slack alerts about jobs by setting up the Slack integration, then configuring the dbt Cloud Slack notification settings. + +:::note +Any account admin can edit the Slack notifications but they'll be limited to configuring the channels that their Slack user has access to. If there has been a change in user roles or Slack permissions where you no longer have access to edit a configured Slack channel, please [contact support](mailto:support@getdbt.com) for assistance. +::: + +### Prerequisites +- You must be an administrator of the Slack workspace. +- You must be an account admin to configure Slack notifications in dbt Cloud. For more details, refer to [Users and licenses](/docs/cloud/manage-access/seats-and-users). + +### Set up the Slack integration + +1. From the gear menu, select **Profile settings**. On your **User profile** page, scroll to the **Linked accounts** section. +1. In the **Linked accounts** section, find the Slack application and click **Link**. + +1. Allow dbt Labs to access the Slack workspace. If you are a member of multiple workspaces, you can select the appropriate workspace from the dropdown menu in the upper right corner. + + +### Configure Slack notifications + +1. From the gear menu, choose **Notification settings**. +1. Select **Slack notifications** in the left sidebar. +1. Select the **Notification channel** you want to receive the job run notifications from the dropdown. + +1. Select the **Environment** for the jobs you want to receive notifications about from the dropdown. +1. Click **Edit** to configure the Slack notification settings. Choose one or more of the run statuses (**Succeeds**, **Fails**, **Is Canceled**) for each job you want to receive notifications about. +1. When you're done with the settings, click **Save**. + + To send alerts to another Slack channel, select another **Notification channel** from the dropdown, **Edit** those job notification settings, and **Save** the changes. + + To set up alerts on jobs from a different environment, select another **Environment** from the dropdown, **Edit** those job notification settings, and **Save** the changes. + + + +### Disable the Slack integration + +1. From the gear menu, select **Profile settings**. On your **User profile** page, scroll to the **Linked accounts** section. +1. Find the Slack application in the **Linked accounts** section, click the trash can icon, and click **Unlink**. Channels that you configured will no longer receive Slack notifications. Channels that are configured by other account admins will continue to receive Slack notifications if they still have active Slack integrations. To migrate ownership of a Slack channel notification configuration, have another account admin edit the configuration. \ No newline at end of file diff --git a/website/docs/faqs/Accounts/slack.md b/website/docs/faqs/Accounts/slack.md deleted file mode 100644 index 4faa60fb09a..00000000000 --- a/website/docs/faqs/Accounts/slack.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -title: How do I set up Slack notifications? -description: "Instructions on how to set up slack notifications" -sidebar_label: 'How to set up Slack' -id: slack ---- - - diff --git a/website/docs/faqs/Project/docs-for-multiple-projects.md b/website/docs/faqs/Project/docs-for-multiple-projects.md deleted file mode 100644 index b7aa1452b39..00000000000 --- a/website/docs/faqs/Project/docs-for-multiple-projects.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -title: Can I render docs for multiple projects? -description: "Using packages to render docs for multiple projects" -sidebar_label: 'Render docs for multiple projects' -id: docs-for-multiple-projects - ---- - -Yes! To do this, you'll need to create a "super project" that lists each project as a dependent [package](/docs/build/packages) in a `packages.yml` file. Then run `dbt deps` to install the projects as packages, prior to running `dbt docs generate`. - -If you are going down the route of multiple projects, be sure to check out our advice [1](https://discourse.getdbt.com/t/should-i-have-an-organisation-wide-project-a-monorepo-or-should-each-work-flow-have-their-own/666) [2](https://discourse.getdbt.com/t/how-to-configure-your-dbt-repository-one-or-many/2121) on the topic. diff --git a/website/docs/guides/orchestration/webhooks/zapier-ms-teams.md b/website/docs/guides/orchestration/webhooks/zapier-ms-teams.md index bb3f03ef0c0..148e16b2469 100644 --- a/website/docs/guides/orchestration/webhooks/zapier-ms-teams.md +++ b/website/docs/guides/orchestration/webhooks/zapier-ms-teams.md @@ -5,7 +5,7 @@ slug: zapier-ms-teams description: Use Zapier and the dbt Cloud API to post to Microsoft Teams --- -This guide will show you how to set up an integration between dbt Cloud jobs and Microsoft Teams using [dbt Cloud Webhooks](/docs/deploy/webhooks) and Zapier, similar to the [native Slack integration](/faqs/accounts/slack). +This guide will show you how to set up an integration between dbt Cloud jobs and Microsoft Teams using [dbt Cloud Webhooks](/docs/deploy/webhooks) and Zapier, similar to the [native Slack integration](/docs/deploy/job-notifications#slack-notifications). When a dbt Cloud job finishes running, the integration will: diff --git a/website/docs/guides/orchestration/webhooks/zapier-slack.md b/website/docs/guides/orchestration/webhooks/zapier-slack.md index c9046ee9943..6ce89eadd12 100644 --- a/website/docs/guides/orchestration/webhooks/zapier-slack.md +++ b/website/docs/guides/orchestration/webhooks/zapier-slack.md @@ -5,7 +5,7 @@ slug: zapier-slack description: Use Zapier and the dbt Cloud API to post error context to Slack --- -This guide will show you how to set up an integration between dbt Cloud jobs and Slack using [dbt Cloud webhooks](/docs/deploy/webhooks) and Zapier. It builds on the native [native Slack integration](/faqs/accounts/slack) by attaching error message details of models and tests in a thread. +This guide will show you how to set up an integration between dbt Cloud jobs and Slack using [dbt Cloud webhooks](/docs/deploy/webhooks) and Zapier. It builds on the native [native Slack integration](/docs/deploy/job-notifications#slack-notifications) by attaching error message details of models and tests in a thread. Note: Because there is not a webhook for Run Cancelled, you may want to keep the standard Slack integration installed to receive those notifications. You could also use the [alternative integration](#alternate-approach) that augments the native integration without replacing it. diff --git a/website/docs/reference/resource-configs/resource-path.md b/website/docs/reference/resource-configs/resource-path.md index 258b83dcd57..86b04430376 100644 --- a/website/docs/reference/resource-configs/resource-path.md +++ b/website/docs/reference/resource-configs/resource-path.md @@ -1,11 +1,28 @@ -The `` nomenclature is used in this documentation when documenting how to configure a model, seed, or snapshot, from your `dbt_project.yml` file. It represents the nested dictionary keys that provide the path to either a directory of models, or a single model. +--- +title: Resource path +description: "Learn how to use resource paths to configure resource types in dbt." +id: resource-path +sidebar_label: "About resource paths" +--- + +The `` nomenclature is used in this documentation when documenting how to configure resource types like models, seeds, snapshots, tests, sources, and others, from your `dbt_project.yml` file. + +It represents the nested dictionary keys that provide the path to a directory of that resource type, or a single instance of that resource type by name. + +```yml +resource_type: + project_name: + directory_name: + subdirectory_name: + instance_of_resource_type (by name): + ... +``` ## Example -:::info -This example is for models, but the same concepts apply for seeds and snapshots. +The following examples are mostly for models and a source, but the same concepts apply for seeds, snapshots, tests, sources, and other resource types. -::: +### Apply config to all models To apply a configuration to all models, do not use a ``: @@ -18,6 +35,8 @@ models: +### Apply config to all models in your project + To apply a configuration to all models in _your_ project only, use your [project name](/reference/project-configs/name) as the ``: @@ -32,6 +51,8 @@ models: +### Apply config to all models in a subdirectory + To apply a configuration to all models in a subdirectory of your project, e.g. `staging`, nest the directory under the project name: @@ -57,6 +78,8 @@ In the following project, this would apply to models in the `staging/` directory ``` +### Apply config to all models in one model + To apply a configuration to one model, nest the full path under the project name. For a model at `/staging/stripe/payments.sql`, this would look like: @@ -92,3 +115,19 @@ In the following project, this would only apply to the `payments` model:    └── payments.sql ``` +### Apply config source nested in a subfolder + +To disable a source nested in a YAML file in a subfolder, you will need to supply the path to that YAML file, as well as the source name in the `dbt_project.yml` file.

+ The following example shows how to disable a source nested in a YAML file in a subfolder: + + + + ```yaml + sources: + your_project_name: + subdirectory_name: + source_yaml_file_name: + source_name: + +enabled: false # This will apply to sources nested in subfolders. + ``` + diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 4e500ed64ea..9a5d513d99b 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -103,7 +103,7 @@ models: contract: enforced: true columns: - - name: customer_id + - name: id data_type: int constraints: - type: not_null diff --git a/website/docs/reference/source-configs.md b/website/docs/reference/source-configs.md index 3f9a19e78ca..c66f41f1cde 100644 --- a/website/docs/reference/source-configs.md +++ b/website/docs/reference/source-configs.md @@ -61,7 +61,24 @@ sources: ## Configuring sources -Sources can be configured via a `config:` block within their `.yml` definitions, or from the `dbt_project.yml` file under the `sources:` key. This configuration is most useful for configuring sources imported from [a package](/docs/build/packages). You can disable sources imported from a package to prevent them from rendering in the documentation, or to prevent [source freshness checks](/docs/build/sources#snapshotting-source-data-freshness) from running on source tables imported from packages. +Sources can be configured via a `config:` block within their `.yml` definitions, or from the `dbt_project.yml` file under the `sources:` key. This configuration is most useful for configuring sources imported from [a package](/docs/build/packages). + +You can disable sources imported from a package to prevent them from rendering in the documentation, or to prevent [source freshness checks](/docs/build/sources#snapshotting-source-data-freshness) from running on source tables imported from packages. + +- **Note**: To disable a source nested in a YAML file in a subfolder, you will need to supply the path to that YAML file plus the source name in the dbt_project.yml file. For example: + + + + ```yml + sources: + your_project_name: + subdirectory_name: + source_yaml_file_name: + source_name: + +enabled: false # This will apply to sources nested in subfolders. + ``` + + ### Examples diff --git a/website/sidebars.js b/website/sidebars.js index 055dfed0e7d..8920a7180d4 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -716,6 +716,7 @@ const sidebarSettings = { label: "Resource configs and properties", items: [ "reference/configs-and-properties", + "reference/resource-configs/resource-path", { type: "category", label: "General properties", diff --git a/website/snippets/slack-notifications-config-steps.md b/website/snippets/slack-notifications-config-steps.md deleted file mode 100644 index da63f7afcc9..00000000000 --- a/website/snippets/slack-notifications-config-steps.md +++ /dev/null @@ -1,28 +0,0 @@ -Setting up Slack notifications in dbt Cloud enables you to receive alerts in a chosen Slack channel when a job run succeeds, fails, or is cancelled. - -:::info Note -Currently, Slack notifications can only be configured by one user to one Slack channel. Additionally, you must be an admin of the Slack workspace in order to configure Slack notifications. - -If there have been changes to the user roles and you need to move ownership, please reach out to support@getdbt.com and provide the support team with the necessary information needed to make this change for you. -::: -### Setup the integration - -1. Click the gear in the top right and select **Profile**. -2. Click **Integrations** to the left. - -3. Click **Link your Slack profile** - -4. Allow dbt Labs to access the Slack workspace. If you are a member of multiple, you can select the appropriate workspace from the dropdown menu in the top right corner. - - -### Configure the notifications - -1. Click the gear in the top right and select **Account Settings**. -2. Click **Slack Notifications** to the left and click **Edit** to the right. - -3. You can find the Slack notification settings at the bottom of the page. - -### Disabling the Slack integration - -To disable the integration entirely, navigate back to the Integrations page and click **Disconnect Account** in the Slack pane. Confirm the disconnect, and the option will revert to its original state. - diff --git a/website/static/img/docs/dbt-cloud/Navigate-to-integrations.png b/website/static/img/docs/dbt-cloud/Navigate-to-integrations.png deleted file mode 100644 index b1e2789f334..00000000000 Binary files a/website/static/img/docs/dbt-cloud/Navigate-to-integrations.png and /dev/null differ diff --git a/website/static/img/docs/dbt-cloud/Navigate-to-notifications.png b/website/static/img/docs/dbt-cloud/Navigate-to-notifications.png deleted file mode 100644 index 13da17d5deb..00000000000 Binary files a/website/static/img/docs/dbt-cloud/Navigate-to-notifications.png and /dev/null differ diff --git a/website/static/img/docs/dbt-cloud/using-dbt-cloud/email-notifications.png b/website/static/img/docs/dbt-cloud/using-dbt-cloud/email-notifications.png deleted file mode 100644 index 7e77098d63e..00000000000 Binary files a/website/static/img/docs/dbt-cloud/using-dbt-cloud/email-notifications.png and /dev/null differ diff --git a/website/static/img/docs/deploy/example-email-notification-settings-page.png b/website/static/img/docs/deploy/example-email-notification-settings-page.png new file mode 100644 index 00000000000..5d834487a43 Binary files /dev/null and b/website/static/img/docs/deploy/example-email-notification-settings-page.png differ diff --git a/website/static/img/docs/deploy/example-notification-external-email.png b/website/static/img/docs/deploy/example-notification-external-email.png new file mode 100644 index 00000000000..5a2326cccd8 Binary files /dev/null and b/website/static/img/docs/deploy/example-notification-external-email.png differ diff --git a/website/static/img/docs/deploy/example-notification-slack-channels.png b/website/static/img/docs/deploy/example-notification-slack-channels.png new file mode 100644 index 00000000000..ecf4ff818bd Binary files /dev/null and b/website/static/img/docs/deploy/example-notification-slack-channels.png differ diff --git a/website/static/img/docs/deploy/example-slack-notification-settings-page.png b/website/static/img/docs/deploy/example-slack-notification-settings-page.png new file mode 100644 index 00000000000..dc72d291e57 Binary files /dev/null and b/website/static/img/docs/deploy/example-slack-notification-settings-page.png differ diff --git a/website/vercel.json b/website/vercel.json index 81d955e2d3d..606e903a700 100644 --- a/website/vercel.json +++ b/website/vercel.json @@ -2,6 +2,16 @@ "cleanUrls": true, "trailingSlash": false, "redirects": [ + { + "source": "/faqs/Project/docs-for-multiple-projects", + "destination": "/docs/collaborate/explore-projects#about-project-level-lineage", + "permanent": true + }, + { + "source": "/faqs/Accounts/slack", + "destination": "/docs/deploy/job-notifications", + "permanent": true + }, { "source": "/guides/migration/versions", "destination": "/docs/dbt-versions/core-upgrade",