Skip to content

Commit

Permalink
Merge branch 'current' into patkearns10-redshift-client-alive-parameters
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Nov 7, 2023
2 parents 010113b + a26e37b commit 5c1fb7c
Show file tree
Hide file tree
Showing 30 changed files with 212 additions and 76 deletions.
33 changes: 33 additions & 0 deletions .github/workflows/crawler.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
name: Algolia Crawler
on:
pull_request:
types:
- closed

jobs:
algolia_recrawl:
# Comment out the if check below if running on every merge to current branch
if: |
contains(github.event.pull_request.labels.*.name, 'trigger-crawl')
&& github.event.pull_request.merged == true
name: Trigger Algolia Crawl
runs-on: ubuntu-latest
steps:
# Checkout repo
- name: Checkout Repo
uses: actions/checkout@v3

# Wait 8 minutes to allow Vercel build to complete
- run: sleep 480

# Once deploy URL is found, trigger Algolia crawl
- name: Run Algolia Crawler
uses: algolia/algoliasearch-crawler-github-actions@v1
id: crawler_push
with:
crawler-user-id: ${{ secrets.CRAWLER_USER_ID }}
crawler-api-key: ${{ secrets.CRAWLER_API_KEY }}
algolia-app-id: ${{ secrets.ALGOLIA_APP_ID }}
algolia-api-key: ${{ secrets.ALGOLIA_API_KEY }}
site-url: 'https://docs.getdbt.com'
crawler-name: ${{ secrets.CRAWLER_NAME }}
4 changes: 4 additions & 0 deletions website/docs/docs/cloud/secure/about-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ description: "Configuring PrivateLink for AWS"
sidebar_label: "About PrivateLink"
---

import SetUpPages from '/snippets/_available-tiers-privatelink.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

PrivateLink enables a private connection from any dbt Cloud Multi-Tenant environment to your data platform hosted on AWS using [AWS PrivateLink](https://aws.amazon.com/privatelink/) technology. PrivateLink allows dbt Cloud customers to meet security and compliance controls as it allows connectivity between dbt Cloud and your data platform without traversing the public internet. This feature is supported in most regions across NA, Europe, and Asia, but [contact us](https://www.getdbt.com/contact/) if you have questions about availability.

### Cross-region PrivateLink
Expand Down
4 changes: 4 additions & 0 deletions website/docs/docs/cloud/secure/databricks-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ sidebar_label: "PrivateLink for Databricks"
pagination_next: null
---

import SetUpPages from '/snippets/_available-tiers-privatelink.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

The following steps will walk you through the setup of a Databricks AWS PrivateLink endpoint in the dbt Cloud multi-tenant environment.

## Configure PrivateLink
Expand Down
3 changes: 3 additions & 0 deletions website/docs/docs/cloud/secure/postgres-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ id: postgres-privatelink
description: "Configuring PrivateLink for Postgres"
sidebar_label: "PrivateLink for Postgres"
---
import SetUpPages from '/snippets/_available-tiers-privatelink.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

A Postgres database, hosted either in AWS or in a properly connected on-prem data center, can be accessed through a private network connection using AWS Interface-type PrivateLink. The type of Target Group connected to the Network Load Balancer (NLB) may vary based on the location and type of Postgres instance being connected, as explained in the following steps.

Expand Down
4 changes: 4 additions & 0 deletions website/docs/docs/cloud/secure/redshift-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ description: "Configuring PrivateLink for Redshift"
sidebar_label: "PrivateLink for Redshift"
---

import SetUpPages from '/snippets/_available-tiers-privatelink.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

AWS provides two different ways to create a PrivateLink VPC endpoint for a Redshift cluster that is running in another VPC:
- [Redshift-managed PrivateLink Endpoints](https://docs.aws.amazon.com/redshift/latest/mgmt/managing-cluster-cross-vpc.html)
- [Redshift Interface-type PrivateLink Endpoints](https://docs.aws.amazon.com/redshift/latest/mgmt/security-private-link.html)
Expand Down
4 changes: 4 additions & 0 deletions website/docs/docs/cloud/secure/snowflake-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ description: "Configuring PrivateLink for Snowflake"
sidebar_label: "PrivateLink for Snowflake"
---

import SetUpPages from '/snippets/_available-tiers-privatelink.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

The following steps will walk you through the setup of a Snowflake AWS PrivateLink endpoint in the dbt Cloud multi-tenant environment.

:::note Snowflake SSO with PrivateLink
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -464,7 +464,7 @@ my-profile:
role: dev
runtime_config:
properties:
spark.executor.instances: 3
spark.executor.instances: "3"
spark.driver.memory: 1g
```

Expand Down
82 changes: 66 additions & 16 deletions website/docs/docs/deploy/job-notifications.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,83 @@
---
title: "Job notifications"
id: "job-notifications"
description: "Set up notifications in dbt Cloud to receive Email or Slack alerts for job run status."
description: "Set up notifications in dbt Cloud to receive email or Slack alerts about job run status."
---

Set up notifications in dbt Cloud to receive email or Slack alerts when a job run succeeds, fails, or is cancelled.

Setting up notifications in dbt Cloud will allow you to receive alerts via Email or a chosen Slack channel when a job run succeeds, fails, or is cancelled.
## Email notifications

### Email
You can receive email alerts about jobs by configuring the dbt Cloud email notification settings.

These are the following options for setting up email notifications. Refer to [Users and licenses](/docs/cloud/manage-access/seats-and-users) for info on license types eligible for email notifications.
### Prerequisites
- You must be either a _developer user_ or an _account admin_ to configure email notifications in dbt Cloud. For more details, refer to [Users and licenses](/docs/cloud/manage-access/seats-and-users).
- As a developer user, you can set up email notifications for yourself.
- As an account admin, you can set up notifications for yourself and other team members.

- As a **user** &mdash; You can set up email notifications for yourself under your Profile.
- As an **admin** &mdash; You can set up notifications on behalf of your team members.
### Configure email notifications

To set up job notifications, follow these steps:
1. From the gear menu, choose **Notification settings**.
1. By default, dbt Cloud sends notifications to the email address that's in your **User profile** page.

1. Click the gear menu in the top right corner and select **Notification Settings**.
If you're an account admin, you can choose a different email address to receive notifications. Select the **Notification email** dropdown and choose another address from the list. The list includes **Internal Users** with access to the account and **External Emails** that have been added.
- To add an external email address, select the **Notification email** dropdown and choose **Add external email**. After you add the external email, it becomes available for selection in the **Notification email** dropdown list. External emails can be addresses that are outside of your dbt Cloud account and also for third-party integrations like [channels in Microsoft Teams](https://support.microsoft.com/en-us/office/tip-send-email-to-a-channel-2c17dbae-acdf-4209-a761-b463bdaaa4ca) and [PagerDuty email integration](https://support.pagerduty.com/docs/email-integration-guide).

2. Select **Edit** to begin editing the **Email Notifications** settings.
- **As a user:** Choose the Notification type (Succeeds, Fails, or Is Cancelled) for each Job you want to receive notifications for.
<Lightbox src="/img/docs/deploy/example-notification-external-email.png" width="50%" title="Example of the Notification email dropdown"/>

- **As an admin:** Under **Configure notifications for**, use the dropdown to select one or more users you'd like to set notifications for. If you only see your own name, then you might not have admin privileges. <br /><br />
Choose the Notification type (Succeeds, Fails, or Is Cancelled) for each Job you want them to receive notifications for.
1. Select the **Environment** for the jobs you want to receive notifications about from the dropdown.

3. Click **Save**.
1. Click **Edit** to configure the email notification settings. Choose one or more of the run statuses (**Succeeds**, **Fails**, **Is Canceled**) for each job you want to receive notifications about.

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/email-notifications.png" width="75%" title="Configuring Email Notifications"/>
1. When you're done with the settings, click **Save**.

### Slack
As an account admin, you can add more email recipients by choosing another **Notification email** from the dropdown, **Edit** the job notification settings, and **Save** the changes.

To set up alerts on jobs from a different environment, select another **Environment** from the dropdown, **Edit** those job notification settings, and **Save** the changes.

<Snippet path="slack-notifications-config-steps" />
<Lightbox src="/img/docs/deploy/example-email-notification-settings-page.png" width="100%" title="Example of the Email notifications page"/>

### Unsubscribe from email notifications
1. From the gear menu, choose **Notification settings**.
1. On the **Email notifications** page, click **Unsubscribe from all email notifications**.

## Slack notifications

You can receive Slack alerts about jobs by setting up the Slack integration, then configuring the dbt Cloud Slack notification settings.

:::note
Any account admin can edit the Slack notifications but they'll be limited to configuring the channels that their Slack user has access to. If there has been a change in user roles or Slack permissions where you no longer have access to edit a configured Slack channel, please [contact support](mailto:[email protected]) for assistance.
:::

### Prerequisites
- You must be an administrator of the Slack workspace.
- You must be an account admin to configure Slack notifications in dbt Cloud. For more details, refer to [Users and licenses](/docs/cloud/manage-access/seats-and-users).

### Set up the Slack integration

1. From the gear menu, select **Profile settings**. On your **User profile** page, scroll to the **Linked accounts** section.
1. In the **Linked accounts** section, find the Slack application and click **Link**.
<Lightbox src="/img/docs/dbt-cloud/Link-your-Slack-Profile.png" width="75%" title="Link for the Slack app"/>
1. Allow dbt Labs to access the Slack workspace. If you are a member of multiple workspaces, you can select the appropriate workspace from the dropdown menu in the upper right corner.
<Lightbox src="/img/docs/dbt-cloud/Allow-dbt-to-access-slack.png" width="75%" title="Allow dbt access to Slack"/>

### Configure Slack notifications

1. From the gear menu, choose **Notification settings**.
1. Select **Slack notifications** in the left sidebar.
1. Select the **Notification channel** you want to receive the job run notifications from the dropdown.
<Lightbox src="/img/docs/deploy/example-notification-slack-channels.png" width="75%" title="Example of the Notification channel dropdown"/>
1. Select the **Environment** for the jobs you want to receive notifications about from the dropdown.
1. Click **Edit** to configure the Slack notification settings. Choose one or more of the run statuses (**Succeeds**, **Fails**, **Is Canceled**) for each job you want to receive notifications about.
1. When you're done with the settings, click **Save**.

To send alerts to another Slack channel, select another **Notification channel** from the dropdown, **Edit** those job notification settings, and **Save** the changes.

To set up alerts on jobs from a different environment, select another **Environment** from the dropdown, **Edit** those job notification settings, and **Save** the changes.

<Lightbox src="/img/docs/deploy/example-slack-notification-settings-page.png" width="100%" title="Example of the Slack notifications page"/>

### Disable the Slack integration

1. From the gear menu, select **Profile settings**. On your **User profile** page, scroll to the **Linked accounts** section.
1. Find the Slack application in the **Linked accounts** section, click the trash can icon, and click **Unlink**. Channels that you configured will no longer receive Slack notifications. Channels that are configured by other account admins will continue to receive Slack notifications if they still have active Slack integrations. To migrate ownership of a Slack channel notification configuration, have another account admin edit the configuration.
5 changes: 3 additions & 2 deletions website/docs/docs/deploy/jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ In dbt Cloud, there are two types of jobs:
- [Deploy jobs](/docs/deploy/deploy-jobs) &mdash; To create and set up triggers for building production data assets
- [Continuous integration (CI) jobs](/docs/deploy/continuous-integration) &mdash; To create and set up triggers for checking code changes

Below is a comparison table that describes how deploy jobs and CI jobs behave differently:
Below is a comparison table that describes how deploy jobs and CI jobs behave:

| | Deploy Jobs | CI Jobs |
| --- | --- | --- |
Expand All @@ -19,4 +19,5 @@ Below is a comparison table that describes how deploy jobs and CI jobs behave di
| Destination | Builds into a production database and schema. | Builds into a staging database and ephemeral schema, lived for the lifetime of the PR. |
| Execution mode | Runs execute sequentially, so as to not have collisions on the underlying DAG. | Runs execute in parallel to promote team velocity. |
| Efficiency run savings | Detects over-scheduled jobs and cancels unnecessary runs to avoid queue clog. | Cancels existing runs when a newer commit is pushed to avoid redundant work. |
| State comparison | Only sometimes needs to detect state. | Almost always needs to compare state against the production environment to build on modified code and its dependents. |
| State comparison | Only sometimes needs to detect state. | Almost always needs to compare state against the production environment to build on modified code and its dependents. |
| Job run duration | Limit is 24 hours. | Limit is 24 hours. |
8 changes: 0 additions & 8 deletions website/docs/faqs/Accounts/slack.md

This file was deleted.

11 changes: 0 additions & 11 deletions website/docs/faqs/Project/docs-for-multiple-projects.md

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@ At a high level, you’ll need to decide:
- Where to draw the lines between your dbt Projects -- i.e. how do you determine where to split your DAG and which models go in which project?
- How to manage your code -- do you want multiple dbt Projects living in the same repository (mono-repo) or do you want to have multiple repos with one repo per project?

### Cycle detection

Like resource dependencies, project dependencies are acyclic, meaning they only move in one direction. This prevents `ref` cycles (or loops), which lead to issues with your data workflows. For example, if project B depends on project A, a new model in project A could not import and use a public model from project B. Refer to [Project dependencies](/docs/collaborate/govern/project-dependencies#how-to-use-ref) for more information.

## Define your project interfaces by splitting your DAG

The first (and perhaps most difficult!) decision when migrating to a multi-project architecture is deciding where to draw the line in your DAG to define the interfaces between your projects. Let's explore some language for discussing the design of these patterns.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ slug: zapier-ms-teams
description: Use Zapier and the dbt Cloud API to post to Microsoft Teams
---

This guide will show you how to set up an integration between dbt Cloud jobs and Microsoft Teams using [dbt Cloud Webhooks](/docs/deploy/webhooks) and Zapier, similar to the [native Slack integration](/faqs/accounts/slack).
This guide will show you how to set up an integration between dbt Cloud jobs and Microsoft Teams using [dbt Cloud Webhooks](/docs/deploy/webhooks) and Zapier, similar to the [native Slack integration](/docs/deploy/job-notifications#slack-notifications).

When a dbt Cloud job finishes running, the integration will:

Expand Down
2 changes: 1 addition & 1 deletion website/docs/guides/orchestration/webhooks/zapier-slack.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ slug: zapier-slack
description: Use Zapier and the dbt Cloud API to post error context to Slack
---

This guide will show you how to set up an integration between dbt Cloud jobs and Slack using [dbt Cloud webhooks](/docs/deploy/webhooks) and Zapier. It builds on the native [native Slack integration](/faqs/accounts/slack) by attaching error message details of models and tests in a thread.
This guide will show you how to set up an integration between dbt Cloud jobs and Slack using [dbt Cloud webhooks](/docs/deploy/webhooks) and Zapier. It builds on the native [native Slack integration](/docs/deploy/job-notifications#slack-notifications) by attaching error message details of models and tests in a thread.

Note: Because there is not a webhook for Run Cancelled, you may want to keep the standard Slack integration installed to receive those notifications. You could also use the [alternative integration](#alternate-approach) that augments the native integration without replacing it.

Expand Down
Loading

0 comments on commit 5c1fb7c

Please sign in to comment.