From f6a80cf142c02cbfb71f7b97f6d685eb20cba83b Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 24 Oct 2024 09:13:11 -0400 Subject: [PATCH 01/18] [DRAFT] Athena quickstart --- website/docs/guides/athena-qs.md | 58 ++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 website/docs/guides/athena-qs.md diff --git a/website/docs/guides/athena-qs.md b/website/docs/guides/athena-qs.md new file mode 100644 index 00000000000..7d70f9bc0b4 --- /dev/null +++ b/website/docs/guides/athena-qs.md @@ -0,0 +1,58 @@ +--- +title: "Quickstart for dbt Cloud and Amazon Athena" +id: "athena" +# time_to_complete: '30 minutes' commenting out until we test +level: 'Beginner' +icon: 'athena' +hide_table_of_contents: true +tags: ['Amazon','Athena', 'dbt Cloud','Quickstart'] +recently_updated: true +--- + +
+ +## Introduction + +In this quickstart guide, you'll learn how to use dbt Cloud with BigQuery. It will show you how to: + +- Create an S3 bucket for Athena query results. +- Creat an Athena database. +- Access sample data in a public dataset. +- Connect dbt Cloud to Amazon Athena. +- Take a sample query and turn it into a model in your dbt project. A model in dbt is a select statement. +- Add tests to your models. +- Document your models. +- Schedule a job to run. + +:::tip Videos for you +You can check out [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) for free if you're interested in course learning with videos. +::: + +### Prerequisites​ + +- You have a [dbt Cloud account](https://www.getdbt.com/signup/). +- You have an [AWS account](https://aws.amazon.com/). +- You have set up [Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/getting-started.html). + +### Related content + +- Learn more with [dbt Learn courses](https://learn.getdbt.com) +- [CI jobs](/docs/deploy/continuous-integration) +- [Deploy jobs](/docs/deploy/deploy-jobs) +- [Job notifications](/docs/deploy/job-notifications) +- [Source freshness](/docs/deploy/source-freshness) + +## Getting started + +Download these CSV files (the Jaffle Shop sample data) that you will need for this guide: + - [jaffle_shop_customers.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_customers.csv) + - [jaffle_shop_orders.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_orders.csv) + - [stripe_payments.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/stripe_payments.csv) + +### Set up the S3 bucket + +You can use an existing or create a new S3 bucket for these steps. + +1. Log into your AWS account and navigate to the **Athena console**. + - If this is your first time in the Athena console (in your current AWS Region), click **Explore the query editor** to open the query editor. Otherwise, Athena opens automatically in the query editor. +1. Open **Settings** and find the **Location of query result box** field. \ No newline at end of file From 2943d007ba1735c2b30233b0ece1c9aaa91a6e9e Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Mon, 4 Nov 2024 12:19:01 -0500 Subject: [PATCH 02/18] Adding more content --- website/docs/guides/athena-qs.md | 293 ++++++++++++++++++++++++++++++- 1 file changed, 288 insertions(+), 5 deletions(-) diff --git a/website/docs/guides/athena-qs.md b/website/docs/guides/athena-qs.md index 7d70f9bc0b4..42861b4e24e 100644 --- a/website/docs/guides/athena-qs.md +++ b/website/docs/guides/athena-qs.md @@ -13,7 +13,7 @@ recently_updated: true ## Introduction -In this quickstart guide, you'll learn how to use dbt Cloud with BigQuery. It will show you how to: +In this quickstart guide, you'll learn how to use dbt Cloud with Amazon Athena. It will show you how to: - Create an S3 bucket for Athena query results. - Creat an Athena database. @@ -49,10 +49,293 @@ Download these CSV files (the Jaffle Shop sample data) that you will need for th - [jaffle_shop_orders.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_orders.csv) - [stripe_payments.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/stripe_payments.csv) -### Set up the S3 bucket +## Set up an Amazon S3 bucket -You can use an existing or create a new S3 bucket for these steps. +For the following guide you can use an existing S3 bucket or create a new one. + +In either case, download the following CSV files (the Jaffle Shop sample data) and load them into your S3 bucket: +- [jaffle_shop_customers.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_customers.csv) +- [jaffle_shop_orders.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_orders.csv) +- [stripe_payments.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/stripe_payments.csv) + + +## Configure Amazon Athena 1. Log into your AWS account and navigate to the **Athena console**. - - If this is your first time in the Athena console (in your current AWS Region), click **Explore the query editor** to open the query editor. Otherwise, Athena opens automatically in the query editor. -1. Open **Settings** and find the **Location of query result box** field. \ No newline at end of file + - If this is your first time in the Athena console (in your current AWS Region), click **Explore the query editor** to open the query editor. Otherwise, Athena opens automatically in the query editor and take one of the following actions: +1. Open **Settings** and find the **Location of query result box** field. + 1. Enter the path of the S3 bucket (prefix it with `s3://`). + 2. Navigate to **Browse S3**, select the S3 bucket you created, and click **Choose**. +1. **Save** the settings configuerations +1. In the **query editor**, create a database by running `create database YOUR_DATABASE_NAME`. +1. To make the database you created your current, select it from the **Database** list on the left side menu. +1. There are 2 options to access the Jaffle Shop data in the S3 bucket: + 1. Manually create the tables + 2. Create a glue crawler to recreate the data as external tables (recommended) +1. Once the tables have been created, you will able to `SELECT` from them. + +## Set up security access to Athena + +To setup the security access for Athena, determine which access method you want to use: + 1. The `aws_access_key_id` and `aws_secret_access_key` (recommended) + 2. An `aws_profile_name` that matches your profile defined in your **AWS credentials** file. + +### AWS access key + +To obtain your `aws_access_key_id` and `aws_secret_access_key`: + +1. Open the **AWS Console**. +1. Click on your **username** near the top right and click **Security Credentials**. +1. Click on **Users** in the sidebar. +1. Click on your **username** (or the name of the user for whom to create the key). +1. Click on the **Security Credentials** tab. +1. Click **Create Access Key**. +1. Click **Show User Security Credentials** and + +Save the `aws_access_key_id` and `aws_secret_access_key` for a future step. + +### AWS profile name + +To obtain your AWS credentials file: +1. Follow the instructions for [configuring the credentials file](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html) usin the AWS CLI +1. Locate the file on your computer + 1. Windows: `%USERPROFILE%\.aws\credentials` + 2. Mac/Linux: `~/.aws/credentials` + +Save the `aws_access_key_id` and `aws_secret_access_key` from the file for a future step. + +## Configure the connection in dbt Cloud + +To configure the Athena connection in dbt Cloud: +1. Click your **account name** on the left-side menu and click **Account settings**. +1. Click **Connections** and click **New connection**. +1. Click **Athena** and fill out the required fields (and any optional fields you want). + 1. **AWS region name** — The AWS region of your environment. + 1. **Database (catalog)** — Enter the database name created in earlier steps (lowercase only). + 1. **AWS S3 staging directory** — Enter the S3 bucket created in earlier steps. +1. Click **Save** + +### Configure your environment + +To configure the Athena credentials in your environment: +1. Click **Deploy** on the left-side menu and click **Environments**. +1. Click **Create environment** and fill out the **General settings**. + - Your **dbt version** must be set to `Versionless` to use the Athena connection. +1. Select the Athena connection from the **Connection** dropdown. +1. Fill out the `aws_access_key` and `aws_access_id` recorded in previous steps, as well as the `Schema` to write to. +1. Click **Test connection** and once it succeeds, **Save** the environment. + +Repeat the process to create a [development environment](https://docs.getdbt.com/docs/dbt-cloud-environments#types-of-environments). + +## Set up a dbt Cloud managed repository + + +## Initialize your dbt project​ and start developing + +Now that you have a repository configured, you can initialize your project and start development in dbt Cloud: + +1. Click **Start developing in the IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. +2. Above the file tree to the left, click **Initialize dbt project**. This builds out your folder structure with example models. +3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit` and click **Commit**. This creates the first commit to your managed repo and allows you to open a branch where you can add new dbt code. +4. You can now directly query data from your warehouse and execute `dbt run`. You can try this out now: + - Click **+ Create new file**, add this query to the new file, and click **Save as** to save the new file: + ```sql + select * from jaffle_shop.customers + ``` + - In the command line bar at the bottom, enter `dbt run` and click **Enter**. You should see a `dbt run succeeded` message. + +## Build your first model + +You have two options for working with files in the dbt Cloud IDE: + +- Create a new branch (recommended) — Create a new branch to edit and commit your changes. Navigate to **Version Control** on the left sidebar and click **Create branch**. +- Edit in the protected primary branch — If you prefer to edit, format, or lint files and execute dbt commands directly in your primary git branch. The dbt Cloud IDE prevents commits to the protected branch, so you will be prompted to commit your changes to a new branch. + +Name the new branch `add-customers-model`. + +1. Click the **...** next to the `models` directory, then select **Create file**. +2. Name the file `customers.sql`, then click **Create**. +3. Copy the following query into the file and click **Save**. + +```sql +with customers as ( + + select + id as customer_id, + first_name, + last_name + + from jaffle_shop.customers + +), + +orders as ( + + select + id as order_id, + user_id as customer_id, + order_date, + status + + from jaffle_shop.orders + +), + +customer_orders as ( + + select + customer_id, + + min(order_date) as first_order_date, + max(order_date) as most_recent_order_date, + count(order_id) as number_of_orders + + from orders + + group by 1 + +), + +final as ( + + select + customers.customer_id, + customers.first_name, + customers.last_name, + customer_orders.first_order_date, + customer_orders.most_recent_order_date, + coalesce(customer_orders.number_of_orders, 0) as number_of_orders + + from customers + + left join customer_orders using (customer_id) + +) + +select * from final +``` + +4. Enter `dbt run` in the command prompt at the bottom of the screen. You should get a successful run and see the three models. + +Later, you can connect your business intelligence (BI) tools to these views and tables so they only read cleaned up data rather than raw data in your BI tool. + +#### FAQs + + + + + + + +## Change the way your model is materialized + + + +## Delete the example models + + + +## Build models on top of other models + + + +1. Create a new SQL file, `models/stg_customers.sql`, with the SQL from the `customers` CTE in our original query. +2. Create a second new SQL file, `models/stg_orders.sql`, with the SQL from the `orders` CTE in our original query. + + + + ```sql + select + id as customer_id, + first_name, + last_name + + from jaffle_shop.customers + ``` + + + + + + ```sql + select + id as order_id, + user_id as customer_id, + order_date, + status + + from jaffle_shop.orders + ``` + + + +3. Edit the SQL in your `models/customers.sql` file as follows: + + + + ```sql + with customers as ( + + select * from {{ ref('stg_customers') }} + + ), + + orders as ( + + select * from {{ ref('stg_orders') }} + + ), + + customer_orders as ( + + select + customer_id, + + min(order_date) as first_order_date, + max(order_date) as most_recent_order_date, + count(order_id) as number_of_orders + + from orders + + group by 1 + + ), + + final as ( + + select + customers.customer_id, + customers.first_name, + customers.last_name, + customer_orders.first_order_date, + customer_orders.most_recent_order_date, + coalesce(customer_orders.number_of_orders, 0) as number_of_orders + + from customers + + left join customer_orders using (customer_id) + + ) + + select * from final + + ``` + + + +4. Execute `dbt run`. + + This time, when you performed a `dbt run`, separate views/tables were created for `stg_customers`, `stg_orders` and `customers`. dbt inferred the order to run these models. Because `customers` depends on `stg_customers` and `stg_orders`, dbt builds `customers` last. You do not need to explicitly define these dependencies. + + +#### FAQs {#faq-2} + + + + + +
+ + + + From e7118341cf4ca97c58d6b53abda9e9f559264029 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 7 Nov 2024 18:39:27 -0500 Subject: [PATCH 03/18] Apply suggestions from code review Co-authored-by: Amy Chen <46451573+amychen1776@users.noreply.github.com> --- website/docs/guides/athena-qs.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/guides/athena-qs.md b/website/docs/guides/athena-qs.md index 42861b4e24e..c2191e767fa 100644 --- a/website/docs/guides/athena-qs.md +++ b/website/docs/guides/athena-qs.md @@ -62,13 +62,13 @@ In either case, download the following CSV files (the Jaffle Shop sample data) a ## Configure Amazon Athena 1. Log into your AWS account and navigate to the **Athena console**. - - If this is your first time in the Athena console (in your current AWS Region), click **Explore the query editor** to open the query editor. Otherwise, Athena opens automatically in the query editor and take one of the following actions: + - If this is your first time in the Athena console (in your current AWS Region), click **Explore the query editor** to open the query editor. Otherwise, Athena opens automatically in the query editor. 1. Open **Settings** and find the **Location of query result box** field. 1. Enter the path of the S3 bucket (prefix it with `s3://`). 2. Navigate to **Browse S3**, select the S3 bucket you created, and click **Choose**. -1. **Save** the settings configuerations +1. **Save** the settings configurations 1. In the **query editor**, create a database by running `create database YOUR_DATABASE_NAME`. -1. To make the database you created your current, select it from the **Database** list on the left side menu. +1. To make the database you created the one you `write` into, select it from the **Database** list on the left side menu. 1. There are 2 options to access the Jaffle Shop data in the S3 bucket: 1. Manually create the tables 2. Create a glue crawler to recreate the data as external tables (recommended) From acdbd0c69d83ff0c1abe4e6e897015902ad3cfbc Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 7 Nov 2024 19:44:23 -0500 Subject: [PATCH 04/18] Apply suggestions from code review Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com> --- website/docs/guides/athena-qs.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/website/docs/guides/athena-qs.md b/website/docs/guides/athena-qs.md index c2191e767fa..3fb224b05c4 100644 --- a/website/docs/guides/athena-qs.md +++ b/website/docs/guides/athena-qs.md @@ -66,19 +66,19 @@ In either case, download the following CSV files (the Jaffle Shop sample data) a 1. Open **Settings** and find the **Location of query result box** field. 1. Enter the path of the S3 bucket (prefix it with `s3://`). 2. Navigate to **Browse S3**, select the S3 bucket you created, and click **Choose**. -1. **Save** the settings configurations +1. **Save** these settings. 1. In the **query editor**, create a database by running `create database YOUR_DATABASE_NAME`. 1. To make the database you created the one you `write` into, select it from the **Database** list on the left side menu. -1. There are 2 options to access the Jaffle Shop data in the S3 bucket: - 1. Manually create the tables - 2. Create a glue crawler to recreate the data as external tables (recommended) +1. Access the Jaffle Shop data in the S3 bucket using one of these options: + 1. Manually create the tables. + 2. Create a glue crawler to recreate the data as external tables (recommended). 1. Once the tables have been created, you will able to `SELECT` from them. ## Set up security access to Athena To setup the security access for Athena, determine which access method you want to use: - 1. The `aws_access_key_id` and `aws_secret_access_key` (recommended) - 2. An `aws_profile_name` that matches your profile defined in your **AWS credentials** file. +* Obtain `aws_access_key_id` and `aws_secret_access_key` (recommended) +* Obtain an `aws_profile_name` that matches your profile defined in your **AWS credentials** file. ### AWS access key @@ -109,7 +109,7 @@ Save the `aws_access_key_id` and `aws_secret_access_key` from the file for a fut To configure the Athena connection in dbt Cloud: 1. Click your **account name** on the left-side menu and click **Account settings**. 1. Click **Connections** and click **New connection**. -1. Click **Athena** and fill out the required fields (and any optional fields you want). +1. Click **Athena** and fill out the required fields (and any optional fields). 1. **AWS region name** — The AWS region of your environment. 1. **Database (catalog)** — Enter the database name created in earlier steps (lowercase only). 1. **AWS S3 staging directory** — Enter the S3 bucket created in earlier steps. From 59ceea94c3fc9b9813d9ecf0e7ce6a45edaa1bc6 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 7 Nov 2024 19:44:34 -0500 Subject: [PATCH 05/18] Update website/docs/guides/athena-qs.md --- website/docs/guides/athena-qs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/guides/athena-qs.md b/website/docs/guides/athena-qs.md index 3fb224b05c4..3842c15a156 100644 --- a/website/docs/guides/athena-qs.md +++ b/website/docs/guides/athena-qs.md @@ -51,7 +51,7 @@ Download these CSV files (the Jaffle Shop sample data) that you will need for th ## Set up an Amazon S3 bucket -For the following guide you can use an existing S3 bucket or create a new one. +For the following guide you can use an existing S3 bucket or [create a new one](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). In either case, download the following CSV files (the Jaffle Shop sample data) and load them into your S3 bucket: - [jaffle_shop_customers.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_customers.csv) From 5205b828ba89f9747e3cc1bbff38ec7eae60eb45 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 7 Nov 2024 19:48:26 -0500 Subject: [PATCH 06/18] Update athena-qs.md Removing duplicate material --- website/docs/guides/athena-qs.md | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/website/docs/guides/athena-qs.md b/website/docs/guides/athena-qs.md index 3842c15a156..70262485639 100644 --- a/website/docs/guides/athena-qs.md +++ b/website/docs/guides/athena-qs.md @@ -44,16 +44,9 @@ You can check out [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundam ## Getting started -Download these CSV files (the Jaffle Shop sample data) that you will need for this guide: - - [jaffle_shop_customers.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_customers.csv) - - [jaffle_shop_orders.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_orders.csv) - - [stripe_payments.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/stripe_payments.csv) - -## Set up an Amazon S3 bucket - For the following guide you can use an existing S3 bucket or [create a new one](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). -In either case, download the following CSV files (the Jaffle Shop sample data) and load them into your S3 bucket: +Download the following CSV files (the Jaffle Shop sample data) and upload them to your S3 bucket: - [jaffle_shop_customers.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_customers.csv) - [jaffle_shop_orders.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_orders.csv) - [stripe_payments.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/stripe_payments.csv) From a36e0bdcdf370265dd1f2551372124dbf5fe9b7a Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 7 Nov 2024 20:07:33 -0500 Subject: [PATCH 07/18] Update website/docs/guides/athena-qs.md --- website/docs/guides/athena-qs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/guides/athena-qs.md b/website/docs/guides/athena-qs.md index 70262485639..335ff85e9ea 100644 --- a/website/docs/guides/athena-qs.md +++ b/website/docs/guides/athena-qs.md @@ -95,7 +95,7 @@ To obtain your AWS credentials file: 1. Windows: `%USERPROFILE%\.aws\credentials` 2. Mac/Linux: `~/.aws/credentials` -Save the `aws_access_key_id` and `aws_secret_access_key` from the file for a future step. +Retrieve the `aws_access_key_id` and `aws_secret_access_key` from the `~/.aws/credentials` file for a future step. ## Configure the connection in dbt Cloud From 5324e97e47146847ac3f44da21af3d54a5c6368b Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 7 Nov 2024 20:09:01 -0500 Subject: [PATCH 08/18] Apply suggestions from code review --- website/docs/guides/athena-qs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/guides/athena-qs.md b/website/docs/guides/athena-qs.md index 335ff85e9ea..ad88b85f4dd 100644 --- a/website/docs/guides/athena-qs.md +++ b/website/docs/guides/athena-qs.md @@ -87,11 +87,11 @@ To obtain your `aws_access_key_id` and `aws_secret_access_key`: Save the `aws_access_key_id` and `aws_secret_access_key` for a future step. -### AWS profile name +### AWS credentials file To obtain your AWS credentials file: 1. Follow the instructions for [configuring the credentials file](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html) usin the AWS CLI -1. Locate the file on your computer +1. Locate the `~/.aws/credentials` file on your computer 1. Windows: `%USERPROFILE%\.aws\credentials` 2. Mac/Linux: `~/.aws/credentials` From fff29492a9f54c213c5f09c1b39525338ffb4afe Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 7 Nov 2024 20:09:45 -0500 Subject: [PATCH 09/18] Update website/docs/guides/athena-qs.md --- website/docs/guides/athena-qs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/guides/athena-qs.md b/website/docs/guides/athena-qs.md index ad88b85f4dd..68de2ddf070 100644 --- a/website/docs/guides/athena-qs.md +++ b/website/docs/guides/athena-qs.md @@ -73,7 +73,7 @@ To setup the security access for Athena, determine which access method you want * Obtain `aws_access_key_id` and `aws_secret_access_key` (recommended) * Obtain an `aws_profile_name` that matches your profile defined in your **AWS credentials** file. -### AWS access key +### AWS access key (recommended) To obtain your `aws_access_key_id` and `aws_secret_access_key`: From 968bf2062a54834778279aca9f73e9843f7fb10f Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 7 Nov 2024 20:09:53 -0500 Subject: [PATCH 10/18] Update website/docs/guides/athena-qs.md --- website/docs/guides/athena-qs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/guides/athena-qs.md b/website/docs/guides/athena-qs.md index 68de2ddf070..b1933bdd076 100644 --- a/website/docs/guides/athena-qs.md +++ b/website/docs/guides/athena-qs.md @@ -71,7 +71,7 @@ Download the following CSV files (the Jaffle Shop sample data) and upload them t To setup the security access for Athena, determine which access method you want to use: * Obtain `aws_access_key_id` and `aws_secret_access_key` (recommended) -* Obtain an `aws_profile_name` that matches your profile defined in your **AWS credentials** file. +* Obtain an **AWS credentials** file. ### AWS access key (recommended) From c8ebe3536b146e13c68e2dad5df45c0fe456f5cd Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Wed, 20 Nov 2024 12:28:57 -0500 Subject: [PATCH 11/18] Update snapshots.md add `description` property to new snapshots example per [internal slack](https://dbt-labs.slack.com/archives/C05FWBP9X1U/p1732121876847939) --- website/docs/docs/build/snapshots.md | 1 + 1 file changed, 1 insertion(+) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index 1ff2298d8f7..f9c7544fbc8 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -54,6 +54,7 @@ Configure your snapshots in YAML files to tell dbt how to detect record changes. snapshots: - name: string relation: relation # source('my_source', 'my_table') or ref('my_model') + description: markdown_string config: [database](/reference/resource-configs/database): string [schema](/reference/resource-configs/schema): string From 291eed9d125c292f5c87eaf7b756427181fa97ac Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Wed, 20 Nov 2024 12:31:18 -0500 Subject: [PATCH 12/18] Update website/docs/docs/build/snapshots.md --- website/docs/docs/build/snapshots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index f9c7544fbc8..7c74b48b296 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -54,7 +54,7 @@ Configure your snapshots in YAML files to tell dbt how to detect record changes. snapshots: - name: string relation: relation # source('my_source', 'my_table') or ref('my_model') - description: markdown_string + [description](/reference/resource-properties/description): markdown_string config: [database](/reference/resource-configs/database): string [schema](/reference/resource-configs/schema): string From 5190cdba58b852bf266a4dd5c0ce451ad771e0d6 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 20 Nov 2024 15:09:13 -0500 Subject: [PATCH 13/18] Limited support of custom incremental strategies (#6514) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What are you changing in this pull request and why? Adding callout for custom incrementals ## Checklist - [ ] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." - [ ] The content in this PR requires a dbt release note, so I added one to the [release notes page](https://docs.getdbt.com/docs/dbt-versions/dbt-cloud-release-notes). --- 🚀 Deployment available! Here are the direct links to the updated files: - https://docs-getdbt-com-git-bq-custom-dbt-labs.vercel.app/docs/build/incremental-strategy --- website/docs/docs/build/incremental-strategy.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/website/docs/docs/build/incremental-strategy.md b/website/docs/docs/build/incremental-strategy.md index 1fb35ba637c..4866279bc01 100644 --- a/website/docs/docs/build/incremental-strategy.md +++ b/website/docs/docs/build/incremental-strategy.md @@ -241,6 +241,12 @@ select * from {{ ref("some_model") }} ### Custom strategies +:::note limited support + +Custom strategies are not currently suppored on the BigQuery and Spark adapters. + +::: + Starting from dbt version 1.2 and onwards, users have an easier alternative to [creating an entirely new materialization](/guides/create-new-materializations). They define and use their own "custom" incremental strategies by: 1. Defining a macro named `get_incremental_STRATEGY_sql`. Note that `STRATEGY` is a placeholder and you should replace it with the name of your custom incremental strategy. From 5494573656c4c35f02ac78703cce24c74db319bf Mon Sep 17 00:00:00 2001 From: Jimmy Zhu Date: Wed, 20 Nov 2024 12:45:16 -0800 Subject: [PATCH 14/18] Mention Snowflake Enterprise or higher is required for query history functionality --- website/docs/docs/collaborate/model-query-history.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/collaborate/model-query-history.md b/website/docs/docs/collaborate/model-query-history.md index 0180757f980..3e85883a86e 100644 --- a/website/docs/docs/collaborate/model-query-history.md +++ b/website/docs/docs/collaborate/model-query-history.md @@ -28,7 +28,7 @@ To access the features, you should meet the following: 1. You have a dbt Cloud account on the [Enterprise plan](https://www.getdbt.com/pricing/). Single-tenant accounts should contact their account representative for setup. 2. You have set up a [production](https://docs.getdbt.com/docs/deploy/deploy-environments#set-as-production-environment) deployment environment for each project you want to explore, with at least one successful job run. 3. You have [admin permissions](/docs/cloud/manage-access/enterprise-permissions) in dbt Cloud to edit project settings or production environment settings. -4. Use Snowflake or BigQuery as your data warehouse and can enable query history permissions or work with an admin to do so. Support for additional data platforms coming soon. +4. Use Snowflake (Enterprise tier or higher) or BigQuery as your data warehouse and can enable query history permissions or work with an admin to do so. Support for additional data platforms coming soon. ## Enable query history in dbt Cloud From 394d55a93ad96c63988e4343ab4aa4c2c02f8174 Mon Sep 17 00:00:00 2001 From: Doug Guthrie Date: Wed, 20 Nov 2024 14:53:46 -0700 Subject: [PATCH 15/18] Update regions-ip-addresses.md for azure us region --- website/docs/docs/cloud/about-cloud/regions-ip-addresses.md | 1 + 1 file changed, 1 insertion(+) diff --git a/website/docs/docs/cloud/about-cloud/regions-ip-addresses.md b/website/docs/docs/cloud/about-cloud/regions-ip-addresses.md index b396ce62feb..d0ba33e95be 100644 --- a/website/docs/docs/cloud/about-cloud/regions-ip-addresses.md +++ b/website/docs/docs/cloud/about-cloud/regions-ip-addresses.md @@ -13,6 +13,7 @@ dbt Cloud is [hosted](/docs/cloud/about-cloud/architecture) in multiple regions | Region | Location | Access URL | IP addresses | Developer plan | Team plan | Enterprise plan | |--------|----------|------------|--------------|----------------|-----------|-----------------| | North America [^1] | AWS us-east-1 (N. Virginia) | **Multi-tenant:** cloud.getdbt.com
**Cell based:** ACCOUNT_PREFIX.us1.dbt.com | 52.45.144.63
54.81.134.249
52.22.161.231
52.3.77.232
3.214.191.130
34.233.79.135 | ✅ | ✅ | ✅ | +| North America [^1] | Azure
East US 2 (Virginia) | **Cell based:** ACCOUNT_PREFIX.us2.dbt.com | 20.10.67.192/26 | ❌ | ❌ | ✅ | | EMEA [^1] | AWS eu-central-1 (Frankfurt) | emea.dbt.com | 3.123.45.39
3.126.140.248
3.72.153.148 | ❌ | ❌ | ✅ | | EMEA [^1] | Azure
North Europe (Ireland) | **Cell based:** ACCOUNT_PREFIX.eu2.dbt.com | 20.13.190.192/26 | ❌ | ❌ | ✅ | | APAC [^1] | AWS ap-southeast-2 (Sydney)| au.dbt.com | 52.65.89.235
3.106.40.33
13.239.155.206
| ❌ | ❌ | ✅ | From 3c63e6f07e98fe6434edfb94c76233f252349f65 Mon Sep 17 00:00:00 2001 From: Katherine Brock <48523663+kbrock91@users.noreply.github.com> Date: Wed, 20 Nov 2024 16:22:12 -0600 Subject: [PATCH 16/18] Update enable-dbt-copilot.md (#6517) CoPilot > Copilot for consistency across docs ## What are you changing in this pull request and why? ## Checklist - [ ] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." - [ ] The content in this PR requires a dbt release note, so I added one to the [release notes page](https://docs.getdbt.com/docs/dbt-versions/dbt-cloud-release-notes). Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com> --- website/docs/docs/cloud/enable-dbt-copilot.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/docs/cloud/enable-dbt-copilot.md b/website/docs/docs/cloud/enable-dbt-copilot.md index 07a9f6294da..67a11fed3fc 100644 --- a/website/docs/docs/cloud/enable-dbt-copilot.md +++ b/website/docs/docs/cloud/enable-dbt-copilot.md @@ -36,7 +36,7 @@ Note: To disable (only after enabled), repeat steps 1 to 3, toggle off in step 4 ### Bringing your own OpenAI API key (BYOK) -Once AI features have been enabled, you can provide your organization's OpenAI API key. dbt Cloud will then leverage your OpenAI account and terms to power dbt CoPilot. This will incur billing charges to your organization from OpenAI for requests made by dbt CoPilot. +Once AI features have been enabled, you can provide your organization's OpenAI API key. dbt Cloud will then leverage your OpenAI account and terms to power dbt Copilot. This will incur billing charges to your organization from OpenAI for requests made by dbt Copilot. Note that Azure OpenAI is not currently supported, but will be in the future. @@ -48,4 +48,4 @@ A dbt Cloud admin can provide their API key by following these steps: 3. Scroll to **AI** and select the toggle for **OpenAI** -4. Enter your API key and click **Save**. \ No newline at end of file +4. Enter your API key and click **Save**. From 4c34faaa8b52553d105c15b804b9dbe539d40c5f Mon Sep 17 00:00:00 2001 From: Jordan Stein Date: Wed, 20 Nov 2024 18:35:51 -0800 Subject: [PATCH 17/18] call out live connection --- .../docs/docs/cloud-integrations/semantic-layer/tableau.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/docs/cloud-integrations/semantic-layer/tableau.md b/website/docs/docs/cloud-integrations/semantic-layer/tableau.md index 15a0a92cf39..adf92496ed3 100644 --- a/website/docs/docs/cloud-integrations/semantic-layer/tableau.md +++ b/website/docs/docs/cloud-integrations/semantic-layer/tableau.md @@ -46,8 +46,8 @@ Alternatively, you can follow these steps to install the Connector: ## Using the integration 1. **Authentication** — Once you authenticate, the system will direct you to the data source page. -2. **Access all Semantic Layer Objects** — Use the "ALL" data source to access all the metrics, dimensions, and entities configured in your dbt Semantic Layer. Note that the "METRICS_AND_DIMENSIONS" data source has been deprecated and replaced by "ALL". -3. **Access saved queries** — You can optionally access individual [saved queries](/docs/build/saved-queries) that you've defined. These will also show up as unique data sources when you log in. +2. **Access all Semantic Layer Objects** — Use the "ALL" data source to access all the metrics, dimensions, and entities configured in your dbt Semantic Layer. Note that the "METRICS_AND_DIMENSIONS" data source has been deprecated and replaced by "ALL". Be sure to use a Live connection as Extracts are not supported at this time. +3. **Access saved queries** — You can optionally access individual [saved queries](/docs/build/saved-queries) that you've defined. These will also show up as unique data sources when you log in. 4. **Access worksheet** — From your data source selection, go directly to a worksheet in the bottom left-hand corner. 5. **Query metrics and dimensions** — Then, you'll find all the metrics, dimensions, and entities that are available to query on the left side of your window based on your selection. From af30a1f773ad4a0e50d0b640031c62c8f603ce93 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Wed, 20 Nov 2024 22:01:32 -0500 Subject: [PATCH 18/18] Update tableau.md --- website/docs/docs/cloud-integrations/semantic-layer/tableau.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud-integrations/semantic-layer/tableau.md b/website/docs/docs/cloud-integrations/semantic-layer/tableau.md index adf92496ed3..1f6755c38fa 100644 --- a/website/docs/docs/cloud-integrations/semantic-layer/tableau.md +++ b/website/docs/docs/cloud-integrations/semantic-layer/tableau.md @@ -46,7 +46,7 @@ Alternatively, you can follow these steps to install the Connector: ## Using the integration 1. **Authentication** — Once you authenticate, the system will direct you to the data source page. -2. **Access all Semantic Layer Objects** — Use the "ALL" data source to access all the metrics, dimensions, and entities configured in your dbt Semantic Layer. Note that the "METRICS_AND_DIMENSIONS" data source has been deprecated and replaced by "ALL". Be sure to use a Live connection as Extracts are not supported at this time. +2. **Access all Semantic Layer Objects** — Use the "ALL" data source to access all the metrics, dimensions, and entities configured in your dbt Semantic Layer. Note that the "METRICS_AND_DIMENSIONS" data source has been deprecated and replaced by "ALL". Be sure to use a live connection since extracts are not supported at this time. 3. **Access saved queries** — You can optionally access individual [saved queries](/docs/build/saved-queries) that you've defined. These will also show up as unique data sources when you log in. 4. **Access worksheet** — From your data source selection, go directly to a worksheet in the bottom left-hand corner. 5. **Query metrics and dimensions** — Then, you'll find all the metrics, dimensions, and entities that are available to query on the left side of your window based on your selection.