From f90ef937a59c0ec67b6c145a8bb24cdda18af170 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Mon, 16 Dec 2024 12:28:18 -0500 Subject: [PATCH 01/18] Private packages beta --- website/docs/docs/build/packages.md | 70 +++++++++++++++++++++++++++-- 1 file changed, 66 insertions(+), 4 deletions(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index 9ba4ceeaff5..f30fbd05741 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -157,9 +157,64 @@ packages: Where `name: 'dbt_utils'` specifies the subfolder of `dbt_packages` that's created for the package source code to be installed within. -### Private packages +## Private packages -#### SSH Key Method (Command Line only) +### Native private packages + +dbt Cloud supports private packages from [supported](#prerequisites) Git repos leveraging an exisiting [configuration](/docs/cloud/git/git-configuration-in-dbt-cloud) in your environment. Previously, you had to configure a [token](#git-token-method) to retrieve packages from your private repos. + +#### Prerequisites + +Native support for private packages is supported on the following services: +- [Github](/docs/cloud/git/connect-github) +- [Azure DevOps](/docs/cloud/git/connect-azure-devops) + +Support for Gitlab is coming soon. + +You must have the Git service configured in the **Integrations** section of your **Account settings** to use this feature. + +#### Configuration + +Use the `private` block in your `packages.yml` or `dependencies.yml` to clone package repos using your existing dbt Cloud Git integration without having to provision an access token or create a dbt Cloud environment variable: + + + +```yaml + +packages: + - private: dbt-labs/awesome_repo + + - package: normal packages + + [...] + +``` + + + +You can pin private packages similar to regular dbt packages: + +```yaml + +packages: + - private: dbt-labs/awesome_repo + revision: branch name | tagged git release | a specific commit (full 40-character hash} + +``` + +If you are using multiple Git integrations, disambiguate by adding the provider key: + +```yaml + +packages: + - private: dbt-labs/awesome_repo + provider: "github" | "azure" + +``` + +With this method, you can retrieve private packages from an integrated Git provider without any additional steps to connect. + +### SSH Key Method (Command Line only) If you're using the Command Line, private packages can be cloned via SSH and an SSH key. When you use SSH keys to authenticate to your git remote server, you don’t need to supply your username and password each time. Read more about SSH keys, how to generate them, and how to add them to your git provider here: [Github](https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh) and [GitLab](https://docs.gitlab.com/ee/user/ssh.html). @@ -177,7 +232,14 @@ packages: If you're using dbt Cloud, the SSH key method will not work, but you can use the [HTTPS Git Token Method](https://docs.getdbt.com/docs/build/packages#git-token-method). -#### Git token method +### Git token method + +:::note + +dbt Cloud has [native support](#native-private-packages) for Git hosted private packages for Github and Azure DevOps (GitLab coming soon). If you are using a supported [integrated Git environment](/docs/cloud/git/git-configuration-in-dbt-cloud), you no longer need to configure Git tokens to retrieve private packages. + +::: + This method allows the user to clone via HTTPS by passing in a git token via an environment variable. Be careful of the expiration date of any token you use, as an expired token could cause a scheduled run to fail. Additionally, user tokens can create a challenge if the user ever loses access to a specific repo. @@ -246,7 +308,7 @@ Read more about creating a Personal Access Token [here](https://confluence.atlas -#### Configure subdirectory for packaged projects +## Configure subdirectory for packaged projects In general, dbt expects `dbt_project.yml` to be located as a top-level file in a package. If the packaged project is instead nested in a subdirectory—perhaps within a much larger mono repo—you can optionally specify the folder path as `subdirectory`. dbt will attempt a [sparse checkout](https://git-scm.com/docs/git-sparse-checkout) of just the files located within that subdirectory. Note that you must be using a recent version of `git` (`>=2.26.0`). From 3b899664290e71ed34c1a425f8952e854586e3df Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Mon, 16 Dec 2024 12:30:20 -0500 Subject: [PATCH 02/18] Update website/docs/docs/build/packages.md --- website/docs/docs/build/packages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index f30fbd05741..6d1c41235b7 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -171,7 +171,7 @@ Native support for private packages is supported on the following services: Support for Gitlab is coming soon. -You must have the Git service configured in the **Integrations** section of your **Account settings** to use this feature. +To use this feature, you must have a supported Git provider configured in the **Integrations** section of your **Account settings**. #### Configuration From ed10f1919a4de4ba304bd152fa1ab0ec001003ab Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Mon, 16 Dec 2024 12:31:16 -0500 Subject: [PATCH 03/18] Update website/docs/docs/build/packages.md --- website/docs/docs/build/packages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index 6d1c41235b7..c3384d952ee 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -214,7 +214,7 @@ packages: With this method, you can retrieve private packages from an integrated Git provider without any additional steps to connect. -### SSH Key Method (Command Line only) +### SSH key method (command line only) If you're using the Command Line, private packages can be cloned via SSH and an SSH key. When you use SSH keys to authenticate to your git remote server, you don’t need to supply your username and password each time. Read more about SSH keys, how to generate them, and how to add them to your git provider here: [Github](https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh) and [GitLab](https://docs.gitlab.com/ee/user/ssh.html). From 80a45d3f2f854bcaacec3a295cd1d82354956245 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 17 Dec 2024 13:12:08 +0000 Subject: [PATCH 04/18] Update website/docs/docs/build/packages.md removing extra spaces --- website/docs/docs/build/packages.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index c3384d952ee..0942f0053a4 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -205,7 +205,6 @@ packages: If you are using multiple Git integrations, disambiguate by adding the provider key: ```yaml - packages: - private: dbt-labs/awesome_repo provider: "github" | "azure" From 71b4813caa0119089cf7a46a32ca02c9cf1c58ec Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 17 Dec 2024 13:12:29 +0000 Subject: [PATCH 05/18] Update website/docs/docs/build/packages.md --- website/docs/docs/build/packages.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index 0942f0053a4..149cdd7bfcf 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -187,7 +187,6 @@ packages: - package: normal packages [...] - ``` From efe5e056f1c5c45077e85c44e6dd9c023aa7fc3b Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 17 Dec 2024 13:12:54 +0000 Subject: [PATCH 06/18] Update website/docs/docs/build/packages.md --- website/docs/docs/build/packages.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index 149cdd7bfcf..bb63751d231 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -194,7 +194,6 @@ packages: You can pin private packages similar to regular dbt packages: ```yaml - packages: - private: dbt-labs/awesome_repo revision: branch name | tagged git release | a specific commit (full 40-character hash} From c17f1442acf2ff1103e014bc97b8260b8ea5d1a5 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 17 Dec 2024 13:13:06 +0000 Subject: [PATCH 07/18] Update website/docs/docs/build/packages.md --- website/docs/docs/build/packages.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index bb63751d231..dcd0fc517a4 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -180,7 +180,6 @@ Use the `private` block in your `packages.yml` or `dependencies.yml` to clone pa ```yaml - packages: - private: dbt-labs/awesome_repo From ed94b8797346555e245ac9d6934dd736adeabd9b Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 17 Dec 2024 16:53:10 -0500 Subject: [PATCH 08/18] Update website/docs/docs/build/packages.md --- website/docs/docs/build/packages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index dcd0fc517a4..a61b30617f1 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -165,7 +165,7 @@ dbt Cloud supports private packages from [supported](#prerequisites) Git repos l #### Prerequisites -Native support for private packages is supported on the following services: +dbt Cloud natively supports private packages for the following services: - [Github](/docs/cloud/git/connect-github) - [Azure DevOps](/docs/cloud/git/connect-azure-devops) From ccda55d57e912eae817781e4d1c8638d5ca48bae Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 17 Dec 2024 16:53:29 -0500 Subject: [PATCH 09/18] Update website/docs/docs/build/packages.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/docs/build/packages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index a61b30617f1..c0ef4cf65b3 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -166,7 +166,7 @@ dbt Cloud supports private packages from [supported](#prerequisites) Git repos l #### Prerequisites dbt Cloud natively supports private packages for the following services: -- [Github](/docs/cloud/git/connect-github) +- [GitHub](/docs/cloud/git/connect-github) - [Azure DevOps](/docs/cloud/git/connect-azure-devops) Support for Gitlab is coming soon. From a034bbdda7e45c5479aa8ea9f295e10eeace96ad Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 17 Dec 2024 16:53:41 -0500 Subject: [PATCH 10/18] Update website/docs/docs/build/packages.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/docs/build/packages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index c0ef4cf65b3..0c59965a3c8 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -168,7 +168,7 @@ dbt Cloud supports private packages from [supported](#prerequisites) Git repos l dbt Cloud natively supports private packages for the following services: - [GitHub](/docs/cloud/git/connect-github) - [Azure DevOps](/docs/cloud/git/connect-azure-devops) - +- Support for GitLab is coming soon. Support for Gitlab is coming soon. To use this feature, you must have a supported Git provider configured in the **Integrations** section of your **Account settings**. From d1fc4378b4db8fd62030de0a9d14966b9c03c0ef Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 17 Dec 2024 16:56:22 -0500 Subject: [PATCH 11/18] Update website/docs/docs/build/packages.md --- website/docs/docs/build/packages.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index 0c59965a3c8..f659427e45b 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -169,7 +169,6 @@ dbt Cloud natively supports private packages for the following services: - [GitHub](/docs/cloud/git/connect-github) - [Azure DevOps](/docs/cloud/git/connect-azure-devops) - Support for GitLab is coming soon. -Support for Gitlab is coming soon. To use this feature, you must have a supported Git provider configured in the **Integrations** section of your **Account settings**. From b34d197e3d5668f4a8f0504fa0318a0a1e0ba342 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 17 Dec 2024 17:08:32 -0500 Subject: [PATCH 12/18] Fixing code spacing --- website/docs/docs/build/packages.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index f659427e45b..88471d67e46 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -180,10 +180,9 @@ Use the `private` block in your `packages.yml` or `dependencies.yml` to clone pa ```yaml packages: - - private: dbt-labs/awesome_repo - - - package: normal packages - + - private: dbt-labs/awesome_repo + - package: normal packages + [...] ``` @@ -193,8 +192,8 @@ You can pin private packages similar to regular dbt packages: ```yaml packages: - - private: dbt-labs/awesome_repo - revision: branch name | tagged git release | a specific commit (full 40-character hash} + - private: dbt-labs/awesome_repo + revision: branch name | tagged git release | a specific commit (full 40-character hash} ``` @@ -202,8 +201,8 @@ If you are using multiple Git integrations, disambiguate by adding the provider ```yaml packages: - - private: dbt-labs/awesome_repo - provider: "github" | "azure" + - private: dbt-labs/awesome_repo + provider: "github" | "azure" ``` From 0e4e05e2b357d29ee3a9daa3028cee2000e32364 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 17 Dec 2024 17:14:23 -0500 Subject: [PATCH 13/18] Update website/docs/docs/build/packages.md --- website/docs/docs/build/packages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index 88471d67e46..ed33071f661 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -193,7 +193,7 @@ You can pin private packages similar to regular dbt packages: ```yaml packages: - private: dbt-labs/awesome_repo - revision: branch name | tagged git release | a specific commit (full 40-character hash} + revision: "0.9.5" # Pin to a tag, branch, or complete 40-character commit hash ``` From e394eb04d0fbb14674bd6c9df3e80466d4cc4f55 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 17 Dec 2024 17:16:07 -0500 Subject: [PATCH 14/18] Update website/docs/docs/build/packages.md --- website/docs/docs/build/packages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index ed33071f661..85d5e8ec96c 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -202,7 +202,7 @@ If you are using multiple Git integrations, disambiguate by adding the provider ```yaml packages: - private: dbt-labs/awesome_repo - provider: "github" | "azure" + provider: "github" # GitHub and Azure are currently supported. GitLab is coming soon. ``` From ef3e6e75038a8aa489e6c3f5dba3f7250e70d8db Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 17 Dec 2024 17:19:32 -0500 Subject: [PATCH 15/18] Update website/docs/docs/build/packages.md --- website/docs/docs/build/packages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index 85d5e8ec96c..e17ce2184f2 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -165,7 +165,7 @@ dbt Cloud supports private packages from [supported](#prerequisites) Git repos l #### Prerequisites -dbt Cloud natively supports private packages for the following services: +To use native private packages, you must have one of the following Git providers configured in the **Integrations** section of your **Account settings**: - [GitHub](/docs/cloud/git/connect-github) - [Azure DevOps](/docs/cloud/git/connect-azure-devops) - Support for GitLab is coming soon. From 121436bcd014e3924381f001650838f417387df6 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 17 Dec 2024 17:19:48 -0500 Subject: [PATCH 16/18] Update website/docs/docs/build/packages.md --- website/docs/docs/build/packages.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index e17ce2184f2..741a001faf7 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -170,7 +170,6 @@ To use native private packages, you must have one of the following Git providers - [Azure DevOps](/docs/cloud/git/connect-azure-devops) - Support for GitLab is coming soon. -To use this feature, you must have a supported Git provider configured in the **Integrations** section of your **Account settings**. #### Configuration From c6220691bf3e2527303ec9b138a8e0b73ebe8cab Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 17 Dec 2024 17:20:52 -0500 Subject: [PATCH 17/18] Update website/docs/docs/build/packages.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/docs/build/packages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index 741a001faf7..4aa1773a258 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -173,7 +173,7 @@ To use native private packages, you must have one of the following Git providers #### Configuration -Use the `private` block in your `packages.yml` or `dependencies.yml` to clone package repos using your existing dbt Cloud Git integration without having to provision an access token or create a dbt Cloud environment variable: +Use the `private` key in your `packages.yml` or `dependencies.yml` to clone package repos using your existing dbt Cloud Git integration without having to provision an access token or create a dbt Cloud environment variable: From 342d913f7592bd2eeb1f5b2cc4b56ce4d9fcf543 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 17 Dec 2024 18:25:27 -0500 Subject: [PATCH 18/18] Update website/docs/docs/build/packages.md Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com> --- website/docs/docs/build/packages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index 4aa1773a258..82ba2c3d74c 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -229,7 +229,7 @@ If you're using dbt Cloud, the SSH key method will not work, but you can use the :::note -dbt Cloud has [native support](#native-private-packages) for Git hosted private packages for Github and Azure DevOps (GitLab coming soon). If you are using a supported [integrated Git environment](/docs/cloud/git/git-configuration-in-dbt-cloud), you no longer need to configure Git tokens to retrieve private packages. +dbt Cloud has [native support](#native-private-packages) for Git hosted private packages with GitHub and Azure DevOps (GitLab coming soon). If you are using a supported [integrated Git environment](/docs/cloud/git/git-configuration-in-dbt-cloud), you no longer need to configure Git tokens to retrieve private packages. :::