From f90ef937a59c0ec67b6c145a8bb24cdda18af170 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Mon, 16 Dec 2024 12:28:18 -0500 Subject: [PATCH] Private packages beta --- website/docs/docs/build/packages.md | 70 +++++++++++++++++++++++++++-- 1 file changed, 66 insertions(+), 4 deletions(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index 9ba4ceeaff5..f30fbd05741 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -157,9 +157,64 @@ packages: Where `name: 'dbt_utils'` specifies the subfolder of `dbt_packages` that's created for the package source code to be installed within. -### Private packages +## Private packages -#### SSH Key Method (Command Line only) +### Native private packages + +dbt Cloud supports private packages from [supported](#prerequisites) Git repos leveraging an exisiting [configuration](/docs/cloud/git/git-configuration-in-dbt-cloud) in your environment. Previously, you had to configure a [token](#git-token-method) to retrieve packages from your private repos. + +#### Prerequisites + +Native support for private packages is supported on the following services: +- [Github](/docs/cloud/git/connect-github) +- [Azure DevOps](/docs/cloud/git/connect-azure-devops) + +Support for Gitlab is coming soon. + +You must have the Git service configured in the **Integrations** section of your **Account settings** to use this feature. + +#### Configuration + +Use the `private` block in your `packages.yml` or `dependencies.yml` to clone package repos using your existing dbt Cloud Git integration without having to provision an access token or create a dbt Cloud environment variable: + + + +```yaml + +packages: + - private: dbt-labs/awesome_repo + + - package: normal packages + + [...] + +``` + + + +You can pin private packages similar to regular dbt packages: + +```yaml + +packages: + - private: dbt-labs/awesome_repo + revision: branch name | tagged git release | a specific commit (full 40-character hash} + +``` + +If you are using multiple Git integrations, disambiguate by adding the provider key: + +```yaml + +packages: + - private: dbt-labs/awesome_repo + provider: "github" | "azure" + +``` + +With this method, you can retrieve private packages from an integrated Git provider without any additional steps to connect. + +### SSH Key Method (Command Line only) If you're using the Command Line, private packages can be cloned via SSH and an SSH key. When you use SSH keys to authenticate to your git remote server, you don’t need to supply your username and password each time. Read more about SSH keys, how to generate them, and how to add them to your git provider here: [Github](https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh) and [GitLab](https://docs.gitlab.com/ee/user/ssh.html). @@ -177,7 +232,14 @@ packages: If you're using dbt Cloud, the SSH key method will not work, but you can use the [HTTPS Git Token Method](https://docs.getdbt.com/docs/build/packages#git-token-method). -#### Git token method +### Git token method + +:::note + +dbt Cloud has [native support](#native-private-packages) for Git hosted private packages for Github and Azure DevOps (GitLab coming soon). If you are using a supported [integrated Git environment](/docs/cloud/git/git-configuration-in-dbt-cloud), you no longer need to configure Git tokens to retrieve private packages. + +::: + This method allows the user to clone via HTTPS by passing in a git token via an environment variable. Be careful of the expiration date of any token you use, as an expired token could cause a scheduled run to fail. Additionally, user tokens can create a challenge if the user ever loses access to a specific repo. @@ -246,7 +308,7 @@ Read more about creating a Personal Access Token [here](https://confluence.atlas -#### Configure subdirectory for packaged projects +## Configure subdirectory for packaged projects In general, dbt expects `dbt_project.yml` to be located as a top-level file in a package. If the packaged project is instead nested in a subdirectory—perhaps within a much larger mono repo—you can optionally specify the folder path as `subdirectory`. dbt will attempt a [sparse checkout](https://git-scm.com/docs/git-sparse-checkout) of just the files located within that subdirectory. Note that you must be using a recent version of `git` (`>=2.26.0`).