Skip to content

Commit

Permalink
Merge branch 'current' into mirnawong1-patch-12
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Oct 10, 2023
2 parents 1c6c207 + c13517e commit af8c9fb
Show file tree
Hide file tree
Showing 13 changed files with 312 additions and 82 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/labeler.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@

name: "Pull Request Labeler"
on:
- pull_request_target

pull_request_target:
types: [opened]
jobs:
triage:
permissions:
Expand Down
15 changes: 9 additions & 6 deletions website/docs/docs/build/semantic-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,20 @@ sidebar_label: Semantic models
tags: [Metrics, Semantic Layer]
---

Semantic models serve as the foundation for defining data in MetricFlow, which powers the dbt Semantic Layer. You can think of semantic models as nodes in your semantic graph, connected via entities as edges. MetricFlow takes semantic models defined in YAML configuration files as inputs and creates a semantic graph that can be used to query metrics.
Semantic models are the foundation for data definition in MetricFlow, which powers the dbt Semantic Layer:

Each semantic model corresponds to a dbt model in your DAG. Therefore you will have one YAML config for each semantic model in your dbt project. You can create multiple semantic models out of a single dbt model, as long as you give each semantic model a unique name.

You can configure semantic models in your dbt project directory in a `YAML` file. Depending on your project structure, you can nest semantic models under a `metrics:` folder or organize them under project sources.
- Think of semantic models as nodes connected by entities in a semantic graph.
- MetricFlow uses YAML configuration files to create this graph for querying metrics.
- Each semantic model corresponds to a dbt model in your DAG, requiring a unique YAML configuration for each semantic model.
- You can create multiple semantic models from a single dbt model, as long as you give each semantic model a unique name.
- Configure semantic models in a YAML file within your dbt project directory.
- Organize them under a `metrics:` folder or within project sources as needed.

Semantic models have 6 components and this page explains the definitions with some examples:

| Component | Description | Type |
| --------- | ----------- | ---- |
| [Name](#name) | Unique name for the semantic model | Required |
| [Name](#name) | Choose a unique name for the semantic model. Avoid using double underscores (__) in the name as they're not supported. | Required |
| [Description](#description) | Includes important details in the description | Optional |
| [Model](#model) | Specifies the dbt model for the semantic model using the `ref` function | Required |
| [Defaults](#defaults) | The defaults for the model, currently only `agg_time_dimension` is supported. | Required |
Expand Down Expand Up @@ -107,7 +110,7 @@ semantic_models:
### Name
Define the name of the semantic model. You must define a unique name for the semantic model. The semantic graph will use this name to identify the model, and you can update it at any time.
Define the name of the semantic model. You must define a unique name for the semantic model. The semantic graph will use this name to identify the model, and you can update it at any time. Avoid using double underscores (__) in the name as they're not supported.
### Description
Expand Down
192 changes: 141 additions & 51 deletions website/docs/docs/cloud/cloud-cli-installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,99 +12,189 @@ These instructions are not intended for general audiences at this time.

:::

## Installing dbt Cloud CLI

### Install and update with Brew on MacOS (recommended)
dbt Cloud natively supports developing using a command line (CLI), empowering team members to contribute with enhanced flexibility and collaboration. The dbt Cloud CLI allows you to run dbt commands against your dbt Cloud development environment from your local command line.

1. Install the dbt Cloud CLI:
dbt commands are run against dbt Cloud's infrastructure and benefit from:

* Secure credential storage in the dbt Cloud platform.
* Automatic deferral of build artifacts to your Cloud project's production environment.
* Speedier, lower-cost builds.
* Support for dbt Mesh ([cross-project `ref`](/docs/collaborate/govern/project-dependencies)),
* Significant platform improvements, to be released over the coming months.


## Install dbt Cloud CLI

You can install the dbt Cloud CLI on the command line by using one of these methods:

<Tabs queryString="install">

<TabItem value="brew" label="macOS">

Before you begin, make sure you have [Homebrew installed](http://brew.sh/) in your code editor or command line terminal. If your operating system runs into path conflicts, refer to the [FAQs](#faqs).

1. Run the following command to verify that there is no conflict with a dbt Core installation on your system:

```bash
which dbt
```
- This should return a `dbt not found`. If the dbt help text appears, use `pip uninstall dbt` to deactivate dbt Core from your machine.

2. Install the dbt Cloud CLI with Homebrew:

```bash
brew tap dbt-labs/dbt-cli
brew install dbt-cloud-cli
```

2. Verify the installation by requesting your homebrew installation path (not your dbt core installs). If the `which dbt` command returns nothing, then you should modify your PATH in `~.zshrc` or create an alias.
3. Verify the installation by running `dbt --help` from the command line. If the help text doesn't indicate that you're using the dbt Cloud CLI, make sure you've deactivated your pyenv or venv and don't have a version of dbt globally installed.

* You no longer need to use the `dbt deps` command. Previously, you had to run that command.

</TabItem>

<TabItem value="windows" label="Windows">

If your operating system runs into path conflicts, refer to the [FAQs](#faqs).

1. Download the latest Windows release for your platform from [GitHub](https://github.com/dbt-labs/dbt-cli/releases).

2. Extract the `dbt.exe` executeable into the same folder as your dbt project.

:::info

Advanced users can configure multiple projects to use the same dbt Cloud CLI by placing the executable in the Program Files folder and [adding it to their Windows PATH environment variable](https://medium.com/@kevinmarkvi/how-to-add-executables-to-your-path-in-windows-5ffa4ce61a53).

Note that if you are using VS Code, you'll need to restart it to pick up modified environment variables.
:::

3. Verify the installation by running `./dbt --help` from the command line. If the help text doesn't indicate that you're using the dbt Cloud CLI, make sure you've deactivated your pyenv or venv and don't have a version of dbt globally installed.

* You no longer need to use the `dbt deps` command. Previously, you had to run that command.

</TabItem>

<TabItem value="linux" label="Linux">

Refer to the [FAQs](#faqs) if your operating system runs into path conflicts.

1. Download the latest Linux release for your platform from [GitHub](https://github.com/dbt-labs/dbt-cli/releases). (Pick the file based on your CPU architecture)

2. Extract the `dbt-cloud-cli` binary to the same folder as your dbt project.

```bash
which dbt
dbt --help
tar -xf dbt_0.29.9_linux_amd64.tar.gz
./dbt --version
```

### Manually install (Windows and Linux)
:::info

1. Download the latest release for your platform from [GitHub](https://github.com/dbt-labs/dbt-cli/releases).
2. Add the `dbt` executable to your path.
3. Move to a directory with a dbt project, and create a `dbt_cloud.yml` file containing your `project-id` from dbt Cloud.
4. Invoke `dbt --help` from your terminal to see a list of supported commands.
Advanced users can configure multiple projects to use the same Cloud CLI executable by adding it to their PATH environment variable in their shell profile.

#### Updating your dbt Cloud installation (Windows + Linux)
:::

Follow the same process in [Installing dbt Cloud CLI](#manually-install-windows-only) and replace the existing `dbt` executable with the new one. You should not have to go through the security steps again.
3. Verify the installation by running `./dbt --help` from the command line. If the help text doesn't indicate that you're using the dbt Cloud CLI, make sure you've deactivated your pyenv or venv and don't have a version of dbt globally installed.

## Setting up the CLI
* You no longer need to use the `dbt deps` command. Previously, you had to run that command.

The following instructions are for setting up the dbt Cloud CLI.
</TabItem>

1. Ensure that you have created a project in [dbt Cloud](https://cloud.getdbt.com/).
</Tabs>

2. Ensure that your personal [development credentials](https://cloud.getdbt.com/settings/profile/credentials) are set on the project.
## Update dbt Cloud CLI

3. Navigate to [your profile](https://cloud.getdbt.com/settings/profile) and enable the **Beta** flag under **Experimental Features.**
The following instructions explain how to update the dbt CLoud CLI to the latest version depending on your operating system.

4. Create an environment variable with your [dbt Cloud API key](https://cloud.getdbt.com/settings/profile#api-access):
<Tabs>

<TabItem value="mac" label="macOS">

```bash
vi ~/.zshrc
To update the dbt Cloud CLI, run `brew upgrade dbt-cloud-cli`.

# dbt Cloud CLI
export DBT_CLOUD_API_KEY="1234" # Replace "1234" with your API key
```
</TabItem>
<TabItem value="windowslinux" label="Windows and Linux">

To update, follow the same process explained in [Install manually (Windows)](/docs/cloud/cloud-cli-installation?install=windows#install-dbt-cloud-cli) and replace the existing `dbt.exe` executable with the new one.

</TabItem>

5. Load the new environment variable. Note: You may need to reactivate your Python virtual environment after sourcing your shell's dot file. Alternatively, restart your shell instead of sourcing the shell's dot file
</Tabs>

## Configure the dbt Cloud CLI

After installation, you can configure the dbt Cloud CLI for your dbt Cloud project and use it to run [dbt commands](/reference/dbt-commands) similar to dbt Core. For example, you can execute the following command to compile a project using dbt Cloud:

```bash
source ~/.zshrc
dbt compile
```

6. Navigate to a dbt project
**Prerequisites**

- You must set up a project in dbt Cloud.
- You must have your [personal development credentials](/docs/dbt-cloud-environments#set-developer-credentials) set for that project. The dbt Cloud CLI will use these credentials, stored securely in dbt Cloud, to communicate with your data platform.
- You must [enroll](/docs/dbt-versions/experimental-features) in the dbt Cloud beta features.
- To enroll, navigate to your **Profile Settings** and enable the **Beta** flag under **Experimental Features**.

Once you install the dbt Cloud CLI, you need to configure it to connect to a dbt Cloud project.

1. Ensure you meet the prerequisites above.
2. Create an environment variable with your [dbt Cloud API key](/docs/dbt-cloud-apis/user-tokens):
- On MacOS, Linux, or Windows add an environment variable:

```bash
export DBT_CLOUD_API_KEY="1234" # Replace 1234 with your API key
```

- In Powershell, add an environment variable:
- Note that this variable resets if you restart your shell. To add an environment variable permanently, add a system environment variable in your platform.

3. Navigate to a dbt project in your terminal:

```bash
cd ~/dbt-projects/jaffle_shop
```

7. Create a `dbt_cloud.yml` in the root project directory. The file is required to have a `project-id` field with a valid [project ID](#glossary). Enter the following commands:
4. In your `dbt_project.yml` file, ensure there is a section titled `dbt-cloud`. This section is required to have a `project-id` field with a valid project ID.

```bash
pwd # Input
/Users/user/dbt-projects/jaffle_shop # Output
```
```yaml
# dbt_project.yml
name:
```bash
echo "project-id: '<your project id>'" > dbt_cloud.yml # Input
```
version:
...
```bash
cat dbt_cloud.yml # Input
project-id: '123456' # Output
dbt-cloud:
project-id: PROJECT_ID
```

You can find your project ID by selecting your project and clicking on **Develop** in the navigation bar. Your project ID is the number in the URL: https://cloud.getdbt.com/develop/26228/projects/PROJECT_ID.
- To find your project ID, go to **Develop** in the navigation menu. Select the dbt Cloud project URL, such as `https://cloud.getdbt.com/develop/26228/projects123456`, where the project ID is `123456`.

If `dbt_cloud.yml` already exists, edit the file, and verify the project ID field uses a valid project ID.

#### Upgrade the CLI with Brew
## Use the dbt Cloud CLI

```bash
brew update
brew upgrade dbt-cloud-cli
```
The dbt Cloud CLI shares the same set of commands as dbt Core. When you invoke a dbt command, that command is sent to dbt Cloud for processing.

The dbt Cloud CLI supports [project dependencies](/docs/collaborate/govern/project-dependencies), which is an exciting way to depend on another project using the metadata service in dbt Cloud. It instantly resolves references (or `ref`) to public models defined in other projects. You don't need to execute or analyze these upstream models yourself. Instead, you treat them as an API that returns a dataset.
Share feedback or request features you'd like to see on the [dbt community Slack](https://getdbt.slack.com/archives/C05M77P54FL).


## FAQs

<details>

<summary>What's the difference between the dbt Cloud CLI and dbt Core?</summary>
The dbt Cloud CLI and <a href="https://github.com/dbt-labs/dbt-core">dbt Core</a>, an open-source project, are both command line tools that enable you to run dbt commands. The key distinction is the dbt Cloud CLI is tailored for dbt Cloud's infrastructure and integrates with all its <a href="https://docs.getdbt.com/docs/cloud/about-cloud/dbt-cloud-features">features</a>.

</details>

## Using dbt Cloud CLI
<details>
<summary>How do I solve for path conflicts</summary>
For compatibility, both the dbt Cloud CLI and dbt Core are invoked by running `dbt`. This can create path conflicts if your operating system selects one over the other based on your $PATH environment variable (settings).

**Coming soon**
If you have dbt Core installed locally, ensure that you deactivate your Python environment or uninstall it using `pip uninstall dbt` before proceeding. Alternatively, advanced users can modify the $PATH environment variable to correctly point to the dbt Cloud CLI binary to use both dbt Cloud CLI and dbt Core together.

## Glossary
You can always uninstall the Cloud CLI to return to using dbt Core.
</details>

- **dbt cloud API key:** Your API key found by navigating to the **gear icon**, clicking **Profile Settings**, and scrolling down to **API**.
- **Project ID:** The ID of the dbt project you're working with. Can be retrieved from the dbt Cloud URL after a project has been selected, for example, `https://cloud.getdbt.com/deploy/{accountID}/projects/{projectID}`
- **Development credentials:** Your personal warehouse credentials for the project you’re working with. They can be set by selecting the project and entering them in dbt Cloud. Navigate to the **gear icon**, click **Profile Settings**, and click **Credentials** from the left-side menu.
1 change: 1 addition & 0 deletions website/docs/docs/collaborate/git-version-control.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,4 @@ When you develop in the command line interface (CLI) or Cloud integrated develo
- [Merge conflicts](/docs/collaborate/git/merge-conflicts)
- [Connect to GitHub](/docs/cloud/git/connect-github)
- [Connect to GitLab](/docs/cloud/git/connect-gitlab)
- [Connect to Azure DevOps](/docs/cloud/git/connect-azure-devops)
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ The following profile fields are optional to set up. They let you configure your
| Profile field | Example | Description |
| ----------------------------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `threads` | `8` | How many threads dbt should use (default is `1`) |
| `roles` | `system: analyst` | Catalog roles |
| `roles` | `system: analyst` | Catalog roles can be set under the optional `roles` parameter using following format: `catalog: role`. |
| `session_properties` | `query_max_run_time: 4h` | Sets Trino session properties used in the connection. Execute `SHOW SESSION` to see available options |
| `prepared_statements_enabled` | `true` or `false` | Enable usage of Trino prepared statements (used in `dbt seed` commands) (default: `true`) |
| `retries` | `10` | Configure how many times all database operation is retried when connection issues arise (default: `3`) |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
title: "Enhancement: Native support for the dbt retry command"
description: "October 2023: Rerun errored jobs from start or from the failure point"
sidebar_label: "Enhancement: Support for dbt retry"
tags: [Oct-2023, Scheduler]
date: 2023-10-06
sidebar_position: 10
---

Previously in dbt Cloud, you could only rerun an errored job from start but now you can also rerun it from its point of failure.

You can view which job failed to complete successully, which command failed in the run step, and choose how to rerun it. To learn more, refer to [Retry jobs](/docs/deploy/retry-jobs).


<Lightbox src="/img/docs/deploy/native-retry.gif" width="70%" title="Example of the Rerun options in dbt Cloud"/>
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
title: "September 2023 product docs updates"
id: "product-docs-sept"
description: "September 2023: The Product docs team merged 107 PRs, made various updates to dbt Cloud and Core, such as GAing continuous integration jobs, Semantic Layer GraphQL API doc, a new community plugin, and more"
sidebar_label: "Update: Product docs changes"
tags: [Sept-2023, product-docs]
date: 2023-10-10
sidebar_position: 09
---

Hello from the dbt Docs team: @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun! First, we’d like to thank the 15 new community contributors to docs.getdbt.com. We merged [107 PRs](https://github.com/dbt-labs/docs.getdbt.com/pulls?q=is%3Apr+merged%3A2023-09-01..2023-09-31) in September.

Here's what's new to [docs.getdbt.com](http://docs.getdbt.com/):

* Migrated docs.getdbt.com from Netlify to Vercel.

## ☁ Cloud projects
- Continuous integration jobs are now generally available and no longer in beta!
- Added [Postgres PrivateLink set up page](/docs/cloud/secure/postgres-privatelink)
- Published beta docs for [dbt Explorer](/docs/collaborate/explore-projects).
- Added a new Semantic Layer [GraphQL API doc](/docs/dbt-cloud-apis/sl-graphql) and updated the [integration docs](/docs/use-dbt-semantic-layer/avail-sl-integrations) to include Hex. Responded to dbt community feedback and clarified Metricflow use cases for dbt Core and dbt Cloud.
- Added an [FAQ](/faqs/Git/git-migration) describing how to migrate from one git provider to another in dbt Cloud.
- Clarified an example and added a [troubleshooting section](/docs/cloud/connect-data-platform/connect-snowflake#troubleshooting) to Snowflake connection docs to address common errors and provide solutions.


## 🎯 Core projects

- Deprecated dbt Core v1.0 and v1.1 from the docs.
- Added configuration instructions for the [AWS Glue](/docs/core/connect-data-platform/glue-setup) community plugin.
- Revised the dbt Core quickstart, making it easier to follow. Divided this guide into steps that align with the [other guides](/quickstarts/manual-install?step=1).

## New 📚 Guides, ✏️ blog posts, and FAQs

Added a [style guide template](/guides/best-practices/how-we-style/6-how-we-style-conclusion#style-guide-template) that you can copy & paste to make sure you adhere to best practices when styling dbt projects!

## Upcoming changes

Stay tuned for a flurry of releases in October and a filterable guides section that will make guides easier to find!
Loading

0 comments on commit af8c9fb

Please sign in to comment.