Skip to content

Commit

Permalink
Merge branch 'current' into bug/prevent-preview-rss
Browse files Browse the repository at this point in the history
  • Loading branch information
breezyfasano authored Sep 19, 2024
2 parents dfb6584 + b6351b1 commit f78eee3
Show file tree
Hide file tree
Showing 18 changed files with 179 additions and 57 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ semantic_models:

Let's review the basics of semantic models:

- 🧱 Consist off **entities, dimensions, and measures**.
- 🧱 Consist of **entities, dimensions, and measures**.
- 🫂 Describe the **semantics and relationships of objects** in the warehouse.
- 1️⃣ Correspond to a **single logical model** in your dbt project.

Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/build/dimensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ mf query --metrics users_created,users_deleted --group-by metric_time__year --or

You can set `is_partition` for time to define specific time spans. Additionally, use the `type_params` section to set `time_granularity` to adjust aggregation details (hourly, daily, weekly, and so on).

<Tabs>
<Tabs queryString="dimension">

<TabItem value="is_partition" label="is_partition">

Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/build/metrics-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ import SLCourses from '/snippets/_sl-course.md';
<SLCourses/>
## Default granularity for metircs
## Default granularity for metrics
It's possible to define a default time granularity for metrics if it's different from the granularity of the default aggregation time dimensions (`metric_time`). This is useful if your time dimension has a very fine grain, like second or hour, but you typically query metrics rolled up at a coarser grain. The granularity can be set using the `time_granularity` parameter on the metric, and defaults to `day`. If day is not available because the dimension is defined at a coarser granularity, it will default to the defined granularity for the dimension.

Expand Down
50 changes: 50 additions & 0 deletions website/docs/docs/cloud/account-settings.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
---
title: "Account settings in dbt Cloud"
sidebar_label: "Account settings"
description: "Learn how to enable account settings for your dbt Cloud users."
---

The following sections describe the different **Account settings** available from your dbt Cloud account in the sidebar (under your account name on the lower left-hand side).

<Lightbox src="/img/docs/dbt-cloud/example-sidebar-account-settings.png" title="Example of Account settings from the sidebar" />

## Git repository caching <Lifecycle status="enterprise" />

At the start of every job run, dbt Cloud clones the project's Git repository so it has the latest versions of your project's code and runs `dbt deps` to install your dependencies.

For improved reliability and performance on your job runs, you can enable dbt Cloud to keep a cache of the project's Git repository. So, if there's a third-party outage that causes the cloning operation to fail, dbt Cloud will instead use the cached copy of the repo so your jobs can continue running as scheduled.

dbt Cloud caches your project's Git repo after each successful run and retains it for 8 days if there are no repo updates. It caches all packages regardless of installation method and does not fetch code outside of the job runs.

dbt Cloud will use the cached copy of your project's Git repo under these circumstances:

- Outages from third-party services (for example, the [dbt package hub](https://hub.getdbt.com/)).
- Git authentication fails.
- There are syntax errors in the `packages.yml` file. You can set up and use [continuous integration (CI)](/docs/deploy/continuous-integration) to find these errors sooner.
- If a package doesn't work with the current dbt version. You can set up and use [continuous integration (CI)](/docs/deploy/continuous-integration) to identify this issue sooner.

To use, select the **Enable repository caching** option from your account settings.

<Lightbox src="/img/docs/deploy/example-account-settings.png" width="85%" title="Example of the Enable repository caching option" />

## Partial parsing

At the start of every dbt invocation, dbt reads all the files in your project, extracts information, and constructs an internal manifest containing every object (model, source, macro, and so on). Among other things, it uses the `ref()`, `source()`, and `config()` macro calls within models to set properties, infer dependencies, and construct your project's DAG. When dbt finishes parsing your project, it stores the internal manifest in a file called `partial_parse.msgpack`.

Parsing projects can be time-consuming, especially for large projects with hundreds of models and thousands of files. To reduce the time it takes dbt to parse your project, use the partial parsing feature in dbt Cloud for your environment. When enabled, dbt Cloud uses the `partial_parse.msgpack` file to determine which files have changed (if any) since the project was last parsed, and then it parses _only_ the changed files and the files related to those changes.

Partial parsing in dbt Cloud requires dbt version 1.4 or newer. The feature does have some known limitations. Refer to [Known limitations](/reference/parsing#known-limitations) to learn more about them.

To use, select the **Enable partial parsing between deployment runs** option from your account settings.

<Lightbox src="/img/docs/deploy/example-account-settings.png" width="85%" title="Example of the Enable partial parsing between deployment runs option" />

## Account access to Advanced CI features <Lifecycle status="beta" />

[Advanced CI](/docs/deploy/advanced-ci) features, such as [compare changes](/docs/deploy/advanced-ci#compare-changes), allow dbt Cloud account members to view details about the changes between what's in the production environment and the pull request.

To use Advanced CI features, your dbt Cloud account must have access to them. Ask your dbt Cloud administrator to enable Advanced CI features on your account, which they can do by choosing the **Enable account access to Advanced CI** option from the account settings.

Once enabled, the **Run compare changes** option becomes available in the CI job settings for you to select.

<Lightbox src="/img/docs/deploy/example-account-settings.png" width="85%" title="Example of the Enable account access to Advanced CI option" />
2 changes: 1 addition & 1 deletion website/docs/docs/dbt-versions/2023-release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ Archived release notes for dbt Cloud from 2023

Now available for dbt Cloud Enterprise plans is a new option to enable Git repository caching for your job runs. When enabled, dbt Cloud caches your dbt project's Git repository and uses the cached copy instead if there's an outage with the Git provider. This feature improves the reliability and stability of your job runs.

To learn more, refer to [Repo caching](/docs/deploy/deploy-environments#git-repository-caching).
To learn more, refer to [Repo caching](/docs/cloud/account-settings#git-repository-caching).

<Lightbox src="/img/docs/deploy/example-account-settings.png" width="85%" title="Example of the Repository caching option" />

Expand Down
4 changes: 2 additions & 2 deletions website/docs/docs/dbt-versions/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -309,7 +309,7 @@ The following features are new or enhanced as part of our [dbt Cloud Launch Show

January also saw some refreshed content, either aligning with new product features or requests from the community:

- Native support for [partial parsing in dbt Cloud](https://docs.getdbt.com/docs/dbt-cloud-environments#partial-parsing)
- Native support for [partial parsing in dbt Cloud](https://docs.getdbt.com/docs/cloud/account-settings#partial-parsing)
- Updated guidance on using dots or underscores in the [Best practice guide for models](https://docs.getdbt.com/best-practices/how-we-style/1-how-we-style-our-dbt-models)
- Updated [PrivateLink for VCS docs](https://docs.getdbt.com/docs/cloud/secure/vcs-privatelink)
- Added a new `job_runner` role in our [Enterprise project role permissions docs](https://docs.getdbt.com/docs/cloud/manage-access/enterprise-permissions#project-role-permissions)
Expand All @@ -326,7 +326,7 @@ The following features are new or enhanced as part of our [dbt Cloud Launch Show

By default, dbt parses all the files in your project at the beginning of every dbt invocation. Depending on the size of your project, this operation can take a long time to complete. With the new partial parsing feature in dbt Cloud, you can reduce the time it takes for dbt to parse your project. When enabled, dbt Cloud parses only the changed files in your project instead of parsing all the project files. As a result, your dbt invocations will take less time to run.

To learn more, refer to [Partial parsing](/docs/dbt-cloud-environments#partial-parsing).
To learn more, refer to [Partial parsing](/docs/cloud/account-settings#partial-parsing).

<Lightbox src="/img/docs/deploy/example-account-settings.png" width="85%" title="Example of the Partial parsing option" />

Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/deploy/about-ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ pagination_next: "docs/deploy/continuous-integration"
hide_table_of_contents: true
---

Use [CI jobs](/docs/deploy/ci-jobs) in dbt Cloud to set up automation for testing code changes before merging to production. Additionally, [enable Advanced CI features](/docs/dbt-cloud-environments#account-access-to-advanced-ci-features) for these jobs to evaluate whether the code changes are producing the appropriate data changes you want by reviewing the comparison differences dbt provides.
Use [CI jobs](/docs/deploy/ci-jobs) in dbt Cloud to set up automation for testing code changes before merging to production. Additionally, [enable Advanced CI features](/docs/cloud/account-settings#account-access-to-advanced-ci-features) for these jobs to evaluate whether the code changes are producing the appropriate data changes you want by reviewing the comparison differences dbt provides.

Refer to the guide [Get started with continuous integration tests](/guides/set-up-ci?step=1) for more information.

Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/deploy/ci-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ dbt Labs recommends that you create your CI job in a dedicated dbt Cloud [deploy
- CI features:
- For both the [concurrent CI checks](/docs/deploy/continuous-integration#concurrent-ci-checks) and [smart cancellation of stale builds](/docs/deploy/continuous-integration#smart-cancellation) features, your dbt Cloud account must be on the [Team or Enterprise plan](https://www.getdbt.com/pricing/).
- [Advanced CI](/docs/deploy/advanced-ci) features:<Lifecycle status="preview" />
- For the [compare changes](/docs/deploy/advanced-ci#compare-changes) feature, your dbt Cloud account must have access to Advanced CI. Please ask your [dbt Cloud administrator to enable](/docs/dbt-cloud-environments#account-access-to-advanced-ci-features) this for you.
- For the [compare changes](/docs/deploy/advanced-ci#compare-changes) feature, your dbt Cloud account must have access to Advanced CI. Please ask your [dbt Cloud administrator to enable](/docs/cloud/account-settings#account-access-to-advanced-ci-features) this for you.
- Set up a [connection with your Git provider](/docs/cloud/git/git-configuration-in-dbt-cloud). This integration lets dbt Cloud run jobs on your behalf for job triggering.
- If you're using a native [GitLab](/docs/cloud/git/connect-gitlab) integration, you need a paid or self-hosted account that includes support for GitLab webhooks and [project access tokens](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html). If you're using GitLab Free, merge requests will trigger CI jobs but CI job status updates (success or failure of the job) will not be reported back to GitLab.

Expand Down
102 changes: 102 additions & 0 deletions website/docs/guides/adapter-creation.md
Original file line number Diff line number Diff line change
Expand Up @@ -556,6 +556,108 @@ While much of dbt's adapter-specific functionality can be modified in adapter ma

See [this GitHub discussion](https://github.com/dbt-labs/dbt-core/discussions/5468) for information on the macros required for `GRANT` statements:

### Behavior change flags

Starting in `dbt-adapters==1.5.0` and `dbt-core==1.8.7`, adapter maintainers can implement their own behavior change flags. Refer to [Behavior changes](https://docs.getdbt.com/reference/global-configs/behavior-changes)for more information.

Behavior Flags are not intended to be long-living feature flags. They should be implemented with the expectation that the behavior will be the default within an expected period of time. To implement a behavior change flag, you must provide a name for the flag, a default setting (`True` / `False`), an optional source, and a description and/or a link to the flag's documentation on docs.getdbt.com.

We recommend having a description and documentation link whenever possible. The description and/or docs should provide end users context for why the flag exists, why they may see a warning, and why they may want to utilize the behavior flag. Behavior change flags can be implemented by overwriting `_behavior_flags()` on the adapter in `impl.py`:

<File name='impl.py'>

```python
class ABCAdapter(BaseAdapter):
...
@property
def _behavior_flags(self) -> List[BehaviorFlag]:
return [
{
"name": "enable_new_functionality_requiring_higher_permissions",
"default": False,
"source": "dbt-abc",
"description": (
"The dbt-abc adapter is implementing a new method for sourcing metadata. "
"This is a more performant way for dbt to source metadata but requires higher permissions on the platform. "
"Enabling this without granting the requisite permissions will result in an error. "
"This feature is expected to be required by Spring 2025."
),
"docs_url": "https://docs.getdbt.com/reference/global-configs/behavior-changes#abc-enable_new_functionality_requiring_higher_permissions",
}
]
```

</File>

Once a behavior change flag has been implemented, it can be referenced on the adapter both in `impl.py` and in Jinja macros:

<File name='impl.py'>

```python
class ABCAdapter(BaseAdapter):
...
def some_method(self, *args, **kwargs):
if self.behavior.enable_new_functionality_requiring_higher_permissions:
# do the new thing
else:
# do the old thing
```

</File>

<File name='adapters.sql'>

```sql
{% macro some_macro(**kwargs) %}
{% if adapter.behavior.enable_new_functionality_requiring_higher_permissions %}
{# do the new thing #}
{% else %}
{# do the old thing #}
{% endif %}
{% endmacro %}
```

</File>

Every time the behavior flag evaluates to `False,` it warns the user, informing them that a change will occur in the future.

This warning doesn't display when the flag evaluates to `True` as the user is already in the new experience.

Recognizing that the warnings can be disruptive and are not always necessary, you can evaluate the flag without triggering the warning. Simply append `.no_warn` to the end of the flag.


<File name='impl.py'>

```python
class ABCAdapter(BaseAdapter):
...
def some_method(self, *args, **kwargs):
if self.behavior.enable_new_functionality_requiring_higher_permissions.no_warn:
# do the new thing
else:
# do the old thing
```

</File>

<File name='adapters.sql'>

```sql
{% macro some_macro(**kwargs) %}
{% if adapter.behavior.enable_new_functionality_requiring_higher_permissions.no_warn %}
{# do the new thing #}
{% else %}
{# do the old thing #}
{% endif %}
{% endmacro %}
```

</File>

It's best practice to evaluate a behavior flag as few times as possible. This will make it easier to remove once the behavior change has matured.

As a result, evaluating the flag earlier in the logic flow is easier. Then, take either the old or the new path. While this may create some duplication in code, using behavior flags in this way provides a safer way to implement a change, which we are already admitting is risky or even breaking in nature.

### Other files

#### `profile_template.yml`
Expand Down
6 changes: 3 additions & 3 deletions website/docs/guides/core-to-cloud-1.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,9 +76,9 @@ This section outlines the steps to set up your dbt Cloud account and configure i
### Additional configuration
Explore these additional configurations for performance and reliability improvements:

1. In **Account settings**, enable [partial parsing](/docs/deploy/deploy-environments#partial-parsing) to only reparse changed files, saving time.
1. In **Account settings**, enable [partial parsing](/docs/cloud/account-settings#partial-parsing) to only reparse changed files, saving time.

2. In **Account settings**, enable [Git repo caching](/docs/deploy/deploy-environments#git-repository-caching) for job reliability & third-party outage protection. <Lifecycle status='enterprise' />
2. In **Account settings**, enable [Git repo caching](/docs/cloud/account-settings#git-repository-caching) for job reliability & third-party outage protection. <Lifecycle status='enterprise' />

## Data platform setup

Expand Down Expand Up @@ -142,7 +142,7 @@ The most common data environments are production, staging, and development. The
- Streamlining the process of switching between development, staging, and production contexts.
- Making it easy to configure environments through the dbt Cloud UI instead of manually editing the `profiles.yml` file. You can also [set up](/reference/dbt-jinja-functions/target) or [customize](/docs/build/custom-target-names) target names in dbt Cloud.
- Adding `profiles.yml` attributes to dbt Cloud environment settings with [Extended Attributes](/docs/dbt-cloud-environments#extended-attributes).
- Using [Git repo caching](/docs/dbt-cloud-environments#git-repository-caching) to protect you from third-party outages, Git auth failures, and more. <Lifecycle status="enterprise"/>
- Using [Git repo caching](/docs/cloud/account-settings#git-repository-caching) to protect you from third-party outages, Git auth failures, and more. <Lifecycle status="enterprise"/>

### Initial setup steps
1. **Set up development environment** &mdash; Set up your [development](/docs/dbt-cloud-environments#create-a-development-environment) environment and [development credentials](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#access-the-cloud-ide). You’ll need this to access your dbt project and start developing.
Expand Down
6 changes: 3 additions & 3 deletions website/docs/guides/core-to-cloud-3.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,11 +99,11 @@ dbt Cloud provides robust orchestration that enables you to schedule, run, and m

### Tips

- Enable [partial parsing](/docs/deploy/deploy-environments#partial-parsing) between jobs in dbt Cloud to significantly speed up project parsing by only processing changed files, optimizing performance for large projects.
- Enable [partial parsing](/docs/cloud/account-settings#partial-parsing) between jobs in dbt Cloud to significantly speed up project parsing by only processing changed files, optimizing performance for large projects.
- [Run multiple CI/CD](/docs/deploy/continuous-integration) jobs at the same time which will not block production runs. The Job scheduler automatically cancels stale runs when a newer commit is pushed. This is because each PR will run in its own schema.
- dbt Cloud automatically [cancels](/docs/deploy/job-scheduler#run-cancellation-for-over-scheduled-jobs) a scheduled run if the existing run is still executing. This prevents unnecessary, duplicative executions.
- Protect you and your data freshness from third-party outages by enabling dbt Cloud’s [Git repository caching](/docs/deploy/deploy-environments#git-repository-caching), which keeps a cache of the project's Git repository. <Lifecycle status="enterprise"/>
- [Link deploy jobs](/docs/deploy/deploy-jobs#trigger-on-job-completion--) across dbt Cloud projects by configuring your job or using the [Create Job API](/dbt-cloud/api-v2#/operations/Create%20Job) to do this. <Lifecycle status="team,enterprise"/>
- Protect you and your data freshness from third-party outages by enabling dbt Cloud’s [Git repository caching](/docs/cloud/account-settings#git-repository-caching), which keeps a cache of the project's Git repository. <Lifecycle status="enterprise"/>
- [Link deploy jobs](/docs/deploy/deploy-jobs#trigger-on-job-completion) across dbt Cloud projects by configuring your job or using the [Create Job API](/dbt-cloud/api-v2#/operations/Create%20Job) to do this. <Lifecycle status="team,enterprise"/>
- [Rerun your jobs](/docs/deploy/retry-jobs) from the start or the point of failure if your dbt job run completed with a status of **`Error.`**

### Caveats
Expand Down
Loading

0 comments on commit f78eee3

Please sign in to comment.