Skip to content

Commit

Permalink
Merge branch 'current' into jerco/oss-expectations-updates
Browse files Browse the repository at this point in the history
  • Loading branch information
jtcohen6 authored Jun 5, 2024
2 parents 8a5d88e + 144ddc5 commit ab9759a
Show file tree
Hide file tree
Showing 114 changed files with 453 additions and 1,120 deletions.
2 changes: 1 addition & 1 deletion website/blog/2021-11-22-dbt-labs-pr-template.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,4 +252,4 @@ Once the file is added, name it whatever you want to make it clear that it’s y

With that, you now have a pull request template in your GitHub repository that can help your team follow analytics engineering best practices.

To dive deeper into how we use it as part of the analytics engineering workflow, check out the free [dbt Fundamentals on-demand course](https://courses.getdbt.com/courses/fundamentals).
To dive deeper into how we use it as part of the analytics engineering workflow, check out the free [dbt Fundamentals on-demand course](https://learn.getdbt.com/courses/dbt-fundamentals).
2 changes: 1 addition & 1 deletion website/blog/2021-11-22-primary-keys.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,4 +140,4 @@ You can query out primary key columns from the `pg_index` and `pg_attribute` adm

## Have you started testing primary keys yet?

If you’re looking for a deeper dive on testing primary keys, definitely check out the [dbt Fundamentals course](https://courses.getdbt.com/courses/fundamentals), which includes a full section with examples + practice on data testing in dbt.
If you’re looking for a deeper dive on testing primary keys, definitely check out the [dbt Fundamentals course](https://learn.getdbt.com/courses/dbt-fundamentals), which includes a full section with examples + practice on data testing in dbt.
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ We’re going to:

**Project Appearance**

Let's check in on the growth of [our project](https://github.com/dbt-labs/dbt-project-maturity/tree/main/2-toddlerhood). We've broken some of our logic into its own model — our original script had repetitive logic in <Term id="subquery">subqueries</Term>, now it's following a key principle of analytics engineering: <Term id="dry">Don't Repeat Yourself (DRY)</Term>. For more information on how to refactor your SQL queries for Modularity - check out our [free on-demand course](https://courses.getdbt.com/courses/refactoring-sql-for-modularity).
Let's check in on the growth of [our project](https://github.com/dbt-labs/dbt-project-maturity/tree/main/2-toddlerhood). We've broken some of our logic into its own model — our original script had repetitive logic in <Term id="subquery">subqueries</Term>, now it's following a key principle of analytics engineering: <Term id="dry">Don't Repeat Yourself (DRY)</Term>. For more information on how to refactor your SQL queries for Modularity - check out our [free on-demand course](https://learn.getdbt.com/courses/refactoring-sql-for-modularity).

We also added our first [YML files](https://circleci.com/blog/what-is-yaml-a-beginner-s-guide/). Here, we have one yml file to [configure our sources](https://github.com/dbt-labs/dbt-project-maturity/blob/main/2-toddlerhood/models/source.yml), and one one yml file to [describe our models](https://github.com/dbt-labs/dbt-project-maturity/blob/main/2-toddlerhood/models/schema.yml). We're just starting with basic declarations of our sources, <Term id="primary-key" /> testing using dbt built in tests, and a model-level description -- these are the first steps of a project just learning to walk!

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -229,7 +229,7 @@ This is where the power of dbt modeling really comes in! dbt allows you to break
The following are some methods I’ve used in order to properly optimize run times, leveraging dbt’s ability to modularize models.

:::note Note
I won’t get into our modeling methodology at dbt Labs in this article, but there are [plenty of resources](https://courses.getdbt.com/) to understand what might be happening in the following DAGs!
I won’t get into our modeling methodology at dbt Labs in this article, but there are [plenty of resources](https://learn.getdbt.com/) to understand what might be happening in the following DAGs!
:::

### Staggered joins
Expand Down
2 changes: 1 addition & 1 deletion website/blog/2022-07-19-migrating-from-stored-procs.md
Original file line number Diff line number Diff line change
Expand Up @@ -221,5 +221,5 @@ dbt Labs has developed a number of related resources you can use to learn more a

- [Refactoring legacy SQL to dbt](https://docs.getdbt.com/tutorial/refactoring-legacy-sql)
- [The case for the ELT workflow](https://www.getdbt.com/analytics-engineering/case-for-elt-workflow/)
- [Refactoring SQL for modularity](https://courses.getdbt.com/courses/refactoring-sql-for-modularity)
- [Refactoring SQL for modularity](https://learn.getdbt.com/courses/refactoring-sql-for-modularity)
- [Data modeling techniques for modularity](https://www.getdbt.com/analytics-engineering/modular-data-modeling-technique/)
2 changes: 1 addition & 1 deletion website/blog/2022-11-30-dbt-project-evaluator.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Throughout these engagements, we began to take note of the common issues many an

Maybe your team is facing some of these issues right now 👀 And that’s okay! We know that building an effective, scalable dbt project takes a lot of effort and brain power. Maybe you’ve inherited a legacy dbt project with a mountain of tech debt. Maybe you’re starting from scratch. Either way it can be difficult to know the best way to set your team up for success. Don’t worry, you’re in the right place!

Through solving these problems over and over, the Professional Services team began to hone our best practices for working with dbt and how analytics engineers could improve their dbt project. We added “solutions reviews'' to our list of service offerings — client engagements in which we evaluate a given dbt project and provide specific recommendations to improve performance, save developer time, and prevent misuse of dbt’s features. And in an effort to share these best practices with the wider dbt community, we developed a *lot* of content. We wrote articles on the Developer Blog (see [1](https://docs.getdbt.com/blog/on-the-importance-of-naming), [2](https://discourse.getdbt.com/t/your-essential-dbt-project-checklist/1377), and [3](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview)), gave [Coalesce talks](https://www.getdbt.com/coalesce-2020/auditing-model-layers-and-modularity-with-your-dag/), and created [training courses](https://courses.getdbt.com/courses/refactoring-sql-for-modularity).
Through solving these problems over and over, the Professional Services team began to hone our best practices for working with dbt and how analytics engineers could improve their dbt project. We added “solutions reviews'' to our list of service offerings — client engagements in which we evaluate a given dbt project and provide specific recommendations to improve performance, save developer time, and prevent misuse of dbt’s features. And in an effort to share these best practices with the wider dbt community, we developed a *lot* of content. We wrote articles on the Developer Blog (see [1](https://docs.getdbt.com/blog/on-the-importance-of-naming), [2](https://discourse.getdbt.com/t/your-essential-dbt-project-checklist/1377), and [3](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview)), gave [Coalesce talks](https://www.getdbt.com/coalesce-2020/auditing-model-layers-and-modularity-with-your-dag/), and created [training courses](https://learn.getdbt.com/courses/refactoring-sql-for-modularity).

TIme and time again, we found that when teams are aligned with these best practices, their projects are more:

Expand Down
2 changes: 1 addition & 1 deletion website/blog/2023-03-30-guide-to-debug-in-jinja.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ date: 2023-03-29
is_featured: true
---

*Editor's note—this post assumes intermediate knowledge of Jinja and macros development in dbt. For an introduction to Jinja in dbt check out [the documentation](https://docs.getdbt.com/docs/build/jinja-macros) and the free self-serve course on [Jinja, Macros, Pacakages](https://courses.getdbt.com/courses/jinja-macros-packages).*
*Editor's note—this post assumes intermediate knowledge of Jinja and macros development in dbt. For an introduction to Jinja in dbt check out [the documentation](https://docs.getdbt.com/docs/build/jinja-macros) and the free self-serve course on [Jinja, Macros, Packages](https://learn.getdbt.com/courses/jinja-macros-and-packages).*

Jinja brings a lot of power to dbt, allowing us to use `ref()`, `source()` , conditional code, and [macros](https://docs.getdbt.com/docs/build/jinja-macros). But, while Jinja brings flexibility, it also brings complexity, and like many times with code, things can run in expected ways.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ Now that we’ve done the pipeline work to set up our metrics for the semantic l
## Our Finance, Operations and GTM teams are all looking at the same metrics 😊
To query to Semantic Layer you have two paths: you can query metrics directly through the Semantic Layer APIs or use one of our [first-class integrations](https://docs.getdbt.com/docs/use-dbt-semantic-layer/avail-sl-integrations). Our analytics team and product teams are big Hex users, while our operations and finance teams live and breathe Google Sheets, so it’s important for us to have the same metric definitions available in both tools.
To query to Semantic Layer you have two paths: you can query metrics directly through the Semantic Layer APIs or use one of our [first-class integrations](https://docs.getdbt.com/docs/cloud-integrations/avail-sl-integrations). Our analytics team and product teams are big Hex users, while our operations and finance teams live and breathe Google Sheets, so it’s important for us to have the same metric definitions available in both tools.
The leg work of building our pipeline and defining metrics is all done, which makes last-mile consumption much easier. First, we set up a launch dashboard in Hex as the source of truth for semantic layer product metrics. This tool is used by cross-functional partners like marketing, sales, and the executive team to easily check product and usage metrics like total semantic layer queries, or weekly active semantic layer users. To set up our Hex connection, we simply enter a few details from our dbt Cloud environment and then we can work with metrics directly in Hex notebooks. We can use the JDBC interface, or use Hex’s GUI metric builder to build reports. We run all our WBRs off this dashboard, which allows us to spot trends in consumption and react quickly to changes in our business.
Expand Down
2 changes: 1 addition & 1 deletion website/blog/2024-05-02-semantic-layer-llm.md
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,6 @@ grant usage on function submit_sl_request(string) to role public;

## Wrapping Up

Building this application has been an absolute blast for multiple reasons. First, we’ve been able to use it internally within the SA org to demonstrate how the semantic layer works. It provides yet another [integration](https://docs.getdbt.com/docs/use-dbt-semantic-layer/avail-sl-integrations) point that further drives home the fundamental value prop of using the Semantic Layer. Secondly, and more importantly, it has served as an example to those customers thinking about (or being pushed to think about) how they can best utilize these technologies to further their goals. Finally, I’ve been able to be heads down, hands on keyboard learning about all of these interesting technologies and stepping back into the role of builder is something I will never turn down!
Building this application has been an absolute blast for multiple reasons. First, we’ve been able to use it internally within the SA org to demonstrate how the semantic layer works. It provides yet another [integration](https://docs.getdbt.com/docs/cloud-integrations/avail-sl-integrations) point that further drives home the fundamental value prop of using the Semantic Layer. Secondly, and more importantly, it has served as an example to those customers thinking about (or being pushed to think about) how they can best utilize these technologies to further their goals. Finally, I’ve been able to be heads down, hands on keyboard learning about all of these interesting technologies and stepping back into the role of builder is something I will never turn down!

Finally, to see the entire code, from Snowflake to Streamlit, check out the repo [here](https://github.com/dpguthrie/dbt-sl-cortex-streamlit-blog/tree/main?tab=readme-ov-file).
16 changes: 0 additions & 16 deletions website/dbt-versions.js
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,6 @@ exports.versions = [
version: "1.5",
EOLDate: "2024-04-27",
},
{
version: "1.4",
EOLDate: "2024-01-25",
},
{
version: "1.3",
EOLDate: "2023-10-12",
},
]

exports.versionedPages = [
Expand Down Expand Up @@ -154,14 +146,6 @@ exports.versionedPages = [
"page": "reference/resource-properties/versions",
"firstVersion": "1.5",
},
{
"page": "reference/dbt-jinja-functions/local-md5",
"firstVersion": "1.4",
},
{
"page": "reference/warehouse-setups/fal-setup",
"firstVersion": "1.3",
},
{
"page": "reference/resource-configs/on_configuration_change",
"firstVersion": "1.6",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ Ready to start transforming your Unity Catalog datasets with dbt?
Check out the resources below for guides, tips, and best practices:

- [How we structure our dbt projects](/best-practices/how-we-structure/1-guide-overview)
- [Self-paced dbt fundamentals training videos](https://courses.getdbt.com/courses/fundamentals)
- [Self-paced dbt fundamentals training course](https://learn.getdbt.com/courses/dbt-fundamentals)
- [Customizing CI/CD](/guides/custom-cicd-pipelines)
- [Debugging errors](/guides/debug-errors)
- [Writing custom generic tests](/best-practices/writing-custom-generic-tests)
Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/build/about-metricflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Before you start, consider the following guidelines:
- Define metrics in YAML and query them using these [new metric specifications](https://github.com/dbt-labs/dbt-core/discussions/7456).
- You must be on [dbt version](/docs/dbt-versions/upgrade-dbt-version-in-cloud) 1.6 or higher to use MetricFlow.
- Use MetricFlow with Snowflake, BigQuery, Databricks, Postgres (dbt Core only), or Redshift.
- Discover insights and query your metrics using the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) and its diverse range of [available integrations](/docs/use-dbt-semantic-layer/avail-sl-integrations).
- Discover insights and query your metrics using the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) and its diverse range of [available integrations](/docs/cloud-integrations/avail-sl-integrations).

## MetricFlow

Expand Down
4 changes: 2 additions & 2 deletions website/docs/docs/build/build-metrics-intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ MetricFlow allows you to:
<Card
title="Available integrations"
body="Discover the diverse range of partners that seamlessly integrate with the powerful dbt Semantic Layer, allowing you to query and unlock valuable insights from your data ecosystem."
link="/docs/use-dbt-semantic-layer/avail-sl-integrations"
link="/docs/cloud-integrations/avail-sl-integrations"
icon="dbt-bit"/>

</div> <br />
Expand All @@ -67,5 +67,5 @@ MetricFlow allows you to:

- [Quickstart guide with the dbt Semantic Layer](/guides/sl-snowflake-qs)
- [The dbt Semantic Layer: what's next](https://www.getdbt.com/blog/dbt-semantic-layer-whats-next/) blog
- [dbt Semantic Layer on-demand courses](https://courses.getdbt.com/courses/semantic-layer)
- [dbt Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer)
- [dbt Semantic Layer FAQs](/docs/use-dbt-semantic-layer/sl-faqs)
34 changes: 0 additions & 34 deletions website/docs/docs/build/custom-aliases.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,31 +73,6 @@ To override dbt's alias name generation, create a macro named `generate_alias_na

The default implementation of `generate_alias_name` simply uses the supplied `alias` config (if present) as the model alias, otherwise falling back to the model name. This implementation looks like this:

<VersionBlock lastVersion="1.4">

<File name='get_custom_alias.sql'>

```jinja2
{% macro generate_alias_name(custom_alias_name=none, node=none) -%}
{%- if custom_alias_name is none -%}
{{ node.name }}
{%- else -%}
{{ custom_alias_name | trim }}
{%- endif -%}
{%- endmacro %}
```

</File>

</VersionBlock>

<VersionBlock firstVersion="1.5">

<File name='get_custom_alias.sql'>
Expand Down Expand Up @@ -176,18 +151,9 @@ If these models should indeed have the same database identifier, you can work ar

#### Model versions

<VersionBlock lastVersion="1.4">

New in v1.5

</VersionBlock>

<VersionBlock firstVersion="1.5">

**Related documentation:**
- [Model versions](/docs/collaborate/govern/model-versions)
- [`versions`](/reference/resource-properties/versions#alias)

By default, dbt will create versioned models with the alias `<model_name>_v<v>`, where `<v>` is that version's unique identifier. You can customize this behavior just like for non-versioned models by configuring a custom `alias` or re-implementing the `generate_alias_name` macro.

</VersionBlock>
47 changes: 1 addition & 46 deletions website/docs/docs/build/exposures.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,6 @@ Exposures make it possible to define and describe a downstream use of your dbt p

Exposures are defined in `.yml` files nested under an `exposures:` key.

<VersionBlock firstVersion="1.4">

<File name='models/<filename>.yml'>

```yaml
Expand Down Expand Up @@ -42,58 +40,15 @@ exposures:
</File>
</VersionBlock>
<VersionBlock lastVersion="1.3">
<File name='models/<filename>.yml'>
```yaml
version: 2

exposures:

- name: weekly_jaffle_report
type: dashboard
maturity: high
url: https://bi.tool/dashboards/1
description: >
Did someone say "exponential growth"?
depends_on:
- ref('fct_orders')
- ref('dim_customers')
- source('gsheets', 'goals')

owner:
name: Callum McData
email: [email protected]
```
</File>
</VersionBlock>
### Available properties
_Required:_
- **name**: a unique exposure name written in [snake case](https://en.wikipedia.org/wiki/Snake_case)
- **type**: one of `dashboard`, `notebook`, `analysis`, `ml`, `application` (used to organize in docs site)
- **owner**: `name` or `email` required; additional properties allowed

<VersionBlock firstVersion="1.4">

_Expected:_
- **depends_on**: list of refable nodes, including `ref`, `source`, and `metric` (While possible, it is highly unlikely you will ever need an `exposure` to depend on a `source` directly)

</VersionBlock>

<VersionBlock lastVersion="1.3">

_Expected:_
- **depends_on**: list of refable nodes, including `ref` and `source` (While possible, it is highly unlikely you will ever need an `exposure` to depend on a `source` directly)

</VersionBlock>
- **depends_on**: list of refable nodes, including `metric`, `ref`, and `source`. While possible, it is highly unlikely you will ever need an `exposure` to depend on a `source` directly.

_Optional:_
- **label**: May contain spaces, capital letters, or special characters.
Expand Down
14 changes: 1 addition & 13 deletions website/docs/docs/build/incremental-strategy.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,16 +47,12 @@ The `merge` strategy is available in dbt-postgres and dbt-redshift beginning in

</VersionBlock>

<VersionBlock firstVersion="1.3">

:::note Snowflake Configurations

dbt v1.3 changed the default materialization for incremental table merges from `temporary table` to `view`. For more information about this change and instructions for setting the configuration to a temp table, please read about [Snowflake temporary tables](/reference/resource-configs/snowflake-configs#temporary-tables).
dbt has changed the default materialization for incremental table merges from `temporary table` to `view`. For more information about this change and instructions for setting the configuration to a temp table, please read about [Snowflake temporary tables](/reference/resource-configs/snowflake-configs#temporary-tables).

:::

</VersionBlock>

### Configuring incremental strategy

The `incremental_strategy` config can either be defined in specific models or
Expand Down Expand Up @@ -90,8 +86,6 @@ select ...

</File>

<VersionBlock firstVersion="1.3">

### Strategy-specific configs

If you use the `merge` strategy and specify a `unique_key`, by default, dbt will entirely overwrite matched rows with new values.
Expand Down Expand Up @@ -134,10 +128,6 @@ select ...

</File>

</VersionBlock>

<VersionBlock firstVersion="1.4">

### About incremental_predicates

`incremental_predicates` is an advanced use of incremental models, where data volume is large enough to justify additional investments in performance. This config accepts a list of any valid SQL expression(s). dbt does not check the syntax of the SQL statements.
Expand Down Expand Up @@ -216,8 +206,6 @@ The syntax depends on how you configure your `incremental_strategy`:
- There's a decent amount of conceptual overlap with the `insert_overwrite` incremental strategy.
:::

</VersionBlock>

### Built-in strategies

Before diving into [custom strategies](#custom-strategies), it's important to understand the built-in incremental strategies in dbt and their corresponding macros:
Expand Down
Loading

0 comments on commit ab9759a

Please sign in to comment.