Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clarify macro properties aren't supported in dbt_project.yml #3989

Merged
merged 45 commits into from
Dec 18, 2023
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
0a8b4bf
clarify macro aren't support in project.yml file
mirnawong1 Aug 30, 2023
907fde4
Merge branch 'current' into mwong-macro-yml
mirnawong1 Aug 31, 2023
cda991a
add snippet and rephrase
mirnawong1 Aug 31, 2023
66ff1df
Update dbt_project.yml.md
mirnawong1 Aug 31, 2023
293e437
Merge branch 'current' into mwong-macro-yml
mirnawong1 Aug 31, 2023
b163c5b
Update website/docs/reference/macro-properties.md
mirnawong1 Aug 31, 2023
ddd2c63
Update website/docs/reference/macro-properties.md
mirnawong1 Aug 31, 2023
237f6da
Merge branch 'mwong-macro-yml' of https://github.com/dbt-labs/docs.ge…
mirnawong1 Aug 31, 2023
6d8ce4e
update lang
mirnawong1 Aug 31, 2023
0b5a949
Merge branch 'current' into mwong-macro-yml
mirnawong1 Sep 4, 2023
49d7b63
Merge branch 'current' into mwong-macro-yml
mirnawong1 Sep 6, 2023
9a9627a
tweak snippet and add versionblock
mirnawong1 Sep 6, 2023
a56d9a8
remove break
mirnawong1 Sep 6, 2023
db04b02
Merge branch 'current' into mwong-macro-yml
mirnawong1 Sep 7, 2023
3d8dbd6
Merge branch 'current' into mwong-macro-yml
mirnawong1 Sep 11, 2023
d22b7e9
Merge branch 'current' into mwong-macro-yml
mirnawong1 Sep 11, 2023
a24c149
Merge branch 'current' into mwong-macro-yml
mirnawong1 Sep 11, 2023
b01215a
Merge branch 'current' into mwong-macro-yml
mirnawong1 Sep 13, 2023
d5c881f
Merge branch 'current' into mwong-macro-yml
mirnawong1 Sep 13, 2023
a453e7a
Merge branch 'current' into mwong-macro-yml
mirnawong1 Sep 13, 2023
f4ccb57
Merge branch 'current' into mwong-macro-yml
mirnawong1 Sep 14, 2023
30c7c4d
Merge branch 'current' into mwong-macro-yml
mirnawong1 Sep 18, 2023
02fb137
adding jerco's great feedback
mirnawong1 Sep 18, 2023
1872ce1
clarify
mirnawong1 Sep 18, 2023
56fbe0a
change yaml to `properties.yml`
mirnawong1 Sep 18, 2023
75b1457
Merge branch 'current' into mwong-macro-yml
mirnawong1 Nov 6, 2023
b5e6f28
Merge branch 'current' into mwong-macro-yml
mirnawong1 Nov 7, 2023
5ec1c05
add table
mirnawong1 Nov 7, 2023
95b430d
Merge branch 'current' into mwong-macro-yml
mirnawong1 Nov 8, 2023
ef8c3bc
Update configs-and-properties.md
mirnawong1 Nov 8, 2023
06740f1
Merge branch 'current' into mwong-macro-yml
mirnawong1 Nov 9, 2023
02a3cae
Remove trailing space
dbeatty10 Nov 10, 2023
fa206ce
Merge branch 'current' into mwong-macro-yml
dbeatty10 Nov 10, 2023
f353599
Merge branch 'current' into mwong-macro-yml
dbeatty10 Nov 10, 2023
29c32df
Merge branch 'current' into mwong-macro-yml
mirnawong1 Nov 13, 2023
6c4a0c6
Merge branch 'current' into mwong-macro-yml
dbeatty10 Nov 13, 2023
5baf269
Merge branch 'current' into mwong-macro-yml
mirnawong1 Nov 15, 2023
ca47c06
Merge branch 'current' into mwong-macro-yml
mirnawong1 Nov 20, 2023
e332c34
Merge branch 'current' into mwong-macro-yml
mirnawong1 Nov 22, 2023
a1a8e7f
Update docs.md
mirnawong1 Nov 22, 2023
169f759
Merge branch 'current' into mwong-macro-yml
mirnawong1 Nov 29, 2023
e27bb9d
Merge branch 'current' into mwong-macro-yml
dbeatty10 Dec 17, 2023
7ad36fd
Update dbt_project.yml.md
dbeatty10 Dec 17, 2023
c26bc39
Remove extra newline
dbeatty10 Dec 17, 2023
9225af1
Merge branch 'current' into mwong-macro-yml
mirnawong1 Dec 18, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion website/docs/docs/build/semantic-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,25 @@ sidebar_label: Semantic models
tags: [Metrics, Semantic Layer]
---

import PropsCallout from '/snippets/_config-prop-callout.md';

Semantic models serve as the foundation for defining data in MetricFlow, which powers the dbt Semantic Layer. You can think of semantic models as nodes in your semantic graph, connected via entities as edges. MetricFlow takes semantic models defined in YAML configuration files as inputs and creates a semantic graph that can be used to query metrics.

Each semantic model corresponds to a dbt model in your DAG. Therefore you will have one YAML config for each semantic model in your dbt project. You can create multiple semantic models out of a single dbt model, as long as you give each semantic model a unique name.

You can configure semantic models in your dbt project directory in a `YAML` file. Depending on your project structure, you can nest semantic models under a `metrics:` folder or organize them under project sources.
<VersionBlock firstVersion="1.7">

You can configure semantic models in your dbt project directory in a `YAML` file, _except_ the `dbt_project.yml` file. Depending on your project structure, you can nest semantic models under a `metrics:` folder or organize them under project sources.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

except the dbt_project.yml file

This will change in v1.7:


Update: I've reread this now, across a few other pages, and I now get your point. You're saying: "You can define properties for sources/models/metrics/etc in a YAML file. You can also configure a number of these resources at once in dbt_project.yml. But you cannot actually define their properties within dbt_project.yml, even though it is also a YAML file associated with your project." (The same holds for a few other "special" YAML files: selectors.yml, packages.yml/dependencies.yml, profiles.yml, ...)

I found this confusing on the first read. I'm trying to think of what else we could do to resolve this ambiguity.

It feels like what we really need is a standard name for the kind of yaml file where these things can be defined. I think we've informally settled on something like "properties yaml file":

Copy link
Contributor Author

@mirnawong1 mirnawong1 Sep 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gotcha, thanks so much and apologies for the confusion! you've summarized my understanding totally right and i think you're right about the standard name -- I'll change it to properties.yml

Copy link
Contributor Author

@mirnawong1 mirnawong1 Sep 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok clarified this by updating the partial:

<span>{props.title} are "special properties" in that you can't configure them in the <code>dbt_project.yml</code> file or using <code>config()</code> blocks. Refer to <a href="https://docs.getdbt.com/reference/configs-and-properties#which-properties-are-not-also-configs" target="_self"> Configs and properties</a> for more info.</span>

which turns content into:

Screenshot 2023-09-18 at 14 24 56


</VersionBlock>

<VersionBlock lastVersion="1.6">

You can configure semantic models in your dbt project directory in a `YAML` file, _except_ the `dbt_project.yml` file. <PropsCallout title={frontMatter.title}/>

Depending on your project structure, you can nest semantic models under a `metrics:` folder or organize them under project sources.

</VersionBlock>

Semantic models have 6 components and this page explains the definitions with some examples:

Expand Down
4 changes: 3 additions & 1 deletion website/docs/reference/analysis-properties.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@
title: Analysis properties
---

We recommend you define analysis properties in your `analyses/` directory, which is illustrated in the [`analysis-paths`](/reference/project-configs/analysis-paths) configuration.
import PropsCallout from '/snippets/_config-prop-callout.md';

We recommend you define analysis properties in your `analyses/` directory, which is illustrated in the [`analysis-paths`](/reference/project-configs/analysis-paths) configuration. <PropsCallout title={frontMatter.title}/> <br />

You can name these files `whatever_you_want.yml`, and nest them arbitrarily deeply in subfolders within the `analyses/` or `models/` directory.

Expand Down
7 changes: 5 additions & 2 deletions website/docs/reference/configs-and-properties.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,15 +64,18 @@ Previous versions of the docs referred to these as `schema.yml` files — we've

### Which properties are _not_ also configs?

dbt has the ability to define node configs in `.yml` files, in addition to `config()` blocks and `dbt_project.yml`. But the reverse isn't always true: there are some things in `.yml` files that can _only_ be defined there.
In dbt, you can define node configs in `.yml` files, in addition to `config()` blocks and `dbt_project.yml`. However, some properties are exclusive to `.yml` files.


Certain properties are special, because:

- They have a unique Jinja rendering context
- They create new project resources
- They don't make sense as hierarchical configuration
- They're older properties that haven't yet been redefined as configs

These properties are:
This means "configs" can be set for multiple resources at once in the `dbt_project.yml` file, while "properties" can only be set for one resource at a time. The following properties _cannot_ be configured from `dbt_project.yml`:

- [`description`](/reference/resource-properties/description)
- [`tests`](/reference/resource-properties/tests)
- [`docs`](/reference/resource-configs/docs)
Expand Down
33 changes: 29 additions & 4 deletions website/docs/reference/dbt_project.yml.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

</Changelog>

Every [dbt project](/docs/build/projects) needs a `dbt_project.yml` file — this is how dbt knows a directory is a dbt project. It also contains important information that tells dbt how to operate on your project.
Every [dbt project](/docs/build/projects) needs a `dbt_project.yml` file — this is how dbt knows a directory is a dbt project. It also contains important information that tells dbt how to operate your project.

<VersionBlock lastVersion="1.4">

Expand All @@ -18,12 +18,37 @@ By default, dbt will look for `dbt_project.yml` in your current working director

</VersionBlock>

The following is a list of all available configurations in the `dbt_project.yml` file.

:::info YAML syntax
dbt uses YAML in a few different places. If you're new to YAML, it would be worth taking the time to learn how arrays, dictionaries and strings are represented.

dbt uses YAML in a few different places. If you're new to YAML, it would be worth taking the time to learn how arrays, dictionaries, and strings are represented.
:::


<VersionBlock firstVersion="1.7">

Something to note, the following properties _aren't_ supported in in the `dbt_project.yml` file:

- [Exposure properties](/reference/exposure-properties)
- [Macro properties](/reference/macro-properties)
- [Analysis properties](/reference/analysis-properties)
- [Source properties](/reference/source-properties)
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

</VersionBlock>

<VersionBlock lastVersion="1.6">

Something to note, the following properties _aren't_ supported in in the `dbt_project.yml` file:

- [Exposure properties](/reference/exposure-properties)
- [Macro properties](/reference/macro-properties)
- [Analysis properties](/reference/analysis-properties)
- [Semantic model configs](/docs/build/semantic-models)
- [Source properties](/reference/source-properties)

</VersionBlock>

The following example is a list of all available configurations in the `dbt_project.yml` file:

<File name='dbt_project.yml'>

```yml
Expand Down
6 changes: 5 additions & 1 deletion website/docs/reference/exposure-properties.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,11 @@ description: "Read this guide to understand exposure properties in dbt."
- [Declaring resource properties](/reference/configs-and-properties)

## Overview
Exposures are defined in `.yml` files nested under an `exposures:` key. You may define `exposures` in YAML files that also define define `sources` or `models`.

import PropsCallout from '/snippets/_config-prop-callout.md';

Exposures are defined in `.yml` files nested under an `exposures:` key. You may define `exposures` in YAML files that also define `sources` or `models`, _except_ in the `dbt_project.yml` file. <PropsCallout title={frontMatter.title}/> <br />
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved


You can name these files `whatever_you_want.yml`, and nest them arbitrarily deeply in subfolders within the `models/` directory.

Expand Down
7 changes: 5 additions & 2 deletions website/docs/reference/macro-properties.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
---
title: Macro properties
id: macro-properties
---

Macro properties can be declared in `.yml` files.
import PropsCallout from '/snippets/_config-prop-callout.md';

You can name these files `whatever_you_want.yml`, and nest them arbitrarily deeply in subfolders.
Macro properties can be declared in any `.yml` file, except for the `dbt_project.yml` file. <PropsCallout title={frontMatter.title}/>

You can name these files `whatever_you_want.yml` and nest them arbitrarily deep in sub-folders.

<File name='macros/<filename>.yml'>

Expand Down
78 changes: 67 additions & 11 deletions website/docs/reference/resource-configs/docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,35 @@ default_value: {show: true}
{ label: 'Macros', value: 'macros', },
]
}>

<TabItem value="models">

You can use the docs property in YAML files, including the `dbt_project.yml`:
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

<File name='dbt_project.yml'>

```yml
name: "your_dbt_project_name"
version: "1.0.0"
config-version: 2

models:
- name: model_name
docs:
show: true | false
```
</File>

<File name='models/schema.yml'>


```yml
version: 2

models:
- name: model_name
docs:
show: true | false

```

</File>
Expand All @@ -43,6 +60,22 @@ This property is not implemented for sources.

<TabItem value="seeds">

You can use the docs property in YAML files, including the `dbt_project.yml`:

<File name='dbt_project.yml'>

```yml
name: "your_dbt_project_name"
version: "1.0.0"
config-version: 2

seeds:
- name: seed_name
docs:
show: true | false
```
</File>

<File name='seeds/schema.yml'>

```yml
Expand All @@ -52,15 +85,29 @@ seeds:
- name: seed_name
docs:
show: true | false

```

</File>

</TabItem>

<TabItem value="snapshots">

You can use the docs property in YAML files, including the `dbt_project.yml`:

<File name='dbt_project.yml'>

```yml
name: "your_dbt_project_name"
version: "1.0.0"
config-version: 2

snapshots:
- name: snapshot_name
docs:
show: true | false
```
</File>

<File name='snapshots/schema.yml'>

```yml
Expand All @@ -70,15 +117,29 @@ snapshots:
- name: snapshot_name
docs:
show: true | false

```

</File>

</TabItem>

<TabItem value="analyses">

You can use the docs property in YAML files, including the `dbt_project.yml`:
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

<File name='dbt_project.yml'>

```yml
name: "your_dbt_project_name"
version: "1.0.0"
config-version: 2

analyses:
- name: analysis_name
docs:
show: true | false
```
</File>
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

<File name='analysis/schema.yml'>

```yml
Expand All @@ -89,16 +150,13 @@ analyses:
docs:
show: true | false
```

</File>

</TabItem>

<TabItem value="macros">

<!----
To-do: check this
--->
You can use the docs property in YAML files, _except_ in `dbt_project.yml`. [Macro properties](/reference/macro-properties) _are not_ supported in the `dbt_project.yml` file.

<File name='macros/schema.yml'>

Expand All @@ -109,9 +167,7 @@ macros:
- name: macro_name
docs:
show: true | false

```

</File>

</TabItem>
Expand Down
10 changes: 7 additions & 3 deletions website/docs/reference/source-properties.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: "About source properties"
title: "Source properties"
description: "Learn how to use source properties in dbt."
---

Expand All @@ -8,9 +8,13 @@ description: "Learn how to use source properties in dbt."
- [Declaring resource properties](/reference/configs-and-properties)

## Overview
Source properties can be declared in `.yml` files in your `models/` directory (as defined by the [`model-paths` config](/reference/project-configs/model-paths)).

You can name these files `whatever_you_want.yml`, and nest them arbitrarily deeply in subfolders within the `models/` directory.
import PropsCallout from '/snippets/_config-prop-callout.md';
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

Source properties can be declared in any `.yml` file in your `models/` directory (as defined by the [`model-paths` config](/reference/project-configs/model-paths)), except for the `dbt_project.yml` file. <PropsCallout title={frontMatter.title}/> <br />


You can name these files `whatever_you_want.yml`, and nest them arbitrarily deeply in subfolders within the `models/` directory:

<File name='models/<filename>.yml'>

Expand Down
1 change: 1 addition & 0 deletions website/snippets/_config-prop-callout.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<span>This is because you can set "configs" for many resources in <code>dbt_project.yml</code>, while you can only set "properties" one resource at a time. Since {props.title} are "properties", this means you cannot configure it from <code>dbt_project.yml</code></span>. Refer to <a href="https://docs.getdbt.com/reference/configs-and-properties#which-properties-are-not-also-configs" target="_self"> Configs and properties</a> for more info.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved