Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adds sl course link and updates fill_nulls_with and join_to_timespine #4912

Merged
merged 44 commits into from
Feb 21, 2024
Merged
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
2f28e2a
snippet and linear tasks
mirnawong1 Feb 13, 2024
1c26f53
updates
mirnawong1 Feb 14, 2024
69e66b8
Merge branch 'current' into add-sl-course-update-fills_nulls_width
mirnawong1 Feb 14, 2024
a145179
Update gsheets.md
mirnawong1 Feb 14, 2024
673c3d4
Update simple.md
mirnawong1 Feb 14, 2024
5b365ba
Merge branch 'current' into add-sl-course-update-fills_nulls_width
mirnawong1 Feb 15, 2024
c0e3fa7
This branch was auto-updated!
github-actions[bot] Feb 15, 2024
3273da0
This branch was auto-updated!
github-actions[bot] Feb 15, 2024
45fac32
This branch was auto-updated!
github-actions[bot] Feb 15, 2024
43b4b4f
This branch was auto-updated!
github-actions[bot] Feb 15, 2024
78fc767
This branch was auto-updated!
github-actions[bot] Feb 15, 2024
b82fca4
This branch was auto-updated!
github-actions[bot] Feb 15, 2024
6306759
This branch was auto-updated!
github-actions[bot] Feb 15, 2024
9ab508b
This branch was auto-updated!
github-actions[bot] Feb 16, 2024
d16b036
This branch was auto-updated!
github-actions[bot] Feb 16, 2024
95b4213
This branch was auto-updated!
github-actions[bot] Feb 16, 2024
03143a0
This branch was auto-updated!
github-actions[bot] Feb 16, 2024
94ddde9
This branch was auto-updated!
github-actions[bot] Feb 16, 2024
72064fe
This branch was auto-updated!
github-actions[bot] Feb 16, 2024
e1cb373
update fill_nulls_with and add_to_time_spine
Jstein77 Feb 17, 2024
4f1f3f7
remove join_to_timespine
Jstein77 Feb 17, 2024
7daa6b4
Merge branch 'current' into add-sl-course-update-fills_nulls_width
mirnawong1 Feb 19, 2024
ffa0aa5
This branch was auto-updated!
github-actions[bot] Feb 19, 2024
3aff58c
Update conversion-metrics.md
mirnawong1 Feb 19, 2024
c32ca90
Update website/docs/docs/build/conversion-metrics.md
mirnawong1 Feb 19, 2024
bd77b3b
Update cumulative-metrics.md
mirnawong1 Feb 19, 2024
53bfc0f
Update cumulative-metrics.md
mirnawong1 Feb 19, 2024
2422fd5
Update website/snippets/_sl-course.md
mirnawong1 Feb 19, 2024
2a3de09
This branch was auto-updated!
github-actions[bot] Feb 19, 2024
372ba3d
This branch was auto-updated!
github-actions[bot] Feb 20, 2024
662a175
This branch was auto-updated!
github-actions[bot] Feb 20, 2024
f83a7c8
This branch was auto-updated!
github-actions[bot] Feb 20, 2024
e7a73b4
This branch was auto-updated!
github-actions[bot] Feb 20, 2024
119bff6
This branch was auto-updated!
github-actions[bot] Feb 21, 2024
e6ea2b7
This branch was auto-updated!
github-actions[bot] Feb 21, 2024
334fc71
This branch was auto-updated!
github-actions[bot] Feb 21, 2024
daa67b2
fix spec
mirnawong1 Feb 21, 2024
af64fb2
fix table
mirnawong1 Feb 21, 2024
9d398d4
Update cumulative-metrics.md
mirnawong1 Feb 21, 2024
e1fd607
Update metrics-overview.md
mirnawong1 Feb 21, 2024
6c82606
Update ratio-metrics.md
mirnawong1 Feb 21, 2024
8018b23
Update semantic-models.md
mirnawong1 Feb 21, 2024
5edce49
Update semantic-models.md
mirnawong1 Feb 21, 2024
a942f8b
Update conversion-metrics.md
mirnawong1 Feb 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 35 additions & 36 deletions website/docs/docs/build/about-metricflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -245,43 +245,42 @@ metrics:

## FAQs

<details>
<summary>Do my datasets need to be normalized?</summary>
<div>
<div>Not at all! While a cleaned and well-modeled data set can be extraordinarily powerful and is the ideal input, you can use any dataset from raw to fully denormalized datasets. <br /><br />It's recommended that you apply quality data consistency, such as filtering bad data, normalizing common objects, and data modeling of keys and tables, in upstream applications. The Semantic Layer is more efficient at doing data denormalization instead of normalization. <br /><br />If you have not invested in data consistency, that is okay. The Semantic Layer can take SQL queries or expressions to define consistent datasets.</div>
</div>
</details>
<details>
<summary>Why is normalized data the ideal input?</summary>
<div>
<div> MetricFlow is built to do denormalization efficiently. There are better tools to take raw datasets and accomplish the various tasks required to build data consistency and organized data models. On the other end, by putting in denormalized data you are potentially creating redundancy which is technically challenging to manage, and you are reducing the potential granularity that MetricFlow can use to aggregate metrics.</div>
</div>
</details>
<details>
<summary>Why not just make metrics the same as measures?</summary>
<div>
<div>One principle of MetricFlow is to reduce the duplication of logic sometimes referred to as Don't Repeat Yourself(DRY).<br /><br />Many metrics are constructed from reused measures and in some cases constructed from measures from different semantic models. This allows for metrics to be built breadth-first (metrics that can stand alone) instead of depth-first (where you have multiple metrics acting as functions of each other).<br /><br />Additionally, not all metrics are constructed off of measures. As an example, a conversion metric is likely defined as the presence or absence of an event record after some other event record.</div>
</div>
</details>
<details>
<summary>How does the Semantic Layer handle joins?</summary>
<div>
<div>MetricFlow builds joins based on the types of keys and parameters that are passed to entities. To better understand how joins are constructed see our documentation on join types.<br /><br />Rather than capturing arbitrary join logic, MetricFlow captures the types of each identifier and then helps the user to navigate to appropriate joins. This allows us to avoid the construction of fan out and chasm joins as well as generate legible SQL.</div>
</div>
</details>
<details>
<summary>Are entities and join keys the same thing?</summary>
<div>
<div>If it helps you to think of entities as join keys, that is very reasonable. Entities in MetricFlow have applications beyond joining two tables, such as acting as a dimension.</div>
</div>
</details>
<details>
<summary>Can a table without a primary or unique entities have dimensions?</summary>
<div>
<div>Yes, but because a dimension is considered an attribute of the primary or unique ent of the table, they are only usable by the metrics that are defined in that table. They cannot be joined to metrics from other tables. This is common in event logs.</div>
</div>
</details>
<detailsToggle alt_header="Do my datasets need to be normalized?">

Not at all! While a cleaned and well-modeled data set can be extraordinarily powerful and is the ideal input, you can use any dataset from raw to fully denormalized datasets.

It's recommended that you apply quality data consistency, such as filtering bad data, normalizing common objects, and data modeling of keys and tables, in upstream applications. The Semantic Layer is more efficient at doing data denormalization instead of normalization.

If you have not invested in data consistency, that is okay. The Semantic Layer can take SQL queries or expressions to define consistent datasets.

</detailsToggle>

<detailsToggle alt_header="Why is normalized data the ideal input?">

MetricFlow is built to do denormalization efficiently. There are better tools to take raw datasets and accomplish the various tasks required to build data consistency and organized data models. On the other end, by putting in denormalized data you are potentially creating redundancy which is technically challenging to manage, and you are reducing the potential granularity that MetricFlow can use to aggregate metrics.
</detailsToggle>

<detailsToggle alt_header="Why not just make metrics the same as measures?">
One principle of MetricFlow is to reduce the duplication of logic sometimes referred to as Don't Repeat Yourself(DRY).

Many metrics are constructed from reused measures and in some cases constructed from measures from different semantic models. This allows for metrics to be built breadth-first (metrics that can stand alone) instead of depth-first (where you have multiple metrics acting as functions of each other).

Additionally, not all metrics are constructed off of measures. As an example, a conversion metric is likely defined as the presence or absence of an event record after some other event record.

</detailsToggle>
<detailsToggle alt_header="How does the dbt Semantic Layer handle joins?">
The dbt Semantic Layer, powered by MetricFlow, builds joins based on the types of keys and parameters that are passed to entities. To better understand how joins are constructed see our documentation on join types.

Rather than capturing arbitrary join logic, MetricFlow captures the types of each identifier and then helps the user to navigate to appropriate joins. This allows us to avoid the construction of fan out and chasm joins as well as generate legible SQL.
</detailsToggle>

<detailsToggle alt_header="Are entities and join keys the same thing?">
If it helps you to think of entities as join keys, that is very reasonable. Entities in MetricFlow have applications beyond joining two tables, such as acting as a dimension.
</detailsToggle>

<detailsToggle alt_header="Can a table without a primary or unique entities have dimensions?">
Yes, but because a dimension is considered an attribute of the primary or unique ent of the table, they are only usable by the metrics that are defined in that table. They cannot be joined to metrics from other tables. This is common in event logs.
</detailsToggle>

## Related docs
- [Joins](/docs/build/join-logic)
Expand Down
5 changes: 1 addition & 4 deletions website/docs/docs/build/build-metrics-intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,12 @@ pagination_prev: null

Use MetricFlow in dbt to centrally define your metrics. As a key component of the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl), MetricFlow is responsible for SQL query construction and defining specifications for dbt semantic models and metrics. It uses familiar constructs like semantic models and metrics to avoid duplicative coding, optimize your development workflow, ensure data governance for company metrics, and guarantee consistency for data consumers.


MetricFlow allows you to:
- Intuitively define metrics in your dbt project
- Develop from your preferred environment, whether that's the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation), [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud), or [dbt Core](/docs/core/installation-overview)
- Use [MetricFlow commands](/docs/build/metricflow-commands) to query and test those metrics in your development environment
- Harness the true magic of the universal dbt Semantic Layer and dynamically query these metrics in downstream tools (Available for dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) accounts only).


<div className="grid--3-col">

<Card
Expand Down Expand Up @@ -57,12 +55,11 @@ MetricFlow allows you to:
link="/docs/use-dbt-semantic-layer/avail-sl-integrations"
icon="dbt-bit"/>


</div> <br />


## Related docs

- [The dbt Semantic Layer: what's next](https://www.getdbt.com/blog/dbt-semantic-layer-whats-next/) blog
- [Get started with MetricFlow](/docs/build/sl-getting-started)
- [dbt Semantic Layer on-demand courses](https://courses.getdbt.com/courses/semantic-layer)
- [dbt Semantic Layer FAQs](/docs/use-dbt-semantic-layer/sl-faqs)
50 changes: 30 additions & 20 deletions website/docs/docs/build/conversion-metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,23 +16,28 @@ Conversion metrics are different from [ratio metrics](/docs/build/ratio) because

The specification for conversion metrics is as follows:

| Parameter | Description | Type | Required/Optional |
| Parameter | Description | Type |
| --- | --- | --- | --- |
| `name` | The name of the metric. | String | Required |
| `description` | The description of the metric. | String | Optional |
| `type` | The type of metric (such as derived, ratio, and so on.). In this case, set as 'conversion' | String | Required |
| `label` | Displayed value in downstream tools. | String | Required |
| `type_params` | Specific configurations for each metric type. | List | Required |
| `conversion_type_params` | Additional configuration specific to conversion metrics. | List | Required |
| `name` | The name of the metric. | Required |
| `description` | The description of the metric. | Optional |
| `type` | The type of metric (such as derived, ratio, and so on.). In this case, set as 'conversion' | Required |
| `label` | Displayed value in downstream tools. | Required |
| `type_params` | Specific configurations for each metric type. | Required |
| `conversion_type_params` | Additional configuration specific to conversion metrics. | Required |
| `entity` | The entity for each conversion event. | Entity | Required |
| `calculation` | Method of calculation. Either `conversion_rate` or `conversions`. Defaults to `conversion_rate`. | String | Optional |
| `base_measure` | The base conversion event measure. | Measure | Required |
| `conversion_measure` | The conversion event measure. | Measure | Required |
| `window` | The time window for the conversion event, such as 7 days, 1 week, 3 months. Defaults to infinity. | String | Optional |
| `constant_properties` | List of constant properties. | List | Optional |
| `base_property` | The property from the base semantic model that you want to hold constant. | Entity or Dimension | Optional |
| `conversion_property` | The property from the conversion semantic model that you want to hold constant. | Entity or Dimension | Optional |
| `fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | String | Optional |
| `calculation` | Method of calculation. Either `conversion_rate` or `conversions`. Defaults to `conversion_rate`. | Optional |
| `base_measure` | A list of base measure inputs | Required |
| `base_measure:name` | The base conversion event measure. | Required |
| `base_measure:fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional |
| `base_measure:join_to_timespine` | Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional |
| `conversion_measure` | A list of conversion measure inputs. | Required |
| `conversion_measure:name` | The base conversion event measure.| Required |
| `conversion_measure:fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional |
| `conversion_measure:join_to_timespine` | Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional |
| `window` | The time window for the conversion event, such as 7 days, 1 week, 3 months. Defaults to infinity. | Optional |
| `constant_properties` | List of constant properties. | Optional |
| `base_property` | The property from the base semantic model that you want to hold constant. | Optional |
| `conversion_property` | The property from the conversion semantic model that you want to hold constant. | Optional |

Refer to [additional settings](#additional-settings) to learn how to customize conversion metrics with settings for null values, calculation type, and constant properties.

Expand All @@ -43,14 +48,19 @@ metrics:
- name: The metric name # Required
description: The metric description # Optional
type: conversion # Required
label: # Required
label: YOUR_LABEL # Required
type_params: # Required
fills_nulls_with: Set the value in your metric definition instead of null (such as zero) # Optional
conversion_type_params: # Required
entity: ENTITY # Required
calculation: CALCULATION_TYPE # Optional. default: conversion_rate. options: conversions(buys) or conversion_rate (buys/visits), and more to come.
base_measure: MEASURE # Required
conversion_measure: MEASURE # Required
base_measure:
name: The name of the measure # Required
fill_nulls_with: Set the value in your metric definition instead of null (such as zero) # Optional
join_to_timespine: Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. # Optional
conversion_measure:
name: The name of the measure # Required
fill_nulls_with: Set the value in your metric definition instead of null (such as zero) # Optional
join_to_timespine: Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. # Optional
window: TIME_WINDOW # Optional. default: infinity. window to join the two events. Follows a similar format as time windows elsewhere (such as 7 days)
constant_properties: # Optional. List of constant properties default: None
- base_property: DIMENSION or ENTITY # Required. A reference to a dimension/entity of the semantic model linked to the base_measure
Expand Down Expand Up @@ -93,7 +103,7 @@ Next, define a conversion metric as follows:
type: conversion
label: Visit to Buy Conversion Rate (7-day window)
type_params:
fills_nulls_with: 0
fill_nulls_with: 0
Copy link
Contributor Author

@mirnawong1 mirnawong1 Feb 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the nesting doesn't match the example so updated to this:

- name: visit_to_buy_conversion_rate_7d
  description: "Conversion rate from visiting to transaction in 7 days"
  type: conversion
  label: Visit to Buy Conversion Rate (7-day window)
  type_params:
    conversion_type_params:
      base_measure: visits
        name: visits
        fill_nulls_with: 0
      conversion_measure: sellers
        name: sellers
      entity: user
      window: 7 days

previously, it was this:

- name: visit_to_buy_conversion_rate_7d
  description: "Conversion rate from visiting to transaction in 7 days"
  type: conversion
  label: Visit to Buy Conversion Rate (7-day window)
  type_params:
    fill_nulls_with: 0
    conversion_type_params:
      base_measure: visits
      conversion_measure: sellers
      entity: user
      window: 7 days

conversion_type_params:
base_measure: visits
conversion_measure: sellers
Expand Down
Loading
Loading