Skip to content

Commit

Permalink
add clarifying info on how MF joins works (dbt-labs#6690)
Browse files Browse the repository at this point in the history
this pr adds more detail about the joins supported by mf. currently it
mentions left joins but there's no info on outer joins or semantic
valiations. this pr adds more info on the types of joins available, when
they're used, semantic validations available, and practical sql examples

raised by [internal
slack](https://dbt-labs.slack.com/archives/C03KHQRQUBX/p1733947531180079)

Resolves dbt-labs#6650

<!-- vercel-deployment-preview -->
---
🚀 Deployment available! Here are the direct links to the updated files:


-
https://docs-getdbt-com-git-update-mf-joins-dbt-labs.vercel.app/docs/build/join-logic

<!-- end-vercel-deployment-preview -->

---------

Co-authored-by: Leona B. Campbell <[email protected]>
  • Loading branch information
mirnawong1 and runleonarun authored Dec 24, 2024
1 parent 9622589 commit 65a46f1
Showing 1 changed file with 70 additions and 12 deletions.
82 changes: 70 additions & 12 deletions website/docs/docs/build/join-logic.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,24 +10,28 @@ Joins are a powerful part of MetricFlow and simplify the process of making all v

Joins use `entities` defined in your semantic model configs as the join keys between tables. Assuming entities are defined in the semantic model, MetricFlow creates a graph using the semantic models as nodes and the join paths as edges to perform joins automatically. MetricFlow chooses the appropriate join type and avoids fan-out or chasm joins with other tables based on the entity types.

<details>
<summary>What are fan-out or chasm joins?</summary>
<div>
<div>&mdash; Fan-out joins are when one row in a table is joined to multiple rows in another table, resulting in more output rows than input rows.<br /><br />
&mdash; Chasm joins are when two tables have a many-to-many relationship through an intermediate table, and the join results in duplicate or missing data. </div>
</div>
</details>

<Expandable alt_header="What are fan-out or chasm joins?" >
- Fan-out joins are when one row in a table is joined to multiple rows in another table, resulting in more output rows than input rows.
- Chasm joins are when two tables have a many-to-many relationship through an intermediate table, and the join results in duplicate or missing data.
</Expandable>

## Types of joins

:::tip Joins are auto-generated
MetricFlow automatically generates the necessary joins to the defined semantic objects, eliminating the need for you to create new semantic models or configuration files.

This document explains the different types of joins that can be used with entities and how to query them using the CLI.
This section explains the different types of joins that can be used with entities and how to query them.
:::

MetricFlow primarily uses left joins for joins, and restricts the use of fan-out and chasm joins. Refer to the table below to identify which joins are or aren't allowed based on specific entity types to prevent the creation of risky joins.
Metricflow uses these specific join strategies:

- Primarily uses left joins when joining `fct` and `dim` models. Left joins make sure all rows from the "base" table are retained, while matching rows are included from the joined table.
- For queries that involve multiple `fct` models, MetricFlow uses full outer joins to ensure all data points are captured, even when some `dim` or `fct` models are missing in certain tables.
- MetricFlow restricts the use of fan-out and chasm joins.

Refer to [SQL examples](#sql-examples) for more information on how MetricFlow handles joins in practice.

The following table identifies which joins are allowed based on specific entity types to prevent the creation of risky joins. This table primarily represents left joins unless otherwise specified. For scenarios involving multiple `fct` models, MetricFlow uses full outer joins.

| entity type - Table A | entity type - Table B | Join type |
|---------------------------|---------------------------|----------------------|
Expand All @@ -39,9 +43,19 @@ MetricFlow primarily uses left joins for joins, and restricts the use of fan-out
| Unique | Foreign | ❌ Fan-out (Not allowed) |
| Foreign | Primary | ✅ Left |
| Foreign | Unique | ✅ Left |
| Foreign | Foreign | ❌ Fan-out (Not allowed) |
| Foreign | Foreign | ❌ Fan-out (Not allowed) |

### Semantic validation

### Example
MetricFlow performs semantic validation by executing `explain` queries in the data platform to ensure that the generated SQL gets executed without errors. This validation includes:

- Verifying that all referenced tables and columns exist.
- Ensuring the data platform supports SQL functions, such as `date_diff(x, y)`.
- Checking for ambiguous joins or paths in multi-hop joins.

If validation fails, MetricFlow surfaces errors for users to address before executing the query.

## Example

The following example uses two semantic models with a common entity and shows a MetricFlow query that requires a join between the two semantic models. The two semantic models are:
- `transactions`
Expand Down Expand Up @@ -83,6 +97,50 @@ dbt sl query --metrics average_purchase_price --group-by metric_time,user_id__ty
mf query --metrics average_purchase_price --group-by metric_time,user_id__type # In dbt Core
```

#### SQL examples

These SQL examples show how MetricFlow handles both left join and full outer join scenarios in practice:

<Tabs>
<TabItem value="SQL example for left join">

Using the previous example for `transactions` and `user_signup` semantic models, this shows a left join between those two semantic models.

```sql
select
transactions.user_id,
transactions.purchase_price,
user_signup.type
from transactions
left outer join user_signup
on transactions.user_id = user_signup.user_id
where transactions.purchase_price is not null
group by
transactions.user_id,
user_signup.type;
```
</TabItem>

<TabItem value="SQL example for outer joins">

If you have multiple `fct` models, let's say `sales` and `returns`, MetricFlow uses full outer joins to ensure all data points are captured.

This example shows a full outer join between the `sales` and `returns` semantic models.

```sql
select
sales.user_id,
sales.total_sales,
returns.total_returns
from sales
full outer join returns
on sales.user_id = returns.user_id
where sales.user_id is not null or returns.user_id is not null;
```

</TabItem>
</Tabs>

## Multi-hop joins

MetricFlow allows users to join measures and dimensions across a graph of entities by moving from one table to another within a graph. This is referred to as "multi-hop join".
Expand Down

0 comments on commit 65a46f1

Please sign in to comment.