Skip to content

Commit

Permalink
Start bullet, bolds, and expansion of mesh guide
Browse files Browse the repository at this point in the history
  • Loading branch information
gwenwindflower committed Sep 15, 2023
1 parent dfbf23f commit f08b0cd
Show file tree
Hide file tree
Showing 4 changed files with 51 additions and 27 deletions.
26 changes: 13 additions & 13 deletions website/docs/guides/best-practices/how-we-mesh/mesh-1-intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,26 +10,26 @@ Historically, building data teams has involved two extremes, building a centrali

dbt Mesh is not a product, but a pattern, enabled a convergence of several features in dbt Cloud. It’s inspired by dbt’s best practices and ideas from [data mesh](https://en.wikipedia.org/wiki/Data_mesh). These features include:

- Cross-project references - this is the core feature that enables a mesh structure. `ref`s now work across projects in dbt Cloud-enabled projects on Enterprise plans.
- Governance - dbt Cloud’s new governance features allow you to manage access and permissions across projects.
- Groups - groups allow you to assign models to subsets of models within a project.
- Access - access configs allow you to control who can view and reference models both within and across projects.
- Versioning - building a dbt Mesh involves treating your data models as stable APIs. To achieve this you need mechanisms to version your models and allow graceful adoption and deprecation of models as they evolve.
- Contracts - data contracts set strict expectations on the shape of the data to ensure data changes upstream of dbt or within a project's logic don't break downstream consumers.
- **Cross-project references** - this is the core feature that enables a mesh structure. `ref`s now work across projects in dbt Cloud-enabled projects on Enterprise plans.
- **Governance** - dbt Cloud’s new governance features allow you to manage access and permissions across projects.
- **Groups** - groups allow you to assign models to subsets of models within a project.
- **Access** - access configs allow you to control who can view and reference models both within and across projects.
- **Versioning** - building a dbt Mesh involves treating your data models as stable APIs. To achieve this you need mechanisms to version your models and allow graceful adoption and deprecation of models as they evolve.
- **Contracts** - data contracts set strict expectations on the shape of the data to ensure data changes upstream of dbt or within a project's logic don't break downstream consumers.

## Who is dbt Mesh for?

dbt Mesh is not for every organization! If you're just starting your dbt journey, don't worry about building a dbt Mesh right away, it increases some meta-complexity around managing your projects that could distract from building initial value in dbt. However, if you're already using dbt and your project has started to experience any of the following, you're likely ready to start exploring a dbt Mesh:

- The number of models in your project is degrading performance and slowing down development.
- Teams have developed separate workflows and need to decouple development.
- Security and governance requirements are increasing and would benefit from increased isolation.
- **The number of models** in your project is degrading performance and slowing down development.
- Teams have developed **separate workflows** and need to decouple development.
- **Security and governance** requirements are increasing and would benefit from increased isolation.

dbt Cloud is designed to coordinate the features above and simplify the meta-complexities (such as scoped CI and multi-project lineage) to solve for these problems.

## Learning goals

- Understand the purpose and tradeoffs of building a dbt Mesh.
- Develop an intuition for various dbt Mesh patterns and how to design a dbt Mesh for your organization.
- Establish recommended steps to incrementally adopt a dbt Mesh pattern in your dbt implementation.
- Offer tooling to help you more quickly and easily implement your dbt Mesh plan.
- Understand the **purpose and tradeoffs** of building a dbt Mesh.
- Develop an intuition for various **dbt Mesh patterns** and how to design a dbt Mesh for your organization.
- Establish recommended steps to **incrementally adopt** a dbt Mesh pattern in your dbt implementation.
- Offer **tooling** to help you more quickly and easily implement your dbt Mesh plan.
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,22 @@ hoverSnippet: Learn how to get started with dbt Mesh

## Exploring mesh patterns

Building a mesh is not a one-size-fits-all process. In fact, it's the opposite, it's about customizing your project structure to fit _your_ team and _your_ data. Often we've had to fit the data team and project structure into our company's org chart, or manage everything in one project to handle the constraints of our data and warehouse. dbt Mesh allows us to mold our organizational knowledge graph to our organizational people graph, bringing people and data closer together rather than compromising one for the other.
Building a dbt Mesh is not a one-size-fits-all process. In fact, it's the opposite, it's about customizing your project structure to fit _your_ team and _your_ data. Often we've had to fit the data team and project structure into our company's org chart, or manage everything in one project to handle the constraints of our data and warehouse. dbt Mesh allows us to mold our organizational knowledge graph to our organizational people graph, bringing people and data closer together rather than compromising one for the other. Let's explore some language for discussing the design of these patterns.

## Vertical splits

Vertical splits are about separating out layers of transformation in the DAG order. For example, splitting up staging and mart layers. Often the vertical separation will be based around security and governance requirements, such as separating out PII data from non-PII data and restricting raw data access to a platform team that's responsible for landing and cleaning data.
Vertical splits are about separating out layers of transformation in the DAG order. Let's look at some examples.

- **Splitting up staging and mart layers.** Creating a more tightly-controlled, shared set of components that other projects build on but can't edit.
- **Isolating earlier models for security and governance requirements.** Separating out and masking PII data so that downstream consumers can't access it is a common use case for a vertical split.
- **Protecting complex or expensive data.** If you have a large or complex model that's expensive to run, you might want to isolate it so that it's safer from accidental selection and easier to debug when it has issues.

## Horizontal splits

Horizonal splits are about splitting up the data based on source or domain. Often the horizontal separation will be based around team consumption patterns, such as splitting out marketing data and financial data. Another common vector of horizontal splitting is data from different sources, such as click event data and transactional ecommerce data. These splits are often based around the shape and size of the data and how it's used, rather than the security or governance requirements.
Horizonal splits are about splitting up the data based on source or domain. Let's consider some possibilites for horizontal splitting.

- Often the horizontal separation will be based around team consumption patterns, such as splitting out marketing data and financial data.
- Another common vector of horizontal splitting is data from different sources, such as click event data and transactional ecommerce data. These splits are often based around the shape and size of the data and how it's used, rather than the security or governance requirements.

## Combining these divisions

Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,9 @@
---
title: "Conclusion"
title: "Go forth and mesh!"
description: Getting started with dbt Mesh patterns
hoverSnippet: Learn how to get started with dbt Mesh
---

## Conclusion

dbt Mesh is a powerful new pattern for data transformation. It helps adapt teams and their data towards each other, rather than making arbitrary decisions based on the constraints of either one. By creating alignment between your people and data flows, developers can move faster, analysts can be more productive, and data consumers can be more confident in the data they use.

You can incrementally adopt the ideas in this guide in your organization as you hit constraints. There's no pressure to adopt this as the _right pattern_ to build with. That said, familiarizing yourself with dbt Mesh concepts and thinking through how they can apply to your organization will help you make better decisions as you grow. We hope this guide has given you a good starting point to do that.
It's important to reiterate that you can _incrementally_ adopt the ideas in this guide in your organization as you hit constraints. The collection of features tha enable dbt Mesh work effectively as independent tools, and there's no pressure to adopt this as the _right pattern_ to build with. That said, familiarizing yourself with the tooling and dbt Mesh concepts, as well as thinking through how they can apply to your organization, will help you make better decisions as you grow. We hope this guide has given you a good starting point to do that.
33 changes: 26 additions & 7 deletions website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -266,7 +266,7 @@ const sidebarSettings = {
"docs/build/validation",
"docs/build/metricflow-time-spine",
"docs/build/metricflow-cli",
]
],
},
{
type: "category",
Expand All @@ -275,8 +275,8 @@ const sidebarSettings = {
items: [
"docs/build/dimensions",
"docs/build/entities",
"docs/build/measures"
]
"docs/build/measures",
],
},
{
type: "category",
Expand All @@ -287,7 +287,7 @@ const sidebarSettings = {
"docs/build/derived",
"docs/build/ratio",
"docs/build/simple",
]
],
},
],
},
Expand Down Expand Up @@ -463,7 +463,10 @@ const sidebarSettings = {
{
type: "category",
label: "Job",
link: { type: "doc", id: "docs/dbt-cloud-apis/discovery-schema-job" },
link: {
type: "doc",
id: "docs/dbt-cloud-apis/discovery-schema-job",
},
items: [
"docs/dbt-cloud-apis/discovery-schema-job-model",
"docs/dbt-cloud-apis/discovery-schema-job-models",
Expand All @@ -486,7 +489,10 @@ const sidebarSettings = {
{
type: "category",
label: "Environment",
link: { type: "doc", id: "docs/dbt-cloud-apis/discovery-schema-environment" },
link: {
type: "doc",
id: "docs/dbt-cloud-apis/discovery-schema-environment",
},
items: [
{
type: "category",
Expand Down Expand Up @@ -925,7 +931,20 @@ const sidebarSettings = {
},
{
type: "category",
label: "Materializations best practices",
label: "How we build our dbt Mesh projects",
link: {
type: "doc",
id: "guides/best-practices/how-we-mesh/mesh-1-intro",
},
items: [
"guides/best-practices/how-we-mesh/mesh-2-structures",
"guides/best-practices/how-we-mesh/mesh-3-implementation",
"guides/best-practices/how-we-mesh/mesh-4-conclusion",
],
},
{
type: "category",
label: "Materialization best practices",
link: {
type: "doc",
id: "guides/best-practices/materializations/materializations-guide-1-guide-overview",
Expand Down

0 comments on commit f08b0cd

Please sign in to comment.