diff --git a/website/blog/2023-12-11-semantic-layer-on-semantic-layer.md b/website/blog/2023-12-11-semantic-layer-on-semantic-layer.md new file mode 100644 index 00000000000..ea77072a6dd --- /dev/null +++ b/website/blog/2023-12-11-semantic-layer-on-semantic-layer.md @@ -0,0 +1,97 @@ +--- +title: "How we built consistent product launch metrics with the dbt Semantic Layer." +description: "We built an end-to-end data pipeline for measuring the launch of the dbt Semantic Layer using the dbt Semantic Layer." +slug: product-analytics-pipeline-with-dbt-semantic-layer + +authors: [jordan_stein] + +tags: [dbt Cloud] +hide_table_of_contents: false + +date: 2023-12-12 +is_featured: false +--- +There’s nothing quite like the feeling of launching a new product. +On launch day emotions can range from excitement, to fear, to accomplishment all in the same hour. +Once the dust settles and the product is in the wild, the next thing the team needs to do is track how the product is doing. +How many users do we have? How is performance looking? What features are customers using? How often? Answering these questions is vital to understanding the success of any product launch. + +At dbt we recently made the [Semantic Layer Generally Available](https://www.getdbt.com/blog/new-dbt-cloud-features-announced-at-coalesce-2023). The Semantic Layer lets teams define business metrics centrally, in dbt, and access them in multiple analytics tools through our semantic layer APIs. +I’m a Product Manager on the Semantic Layer team, and the launch of the Semantic Layer put our team in an interesting, somewhat “meta,” position: we need to understand how a product launch is doing, and the product we just launched is designed to make defining and consuming metrics much more efficient. It’s the perfect opportunity to put the semantic layer through its paces for product analytics. This blog post walks through the end-to-end process we used to set up product analytics for the dbt Semantic Layer using the dbt Semantic Layer. + +## Getting your data ready for metrics + +The first steps to building a product analytics pipeline with the Semantic Layer look the same as just using dbt - it’s all about data transformation. The steps we followed were broadly: + +1. Work with engineering to understand the data sources. In our case, it’s db exports from Semantic Layer Server. +2. Load the data into our warehouse. We use Fivetran and Snowflake. +3. Transform the data into normalized tables with dbt. This step is a classic. dbt’s bread and butter. You probably know the drill by now. + +There are [plenty of other great resources](https://docs.getdbt.com/docs/build/projects) on how to accomplish the above steps, I’m going to skip that in this post and focus on how we built business metrics using the Semantic Layer. Once the data is loaded and modeling is complete, our DAG for the Semantic Layer data looks like the following: + + + + + + + + +Let’s walk through the DAG from left to right: First, we have raw tables from the Semantic Layer Server loaded into our warehouse, next we have staging models where we apply business logic and finally a clean, normalized `fct_semantic_layer_queries` model. Finally, we built a semantic model named `semantic_layer_queries` on top of our normalized fact model. This is a typical DAG for a dbt project that contains semantic objects. Now let’s zoom in to the section of the DAG that contains our semantic layer objects and look in more detail at how we defined our semantic layer product metrics. + +## [How we build semantic models and metrics](https://docs.getdbt.com/best-practices/how-we-build-our-metrics/semantic-layer-1-intro) + +What [is a semantic model](https://docs.getdbt.com/docs/build/semantic-models)? Put simply, semantic models contain the components we need to build metrics. Semantic models are YAML files that live in your dbt project. They contain metadata about your dbt models in a format that MetricFlow, the query builder that powers the semantic layer, can understand. The DAG below in [dbt Explorer](https://docs.getdbt.com/docs/collaborate/explore-projects) shows the metrics we’ve built off of `semantic_layer_queries`. + + + + +Let’s dig into semantic models and metrics a bit more, and explain some of the data modeling decisions we made. First, we needed to decide what model to use as a base for our semantic model. We decide to use`fct_semantic_layer`queries as our base model because defining a semantic model on top of a normalized fact table gives us maximum flexibility to join to other tables. This increased the number of dimensions available, which means we can answer more questions. + +You may wonder: why not just build our metrics on top of raw tables and let MetricFlow figure out the rest? The reality is, that you will almost almost always need to do some form of data modeling to create the data set you want to build your metrics off of. MetricFlow’s job isn’t to do data modeling. The transformation step is done with dbt. + +Next, we had to decide what we wanted to put into our semantic models. Semantic models contain [dimensions](https://docs.getdbt.com/docs/build/dimensions), [measures](https://docs.getdbt.com/docs/build/measures), and [entities](https://docs.getdbt.com/docs/build/entities). We took the following approach to add each of these components: + +- Dimensions: We included all the relevant dimensions in our semantic model that stakeholders might ask for, like the time a query was created, the query status, and booleans showing if a query contained certain elements like a where filter or multiple metrics. +- Entities: We added entities to our semantic model, like dbt cloud environment id. Entities function as join keys in semantic models, which means any other semantic models that have a j[oinable entity](https://docs.getdbt.com/docs/build/join-logic) can be used when querying metrics. +- Measures: Next we added Measures. Measures define the aggregation you want to run on your data. I think of measures as a metric primitive, we’ll use them to build metrics and can reuse them to keep our code [DRY](https://docs.getdbt.com/terms/dry). + +Finally, we reference the measures defined in our semantic model to create metrics. Our initial set of usage metrics are all relatively simple aggregations. For example, the total number of queries run. + +```yaml +## Example of a metric definition +metrics: + - name: queries + description: The total number of queries run + type: simple + label: Semantic Layer Queries + type_params: + measure: queries +``` + +Having our metrics in the semantic layer is powerful in a few ways. Firstly, metric definitions and the generated SQL are centralized, and live in our dbt project, instead of being scattered across BI tools or sql clients. Secondly, the types of queries I can run are dynamic and flexible. Traditionally, I would materialize a cube or rollup table which needs to contain all the different dimensional slices my users might be curious about. Now, users can join tables and add dimensionality to their metrics queries on the fly at query time, saving our data team cycles of updating and adding new fields to rollup tables. Thirdly, we can expose these metrics to a variety of downstream BI tools so stakeholders in product, finance, or GTM can understand product performance regardless of their technical skills. + +Now that we’ve done the pipeline work to set up our metrics for the semantic layer launch we’re ready to analyze how the launch went! + +## Our Finance, Operations and GTM teams are all looking at the same metrics 😊 + +To query to Semantic Layer you have two paths: you can query metrics directly through the Semantic Layer APIs or use one of our [first-class integrations](https://docs.getdbt.com/docs/use-dbt-semantic-layer/avail-sl-integrations). Our analytics team and product teams are big Hex users, while our operations and finance teams live and breathe Google Sheets, so it’s important for us to have the same metric definitions available in both tools. + +The leg work of building our pipeline and defining metrics is all done, which makes last-mile consumption much easier. First, we set up a launch dashboard in Hex as the source of truth for semantic layer product metrics. This tool is used by cross-functional partners like marketing, sales, and the executive team to easily check product and usage metrics like total semantic layer queries, or weekly active semantic layer users. To set up our Hex connection, we simply enter a few details from our dbt Cloud environment and then we can work with metrics directly in Hex notebooks. We can use the JDBC interface, or use Hex’s GUI metric builder to build reports. We run all our WBRs off this dashboard, which allows us to spot trends in consumption and react quickly to changes in our business. + + + + + +On the finance and operations side, product usage data is crucial to making informed pricing decisions. All our pricing models are created in spreadsheets, so we leverage the Google Sheets integration to give those teams access to consistent data sets without the need to download CSVs from the Hex dashboard. This lets the Pricing team add dimensional slices, like tier and company size, to the data in a self-serve manner without having to request data team resources to generate those insights. This allows our finance team to iteratively build financial models and be more self-sufficient in pulling data, instead of relying on data team resources. + + + + + +As a former data scientist and data engineer, I personally think this is a huge improvement over the approach I would have used without the semantic layer. My old approach would have been to materialize One Big Table with all the numeric and categorical columns I needed for analysis. Then write a ton of SQL in Hex or various notebooks to create reports for stakeholders. Inevitably I’m signing up for more development cycles to update the pipeline whenever a new dimension needs to be added or the data needs to be aggregated in a slightly different way. From a data team management perspective, using a central semantic layer saves data analysts cycles since users can more easily self-serve. At every company I’ve ever worked at, data analysts are always in high demand, with more requests than they can reasonably accomplish. This means any time a stakeholder can self-serve their data without pulling us in is a huge win. + +## The Result: Consistent Governed Metrics + +And just like that, we have an end-to-end pipeline for product analytics on the dbt Semantic Layer using the dbt Semantic Layer 🤯. Part of the foundational work to build this pipeline will be familiar to you, like building out a normalized fact table using dbt. Hopefully walking through the next step of adding semantic models and metrics on top of those dbt models helped give you some ideas about how you can use the semantic layer for your team. Having launch metrics defined in dbt made keeping the entire organization up to date on product adoption and performance much easier. Instead of a rollup table or static materialized cubes, we added flexible metrics without rewriting logic in SQL, or adding additional tables to the end of our DAG. + +The result is access to consistent and governed metrics in the tool our stakeholders are already using to do their jobs. We are able to keep the entire organization aligned and give them access to consistent, accurate data they need to do their part to make the semantic layer product successful. Thanks for reading! If you’re thinking of using the semantic layer, or have questions we’re always happy to keep the conversation going in the [dbt community slack.](https://www.getdbt.com/community/join-the-community) Drop us a note in #dbt-cloud-semantic-layer. We’d love to hear from you! diff --git a/website/blog/authors.yml b/website/blog/authors.yml index 31d69824ed4..cd2bd162935 100644 --- a/website/blog/authors.yml +++ b/website/blog/authors.yml @@ -287,6 +287,14 @@ jonathan_natkins: url: https://twitter.com/nattyice name: Jon "Natty" Natkins organization: dbt Labs +jordan_stein: + image_url: /img/blog/authors/jordan.jpeg + job_title: Product Manager + links: + - icon: fa-linkedin + url: https://www.linkedin.com/in/jstein5/ + name: Jordan Stein + organization: dbt Labs josh_fell: image_url: /img/blog/authors/josh-fell.jpeg diff --git a/website/docs/best-practices/how-we-build-our-metrics/semantic-layer-1-intro.md b/website/docs/best-practices/how-we-build-our-metrics/semantic-layer-1-intro.md index 19c6717063c..ee3d4262882 100644 --- a/website/docs/best-practices/how-we-build-our-metrics/semantic-layer-1-intro.md +++ b/website/docs/best-practices/how-we-build-our-metrics/semantic-layer-1-intro.md @@ -4,10 +4,6 @@ description: Getting started with the dbt and MetricFlow hoverSnippet: Learn how to get started with the dbt and MetricFlow --- -:::tip -**This is a guide for a beta product.** We anticipate this guide will evolve alongside the Semantic Layer through community collaboration. We welcome discussions, ideas, issues, and contributions to refining best practices. -::: - Flying cars, hoverboards, and true self-service analytics: this is the future we were promised. The first two might still be a few years out, but real self-service analytics is here today. With dbt Cloud's Semantic Layer, you can resolve the tension between accuracy and flexibility that has hampered analytics tools for years, empowering everybody in your organization to explore a shared reality of metrics. Best of all for analytics engineers, building with these new tools will significantly [DRY](https://docs.getdbt.com/terms/dry) up and simplify your codebase. As you'll see, the deep interaction between your dbt models and the Semantic Layer make your dbt project the ideal place to craft your metrics. ## Learning goals diff --git a/website/docs/community/resources/oss-expectations.md b/website/docs/community/resources/oss-expectations.md index 9c916de1240..a57df7fe67f 100644 --- a/website/docs/community/resources/oss-expectations.md +++ b/website/docs/community/resources/oss-expectations.md @@ -94,6 +94,8 @@ PRs are your surest way to make the change you want to see in dbt / packages / d **Every PR should be associated with an issue.** Why? Before you spend a lot of time working on a contribution, we want to make sure that your proposal will be accepted. You should open an issue first, describing your desired outcome and outlining your planned change. If you've found an older issue that's already open, comment on it with an outline for your planned implementation. Exception to this rule: If you're just opening a PR for a cosmetic fix, such as a typo in documentation, an issue isn't needed. +**PRs should include robust testing.** Comprehensive testing within pull requests is crucial for the stability of our project. By prioritizing robust testing, we ensure the reliability of our codebase, minimize unforeseen issues and safeguard against potential regressions. We understand that creating thorough tests often requires significant effort, and your dedication to this process greatly contributes to the project's overall reliability. Thank you for your commitment to maintaining the integrity of our codebase!" + **Our goal is to review most new PRs within 7 days.** The first review will include some high-level comments about the implementation, including (at a high level) whether it’s something we think suitable to merge. Depending on the scope of the PR, the first review may include line-level code suggestions, or we may delay specific code review until the PR is more finalized / until we have more time. **Automation that can help us:** Many repositories have a template for pull request descriptions, which will include a checklist that must be completed before the PR can be merged. You don’t have to do all of these things to get an initial PR, but they definitely help. Those many include things like: diff --git a/website/docs/docs/build/incremental-models.md b/website/docs/docs/build/incremental-models.md index 3a597499f04..01a392c12fe 100644 --- a/website/docs/docs/build/incremental-models.md +++ b/website/docs/docs/build/incremental-models.md @@ -255,10 +255,10 @@ The `merge` strategy is available in dbt-postgres and dbt-redshift beginning in | [dbt-postgres](/reference/resource-configs/postgres-configs#incremental-materialization-strategies) | `append` | `delete+insert` | | [dbt-redshift](/reference/resource-configs/redshift-configs#incremental-materialization-strategies) | `append` | `delete+insert` | | [dbt-bigquery](/reference/resource-configs/bigquery-configs#merge-behavior-incremental-models) | `merge` | `insert_overwrite` | -| [dbt-spark](/reference/resource-configs/spark-configs#incremental-models) | `append` | `merge` (Delta only) `insert_overwrite` | -| [dbt-databricks](/reference/resource-configs/databricks-configs#incremental-models) | `append` | `merge` (Delta only) `insert_overwrite` | +| [dbt-spark](/reference/resource-configs/spark-configs#incremental-models) | `append` | `merge`, `insert_overwrite` | +| [dbt-databricks](/reference/resource-configs/databricks-configs#incremental-models) | `merge` | `append`, `insert_overwrite` | | [dbt-snowflake](/reference/resource-configs/snowflake-configs#merge-behavior-incremental-models) | `merge` | `append`, `delete+insert` | -| [dbt-trino](/reference/resource-configs/trino-configs#incremental) | `append` | `merge` `delete+insert` | +| [dbt-trino](/reference/resource-configs/trino-configs#incremental) | `append` | `merge`, `delete+insert` | @@ -267,13 +267,13 @@ The `merge` strategy is available in dbt-postgres and dbt-redshift beginning in | data platform adapter | default strategy | additional supported strategies | | :----------------- | :----------------| : ---------------------------------- | -| [dbt-postgres](/reference/resource-configs/postgres-configs#incremental-materialization-strategies) | `append` | `merge` , `delete+insert` | -| [dbt-redshift](/reference/resource-configs/redshift-configs#incremental-materialization-strategies) | `append` | `merge`, `delete+insert` | +| [dbt-postgres](/reference/resource-configs/postgres-configs#incremental-materialization-strategies) | `append` | `merge` , `delete+insert` | +| [dbt-redshift](/reference/resource-configs/redshift-configs#incremental-materialization-strategies) | `append` | `merge`, `delete+insert` | | [dbt-bigquery](/reference/resource-configs/bigquery-configs#merge-behavior-incremental-models) | `merge` | `insert_overwrite` | -| [dbt-spark](/reference/resource-configs/spark-configs#incremental-models) | `append` | `merge` (Delta only) `insert_overwrite` | -| [dbt-databricks](/reference/resource-configs/databricks-configs#incremental-models) | `append` | `merge` (Delta only) `insert_overwrite` | +| [dbt-spark](/reference/resource-configs/spark-configs#incremental-models) | `append` | `merge`, `insert_overwrite` | +| [dbt-databricks](/reference/resource-configs/databricks-configs#incremental-models) | `merge` | `append`, `insert_overwrite` | | [dbt-snowflake](/reference/resource-configs/snowflake-configs#merge-behavior-incremental-models) | `merge` | `append`, `delete+insert` | -| [dbt-trino](/reference/resource-configs/trino-configs#incremental) | `append` | `merge` `delete+insert` | +| [dbt-trino](/reference/resource-configs/trino-configs#incremental) | `append` | `merge`, `delete+insert` | diff --git a/website/docs/docs/core/connect-data-platform/infer-setup.md b/website/docs/docs/core/connect-data-platform/infer-setup.md index 7642c553cc4..c04fba59a56 100644 --- a/website/docs/docs/core/connect-data-platform/infer-setup.md +++ b/website/docs/docs/core/connect-data-platform/infer-setup.md @@ -12,16 +12,21 @@ meta: slack_channel_name: n/a slack_channel_link: platform_name: 'Infer' - config_page: '/reference/resource-configs/no-configs' + config_page: '/reference/resource-configs/infer-configs' min_supported_version: n/a --- +:::info Vendor-supported plugin + +Certain core functionality may vary. If you would like to report a bug, request a feature, or contribute, you can check out the linked repository and open an issue. + +::: + import SetUpPages from '/snippets/_setup-pages-intro.md'; - ## Connecting to Infer with **dbt-infer** Infer allows you to perform advanced ML Analytics within SQL as if native to your data warehouse. @@ -30,10 +35,18 @@ you can build advanced analysis for any business use case. Read more about SQL-inf and Infer in the [Infer documentation](https://docs.getinfer.io/). The `dbt-infer` package allow you to use SQL-inf easily within your DBT models. -You can read more about the `dbt-infer` package itself and how it connecst to Infer in the [dbt-infer documentation](https://dbt.getinfer.io/). +You can read more about the `dbt-infer` package itself and how it connects to Infer in the [dbt-infer documentation](https://dbt.getinfer.io/). + +The dbt-infer adapter is maintained via PyPi and installed with pip. +To install the latest dbt-infer package simply run the following within the same shell as you run dbt. +```python +pip install dbt-infer +``` + +Versioning of dbt-infer follows the standard dbt versioning scheme - meaning if you are using dbt 1.2 the corresponding dbt-infer will be named 1.2.x where is the latest minor version number. Before using SQL-inf in your DBT models you need to setup an Infer account and generate an API-key for the connection. -You can read how to do that in the [Getting Started Guide](https://dbt.getinfer.io/docs/getting_started#sign-up-to-infer). +You can read how to do that in the [Getting Started Guide](https://docs.getinfer.io/docs/reference/integrations/dbt). The profile configuration in `profiles.yml` for `dbt-infer` should look something like this: @@ -101,10 +114,10 @@ as native SQL functions. Infer supports a number of SQL-inf commands, including `PREDICT`, `EXPLAIN`, `CLUSTER`, `SIMILAR_TO`, `TOPICS`, `SENTIMENT`. -You can read more about SQL-inf and the commands it supports in the [SQL-inf Reference Guide](https://docs.getinfer.io/docs/reference). +You can read more about SQL-inf and the commands it supports in the [SQL-inf Reference Guide](https://docs.getinfer.io/docs/category/commands). To get you started we will give a brief example here of what such a model might look like. -You can find other more complex examples on the [dbt-infer examples page](https://dbt.getinfer.io/docs/examples). +You can find other more complex examples in the [dbt-infer examples repo](https://github.com/inferlabs/dbt-infer-examples). In our simple example, we will show how to use a previous model 'user_features' to predict churn by predicting the column `has_churned`. diff --git a/website/docs/docs/dbt-versions/core-upgrade/11-Older versions/upgrading-to-0-14-0.md b/website/docs/docs/dbt-versions/core-upgrade/11-Older versions/upgrading-to-0-14-0.md index 036a9a2aedf..48aa14a42e5 100644 --- a/website/docs/docs/dbt-versions/core-upgrade/11-Older versions/upgrading-to-0-14-0.md +++ b/website/docs/docs/dbt-versions/core-upgrade/11-Older versions/upgrading-to-0-14-0.md @@ -189,7 +189,7 @@ models: -**Configuring the `incremental_strategy for a single model:** +**Configuring the `incremental_strategy` for a single model:** diff --git a/website/docs/docs/deploy/deploy-jobs.md b/website/docs/docs/deploy/deploy-jobs.md index e43020bf66e..cee6e245359 100644 --- a/website/docs/docs/deploy/deploy-jobs.md +++ b/website/docs/docs/deploy/deploy-jobs.md @@ -84,15 +84,22 @@ To fully customize the scheduling of your job, choose the **Custom cron schedule Use tools such as [crontab.guru](https://crontab.guru/) to generate the correct cron syntax. This tool allows you to input cron snippets and returns their plain English translations. -Refer to the following example snippets: +Here are examples of cron job schedules. The dbt Cloud job scheduler supports using `L` to schedule jobs on the last day of the month: -- `0 * * * *`: Every hour, at minute 0 -- `*/5 * * * *`: Every 5 minutes -- `5 4 * * *`: At exactly 4:05 AM UTC -- `30 */4 * * *`: At minute 30 past every 4th hour (e.g. 4:30AM, 8:30AM, 12:30PM, etc., all UTC) -- `0 0 */2 * *`: At midnight UTC every other day +- `0 * * * *`: Every hour, at minute 0. +- `*/5 * * * *`: Every 5 minutes. +- `5 4 * * *`: At exactly 4:05 AM UTC. +- `30 */4 * * *`: At minute 30 past every 4th hour (such as 4:30 AM, 8:30 AM, 12:30 PM, and so on, all UTC). +- `0 0 */2 * *`: At 12:00 AM (midnight) UTC every other day. - `0 0 * * 1`: At midnight UTC every Monday. +- `0 0 L * *`: At 12:00 AM (midnight), on the last day of the month. +- `0 0 L 1,2,3,4,5,6,8,9,10,11,12 *`: At 12:00 AM, on the last day of the month, only in January, February, March, April, May, June, August, September, October, November, and December. +- `0 0 L 7 *`: At 12:00 AM, on the last day of the month, only in July. +- `0 0 L * FRI,SAT`: At 12:00 AM, on the last day of the month, and on Friday and Saturday. +- `0 12 L * *`: At 12:00 PM (afternoon), on the last day of the month. +- `0 7 L * 5`: At 07:00 AM, on the last day of the month, and on Friday. +- `30 14 L * *`: At 02:30 PM, on the last day of the month. ## Related docs diff --git a/website/docs/reference/node-selection/syntax.md b/website/docs/reference/node-selection/syntax.md index d0ea4a9acd8..22946903b7d 100644 --- a/website/docs/reference/node-selection/syntax.md +++ b/website/docs/reference/node-selection/syntax.md @@ -35,7 +35,7 @@ To follow [POSIX standards](https://pubs.opengroup.org/onlinepubs/9699919799/bas 3. dbt now has a list of still-selected resources of varying types. As a final step, it tosses away any resource that does not match the resource type of the current task. (Only seeds are kept for `dbt seed`, only models for `dbt run`, only tests for `dbt test`, and so on.) -### Shorthand +## Shorthand Select resources to build (run, test, seed, snapshot) or check freshness: `--select`, `-s` @@ -43,6 +43,9 @@ Select resources to build (run, test, seed, snapshot) or check freshness: `--sel By default, `dbt run` will execute _all_ of the models in the dependency graph. During development (and deployment), it is useful to specify only a subset of models to run. Use the `--select` flag with `dbt run` to select a subset of models to run. Note that the following arguments (`--select`, `--exclude`, and `--selector`) also apply to other dbt tasks, such as `test` and `build`. + + + The `--select` flag accepts one or more arguments. Each argument can be one of: 1. a package name @@ -52,8 +55,7 @@ The `--select` flag accepts one or more arguments. Each argument can be one of: Examples: - - ```bash +```bash dbt run --select "my_dbt_project_name" # runs all models in your project dbt run --select "my_dbt_model" # runs a specific model dbt run --select "path.to.my.models" # runs all models in a specific directory @@ -61,14 +63,30 @@ dbt run --select "my_package.some_model" # run a specific model in a specific pa dbt run --select "tag:nightly" # run models with the "nightly" tag dbt run --select "path/to/models" # run models contained in path/to/models dbt run --select "path/to/my_model.sql" # run a specific model by its path - ``` +``` -dbt supports a shorthand language for defining subsets of nodes. This language uses the characters `+`, `@`, `*`, and `,`. + + - ```bash +dbt supports a shorthand language for defining subsets of nodes. This language uses the following characters: + +- plus operator [(`+`)](/reference/node-selection/graph-operators#the-plus-operator) +- at operator [(`@`)](/reference/node-selection/graph-operators#the-at-operator) +- asterisk operator (`*`) +- comma operator (`,`) + +Examples: + +```bash # multiple arguments can be provided to --select - dbt run --select "my_first_model my_second_model" +dbt run --select "my_first_model my_second_model" + +# select my_model and all of its children +dbt run --select "my_model+" + +# select my_model, its children, and the parents of its children +dbt run --models @my_model # these arguments can be projects, models, directory paths, tags, or sources dbt run --select "tag:nightly my_model finance.base.*" @@ -77,6 +95,10 @@ dbt run --select "tag:nightly my_model finance.base.*" dbt run --select "path:marts/finance,tag:nightly,config.materialized:table" ``` + + + + As your selection logic gets more complex, and becomes unwieldly to type out as command-line arguments, consider using a [yaml selector](/reference/node-selection/yaml-selectors). You can use a predefined definition with the `--selector` flag. Note that when you're using `--selector`, most other flags (namely `--select` and `--exclude`) will be ignored. diff --git a/website/docs/reference/resource-configs/bigquery-configs.md b/website/docs/reference/resource-configs/bigquery-configs.md index ffbaa37c059..d3497a02caf 100644 --- a/website/docs/reference/resource-configs/bigquery-configs.md +++ b/website/docs/reference/resource-configs/bigquery-configs.md @@ -718,3 +718,188 @@ Views with this configuration will be able to select from objects in `project_1. The `grant_access_to` config is not thread-safe when multiple views need to be authorized for the same dataset. The initial `dbt run` operation after a new `grant_access_to` config is added should therefore be executed in a single thread. Subsequent runs using the same configuration will not attempt to re-apply existing access grants, and can make use of multiple threads. + + +## Materialized views + +The BigQuery adapter supports [materialized views](https://cloud.google.com/bigquery/docs/materialized-views-intro) +with the following configuration parameters: + +| Parameter | Type | Required | Default | Change Monitoring Support | +|-------------------------------------------------------------|------------------------|----------|---------|---------------------------| +| `on_configuration_change` | `` | no | `apply` | n/a | +| [`cluster_by`](#clustering-clause) | `[]` | no | `none` | drop/create | +| [`partition_by`](#partition-clause) | `{}` | no | `none` | drop/create | +| [`enable_refresh`](#auto-refresh) | `` | no | `true` | alter | +| [`refresh_interval_minutes`](#auto-refresh) | `` | no | `30` | alter | +| [`max_staleness`](#auto-refresh) (in Preview) | `` | no | `none` | alter | +| [`description`](/reference/resource-properties/description) | `` | no | `none` | alter | +| [`labels`](#specifying-labels) | `{: }` | no | `none` | alter | +| [`hours_to_expiration`](#controlling-table-expiration) | `` | no | `none` | alter | +| [`kms_key_name`](#using-kms-encryption) | `` | no | `none` | alter | + + + + + + + + +```yaml +models: + [](/reference/resource-configs/resource-path): + [+](/reference/resource-configs/plus-prefix)[materialized](/reference/resource-configs/materialized): materialized_view + [+](/reference/resource-configs/plus-prefix)on_configuration_change: apply | continue | fail + [+](/reference/resource-configs/plus-prefix)[cluster_by](#clustering-clause): | [] + [+](/reference/resource-configs/plus-prefix)[partition_by](#partition-clause): + - field: + - data_type: timestamp | date | datetime | int64 + # only if `data_type` is not 'int64' + - granularity: hour | day | month | year + # only if `data_type` is 'int64' + - range: + - start: + - end: + - interval: + [+](/reference/resource-configs/plus-prefix)[enable_refresh](#auto-refresh): true | false + [+](/reference/resource-configs/plus-prefix)[refresh_interval_minutes](#auto-refresh): + [+](/reference/resource-configs/plus-prefix)[max_staleness](#auto-refresh): + [+](/reference/resource-configs/plus-prefix)[description](/reference/resource-properties/description): + [+](/reference/resource-configs/plus-prefix)[labels](#specifying-labels): {: } + [+](/reference/resource-configs/plus-prefix)[hours_to_expiration](#acontrolling-table-expiration): + [+](/reference/resource-configs/plus-prefix)[kms_key_name](##using-kms-encryption): +``` + + + + + + + + + + +```yaml +version: 2 + +models: + - name: [] + config: + [materialized](/reference/resource-configs/materialized): materialized_view + on_configuration_change: apply | continue | fail + [cluster_by](#clustering-clause): | [] + [partition_by](#partition-clause): + - field: + - data_type: timestamp | date | datetime | int64 + # only if `data_type` is not 'int64' + - granularity: hour | day | month | year + # only if `data_type` is 'int64' + - range: + - start: + - end: + - interval: + [enable_refresh](#auto-refresh): true | false + [refresh_interval_minutes](#auto-refresh): + [max_staleness](#auto-refresh): + [description](/reference/resource-properties/description): + [labels](#specifying-labels): {: } + [hours_to_expiration](#acontrolling-table-expiration): + [kms_key_name](##using-kms-encryption): +``` + + + + + + + + + + +```jinja +{{ config( + [materialized](/reference/resource-configs/materialized)='materialized_view', + on_configuration_change="apply" | "continue" | "fail", + [cluster_by](#clustering-clause)="" | [""], + [partition_by](#partition-clause)={ + "field": "", + "data_type": "timestamp" | "date" | "datetime" | "int64", + + # only if `data_type` is not 'int64' + "granularity": "hour" | "day" | "month" | "year, + + # only if `data_type` is 'int64' + "range": { + "start": , + "end": , + "interval": , + } + }, + + # auto-refresh options + [enable_refresh](#auto-refresh)= true | false, + [refresh_interval_minutes](#auto-refresh)=, + [max_staleness](#auto-refresh)="", + + # additional options + [description](/reference/resource-properties/description)="", + [labels](#specifying-labels)={ + "": "", + }, + [hours_to_expiration](#acontrolling-table-expiration)=, + [kms_key_name](##using-kms-encryption)="", +) }} +``` + + + + + + + +Many of these parameters correspond to their table counterparts and have been linked above. +The set of parameters unique to materialized views covers [auto-refresh functionality](#auto-refresh). + +Find more information about these parameters in the BigQuery docs: +- [CREATE MATERIALIZED VIEW statement](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#create_materialized_view_statement) +- [materialized_view_option_list](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#materialized_view_option_list) + +### Auto-refresh + +| Parameter | Type | Required | Default | Change Monitoring Support | +|------------------------------|--------------|----------|---------|---------------------------| +| `enable_refresh` | `` | no | `true` | alter | +| `refresh_interval_minutes` | `` | no | `30` | alter | +| `max_staleness` (in Preview) | `` | no | `none` | alter | + +BigQuery supports [automatic refresh](https://cloud.google.com/bigquery/docs/materialized-views-manage#automatic_refresh) configuration for materialized views. +By default, a materialized view will automatically refresh within 5 minutes of changes in the base table, but not more frequently than once every 30 minutes. +BigQuery only officially supports the configuration of the frequency (the "once every 30 minutes" frequency); +however, there is a feature in preview that allows for the configuration of the staleness (the "5 minutes" refresh). +dbt will monitor these parameters for changes and apply them using an `ALTER` statement. + +Find more information about these parameters in the BigQuery docs: +- [materialized_view_option_list](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#materialized_view_option_list) +- [max_staleness](https://cloud.google.com/bigquery/docs/materialized-views-create#max_staleness) + +### Limitations + +As with most data platforms, there are limitations associated with materialized views. Some worth noting include: + +- Materialized view SQL has a [limited feature set](https://cloud.google.com/bigquery/docs/materialized-views-create#supported-mvs). +- Materialized view SQL cannot be updated; the materialized view must go through a `--full-refresh` (DROP/CREATE). +- The `partition_by` clause on a materialized view must match that of the underlying base table. +- While materialized views can have descriptions, materialized view *columns* cannot. +- Recreating/dropping the base table requires recreating/dropping the materialized view. + +Find more information about materialized view limitations in Google's BigQuery [docs](https://cloud.google.com/bigquery/docs/materialized-views-intro#limitations). + + diff --git a/website/docs/reference/resource-configs/infer-configs.md b/website/docs/reference/resource-configs/infer-configs.md new file mode 100644 index 00000000000..c4837806935 --- /dev/null +++ b/website/docs/reference/resource-configs/infer-configs.md @@ -0,0 +1,39 @@ +--- +title: "Infer configurations" +description: "Read this guide to understand how to configure Infer with dbt." +id: "infer-configs" +--- + + +## Authentication + +To connect to Infer from your dbt instance you need to set up a correct profile in your `profiles.yml`. + +The format of this should look like this: + + + +```yaml +: + target: + outputs: + : + type: infer + url: "" + username: "" + apikey: "" + data_config: + [configuration for your underlying data warehouse] +``` + + + +### Description of Infer Profile Fields + +| Field | Required | Description | +|------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------| +| `type` | Yes | Must be set to `infer`. This must be included either in `profiles.yml` or in the `dbt_project.yml` file. | +| `url` | Yes | The host name of the Infer server to connect to. Typically this is `https://app.getinfer.io`. | +| `username` | Yes | Your Infer username - the one you use to login. | +| `apikey` | Yes | Your Infer api key. | +| `data_config` | Yes | The configuration for your underlying data warehouse. The format of this follows the format of the configuration for your data warehouse adapter. | diff --git a/website/docs/reference/resource-configs/postgres-configs.md b/website/docs/reference/resource-configs/postgres-configs.md index 97a695ee12e..8465a5cbb31 100644 --- a/website/docs/reference/resource-configs/postgres-configs.md +++ b/website/docs/reference/resource-configs/postgres-configs.md @@ -108,21 +108,100 @@ models: ## Materialized views -The Postgres adapter supports [materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html). -Indexes are the only configuration that is specific to `dbt-postgres`. -The remaining configuration follows the general [materialized view](/docs/build/materializations#materialized-view) configuration. -There are also some limitations that we hope to address in the next version. +The Postgres adapter supports [materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html) +with the following configuration parameters: -### Monitored configuration changes +| Parameter | Type | Required | Default | Change Monitoring Support | +|---------------------------|--------------------|----------|---------|---------------------------| +| `on_configuration_change` | `` | no | `apply` | n/a | +| [`indexes`](#indexes) | `[{}]` | no | `none` | alter | -The settings below are monitored for changes applicable to `on_configuration_change`. + -#### Indexes -Index changes (`CREATE`, `DROP`) can be applied without the need to rebuild the materialized view. -This differs from a table model, where the table needs to be dropped and re-created to update the indexes. -If the `indexes` portion of the `config` block is updated, the changes will be detected and applied -directly to the materialized view in place. + + + + +```yaml +models: + [](/reference/resource-configs/resource-path): + [+](/reference/resource-configs/plus-prefix)[materialized](/reference/resource-configs/materialized): materialized_view + [+](/reference/resource-configs/plus-prefix)on_configuration_change: apply | continue | fail + [+](/reference/resource-configs/plus-prefix)[indexes](#indexes): + - columns: [] + unique: true | false + type: hash | btree +``` + + + + + + + + + + +```yaml +version: 2 + +models: + - name: [] + config: + [materialized](/reference/resource-configs/materialized): materialized_view + on_configuration_change: apply | continue | fail + [indexes](#indexes): + - columns: [] + unique: true | false + type: hash | btree +``` + + + + + + + + + + +```jinja +{{ config( + [materialized](/reference/resource-configs/materialized)="materialized_view", + on_configuration_change="apply" | "continue" | "fail", + [indexes](#indexes)=[ + { + "columns": [""], + "unique": true | false, + "type": "hash" | "btree", + } + ] +) }} +``` + + + + + + + +The [`indexes`](#indexes) parameter corresponds to that of a table, as explained above. +It's worth noting that, unlike tables, dbt monitors this parameter for changes and applies the changes without dropping the materialized view. +This happens via a `DROP/CREATE` of the indexes, which can be thought of as an `ALTER` of the materialized view. + +Find more information about materialized view parameters in the Postgres docs: +- [CREATE MATERIALIZED VIEW](https://www.postgresql.org/docs/current/sql-creatematerializedview.html) + + ### Limitations @@ -138,3 +217,5 @@ If the user changes the model's config to `materialized="materialized_view"`, th The solution is to execute `DROP TABLE my_model` on the data warehouse before trying the model again. + + diff --git a/website/docs/reference/resource-configs/redshift-configs.md b/website/docs/reference/resource-configs/redshift-configs.md index 9bd127a1e1a..b559c0451b0 100644 --- a/website/docs/reference/resource-configs/redshift-configs.md +++ b/website/docs/reference/resource-configs/redshift-configs.md @@ -111,40 +111,138 @@ models: ## Materialized views -The Redshift adapter supports [materialized views](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-overview.html). -Redshift-specific configuration includes the typical `dist`, `sort_type`, `sort`, and `backup`. -For materialized views, there is also the `auto_refresh` setting, which allows Redshift to [automatically refresh](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-refresh.html) the materialized view for you. -The remaining configuration follows the general [materialized view](/docs/build/materializations#Materialized-View) configuration. -There are also some limitations that we hope to address in the next version. +The Redshift adapter supports [materialized views](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-overview.html) +with the following configuration parameters: + +| Parameter | Type | Required | Default | Change Monitoring Support | +|-------------------------------------------|--------------|----------|------------------------------------------------|---------------------------| +| `on_configuration_change` | `` | no | `apply` | n/a | +| [`dist`](#using-sortkey-and-distkey) | `` | no | `even` | drop/create | +| [`sort`](#using-sortkey-and-distkey) | `[]` | no | `none` | drop/create | +| [`sort_type`](#using-sortkey-and-distkey) | `` | no | `auto` if no `sort`
`compound` if `sort` | drop/create | +| [`auto_refresh`](#auto-refresh) | `` | no | `false` | alter | +| [`backup`](#backup) | `` | no | `true` | n/a | + + + + + + + + +```yaml +models: + [](/reference/resource-configs/resource-path): + [+](/reference/resource-configs/plus-prefix)[materialized](/reference/resource-configs/materialized): materialized_view + [+](/reference/resource-configs/plus-prefix)on_configuration_change: apply | continue | fail + [+](/reference/resource-configs/plus-prefix)[dist](#using-sortkey-and-distkey): all | auto | even | + [+](/reference/resource-configs/plus-prefix)[sort](#using-sortkey-and-distkey): | [] + [+](/reference/resource-configs/plus-prefix)[sort_type](#using-sortkey-and-distkey): auto | compound | interleaved + [+](/reference/resource-configs/plus-prefix)[auto_refresh](#auto-refresh): true | false + [+](/reference/resource-configs/plus-prefix)[backup](#backup): true | false +``` + + + + -### Monitored configuration changes -The settings below are monitored for changes applicable to `on_configuration_change`. + -#### Dist + -Changes to `dist` will result in a full refresh of the existing materialized view (applied at the time of the next `dbt run` of the model). Redshift requires a materialized view to be -dropped and recreated to apply a change to the `distkey` or `diststyle`. +```yaml +version: 2 -#### Sort type, sort +models: + - name: [] + config: + [materialized](/reference/resource-configs/materialized): materialized_view + on_configuration_change: apply | continue | fail + [dist](#using-sortkey-and-distkey): all | auto | even | + [sort](#using-sortkey-and-distkey): | [] + [sort_type](#using-sortkey-and-distkey): auto | compound | interleaved + [auto_refresh](#auto-refresh): true | false + [backup](#backup): true | false +``` + + -Changes to `sort_type` or `sort` will result in a full refresh. Redshift requires a materialized -view to be dropped and recreated to apply a change to the `sortkey` or `sortstyle`. + + + + + + + +```jinja +{{ config( + [materialized](/reference/resource-configs/materialized)="materialized_view", + on_configuration_change="apply" | "continue" | "fail", + [dist](#using-sortkey-and-distkey)="all" | "auto" | "even" | "", + [sort](#using-sortkey-and-distkey)=[""], + [sort_type](#using-sortkey-and-distkey)="auto" | "compound" | "interleaved", + [auto_refresh](#auto-refresh)=true | false, + [backup](#backup)=true | false, +) }} +``` + + + + + + + +Many of these parameters correspond to their table counterparts and have been linked above. +The parameters unique to materialized views are the [auto-refresh](#auto-refresh) and [backup](#backup) functionality, which are covered below. + +Find more information about the [CREATE MATERIALIZED VIEW](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-create-sql-command.html) parameters in the Redshift docs. + +#### Auto-refresh + +| Parameter | Type | Required | Default | Change Monitoring Support | +|----------------|-------------|----------|---------|---------------------------| +| `auto_refresh` | `` | no | `false` | alter | + +Redshift supports [automatic refresh](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-refresh.html#materialized-view-auto-refresh) configuration for materialized views. +By default, a materialized view does not automatically refresh. +dbt monitors this parameter for changes and applies them using an `ALTER` statement. + +Learn more information about the [parameters](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-create-sql-command.html#mv_CREATE_MATERIALIZED_VIEW-parameters) in the Redshift docs. #### Backup -Changes to `backup` will result in a full refresh. Redshift requires a materialized -view to be dropped and recreated to apply a change to the `backup` setting. +| Parameter | Type | Required | Default | Change Monitoring Support | +|-----------|-------------|----------|---------|---------------------------| +| `backup` | `` | no | `true` | n/a | -#### Auto refresh +Redshift supports [backup](https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-snapshots.html) configuration of clusters at the object level. +This parameter identifies if the materialized view should be backed up as part of the cluster snapshot. +By default, a materialized view will be backed up during a cluster snapshot. +dbt cannot monitor this parameter as it is not queryable within Redshift. +If the value is changed, the materialized view will need to go through a `--full-refresh` in order to set it. -The `auto_refresh` setting can be updated via an `ALTER` statement. This setting effectively toggles -automatic refreshes on or off. The default setting for this config is off (`False`). If this -is the only configuration change for the materialized view, dbt will choose to apply -an `ALTER` statement instead of issuing a full refresh, +Learn more about the [parameters](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-create-sql-command.html#mv_CREATE_MATERIALIZED_VIEW-parameters) in the Redshift docs. ### Limitations +As with most data platforms, there are limitations associated with materialized views. Some worth noting include: + +- Materialized views cannot reference views, temporary tables, user-defined functions, or late-binding tables. +- Auto-refresh cannot be used if the materialized view references mutable functions, external schemas, or another materialized view. + +Find more information about materialized view limitations in Redshift's [docs](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-create-sql-command.html#mv_CREATE_MATERIALIZED_VIEW-limitations). + + + #### Changing materialization from "materialized_view" to "table" or "view" Swapping a materialized view to a table or view is not supported. @@ -157,3 +255,5 @@ If the user changes the model's config to `materialized="table"`, they will get The workaround is to execute `DROP MATERIALIZED VIEW my_mv CASCADE` on the data warehouse before trying the model again. + + diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md index 30c7966ec68..aa2bf370df6 100644 --- a/website/docs/reference/resource-configs/snowflake-configs.md +++ b/website/docs/reference/resource-configs/snowflake-configs.md @@ -346,82 +346,122 @@ In the configuration format for the model SQL file: ## Dynamic tables -The Snowflake adapter supports [dynamic tables](https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table). +The Snowflake adapter supports [dynamic tables](https://docs.snowflake.com/en/user-guide/dynamic-tables-about). This materialization is specific to Snowflake, which means that any model configuration that would normally come along for the ride from `dbt-core` (e.g. as with a `view`) may not be available for dynamic tables. This gap will decrease in future patches and versions. While this materialization is specific to Snowflake, it very much follows the implementation of [materialized views](/docs/build/materializations#Materialized-View). In particular, dynamic tables have access to the `on_configuration_change` setting. -There are also some limitations that we hope to address in the next version. +Dynamic tables are supported with the following configuration parameters: -### Parameters +| Parameter | Type | Required | Default | Change Monitoring Support | +|----------------------------------------------------------|------------|----------|---------|---------------------------| +| `on_configuration_change` | `` | no | `apply` | n/a | +| [`target_lag`](#target-lag) | `` | yes | | alter | +| [`snowflake_warehouse`](#configuring-virtual-warehouses) | `` | yes | | alter | -Dynamic tables in `dbt-snowflake` require the following parameters: -- `target_lag` -- `snowflake_warehouse` -- `on_configuration_change` + -To learn more about each parameter and what values it can take, see -the Snowflake docs page: [`CREATE DYNAMIC TABLE: Parameters`](https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table). -### Usage + -You can create a dynamic table by editing _one_ of these files: + -- the SQL file for your model -- the `dbt_project.yml` configuration file +```yaml +models: + [](/reference/resource-configs/resource-path): + [+](/reference/resource-configs/plus-prefix)[materialized](/reference/resource-configs/materialized): dynamic_table + [+](/reference/resource-configs/plus-prefix)on_configuration_change: apply | continue | fail + [+](/reference/resource-configs/plus-prefix)[target_lag](#target-lag): downstream | + [+](/reference/resource-configs/plus-prefix)[snowflake_warehouse](#configuring-virtual-warehouses): +``` -The following examples create a dynamic table: + - + -```sql -{{ config( - materialized = 'dynamic_table', - snowflake_warehouse = 'snowflake_warehouse', - target_lag = '10 minutes', -) }} -``` - + - + ```yaml +version: 2 + models: - path: - materialized: dynamic_table - snowflake_warehouse: snowflake_warehouse - target_lag: '10 minutes' + - name: [] + config: + [materialized](/reference/resource-configs/materialized): dynamic_table + on_configuration_change: apply | continue | fail + [target_lag](#target-lag): downstream | + [snowflake_warehouse](#configuring-virtual-warehouses): ``` -### Monitored configuration changes + -The settings below are monitored for changes applicable to `on_configuration_change`. -#### Target lag + -Changes to `target_lag` can be applied by running an `ALTER` statement. Refreshing is essentially -always on for dynamic tables; this setting changes how frequently the dynamic table is updated. + -#### Warehouse +```jinja +{{ config( + [materialized](/reference/resource-configs/materialized)="dynamic_table", + on_configuration_change="apply" | "continue" | "fail", + [target_lag](#target-lag)="downstream" | " seconds | minutes | hours | days", + [snowflake_warehouse](#configuring-virtual-warehouses)="", +) }} +``` + + + + + + + +Find more information about these parameters in Snowflake's [docs](https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table): -Changes to `snowflake_warehouse` can be applied via an `ALTER` statement. +### Target lag + +Snowflake allows two configuration scenarios for scheduling automatic refreshes: +- **Time-based** — Provide a value of the form ` { seconds | minutes | hours | days }`. For example, if the dynamic table needs to be updated every 30 minutes, use `target_lag='30 minutes'`. +- **Downstream** — Applicable when the dynamic table is referenced by other dynamic tables. In this scenario, `target_lag='downstream'` allows for refreshes to be controlled at the target, instead of at each layer. + +Find more information about `target_lag` in Snowflake's [docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh#understanding-target-lag). ### Limitations +As with materialized views on most data platforms, there are limitations associated with dynamic tables. Some worth noting include: + +- Dynamic table SQL has a [limited feature set](https://docs.snowflake.com/en/user-guide/dynamic-tables-tasks-create#query-constructs-not-currently-supported-in-dynamic-tables). +- Dynamic table SQL cannot be updated; the dynamic table must go through a `--full-refresh` (DROP/CREATE). +- Dynamic tables cannot be downstream from: materialized views, external tables, streams. +- Dynamic tables cannot reference a view that is downstream from another dynamic table. + +Find more information about dynamic table limitations in Snowflake's [docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-tasks-create#dynamic-table-limitations-and-supported-functions). + + + #### Changing materialization to and from "dynamic_table" -Swapping an already materialized model to be a dynamic table and vice versa. -The workaround is manually dropping the existing materialization in the data warehouse prior to calling `dbt run`. -Normally, re-running with the `--full-refresh` flag would resolve this, but not in this case. -This would only need to be done once as the existing object would then be a dynamic table. +Version `1.6.x` does not support altering the materialization from a non-dynamic table be a dynamic table and vice versa. +Re-running with the `--full-refresh` does not resolve this either. +The workaround is manually dropping the existing model in the warehouse prior to calling `dbt run`. +This only needs to be done once for the conversion. For example, assume for the example model below, `my_model`, has already been materialized to the underlying data platform via `dbt run`. -If the user changes the model's config to `materialized="dynamic_table"`, they will get an error. +If the model config is updated to `materialized="dynamic_table"`, dbt will return an error. The workaround is to execute `DROP TABLE my_model` on the data warehouse before trying the model again. @@ -429,7 +469,7 @@ The workaround is to execute `DROP TABLE my_model` on the data warehouse before ```yaml {{ config( - materialized="table" # or any model type eg view, incremental + materialized="table" # or any model type (e.g. view, incremental) ) }} ``` @@ -437,3 +477,5 @@ The workaround is to execute `DROP TABLE my_model` on the data warehouse before + + diff --git a/website/sidebars.js b/website/sidebars.js index 598fffc7f0d..ba00ba582b2 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -723,6 +723,7 @@ const sidebarSettings = { "reference/resource-configs/oracle-configs", "reference/resource-configs/upsolver-configs", "reference/resource-configs/starrocks-configs", + "reference/resource-configs/infer-configs", ], }, { diff --git a/website/static/img/blog/2023-12-11-semantic-layer-on-semantic-layer/Screenshot-dag.png b/website/static/img/blog/2023-12-11-semantic-layer-on-semantic-layer/Screenshot-dag.png new file mode 100644 index 00000000000..c053e595b77 Binary files /dev/null and b/website/static/img/blog/2023-12-11-semantic-layer-on-semantic-layer/Screenshot-dag.png differ diff --git a/website/static/img/blog/2023-12-11-semantic-layer-on-semantic-layer/Screenshot-gsheets.png b/website/static/img/blog/2023-12-11-semantic-layer-on-semantic-layer/Screenshot-gsheets.png new file mode 100644 index 00000000000..b34fb11a678 Binary files /dev/null and b/website/static/img/blog/2023-12-11-semantic-layer-on-semantic-layer/Screenshot-gsheets.png differ diff --git a/website/static/img/blog/2023-12-11-semantic-layer-on-semantic-layer/Screenshot-hex.png b/website/static/img/blog/2023-12-11-semantic-layer-on-semantic-layer/Screenshot-hex.png new file mode 100644 index 00000000000..16affa42ea3 Binary files /dev/null and b/website/static/img/blog/2023-12-11-semantic-layer-on-semantic-layer/Screenshot-hex.png differ diff --git a/website/static/img/blog/2023-12-11-semantic-layer-on-semantic-layer/Screenshot-metrics-dag.png b/website/static/img/blog/2023-12-11-semantic-layer-on-semantic-layer/Screenshot-metrics-dag.png new file mode 100644 index 00000000000..ed141da4679 Binary files /dev/null and b/website/static/img/blog/2023-12-11-semantic-layer-on-semantic-layer/Screenshot-metrics-dag.png differ diff --git a/website/static/img/blog/authors/jordan.jpeg b/website/static/img/blog/authors/jordan.jpeg new file mode 100644 index 00000000000..16252e3161f Binary files /dev/null and b/website/static/img/blog/authors/jordan.jpeg differ