Skip to content

Commit

Permalink
remove capital letter on directories
Browse files Browse the repository at this point in the history
  • Loading branch information
john-rock committed Sep 18, 2023
1 parent 85ee486 commit 317d17f
Show file tree
Hide file tree
Showing 51 changed files with 1,316 additions and 0 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
---
title: "Update: Improvements to dbt Cloud continuous integration"
description: "September 2023: dbt Cloud now has two types of jobs — deploy jobs and CI jobs — with streamlined setup and improved efficiency. "
sidebar_label: "Update: Improvements to dbt jobs"
tags: [Sept-2023, CI]
date: 2023-09-11
sidebar_position: 10
---

dbt Cloud now has two distinct job types: [deploy jobs](/docs/deploy/deploy-jobs) for building production data assets, and [CI jobs](/docs/deploy/ci-jobs) for checking code changes. These jobs perform fundamentally different tasks so dbt Labs improved the setup experience with better defaults for each.

With two types of jobs, instead of one generic type, we can better guide you through the setup flow. Best practices are built into the default settings so you can go from curious to being set up in seconds.

<Lightbox src="/img/docs/release-notes/ci-job-setup.gif" title="Example of setting up a CI job"/>

And, we now have more efficient state comparisons on CI checks: never waste a build or test on code that hasn’t been changed. We now diff between the Git PR code and what’s running in production more efficiently with the introduction of deferral to an environment versus a job. To learn more, refer to [Continuous integration in dbt Cloud](/docs/deploy/continuous-integration) and [Get started with continuous integration tests](/guides/orchestration/set-up-ci/overview).

Below is a comparison table that describes how deploy jobs and CI jobs behave differently:

| | Deploy Jobs | CI Jobs |
| --- | --- | --- |
| Purpose | Builds production data assets. | Builds and tests new code before merging changes into production. |
| Trigger types | Triggered by a schedule or by API. | Triggered by a commit to a PR or by API. |
| Destination | Builds into a production database and schema. | Builds into a staging database and ephemeral schema, lived for the lifetime of the PR. |
| Execution mode | Runs execute sequentially, so as to not have collisions on the underlying DAG. | Runs execute in parallel to promote team velocity. |
| Efficiency run savings | Detects over-scheduled jobs and cancels unnecessary runs to avoid queue clog. | Cancels existing runs when a newer commit is pushed to avoid redundant work. |
| State comparison | Only sometimes needs to detect state. | Almost always needs to compare state against the production environment to build on modified code and its dependents. |


## What you need to update

If you previously set up a job using the [Create Job](/dbt-cloud/api-v2#/operations/Create%20Job) API endpoint before September 11, 2023, you must re-create the job as described in [Trigger a CI job with the API](/docs/deploy/ci-jobs#trigger-a-ci-job-with-the-api). This is because you must set the `job_type` to be `ci`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
---
title: "Summer 2023 product docs updates"
id: "product-docs-summer"
description: "Summer 2023: The Product docs team merged 256 PRs, made various updates to dbt Cloud and Core, such as adding What's New, writing Semantic Layer beta docs, releasing dbt 1.6 docs, and more!"
sidebar_label: "Update: Product docs changes"
tags: [July-2023, Aug-2023, product-docs]
date: 2023-09-13
sidebar_position: 09
---

Hello from dbt's Product Documentation team (the stewards of the docs.getdbt.com site): @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun. What a busy summer! We merged 256 PRs between July 1st and August 31.

We'd like to recognize all of the docs and support from our partner team, Developer Experience: @jasnonaz @gwenwindflower @dbeatty10 @dataders @joellabes @Jstein77 @dave-connors-3!

We'd also like to give a special thanks to the 22 community members who contributed to the [dbt Product docs](https://docs.getdbt.com) for the first time. :pray: Based on feedback from the dbt community, we made these changes:

- Added a [permissions table](/docs/cloud/manage-access/enterprise-permissions) for Enterprise accounts
- Added a [browser session page](/docs/cloud/about-cloud/browsers#browser-sessions) that clarifies dbt Cloud’s browser session time and when it logs users off.

You can provide feedback by opening a pull request or issue in [our repo](https://github.com/dbt-labs/docs.getdbt.com) or reaching out in the dbt community Slack channel [#dbt-product-docs](https://getdbt.slack.com/archives/C0441GSRU04)).

## :zap: General docs projects

* Added the ability to collapse sections you’re not currently looking at. There were quite a few people who wanted this, and it bugged us too, so we were happy to get this shipped!
* Introduced the idea of [“Trusted” adapters](/docs/supported-data-platforms#types-of-adapters).

## ☁ Cloud projects

* The **What’s new?** product update widget is back in the dbt Cloud UI! The Docs team will begin updating the content to keep you informed about new features.
* Launched the re-released [Semantic Layer beta docs](/docs/use-dbt-semantic-layer/dbt-sl), which introduces users to the new API, new guide to set up MetricFlow and the new Semantic Layer, as well as revamp the ‘Use the dbt Semantic Layer’ section for users.
* Updated [Admin API v2 and v3](/docs/dbt-cloud-apis/admin-cloud-api) to help you understand the differences between them and which version includes the endpoints you use.
* To improve discoverability, the docs team made changes to the [deploy dbt sidebar](/docs/deploy/deployments). We added cards and aligned better with the dbt Cloud UI and the way it’s used.
* Deprecated legacy job schemas in the [Discovery API](/docs/dbt-cloud-apis/discovery-api).
* Added a page to describe [experimental and beta features](/docs/dbt-versions/experimental-features) in dbt Cloud and what you need to know about them.
* Added a section to introduce a new beta feature [**Extended Attributes**](/docs/dbt-cloud-environments#extended-attributes-beta), which allows users to set a flexible `profiles.yml` snippet in their dbt Cloud Environment settings.
## 🎯 Core projects

* We released [dbt 1.6](/guides/migration/versions/upgrading-to-v1.6)! We added docs for the new commands `dbt retry` and `dbt clone`

## New 📚 Guides, ✏️ blog posts, and FAQs
* Check out how these community members use the dbt community in the [Community spotlight](/community/spotlight).
* Blog posts published this summer include [Optimizing Materialized Views with dbt](/blog/announcing-materialized-views), [Data Vault 2.0 with dbt Cloud](/blog/data-vault-with-dbt-cloud), and [Create dbt Documentation and Tests 10x faster with ChatGPT](/blog/create-dbt-documentation-10x-faster-with-chatgpt)
* We now have two new best practice guides: [How we build our metrics](/guides/best-practices/how-we-build-our-metrics/semantic-layer-1-intro) and [Set up Continuous Integration](/guides/orchestration/set-up-ci/overview).
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
---
title: "Deprecation: Query patterns and endpoints in the dbt Cloud Discovery API"
description: "August 2023: Learn about the upcoming deprecation of certain endpoints and query patterns in the Discovery API."
sidebar_position: 6
sidebar_label: "Deprecation: Certain Discovery API endpoints and query patterns"
tags: [Aug-2023, API]
date: 2023-08-31
---

dbt Labs has deprecated and will be deprecating certain query patterns and replacing them with new conventions to enhance the performance of the dbt Cloud [Discovery API](/docs/dbt-cloud-apis/discovery-api).

All these changes will be in effect on _September 7, 2023_.

We understand that these changes might require adjustments to your existing integration with the Discovery API. Please [contact us](mailto:[email protected]) with any questions. We're here to help you during this transition period.

## Job-based queries

Job-based queries that use the data type `Int` for IDs will be deprecated. They will be marked as deprecated in the [GraphQL explorer](https://metadata.cloud.getdbt.com/graphql). The new convention will be for you to use the data type `BigInt` instead.

This change will be in effect starting September 7, 2023.


Example of query before deprecation:

```graphql
query ($jobId: Int!) {
models(jobId: $jobId){
uniqueId
}
}
```

Example of query after deprecation:

```graphql
query ($jobId: BigInt!) {
job(id: $jobId) {
models {
uniqueId
}
}
}
```

## modelByEnvironment queries

The `modelByEnvironment` object has been renamed and moved into the `environment` object. This change is in effect and has been since August 15, 2023.

Example of query before deprecation:

```graphql
query ($environmentId: Int!, $uniqueId: String) {
modelByEnvironment(environmentId: $environmentId, uniqueId: $uniqueId) {
uniqueId
executionTime
executeCompletedAt
}
}
```

Example of query after deprecation:

```graphql
query ($environmentId: BigInt!, $uniqueId: String) {
environment(id: $environmentId) {
applied {
modelHistoricalRuns(uniqueId: $uniqueId) {
uniqueId
executionTime
executeCompletedAt
}
}
}
}
```


## Environment and account queries

Environment and account queries that use `Int` as a data type for ID have been deprecated. IDs must now be in `BigInt`. This change is in effect and has been since August 15, 2023.


Example of query before deprecation:

```graphql
query ($environmentId: Int!, $first: Int!) {
environment(id: $environmentId) {
applied {
models(first: $first) {
edges {
node {
uniqueId
executionInfo {
lastRunId
}
}
}
}
}
}
}
```


Example of query after deprecation:

```graphql
query ($environmentId: BigInt!, $first: Int!) {
environment(id: $environmentId) {
applied {
models(first: $first) {
edges {
node {
uniqueId
executionInfo {
lastRunId
}
}
}
}
}
}
}
```


Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
title: "Update: Cloud IDE v1.2 includes a new service"
description: "August 2023: Cloud IDE now uses dbt-server to provide more reliable service and dbt Core feature parity, including support for commands like `dbt list`."
sidebar_label: "Update: Cloud IDE v1.2"
tags: [Aug-2023, IDE]
date: 2023-08-03
sidebar_position: 8
---

We're excited to announce that we replaced the backend service that powers the Cloud IDE with a more reliable server -- dbt-server. Because this release contains foundational changes, IDE v1.2 requires dbt v1.6 or higher. This significant update follows the rebuild of the IDE frontend last year. We're committed to improving the IDE to provide you with a better experience.

Previously, the Cloud IDE used dbt-rpc, an outdated service that was unable to stay up-to-date with changes from dbt-core. The dbt-rpc integration used legacy dbt-core entry points and logging systems, causing it to be sluggish, brittle, and poorly tested. The Core team had been working around this outdated technology to avoid breaking it, which prevented them from developing with velocity and confidence.

## New features

- **Better dbt-core parity:** The Cloud IDE has better command parity with dbt-core, including support for commands like `dbt list` and improved treatment of flags like `--vars`, `--fail-fast`, etc.
- **Improved maintainability:** With the new dbt-server, it's easier to fix bugs and improve the overall quality of the product. With dbt-rpc, fixing bugs was a time-consuming and challenging process that required extensive testing. With the new service, we can identify and fix bugs more quickly, resulting in a more stable and reliable IDE.
- **A more reliable service:** Simplified architecture that's less prone to failure.

### Product refinements

- Improved `Preview` capabilities with Core v1.6 + IDE v1.2. [This Loom](https://www.loom.com/share/12838feb77bf463c8585fc1fc6aa161b) provides more information.

### Bug fixes

- Global page can become "inert" and stop handling clicks
- Switching back and forth between files in the git diff view can cause overwrite
- Browser gets stuck during markdown preview for doc with large table
- Editor right click menu is offset
- Unable to Cancel on the Save New File component when Closing All Files in the IDE
- Mouse flicker in the modal's file tree makes it difficult to select a folder where you want to save a new file
- Snapshots not showing in Lineage when inside a subfolder and is mixed cased named
- Tooltips do not work for Format and Save
- When a dbt invocation is in progress or if parsing is ongoing, attempting to switch branches will cause the `Git Branch` dropdown to close automatically

### Known issues

- `{{this}}` function does not display properly in preview/compile with dbt-server
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
---
title: "Enhancement: Revamped dbt Semantic Layer available in public beta"
description: "August 2023: The revamped dbt Semantic Layer, now available in public beta, introduces new semantic components and evolves the semantic layer's capability."
sidebar_label: "Enhancement: Revamped dbt Semantic Layer in public beta"
tags: [Aug-2023, dbt Semantic Layer]
date: 2023-08-03
sidebar_position: 7
---

:::important
If you're using the legacy Semantic Layer, we **highly** recommend you [upgrade your dbt version](/docs/dbt-versions/upgrade-core-in-cloud) to dbt v1.6 or higher to use the new dbt Semantic Layer. To migrate to the new Semantic Layer, refer to the dedicated [migration guide](/guides/migration/sl-migration) for more info.
:::

dbt Labs are thrilled to announce the re-release of the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl), now available in [public beta](#public-beta). It aims to bring the best of modeling and semantics to downstream applications by introducing:

- [MetricFlow](/docs/build/about-metricflow) is a framework for constructing performant and legible SQL from an all new set of semantic constructs which include semantic models, entities, and metrics.
- New Semantic Layer infrastructure that enables support for more data platforms (Snowflake, Databricks, BigQuery, Redshift, and soon more), along with improved performance.
- New and improved [developer workflows](/guides/migration/sl-migration), governance, and collaboration features.
- New [Semantic Layer API](/docs/dbt-cloud-apis/sl-api-overview) using JDBC to query metrics and build integrations.

With semantics at its core, the dbt Semantic Layer marks a crucial milestone towards a new era of centralized logic and data applications.

<Lightbox src="/img/docs/dbt-cloud/semantic-layer/sl-architecture.jpg" width="75%" title="The universal dbt Semantic Layer connecting to integration tools."/>

## Enhanced dbt Semantic Layer

What sets the dbt Semantic Layer apart is its ability to centralize logic for many downstream data applications, streamlining access and governance and enabling more efficient utilization of data models. It provides a consistent view of data while simplifying complex tasks in downstream applications and reducing the costs of and barriers to data access.

We are excited to present several important capabilities with the enhanced dbt Semantic Layer:

- **Consistent organization**: Provides a consistent view of data, ensuring that metrics and definitions match across the organization and the breadth of interfaces where data is consumed. This fosters trust in data and drives better decision-making by eliminating inconsistencies and errors that come up when individual users define metrics independently.

- **Improved governance**: The dbt Semantic Layer ensures proper governance and auditing of data changes, providing an auditable record of modifications and clear ownership. This saves time by making it clear who can create and manage new metrics, ensuring accountability and data integrity.

- **Reduce costs**: The dbt Semantic Layer simplifies complex tasks, such as bridging entities across a semantic graph. Often users duplicate slices and dice of data and make them available in a data platform, making it difficult to manage and causing high computation. The dbt Semantic Layer minimizes duplication of work and reduces computational costs - allowing users to focus on analyzing data rather than navigating intricate technical processes or duplicating work.

- **Enhanced efficiency**: With the dbt Semantic Layer, data teams can create and update metrics using a new set of validations that make defining and iterating on metrics efficient. The streamlined development workflows makes it simpler for a data team to serve large organizations with broad data needs.

- **Accessible data**: Defining common metrics and dimensions and making them joinable, makes access simpler for users with less expertise in the specifics of a company's data modeling work. This creates opportunities to leverage data insights, fostering collaboration and driving innovation in a more inclusive data environment.

By bringing these enhancements to the dbt Semantic Layer, we enable organizations of all sizes and industries to leverage the power of semantics in their data workflows.

## Public beta

The dbt Semantic Layer is currently available as a public beta, which means:

- **Who** &mdash; To experience the new dbt Semantic Layer, you must be on a dbt Cloud [Team and Enterprise](https://www.getdbt.com/pricing/) multi-tenant dbt Cloud plan, [hosted](/docs/cloud/about-cloud/regions-ip-addresses) in North America and on dbt v1.6 and higher. Look out for announcements on removing the location requirement soon.

- Developer plans or dbt Core users can use MetricFlow to define and test metrics using the dbt MetricFlow CLI only.

- **What** &mdash; Public beta provides early access to new features. The dbt Semantic Layer is stable and you can use it for production deployments, but there may still be some planned additions and modifications to product behaviors before moving to general availability later this year. We may also introduce new functionality that isn't backwards compatible. We provide support, and relevant service level objectives (SLOs) apply. If you have any questions on pricing, please reach out to your account representative.

- **When** &mdash; Public beta starts on August 1st, 2023.

- **Where** &mdash; You can experience the dbt Semantic Layer in dbt Cloud. Public beta is enabled at the account level so you don’t need to worry about enabling it per user.

## Next steps

To experience the universal dbt Semantic Layer and its enhanced beta capabilities, check out:

- [Introducing the new dbt Semantic Layer](https://www.getdbt.com/blog/introducing-new-look-dbt-semantic-layer)
- [dbt Semantic Layer docs](/docs/use-dbt-semantic-layer/dbt-sl)
- [dbt Semantic Layer get started guide](/docs/use-dbt-semantic-layer/quickstart-sl)
- [Build your metrics with MetricFlow](/docs/build/build-metrics-intro)

Loading

0 comments on commit 317d17f

Please sign in to comment.