diff --git a/website/docs/docs/dbt-versions/release-notes/04-sept-2023/ci-updates-phase2-rn.md b/website/docs/docs/dbt-versions/release-notes/04-sept-2023/ci-updates-phase2-rn.md new file mode 100644 index 00000000000..fefa07e6d6c --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/04-sept-2023/ci-updates-phase2-rn.md @@ -0,0 +1,32 @@ +--- +title: "Update: Improvements to dbt Cloud continuous integration" +description: "September 2023: dbt Cloud now has two types of jobs — deploy jobs and CI jobs — with streamlined setup and improved efficiency. " +sidebar_label: "Update: Improvements to dbt jobs" +tags: [Sept-2023, CI] +date: 2023-09-11 +sidebar_position: 10 +--- + +dbt Cloud now has two distinct job types: [deploy jobs](/docs/deploy/deploy-jobs) for building production data assets, and [CI jobs](/docs/deploy/ci-jobs) for checking code changes. These jobs perform fundamentally different tasks so dbt Labs improved the setup experience with better defaults for each. + +With two types of jobs, instead of one generic type, we can better guide you through the setup flow. Best practices are built into the default settings so you can go from curious to being set up in seconds. + + + +And, we now have more efficient state comparisons on CI checks: never waste a build or test on code that hasn’t been changed. We now diff between the Git PR code and what’s running in production more efficiently with the introduction of deferral to an environment versus a job. To learn more, refer to [Continuous integration in dbt Cloud](/docs/deploy/continuous-integration) and [Get started with continuous integration tests](/guides/orchestration/set-up-ci/overview). + +Below is a comparison table that describes how deploy jobs and CI jobs behave differently: + +| | Deploy Jobs | CI Jobs | +| --- | --- | --- | +| Purpose | Builds production data assets. | Builds and tests new code before merging changes into production. | +| Trigger types | Triggered by a schedule or by API. | Triggered by a commit to a PR or by API. | +| Destination | Builds into a production database and schema. | Builds into a staging database and ephemeral schema, lived for the lifetime of the PR. | +| Execution mode | Runs execute sequentially, so as to not have collisions on the underlying DAG. | Runs execute in parallel to promote team velocity. | +| Efficiency run savings | Detects over-scheduled jobs and cancels unnecessary runs to avoid queue clog. | Cancels existing runs when a newer commit is pushed to avoid redundant work. | +| State comparison | Only sometimes needs to detect state. | Almost always needs to compare state against the production environment to build on modified code and its dependents. | + + +## What you need to update + +If you previously set up a job using the [Create Job](/dbt-cloud/api-v2#/operations/Create%20Job) API endpoint before September 11, 2023, you must re-create the job as described in [Trigger a CI job with the API](/docs/deploy/ci-jobs#trigger-a-ci-job-with-the-api). This is because you must set the `job_type` to be `ci`. \ No newline at end of file diff --git a/website/docs/docs/dbt-versions/release-notes/04-sept-2023/product-docs-summer-rn.md b/website/docs/docs/dbt-versions/release-notes/04-sept-2023/product-docs-summer-rn.md new file mode 100644 index 00000000000..a647bb5f585 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/04-sept-2023/product-docs-summer-rn.md @@ -0,0 +1,43 @@ +--- +title: "Summer 2023 product docs updates" +id: "product-docs-summer" +description: "Summer 2023: The Product docs team merged 256 PRs, made various updates to dbt Cloud and Core, such as adding What's New, writing Semantic Layer beta docs, releasing dbt 1.6 docs, and more!" +sidebar_label: "Update: Product docs changes" +tags: [July-2023, Aug-2023, product-docs] +date: 2023-09-13 +sidebar_position: 09 +--- + +Hello from dbt's Product Documentation team (the stewards of the docs.getdbt.com site): @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun. What a busy summer! We merged 256 PRs between July 1st and August 31. + +We'd like to recognize all of the docs and support from our partner team, Developer Experience: @jasnonaz @gwenwindflower @dbeatty10 @dataders @joellabes @Jstein77 @dave-connors-3! + +We'd also like to give a special thanks to the 22 community members who contributed to the [dbt Product docs](https://docs.getdbt.com) for the first time. :pray: Based on feedback from the dbt community, we made these changes: + +- Added a [permissions table](/docs/cloud/manage-access/enterprise-permissions) for Enterprise accounts +- Added a [browser session page](/docs/cloud/about-cloud/browsers#browser-sessions) that clarifies dbt Cloud’s browser session time and when it logs users off. + +You can provide feedback by opening a pull request or issue in [our repo](https://github.com/dbt-labs/docs.getdbt.com) or reaching out in the dbt community Slack channel [#dbt-product-docs](https://getdbt.slack.com/archives/C0441GSRU04)). + +## :zap: General docs projects + +* Added the ability to collapse sections you’re not currently looking at. There were quite a few people who wanted this, and it bugged us too, so we were happy to get this shipped! +* Introduced the idea of [“Trusted” adapters](/docs/supported-data-platforms#types-of-adapters). + +## ☁ Cloud projects + +* The **What’s new?** product update widget is back in the dbt Cloud UI! The Docs team will begin updating the content to keep you informed about new features. +* Launched the re-released [Semantic Layer beta docs](/docs/use-dbt-semantic-layer/dbt-sl), which introduces users to the new API, new guide to set up MetricFlow and the new Semantic Layer, as well as revamp the ‘Use the dbt Semantic Layer’ section for users. +* Updated [Admin API v2 and v3](/docs/dbt-cloud-apis/admin-cloud-api) to help you understand the differences between them and which version includes the endpoints you use. +* To improve discoverability, the docs team made changes to the [deploy dbt sidebar](/docs/deploy/deployments). We added cards and aligned better with the dbt Cloud UI and the way it’s used. +* Deprecated legacy job schemas in the [Discovery API](/docs/dbt-cloud-apis/discovery-api). +* Added a page to describe [experimental and beta features](/docs/dbt-versions/experimental-features) in dbt Cloud and what you need to know about them. +* Added a section to introduce a new beta feature [**Extended Attributes**](/docs/dbt-cloud-environments#extended-attributes-beta), which allows users to set a flexible `profiles.yml` snippet in their dbt Cloud Environment settings. +## 🎯 Core projects + +* We released [dbt 1.6](/guides/migration/versions/upgrading-to-v1.6)! We added docs for the new commands `dbt retry` and `dbt clone` + +## New 📚 Guides, ✏️ blog posts, and FAQs +* Check out how these community members use the dbt community in the [Community spotlight](/community/spotlight). +* Blog posts published this summer include [Optimizing Materialized Views with dbt](/blog/announcing-materialized-views), [Data Vault 2.0 with dbt Cloud](/blog/data-vault-with-dbt-cloud), and [Create dbt Documentation and Tests 10x faster with ChatGPT](/blog/create-dbt-documentation-10x-faster-with-chatgpt) +* We now have two new best practice guides: [How we build our metrics](/guides/best-practices/how-we-build-our-metrics/semantic-layer-1-intro) and [Set up Continuous Integration](/guides/orchestration/set-up-ci/overview). diff --git a/website/docs/docs/dbt-versions/release-notes/05-aug-2023/deprecation-endpoints-discovery.md b/website/docs/docs/dbt-versions/release-notes/05-aug-2023/deprecation-endpoints-discovery.md new file mode 100644 index 00000000000..cd088b92fab --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/05-aug-2023/deprecation-endpoints-discovery.md @@ -0,0 +1,126 @@ +--- +title: "Deprecation: Query patterns and endpoints in the dbt Cloud Discovery API" +description: "August 2023: Learn about the upcoming deprecation of certain endpoints and query patterns in the Discovery API." +sidebar_position: 6 +sidebar_label: "Deprecation: Certain Discovery API endpoints and query patterns" +tags: [Aug-2023, API] +date: 2023-08-31 +--- + +dbt Labs has deprecated and will be deprecating certain query patterns and replacing them with new conventions to enhance the performance of the dbt Cloud [Discovery API](/docs/dbt-cloud-apis/discovery-api). + +All these changes will be in effect on _September 7, 2023_. + +We understand that these changes might require adjustments to your existing integration with the Discovery API. Please [contact us](mailto:support@getdbt.com) with any questions. We're here to help you during this transition period. + +## Job-based queries + +Job-based queries that use the data type `Int` for IDs will be deprecated. They will be marked as deprecated in the [GraphQL explorer](https://metadata.cloud.getdbt.com/graphql). The new convention will be for you to use the data type `BigInt` instead. + +This change will be in effect starting September 7, 2023. + + +Example of query before deprecation: + +```graphql +query ($jobId: Int!) { + models(jobId: $jobId){ + uniqueId + } +} +``` + +Example of query after deprecation: + +```graphql +query ($jobId: BigInt!) { + job(id: $jobId) { + models { + uniqueId + } + } +} +``` + +## modelByEnvironment queries + +The `modelByEnvironment` object has been renamed and moved into the `environment` object. This change is in effect and has been since August 15, 2023. + +Example of query before deprecation: + +```graphql +query ($environmentId: Int!, $uniqueId: String) { + modelByEnvironment(environmentId: $environmentId, uniqueId: $uniqueId) { + uniqueId + executionTime + executeCompletedAt + } +} +``` + +Example of query after deprecation: + +```graphql +query ($environmentId: BigInt!, $uniqueId: String) { + environment(id: $environmentId) { + applied { + modelHistoricalRuns(uniqueId: $uniqueId) { + uniqueId + executionTime + executeCompletedAt + } + } + } +} +``` + + +## Environment and account queries + +Environment and account queries that use `Int` as a data type for ID have been deprecated. IDs must now be in `BigInt`. This change is in effect and has been since August 15, 2023. + + +Example of query before deprecation: + +```graphql +query ($environmentId: Int!, $first: Int!) { + environment(id: $environmentId) { + applied { + models(first: $first) { + edges { + node { + uniqueId + executionInfo { + lastRunId + } + } + } + } + } + } +} +``` + + +Example of query after deprecation: + +```graphql +query ($environmentId: BigInt!, $first: Int!) { + environment(id: $environmentId) { + applied { + models(first: $first) { + edges { + node { + uniqueId + executionInfo { + lastRunId + } + } + } + } + } + } +} +``` + + diff --git a/website/docs/docs/dbt-versions/release-notes/05-aug-2023/ide-v1.2.md b/website/docs/docs/dbt-versions/release-notes/05-aug-2023/ide-v1.2.md new file mode 100644 index 00000000000..10baa5cd6d7 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/05-aug-2023/ide-v1.2.md @@ -0,0 +1,38 @@ +--- +title: "Update: Cloud IDE v1.2 includes a new service" +description: "August 2023: Cloud IDE now uses dbt-server to provide more reliable service and dbt Core feature parity, including support for commands like `dbt list`." +sidebar_label: "Update: Cloud IDE v1.2" +tags: [Aug-2023, IDE] +date: 2023-08-03 +sidebar_position: 8 +--- + +We're excited to announce that we replaced the backend service that powers the Cloud IDE with a more reliable server -- dbt-server. Because this release contains foundational changes, IDE v1.2 requires dbt v1.6 or higher. This significant update follows the rebuild of the IDE frontend last year. We're committed to improving the IDE to provide you with a better experience. + +Previously, the Cloud IDE used dbt-rpc, an outdated service that was unable to stay up-to-date with changes from dbt-core. The dbt-rpc integration used legacy dbt-core entry points and logging systems, causing it to be sluggish, brittle, and poorly tested. The Core team had been working around this outdated technology to avoid breaking it, which prevented them from developing with velocity and confidence. + +## New features + +- **Better dbt-core parity:** The Cloud IDE has better command parity with dbt-core, including support for commands like `dbt list` and improved treatment of flags like `--vars`, `--fail-fast`, etc. +- **Improved maintainability:** With the new dbt-server, it's easier to fix bugs and improve the overall quality of the product. With dbt-rpc, fixing bugs was a time-consuming and challenging process that required extensive testing. With the new service, we can identify and fix bugs more quickly, resulting in a more stable and reliable IDE. +- **A more reliable service:** Simplified architecture that's less prone to failure. + +### Product refinements + +- Improved `Preview` capabilities with Core v1.6 + IDE v1.2. [This Loom](https://www.loom.com/share/12838feb77bf463c8585fc1fc6aa161b) provides more information. + +### Bug fixes + +- Global page can become "inert" and stop handling clicks +- Switching back and forth between files in the git diff view can cause overwrite +- Browser gets stuck during markdown preview for doc with large table +- Editor right click menu is offset +- Unable to Cancel on the Save New File component when Closing All Files in the IDE +- Mouse flicker in the modal's file tree makes it difficult to select a folder where you want to save a new file +- Snapshots not showing in Lineage when inside a subfolder and is mixed cased named +- Tooltips do not work for Format and Save +- When a dbt invocation is in progress or if parsing is ongoing, attempting to switch branches will cause the `Git Branch` dropdown to close automatically + +### Known issues + +- `{{this}}` function does not display properly in preview/compile with dbt-server diff --git a/website/docs/docs/dbt-versions/release-notes/05-aug-2023/sl-revamp-beta.md b/website/docs/docs/dbt-versions/release-notes/05-aug-2023/sl-revamp-beta.md new file mode 100644 index 00000000000..921ed6dcd79 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/05-aug-2023/sl-revamp-beta.md @@ -0,0 +1,65 @@ +--- +title: "Enhancement: Revamped dbt Semantic Layer available in public beta" +description: "August 2023: The revamped dbt Semantic Layer, now available in public beta, introduces new semantic components and evolves the semantic layer's capability." +sidebar_label: "Enhancement: Revamped dbt Semantic Layer in public beta" +tags: [Aug-2023, dbt Semantic Layer] +date: 2023-08-03 +sidebar_position: 7 +--- + +:::important +If you're using the legacy Semantic Layer, we **highly** recommend you [upgrade your dbt version](/docs/dbt-versions/upgrade-core-in-cloud) to dbt v1.6 or higher to use the new dbt Semantic Layer. To migrate to the new Semantic Layer, refer to the dedicated [migration guide](/guides/migration/sl-migration) for more info. +::: + +dbt Labs are thrilled to announce the re-release of the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl), now available in [public beta](#public-beta). It aims to bring the best of modeling and semantics to downstream applications by introducing: + +- [MetricFlow](/docs/build/about-metricflow) is a framework for constructing performant and legible SQL from an all new set of semantic constructs which include semantic models, entities, and metrics. +- New Semantic Layer infrastructure that enables support for more data platforms (Snowflake, Databricks, BigQuery, Redshift, and soon more), along with improved performance. +- New and improved [developer workflows](/guides/migration/sl-migration), governance, and collaboration features. +- New [Semantic Layer API](/docs/dbt-cloud-apis/sl-api-overview) using JDBC to query metrics and build integrations. + +With semantics at its core, the dbt Semantic Layer marks a crucial milestone towards a new era of centralized logic and data applications. + + + +## Enhanced dbt Semantic Layer + +What sets the dbt Semantic Layer apart is its ability to centralize logic for many downstream data applications, streamlining access and governance and enabling more efficient utilization of data models. It provides a consistent view of data while simplifying complex tasks in downstream applications and reducing the costs of and barriers to data access. + +We are excited to present several important capabilities with the enhanced dbt Semantic Layer: + +- **Consistent organization**: Provides a consistent view of data, ensuring that metrics and definitions match across the organization and the breadth of interfaces where data is consumed. This fosters trust in data and drives better decision-making by eliminating inconsistencies and errors that come up when individual users define metrics independently. + +- **Improved governance**: The dbt Semantic Layer ensures proper governance and auditing of data changes, providing an auditable record of modifications and clear ownership. This saves time by making it clear who can create and manage new metrics, ensuring accountability and data integrity. + +- **Reduce costs**: The dbt Semantic Layer simplifies complex tasks, such as bridging entities across a semantic graph. Often users duplicate slices and dice of data and make them available in a data platform, making it difficult to manage and causing high computation. The dbt Semantic Layer minimizes duplication of work and reduces computational costs - allowing users to focus on analyzing data rather than navigating intricate technical processes or duplicating work. + +- **Enhanced efficiency**: With the dbt Semantic Layer, data teams can create and update metrics using a new set of validations that make defining and iterating on metrics efficient. The streamlined development workflows makes it simpler for a data team to serve large organizations with broad data needs. + +- **Accessible data**: Defining common metrics and dimensions and making them joinable, makes access simpler for users with less expertise in the specifics of a company's data modeling work. This creates opportunities to leverage data insights, fostering collaboration and driving innovation in a more inclusive data environment. + +By bringing these enhancements to the dbt Semantic Layer, we enable organizations of all sizes and industries to leverage the power of semantics in their data workflows. + +## Public beta + +The dbt Semantic Layer is currently available as a public beta, which means: + +- **Who** — To experience the new dbt Semantic Layer, you must be on a dbt Cloud [Team and Enterprise](https://www.getdbt.com/pricing/) multi-tenant dbt Cloud plan, [hosted](/docs/cloud/about-cloud/regions-ip-addresses) in North America and on dbt v1.6 and higher. Look out for announcements on removing the location requirement soon. + + - Developer plans or dbt Core users can use MetricFlow to define and test metrics using the dbt MetricFlow CLI only. + +- **What** — Public beta provides early access to new features. The dbt Semantic Layer is stable and you can use it for production deployments, but there may still be some planned additions and modifications to product behaviors before moving to general availability later this year. We may also introduce new functionality that isn't backwards compatible. We provide support, and relevant service level objectives (SLOs) apply. If you have any questions on pricing, please reach out to your account representative. + +- **When** — Public beta starts on August 1st, 2023. + +- **Where** — You can experience the dbt Semantic Layer in dbt Cloud. Public beta is enabled at the account level so you don’t need to worry about enabling it per user. + +## Next steps + +To experience the universal dbt Semantic Layer and its enhanced beta capabilities, check out: + +- [Introducing the new dbt Semantic Layer](https://www.getdbt.com/blog/introducing-new-look-dbt-semantic-layer) +- [dbt Semantic Layer docs](/docs/use-dbt-semantic-layer/dbt-sl) +- [dbt Semantic Layer get started guide](/docs/use-dbt-semantic-layer/quickstart-sl) +- [Build your metrics with MetricFlow](/docs/build/build-metrics-intro) + diff --git a/website/docs/docs/dbt-versions/release-notes/06-july-2023/faster-run.md b/website/docs/docs/dbt-versions/release-notes/06-july-2023/faster-run.md new file mode 100644 index 00000000000..ba82234c0b5 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/06-july-2023/faster-run.md @@ -0,0 +1,34 @@ +--- +title: "Enhancement: Faster run starts and unlimited job concurrency" +description: "We have enhanced the dbt Cloud Scheduler by reducing prep time for all accounts and provided unlimited job concurrency for Enterprise accounts." +sidebar_label: "Enhancement: Faster run starts and unlimited job concurrency" +tags: [July-2023, scheduler] +date: 2023-07-06 +sidebar_position: 10 +--- + +We’ve introduced significant improvements to the dbt Cloud Scheduler, offering improved performance, durability, and scalability. + +Read more on how you can experience faster run start execution and how enterprise users can now run as many jobs concurrently as they want to. + +## Faster run starts + +The Scheduler takes care of preparing each dbt Cloud job to run in your cloud data platform. This [prep](/docs/deploy/job-scheduler#scheduler-queue) involves readying a Kubernetes pod with the right version of dbt installed, setting environment variables, loading data platform credentials, and git provider authorization, amongst other environment-setting tasks. Only after the environment is set up, can dbt execution begin. We display this time to the user in dbt Cloud as “prep time”. + + + +For all its strengths, Kubernetes has challenges, especially with pod management impacting run execution time. We’ve rebuilt our scheduler by ensuring faster job execution with a ready pool of pods to execute customers’ jobs. This means you won't experience long prep times at the top of the hour, and we’re determined to keep runs starting near instantaneously. Don’t just take our word, review the data yourself. + + + +Jobs scheduled at the top of the hour used to take over 106 seconds to prepare because of the volume of runs the scheduler has to process. Now, even with increased runs, we have reduced prep time to 27 secs (at a maximum) — a 75% speed improvement for runs at peak traffic times! + +## Unlimited job concurrency for Enterprise accounts + +Our enhanced scheduler offers more durability and empowers users to run jobs effortlessly. + +This means Enterprise, multi-tenant accounts can now enjoy the advantages of unlimited job concurrency. Previously limited to a fixed number of run slots, Enterprise accounts now have the freedom to operate without constraints. Single-tenant support will be coming soon. Team plan customers will continue to have only 2 run slots. + +Something to note, each running job occupies a run slot for its duration, and if all slots are occupied, jobs will queue accordingly. + +For more feature details, refer to the [dbt Cloud pricing page](https://www.getdbt.com/pricing/). diff --git a/website/docs/docs/dbt-versions/release-notes/07-june-2023/admin-api-rn.md b/website/docs/docs/dbt-versions/release-notes/07-june-2023/admin-api-rn.md new file mode 100644 index 00000000000..2008331ebe6 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/07-june-2023/admin-api-rn.md @@ -0,0 +1,15 @@ +--- +title: "Update: dbt Cloud Administrative API docs for v2 and v3" +description: "June 2023 release note: The Administrative API docs are now available for v2 and v3 with a different UI." +sidebar_label: "Update: Admin API docs for v2 and v3 " +tags: [June-2023, API] +sidebar_position: 9 +--- + +dbt Labs updated the docs for the [dbt Cloud Administrative API](/docs/dbt-cloud-apis/admin-cloud-api) and they are now available for both [v2](/dbt-cloud/api-v2#/) and [v3](/dbt-cloud/api-v3#/). + +- Now using Spotlight for improved UI and UX. +- All endpoints are now documented for v2 and v3. Added automation to the docs so they remain up to date. +- Documented many of the request and response bodies. +- You can now test endpoints directly from within the API docs. And, you can choose which [regional server](/docs/cloud/about-cloud/regions-ip-addresses) to use (North America, APAC, or EMEA). +- With the new UI, you can more easily generate code for any endpoint. diff --git a/website/docs/docs/dbt-versions/release-notes/07-june-2023/ci-updates-phase1-rn.md b/website/docs/docs/dbt-versions/release-notes/07-june-2023/ci-updates-phase1-rn.md new file mode 100644 index 00000000000..fa02a6d9bd8 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/07-june-2023/ci-updates-phase1-rn.md @@ -0,0 +1,24 @@ +--- +title: "Update: Improvements to dbt Cloud continuous integration" +description: "dbt Cloud's CI checks now run in parallel, will not block production runs, and stale runs are automatically canceled when a newer commit is pushed." +sidebar_label: "Update: Improvements to continuous integration" +tags: [June-2023, CI] +date: 2023-06-20 +sidebar_position: 8 +--- + +dbt Cloud CI is a critical part of the analytics engineering workflow. Large teams rely on process to ensure code quality is high, and they look to dbt Cloud CI to automate testing code changes in an efficient way, enabling speed while keep the bar high. With status checks directly posted to their dbt PRs, developers gain the confidence that their code changes will work as expected in production, and once you’ve grown accustomed to seeing that green status check in your PR, you won’t be able to work any other way. + + + +What separates dbt Cloud CI from other CI providers is its ability to keep track of state of what’s running in your production environment, so that when you run a CI job, only the modified data assets in your pull request and their downstream dependencies get built and tested in a staging schema. dbt Cloud aims to make each CI check as efficient as possible, so as to not waste any data warehouse resources. As soon as the CI run completes, its status posts directly back to the PR in GitHub, GitLab, or Azure DevOps, depending on which Git provider you’re using. Teams can set up guardrails to let only PRs with successful CI checks be approved for merging, and the peer review process is greatly streamlined because dbt Cloud does the first testing pass. + +We're excited to introduce a few critical capabilities to dbt Cloud CI that will improve productivity and collaboration in your team’s testing and integration workflow. As of this week, you can now: + +- **Run multiple CI checks in parallel**. If more than one contributor makes changes to the same dbt project in dbt Cloud in short succession, the later arriving CI check no longer has to wait for the first check to complete. Both checks will execute concurrently. + +- **Automatically cancel stale CI runs**. If you push multiple commits to the same PR, dbt Cloud will cancel older, now-out-of-date CI checks automatically. No resources wasted on checking stale code. + +- **Run CI checks without blocking production runs**. CI checks will no longer consume run slots, meaning you can have as many CI checks running as you want, without impeding your production jobs. + +To learn more, refer to [Continuous integration](/docs/deploy/continuous-integration) and [CI jobs](/docs/deploy/ci-jobs). diff --git a/website/docs/docs/dbt-versions/release-notes/07-june-2023/lint-format-rn.md b/website/docs/docs/dbt-versions/release-notes/07-june-2023/lint-format-rn.md new file mode 100644 index 00000000000..e99d1fe3e0b --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/07-june-2023/lint-format-rn.md @@ -0,0 +1,26 @@ +--- +title: "New: You can now lint and format your code in the IDE" +id: "lint-format-rn" +description: "June 2023 release note: In the dbt Cloud IDE, you can perform linting and formatting on SQL, YAML, Markdown, Python, and JSON files using tools like SQLFluff, sqlfmt, Prettier, and Black." +sidebar_label: "New: Lint and format in the IDE" +sidebar_position: 10 +tags: [June-2023, IDE] +--- + +dbt Labs is excited to announce you can now lint and format your dbt code in the dbt Cloud IDE. This is an enhanced development workflow which empowers you to effortlessly prioritize code quality. + +You can perform linting and formatting on five different file types: SQL, YAML, Markdown, Python, and JSON. + +For SQL files, you can easily lint and format your code using [SQLFluff](https://sqlfluff.com/) and apply consistent formatting using [sqlfmt](http://sqlfmt.com/). Additionally, for other file types like YAML, Markdown, JSON, and Python, you can utilize the respective tools powered by [Prettier](https://prettier.io/) and [Black](https://black.readthedocs.io/en/latest/) to ensure clean and standardized code formatting. + +For more info, read [Lint and format your code](/docs/cloud/dbt-cloud-ide/lint-format). + + + + + + + + + + diff --git a/website/docs/docs/dbt-versions/release-notes/07-june-2023/product-docs-jun.md b/website/docs/docs/dbt-versions/release-notes/07-june-2023/product-docs-jun.md new file mode 100644 index 00000000000..469d2ac362b --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/07-june-2023/product-docs-jun.md @@ -0,0 +1,35 @@ +--- +title: "June 2023 product docs updates" +description: "June 2023: The Product docs team merged 132 PRs, made various updates to dbt Cloud and Core, such as the Deploy sidebar, Supported platforms page, added a landing page on the References section, added an ADO example to the CI/CD guide, and more" +sidebar_label: "Update: Product docs changes" +tags: [June-2023, product-docs] +date: 2023-07-04 +sidebar_position: 10 +--- + +Hello from the dbt Docs team: @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun! First, we’d like to thank the 17 new community contributors to docs.getdbt.com — ✨ @aaronbini, @sjaureguimodo, @aranke, @eiof, @tlochner95, @mani-dbt, @iamtodor, @monilondo, @vrfn, @raginjason, @AndrewRTsao, @MitchellBarker, @ajaythomas, @smitsrr, @leoguyaux, @GideonShils, @michaelmherrera! + +Here's what's new to [docs.getdbt.com](http://docs.getdbt.com/) in June: + +## ☁ Cloud projects + +- We clarified the nuances of [CI and CI jobs](/docs/deploy/continuous-integration), updated the [Scheduler content](/docs/deploy/job-scheduler), added two new pages for the job settings and run visibility, moved the project state page to the [Syntax page](/reference/node-selection/syntax), and provided a landing page for [Deploying with Cloud](/docs/deploy/jobs) to help readers navigate the content better. +- We reformatted the [Supported data platforms page](/docs/supported-data-platforms) by adding dbt Cloud to the page, splitting it into multiple pages, using cards to display verified adapters, and moving the [Warehouse setup pages](/docs/core/connect-data-platform/about-core-connections) to the Docs section. +- We launched a new [Lint and format page](/docs/cloud/dbt-cloud-ide/lint-format), which highlights the awesome new dbt Cloud IDE linting/formatting function. +- We enabled a connection between [dbt Cloud release notes](/docs/dbt-versions/dbt-cloud-release-notes) and the dbt Slack community. This means new dbt Cloud release notes are automatically sent to the slack community [#dbt-cloud channel](https://getdbt.slack.com/archives/CMZ2V0X8V) via RSS feed, keeping users up to date with changes that may affect them. +- We’ve added two new docs links in the dbt Cloud Job settings user interface (UI). This will provide additional guidance and help users succeed when setting up a dbt Cloud job: [job commands](/docs/deploy/job-commands) and job triggers. +- We added information related to the newly created [IT license](/docs/cloud/manage-access/about-user-access#license-based-access-control), available for Team and Enterprise plans. +- We added a new [Supported browser page](/docs/cloud/about-cloud/browsers), which lists the recommended browsers for dbt Cloud. +- We launched a new page informing users of [new Experimental features option](/docs/dbt-versions/experimental-features) in dbt Cloud. +- We worked with dbt Engineering to help publish new beta versions of the dbt [dbt Cloud Administrative API docs](/docs/dbt-cloud-apis/admin-cloud-api). + + +## 🎯 Core projects + +- We launched the new [MetricFlow docs](/docs/build/build-metrics-intro) on dbt Core v1.6 beta. +- Split [Global configs](reference/global-configs/about-global-configs) into individual pages, making it easier to find, especially using search. + + +## New 📚 Guides, ✏️ blog posts, and FAQs + +- Add an Azure DevOps example to the [Customizing CI/CD guide](/guides/orchestration/custom-cicd-pipelines/3-dbt-cloud-job-on-merge). diff --git a/website/docs/docs/dbt-versions/release-notes/08-may-2023/discovery-api-public-preview.md b/website/docs/docs/dbt-versions/release-notes/08-may-2023/discovery-api-public-preview.md new file mode 100644 index 00000000000..abe0e06f164 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/08-may-2023/discovery-api-public-preview.md @@ -0,0 +1,24 @@ +--- +title: "Latest environment state in the dbt Cloud Discovery API" +id: "discovery-api-public-preview" +description: "Public preview of the dbt Cloud Discovery API is now available." +sidebar_label: "New: Latest environment state in the Discovery API" +sidebar_position: 5 +tags: [May-2023, API] +--- + +Users of the Discovery API can now query the latest state of their environment, meaning there's no need to consolidate results across jobs or artifact files. The environment essentially represents the latest production state of a dbt project. The new `environment` endpoint is in public preview and can be used with the existing `modelByEnvironment` endpoint for historical analysis. For details, refer to these docs: + +- [About the Discovery API](/docs/dbt-cloud-apis/discovery-api) +- [Use cases and examples for the Discovery API](/docs/dbt-cloud-apis/discovery-use-cases-and-examples) +- [Query the Discovery API](/docs/dbt-cloud-apis/discovery-querying) +- [Project state in dbt Cloud](/docs/dbt-cloud-apis/project-state) + +## Roadmap + +dbt Labs is continually enhancing the dbt Cloud Discovery API to ensure you have access to timely, rich, and reliable metadata about your dbt runs. In the coming year, we’ll expand the range of questions the API helps answer so you can more easily understand the state, meaning, and structure of your data to inform data development and analysis experiences. + +- **[Now] Query across jobs & lineage** — Get the latest state of a dbt DAG (production environment) to find, understand, and trust the right dataset to analyze. + - **[Ongoing] Improvements** — Enhanced developer ergonomics, state fidelity, and metadata timeliness. +- **[Soon] Query across projects** — View and manage cross-project lineage using public models to define, use, and manage governed datasets for enhanced collaboration across teams. +- **[Later] Query over time** — Understand longer-term dbt Cloud execution result trends to optimize pipeline performance and costs, such as improving costly, error-prone, or slow datasets. diff --git a/website/docs/docs/dbt-versions/release-notes/08-may-2023/may-ide-updates.md b/website/docs/docs/dbt-versions/release-notes/08-may-2023/may-ide-updates.md new file mode 100644 index 00000000000..d85ffa154dd --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/08-may-2023/may-ide-updates.md @@ -0,0 +1,46 @@ +--- +title: "May IDE updates and fixes" +id: "may-ide-updates" +description: "May 2023 release note: We've launched SQLFluff in beta, released an IDE UI page, significantly improved IDE performance, improved error messages, fixed bugs, and more." +sidebar_label: "Update and fixes: IDE" +sidebar_position: 2 +tags: [May-2023, IDE] +--- + +To continue improving your [Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) development experience, the dbt Labs team continues to work on adding new features, fixing bugs, and increasing reliability ✨. + +Stay up-to-date with [IDE-related changes](/tags/ide). + +## New features +- Lint via SQL Fluff is now available in beta (GA over the next 2-3 weeks) +- Format markdown files with prettier +- Leverage developer experience shortcuts, including ``Ctrl + ` `` (toggle history drawer), `CMD + Option + /` (toggle block comment), `CMD + Shift + P` (open command palette), `Option + W` (close editor tab) +- Display parent folder name for files with same name in Changes section +- Navigate the new IDE features quickly using [the IDE User Interface](/docs/cloud/dbt-cloud-ide/ide-user-interface) help page +- Use `top X` in SQL when previewing in the IDE +- Opt into the new IDE backend layer over the past month (still with dbt-rpc). Ready for beta later in June! + + +## Product refinements + +- Performance-related upgrades: + - Reduced cold start time by 60+% + - Improved render time of modals in the IDE by 98% + - Improved IDE performance with dbt Core v1.5+ (faster and snappier – highly encourage you to [upgrade your dbt version](/docs/dbt-versions/upgrade-core-in-cloud)!) +- Upgraded sqlfmt (which powers the Format button) to 0.18.0 +- Updated Build button to change menu options based on file/model type (snapshot, macro, etc.) +- Display message to disable adblocker for file contents error +- Moved Format button to console bar +- Made many security enhancements in the IDE +## Bug fixes + +- File icon sizes no longer get wonky in small screen +- Toast notifications no longer take over command bar menu +- Hover info inside the text editor no longer gets cut off +- Transition between a file and a recently modified scratchpad no longer triggers a console error +- dbt v1.5+ now can access the IDE +- Confirm button on the Unsaved Changes modal now closes after clicking it +- Long node names no longer overflow in the parsed logs section in history drawer +- Status pill in history drawer no longer scales with longer command +- Tooltip for tab name with a long file name is no longer cut off +- Lint button should no longer available in main branch diff --git a/website/docs/docs/dbt-versions/release-notes/08-may-2023/product-docs-may.md b/website/docs/docs/dbt-versions/release-notes/08-may-2023/product-docs-may.md new file mode 100644 index 00000000000..762a6a723f8 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/08-may-2023/product-docs-may.md @@ -0,0 +1,43 @@ +--- +title: "May 2023 product docs updates" +id: "May-product-docs" +description: "May 2023: Find out what the product docs team has been busy doing in the month of May." +sidebar_label: "Update: Product docs changes" +sidebar_position: 1 +tags: [May-2023, product-docs] +date: 2023-06-01 +--- + +Hello from the dbt Docs team: @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun! First, we’d like to thank the 13 new community contributors to docs.getdbt.com! + +Here's what's new to [docs.getdbt.com](http://docs.getdbt.com/) in May: + +## 🔎 Discoverability + +- We made sure everyone knows that Cloud-users don’t need a [profiles.yml file](/docs/core/connect-data-platform/profiles.yml) by adding a callout on several key pages. +- Fleshed out the [model jinja variable page](/reference/dbt-jinja-functions/model), which originally lacked conceptual info and didn’t link to the schema page. +- Added a new [Quickstarts landing page](/quickstarts). This new format sets up for future iterations that will include filtering! But for now, we are excited you can step through quickstarts in a focused way. + +## ☁ Cloud projects + +- We launched [dbt Cloud IDE user interface doc](/docs/cloud/dbt-cloud-ide/ide-user-interface), which provides a thorough walkthrough of the IDE UI elements and their definitions. +- Launched a sparkling new [dbt Cloud Scheduler page](/docs/deploy/job-scheduler) ✨! We went from previously having little content around the scheduler to a subsection that breaks down the awesome scheduler features and how it works. +- Updated the [dbt Cloud user license page](/docs/cloud/manage-access/seats-and-users#licenses) to clarify how to add or remove cloud users. +- Shipped these Discovery API docs to coincide with the launch of the Discovery API: + - [About the Discovery API](/docs/dbt-cloud-apis/discovery-api) + - [Use cases and examples for the Discovery API](/docs/dbt-cloud-apis/discovery-use-cases-and-examples) + - [Query the Discovery API](/docs/dbt-cloud-apis/discovery-querying) + +## 🎯 Core projects + +- See what’s coming up [in Core v 1.6](https://github.com/dbt-labs/docs.getdbt.com/issues?q=is%3Aissue+label%3A%22dbt-core+v1.6%22)! +- We turned the `profiles.yml` [page](/docs/core/connect-data-platform/profiles.yml) into a landing page, added more context to profile.yml page, and moved the ‘About CLI’ higher up in the `Set up dbt` section. + +## New 📚 Guides, ✏️ blog posts, and FAQs + +If you want to contribute to a blog post, we’re focusing on content + +- Published a blog post: [Accelerate your documentation workflow: Generate docs for whole folders at once](/blog/generating-dynamic-docs-dbt) +- Published a blog post: [Data engineers + dbt v1.5: Evolving the craft for scale](/blog/evolving-data-engineer-craft) +- Added an [FAQ](/faqs/Warehouse/db-connection-dbt-compile) to clarify the common question users have on *Why does dbt compile needs to connect to the database?* +- Published a [discourse article](https://discourse.getdbt.com/t/how-to-configure-external-user-email-notifications-in-dbt-cloud/8393) about configuring job notifications for non-dbt Cloud users diff --git a/website/docs/docs/dbt-versions/release-notes/08-may-2023/run-details-and-logs-improvements.md b/website/docs/docs/dbt-versions/release-notes/08-may-2023/run-details-and-logs-improvements.md new file mode 100644 index 00000000000..1aabe517076 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/08-may-2023/run-details-and-logs-improvements.md @@ -0,0 +1,19 @@ +--- +title: "Run details and logs usability and design improvements" +sidebar_label: "Update: Improvements to run details and logs" +sidebar_position: 3 +tags: [May-2023, Scheduler] +--- + +New usability and design improvements to the run details and logs in dbt Cloud are now available. The ability to triage errors in logs is a big benefit of using dbt Cloud's job and scheduler functionality. These updates help make the process of finding the root cause much easier. + +Highlights include: + +- Surfacing a warn state on a run step +- Search in logs +- Easier discoverability of errors and warnings in logs +- Lazy loading of logs, making the whole run details page load faster and feel more performant +- Cleaner look and feel with iconography +- Helpful tool tips + + diff --git a/website/docs/docs/dbt-versions/release-notes/08-may-2023/run-history-endpoint.md b/website/docs/docs/dbt-versions/release-notes/08-may-2023/run-history-endpoint.md new file mode 100644 index 00000000000..050fd8339a2 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/08-may-2023/run-history-endpoint.md @@ -0,0 +1,20 @@ +--- +title: "Older Run History retrieval change" +id: "run-history-endpoint" +sidebar_label: "Update: Older Run History retrieval change" +sidebar_position: 6 +tags: [May-2023, API, Scheduler] +--- + +dbt Labs is making a change to the metadata retrieval policy for Run History in dbt Cloud. + + +**Beginning June 1, 2023,** developers on the dbt Cloud multi-tenant application will be able to self-serve access to their account’s run history through the dbt Cloud user interface (UI) and API for only 365 days, on a rolling basis. Older run history will be available for download by reaching out to Customer Support. We're seeking to minimize the amount of metadata we store while maximizing application performance. + + +Specifically, all `GET` requests to the dbt Cloud [Runs endpoint](https://docs.getdbt.com/dbt-cloud/api-v2-legacy#tag/Runs) will return information on runs, artifacts, logs, and run steps only for the past 365 days. Additionally, the run history displayed in the dbt Cloud UI will only show runs for the past 365 days. + + + +We will retain older run history in cold storage and can make it available to customers who reach out to our Support team. To request older run history info, contact the Support team at [support@getdbt.com](mailto:support@getdbt.com) or use the dbt Cloud application chat by clicking the `?` icon in the dbt Cloud UI. + diff --git a/website/docs/docs/dbt-versions/release-notes/08-may-2023/run-history-improvements.md b/website/docs/docs/dbt-versions/release-notes/08-may-2023/run-history-improvements.md new file mode 100644 index 00000000000..d4d299b1d36 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/08-may-2023/run-history-improvements.md @@ -0,0 +1,21 @@ +--- +title: "Run History usability and design improvements" +sidebar_label: "Update: Run History improvements" +sidebar_position: 4 +tags: [May-2023, Scheduler] +--- + +New usability and design improvements to the **Run History** dashboard in dbt Cloud are now available. These updates allow people to discover the information they need more easily by reducing the number of clicks, surfacing more relevant information, keeping people in flow state, and designing the look and feel to be more intuitive to use. + + + + +Highlights include: + +- Usability improvements for CI runs with hyperlinks to the branch, PR, and commit SHA, along with more discoverable temporary schema names +- Preview of runs' error messages on hover +- Hyperlinks to the environment +- Better iconography on run status +- Clearer run trigger cause (API, scheduled, pull request, triggered by user) +- More details on the schedule time on hover +- Run timeout visibility diff --git a/website/docs/docs/dbt-versions/release-notes/09-april-2023/api-endpoint-restriction.md b/website/docs/docs/dbt-versions/release-notes/09-april-2023/api-endpoint-restriction.md new file mode 100644 index 00000000000..8507fe3dbbb --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/09-april-2023/api-endpoint-restriction.md @@ -0,0 +1,23 @@ +--- +title: "List Runs API Endpoint `order_by` restrictions" +id: "api-endpoint-restriction" +sidebar_label: "Deprecation: List Runs API Endpoint order_by restrictions" +sidebar_position: 10 +tags: [Apr-2023, API] +--- + +Starting May 15, 2023, we will support only the following `order_by` functionality for the List Runs endpoint: + +- `id` and `-id` +- `created_at` and `-created_at` +- `finished_at` and `-finished_at` + +We recommend that you change your API requests to https:///api/v2/accounts/{accountId}/runs/ to use a supported `order_by` before this date. + +:::info Access URLs + +dbt Cloud is hosted in multiple regions around the world, and each region has a different access URL. Users on Enterprise plans can choose to have their account hosted in any one of these regions. For a complete list of available dbt Cloud access URLs, refer to [Regions & IP addresses](/docs/cloud/about-cloud/regions-ip-addresses). + +::: + +For more info, refer to our [documentation](/dbt-cloud/api-v2-legacy#tag/Runs/operation/listRunsForAccount). diff --git a/website/docs/docs/dbt-versions/release-notes/09-april-2023/apr-ide-updates.md b/website/docs/docs/dbt-versions/release-notes/09-april-2023/apr-ide-updates.md new file mode 100644 index 00000000000..0960feb0e2f --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/09-april-2023/apr-ide-updates.md @@ -0,0 +1,45 @@ +--- +title: "April IDE updates and fixes" +id: "apr-ide-updates" +description: "Apr 2023 release note: We've enhanced the IDE by displaying load times when previewing models, showing live previews of Markdown and CSV files, adding the ability to duplicate files in the File Tree, and more." +sidebar_label: "Update and fixes: IDE" +sidebar_position: 7 +tags: [Apr-2023, IDE] +--- + +To continue improving your [Cloud IDE](https://docs.getdbt.com/docs/cloud/develop-in-the-cloud) development experience, the dbt Labs team continue to work on adding new features, fixing bugs, and increasing reliability ✨. + +Read more about the [upcoming improvements to the Cloud IDE](https://www.getdbt.com/blog/improvements-to-the-dbt-cloud-ide/) and stay up-to-date with [IDE-related changes](https://docs.getdbt.com/tags/ide). + +## New features + +* New warning message suggests you invoke `dbt deps` when it's needed (as informed by `dbt-score`). +* New warning message appears when you select models but don't save them before clicking **Build** or invoking dbt (like, dbt build/run/test). +* Previews of Markdown and CSV files are now available in the IDE console. +* The file tree menu now includes a Duplicate File option. +* Display loading time when previewing a model + +## Product refinements + +* Enhance autocomplete experience which has performed slowly for people with large projects and who implement a limit to max `manifest.json` for this feature +* Introduce pagination for invocation node summary view (displaying 100 nodes at a time) +* Improve rendering for the Changes / Version Control section of the IDE +* Update icons to be consistent in dbt Cloud +* Add table support to the Markdown preview +* Add the lineage tab back to seed resources in the IDE +* Implement modal priority when there are multiple warning modals +* Improve a complex command's description in the command palette + +## Bug fixes + +* File tree no longer collapses on first click when there's a project subdirectory defined +* **Revert all** button now works as expected +* CSV preview no longer fails with only one column +* Cursor and scroll bar location are now persistent with their positions +* `git diff` view now shows just change diffs and no longer shows full diff (as if file is new) until page refreshes +* ToggleMinimap Command no longer runs another Command at the same time +* `git diff` view no longer shows infinite spins in specific scenarios (new file, etc.) +* File contents no longer get mixed up when using diff view and one file has unsaved changes +* YML lineage now renders model without tests (in dbt Core v1.5 and above) +* Radio buttons for **Summary** and **Details** in the logs section now consistently update to show the accurate tab selection +* IDE no longer throws the console error `Error: Illegal argument` and redirects to the `Something went wrong` page diff --git a/website/docs/docs/dbt-versions/release-notes/09-april-2023/product-docs.md b/website/docs/docs/dbt-versions/release-notes/09-april-2023/product-docs.md new file mode 100644 index 00000000000..d30bcf85b99 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/09-april-2023/product-docs.md @@ -0,0 +1,40 @@ +--- +title: "April 2023 product docs updates" +id: "april-product-docs" +description: "April 2023: " +sidebar_label: "Update: product docs changes" +sidebar_position: 8 +tags: [Apr-2023, product-docs] +--- + +Hello from the dbt Docs team: @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun! We want to share some highlights introduced to docs.getdbt.com in the last month: + +## 🔎 Discoverability + +- [API docs](/docs/dbt-cloud-apis/overview) now live in the left sidebar to improve discoverability. +- [The deploy dbt jobs sidebar](/docs/deploy/deployments) has had a glow up 💅 that splits the ‘about deployment’ into two paths (deploy w dbt cloud and deploy w other tools), adds more info about the dbt cloud scheduler, its features, and how to create a job, adds ADF deployment guidance. We hope the changes improve the user experience and provide users with guidance when deploying with other tools. + +## ☁ Cloud projects + +- Added Starburst/Trino adapter docs, including: + * [dbt Cloud quickstart guide](/quickstarts/starburst-galaxy),  + * [connection page](/docs/cloud/connect-data-platform/connect-starburst-trino),  + * [set up page](/docs/core/connect-data-platform/trino-setup), and [config page](/reference/resource-configs/trino-configs). +- Enhanced [dbt Cloud jobs page](/docs/deploy/jobs) and section to include conceptual info on the queue time, improvements made around it, and about failed jobs. +- Check out the April dbt [Cloud release notes](/docs/dbt-versions/dbt-cloud-release-notes) + +## 🎯 Core projects + +- Clearer descriptions in the [Jinja functions page](/reference/dbt-jinja-functions), that improve content for each card.  +- [1.5 Docs](/guides/migration/versions/upgrading-to-v1.5) have been released as an RC! +- See the beautiful [work captured in Core v 1.5](https://github.com/dbt-labs/docs.getdbt.com/issues?q=is%3Aissue+label%3A%22dbt-core+v1.5%22+is%3Aclosed). + +## New 📚 Guides and ✏️ blog posts + +- [Use Databricks workflows to run dbt Cloud jobs](/guides/orchestration/how-to-use-databricks-workflows-to-run-dbt-cloud-jobs) +- [Refresh Tableau workbook with extracts after a job finishes](/guides/orchestration/webhooks/zapier-refresh-tableau-workbook) +- [dbt Python Snowpark workshop/tutorial](/guides/dbt-ecosystem/dbt-python-snowpark/1-overview-dbt-python-snowpark) +- [How to optimize and troubleshoot dbt Models on Databricks](/guides/dbt-ecosystem/databricks-guides/how_to_optimize_dbt_models_on_databricks) +- [The missing guide to debug() in dbt](https://docs.getdbt.com/blog/guide-to-jinja-debug) +- [dbt Squared: Leveraging dbt Core and dbt Cloud together at scale](https://docs.getdbt.com/blog/dbt-squared) +- [Audit_helper in dbt: Bringing data auditing to a higher level](https://docs.getdbt.com/blog/audit-helper-for-migration) diff --git a/website/docs/docs/dbt-versions/release-notes/09-april-2023/scheduler-optimized.md b/website/docs/docs/dbt-versions/release-notes/09-april-2023/scheduler-optimized.md new file mode 100644 index 00000000000..e552c449808 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/09-april-2023/scheduler-optimized.md @@ -0,0 +1,16 @@ +--- +title: "The dbt Cloud scheduler now prevents queue clog by canceling unnecessary runs of over-scheduled jobs" +id: "scheduler-optimized" +description: "April 2023: " +sidebar_label: "Update: Scheduler optimizes job queue" +sidebar_position: 9 +tags: [Apr-2023, scheduler] +--- + +The dbt Cloud Scheduler now prevents queue clog by canceling unnecessary runs of over-scheduled jobs. + +The duration of a job run tends to grow over time, usually caused by growing amounts of data in the warehouse. If the run duration becomes longer than the frequency of the job’s schedule, the queue will grow faster than the scheduler can process the job’s runs, leading to a runaway queue with runs that don’t need to be processed. + +Previously, when a job was in this over-scheduled state, the scheduler would stop queuing runs after 50 were already in the queue. This led to a poor user experience where the scheduler canceled runs indiscriminately. You’d have to log into dbt Cloud to manually cancel all the queued runs and change the job schedule to "unclog" the scheduler queue. + +Now, the dbt Cloud scheduler detects when a scheduled job is set to run too frequently and appropriately cancels runs that don’t need to be processed. Specifically, scheduled jobs can only ever have one run of the job in the queue, and if a more recent run gets queued, the early queued run will get canceled with a helpful error message. Users will still need to either refactor the job so it runs faster or change the job schedule to run less often if the job often gets into an over-scheduled state. diff --git a/website/docs/docs/dbt-versions/release-notes/09-april-2023/starburst-trino-ga.md b/website/docs/docs/dbt-versions/release-notes/09-april-2023/starburst-trino-ga.md new file mode 100644 index 00000000000..613a0c02432 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/09-april-2023/starburst-trino-ga.md @@ -0,0 +1,12 @@ +--- +title: "Integrate Starburst and Trino with dbt-trino in dbt Cloud" +id: "starburst-trino-ga" +sidebar_label: "New: Integrate Starburst and Trino in dbt Cloud" +sidebar_position: 8 +tags: [Apr-2023] +--- + +The Starburst (Trino compatible) connection is now generally available in dbt Cloud. This means you can now use dbt Cloud to connect with Starburst Galaxy, Starburst Enterprise, and self-hosted Trino. This feature is powered by the [`dbt-trino`](https://github.com/starburstdata/dbt-trino) adapter. + +To learn more, check out our Quickstart guide for [dbt Cloud and Starburst Galaxy](https://docs.getdbt.com/quickstarts/starburst-galaxy). + diff --git a/website/docs/docs/dbt-versions/release-notes/10-mar-2023/1.0-deprecation.md b/website/docs/docs/dbt-versions/release-notes/10-mar-2023/1.0-deprecation.md new file mode 100644 index 00000000000..b11bf702330 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/10-mar-2023/1.0-deprecation.md @@ -0,0 +1,21 @@ +--- +title: "dbt Cloud requires dbt version 1.0 or later" +id: "1.0-deprecation" +description: "Mar 2023: dbt Cloud now requires dbt version 1.0 or later and non-compliant jobs or environments were automatically upgraded to v1.4" +sidebar_label: "Deprecation: dbt Cloud requires dbt v1.0 or later." +tags: [Mar-2023] +--- + + +dbt Cloud now requires dbt version 1.0 or later. As of March 1, 2023, we removed all instances of older dbt versions from dbt Cloud. + +Any environments or jobs configured with a dbt version lower than 1.0 were automatically updated to dbt v1.4, which is the latest minor version available on dbt Cloud. + +For more info on dbt versions, releases, and dbt Cloud support timeline, refer to [About dbt Core versions](/docs/dbt-versions/core#latest-releases). + +Refer to some additional info and resources to help you upgrade your dbt version: + +- [How to upgrade dbt without fear](https://docs.getdbt.com/blog/upgrade-dbt-without-fear) +- [Upgrade Q&A on breaking changes](/docs/dbt-versions/upgrade-core-in-cloud#upgrading-legacy-versions-under-10) +- [Version migration guides](/guides/migration/versions) + diff --git a/website/docs/docs/dbt-versions/release-notes/10-mar-2023/apiv2-limit.md b/website/docs/docs/dbt-versions/release-notes/10-mar-2023/apiv2-limit.md new file mode 100644 index 00000000000..85c4af48b54 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/10-mar-2023/apiv2-limit.md @@ -0,0 +1,14 @@ +--- +title: "API requests have a maximum limit of `100`" +id: apiv2-limit" +description: "Mar 2023: In order to ease pressure on our API, we have implemented a maximum limit of `100` for all API requests to our `list` endpoints. This limit is applicable to multi-tenant instances only." +sidebar_label: "Update: API requests have a maximum limit of `100`" +tags: [Mar-2023, API] +--- + + +To make the API more scalable and reliable, we've implemented a maximum limit of `100` for all API requests to our `list` endpoints. If API requests exceed the maximum limit parameter of `100`, a user will receive an API error message. + +This maximum limit applies to [multi-tenant instances](/docs/cloud/about-cloud/regions-ip-addresses) only, and _does not_ apply to single tenant instances. + +Refer to the [Pagination](https://docs.getdbt.com/dbt-cloud/api-v2-legacy#section/Pagination) section for more information on this change. diff --git a/website/docs/docs/dbt-versions/release-notes/10-mar-2023/mar-ide-updates.md b/website/docs/docs/dbt-versions/release-notes/10-mar-2023/mar-ide-updates.md new file mode 100644 index 00000000000..d72d95e9b23 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/10-mar-2023/mar-ide-updates.md @@ -0,0 +1,50 @@ +--- +title: "March IDE updates and fixes" +id: "mar-ide-updates" +description: "Mar 2023 release note: We've enhanced the IDE by adding add common dbt commands to the command palette, creating PRs even if you have uncommitted changes, autocompleting suggestions when editing a yml file, editing directly in the git diff view, improved the DAG selector, upgraded sqlfmt, improved syntax error messaging, and more." +sidebar_label: "Update and fixes: IDE" +tags: [Mar-2023, IDE] +--- + +To continue improving your [Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) development experience, the dbt Labs team continue to work on adding new features, fixing bugs, and increasing reliability ✨. + +Read more about the [upcoming improvements to the Cloud IDE](https://www.getdbt.com/blog/improvements-to-the-dbt-cloud-ide/) and stay up-to-date with [IDE-related changes](https://docs.getdbt.com/tags/ide). + + +## New features + +- Commit and revert individual files under **Version Control**. +- Use the [command palette](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#cloud-ide-features) to invoke common complex dbt commands, such as resuming from the last failure. +- Create PRs even when there are uncommitted changes (under the **git** dropdown). +- The IDE will display more autocomplete suggestions when editing a YML file, powered by [dbt-jsonschema](https://github.com/dbt-labs/dbt-jsonschema). +- The file tree now has additional options in the right-click menu, such as Copy model as ref or Copy file path. +- The DAG view has been adjusted to a default of `2+model+2`. +- A lineage selector has been implemented in the DAG/lineage sub-tab. +- Edit directly in the git diff view located in the right pane. +- A warning message will now appear when users press Command-W/Control-W when there are unsaved changes. +- A new onboarding flow guide is now available. + +## Product refinements + +- The DAG selector now uses `name` instead of `file_uri` to build selectors. +- The DAG is now vertically centered under the new Selector Input element +- sqlfmt has been upgraded to v0.17.0. +- When the Format button fails, a toast notification will display a syntax error. +- The editor now has the option to toggle minimap/word-wrap via right-click. +- The history drawer displays elapsed time in real-time and s/m/h increments. +- When deleting development environments, the delete modal will now warn users that any uncommitted changes will be lost. +- The context for the Git button has been adjusted to show that it will link to an external site (such as GitHub or GitLab) when users create a pull request. + +## Bug fixes + +- The IDE now displays an error message when the git repository is not reachable. Previously, it failed silently. +- The kebab menu is now visible when the invocation history drawer is open. Previously, it wasn't showing. +- DAGs are now updated/populated consistently. Previously, it occasionally failed. +- The purple highlight for DAG selection is now consistent across files. Previously, it was inconsistent. +- Users can now rename files back to their original name. Previously, this wasn't possible. +- The link to the IDE from the project setup page has been corrected. +- The IDE no longer has issues with single-space file names. +- Adding invalid characters in the sub-directory config no longer causes the IDE to fail. +- YML autocomplete triggers consistently now. Previously, it occasionally didn't trigger. +- Reverting single files now reloads the file contents in the tab. Previously, it didn't reload. +- The file tree no longer collapses on the first click when there is a project subdirectory defined. diff --git a/website/docs/docs/dbt-versions/release-notes/10-mar-2023/public-preview-trino-in-dbt-cloud.md b/website/docs/docs/dbt-versions/release-notes/10-mar-2023/public-preview-trino-in-dbt-cloud.md new file mode 100644 index 00000000000..bf3840a8b02 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/10-mar-2023/public-preview-trino-in-dbt-cloud.md @@ -0,0 +1,17 @@ +--- +title: "Starbust integration with public preview of dbt-trino in dbt Cloud" +id: "public-preview-trino-in-dbt-cloud" +description: "Public preview of dbt Cloud integrations with Starburst and Trino clusters is now available." +sidebar_label: "New: Starbust integration with public preview of dbt-trino in dbt Cloud" +tags: [Mar-2023] +--- + +dbt Labs is introducing the newest connection option in dbt Cloud: the `dbt-trino` adapter is now available in Public Preview. This allows you to connect to Starburst Galaxy, Starburst Enterprise, and self-hosted Trino from dbt Cloud. + +Check out our [Quickstart for dbt Cloud and Starburst Galaxy](/quickstarts/starburst-galaxy) to explore more. + +## What’s the reason users should be excited about this? + +By using Starburst or Trino as your data platform in dbt Cloud, you can now seamlessly transform data that's distributed across multiple sources, including object storage, data lakes, and data warehouses, with no data movement required. + +The seamless integration means you can now get up and running within minutes. diff --git a/website/docs/docs/dbt-versions/release-notes/11-feb-2023/feb-ide-updates.md b/website/docs/docs/dbt-versions/release-notes/11-feb-2023/feb-ide-updates.md new file mode 100644 index 00000000000..d52ad2d4081 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/11-feb-2023/feb-ide-updates.md @@ -0,0 +1,41 @@ +--- +title: "Feb IDE updates and fixes" +id: "feb-ide-updates" +description: "Feb 2023 release note: We've enhanced the IDE by adding custom node colors in the DAG, ref autocomplete, double-click files to rename them, add link to repo from the branch name, enabled syntax highlighting for jinja, improve file tree render time, and more." +sidebar_label: "Update and fixes: IDE" +tags: [Feb-2023, IDE] +--- + +To continue improving our [Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) experience, the dbt Labs team worked on fixing bugs, increasing reliability, and adding new features ✨. + +Learn more about the [February changes](https://getdbt.slack.com/archives/C03SAHKKG2Z/p1677605383451109). + +## New features + +- Support for custom node colors in the IDE DAG visualization +- Autosave prototype is now available under feature flag. [Contact](mailto:cloud-ide-feedback@dbtlabs.com) the dbt Labs IDE team to try this out +- Ref autocomplete includes models from seeds and snapshots +- Prevent menus from getting cropped (git controls dropdown, file tree dropdown, build button, editor tab options) +- Additional option to access the file menu by right-clicking on the files and folders in the file tree +- Rename files by double-clicking on files in the file tree and the editor tabs +- Right-clicking on file tabs has new options and will now open at your cursor instead of in the middle of the tab +- The git branch name above **Version Control** links to the repo for specific git providers + * Currently available for all [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) instances using GitHub or GitLab providers + +## Product refinements + +- Added an error modal for RPC parsing errors when users attempt to invoke dbt commands (preview, compile, or general dbt invocations) +- Enabled syntax highlighting for Jinja expression and statement delimiters +- Clarified and renamed the options under the **Build** button +- Changed the term for RPC status from `Compiling` to `Parsing` to match dbt-core construct +- Implemented a new File Tree component to improve render time by 60% +- Disabled the Local Storage of File Tree to prevent users from running into max LocalStorage issue for large projects +- Changed snapshot snippet template (`__snapshot`) to a select from source + +## Bug fixes + +- You no longer have file contents carrying over when you switch to a different project that has the same file name +- The preview max limit no longer allows you to override the maximum +- You no longer encounter node statuses failing to update in the history drawer for those on version 1.4 core. (This is a partial fix that may be fully addressed by core version 1.5) +- You can now use the **Copy File Name** option to copy up to the last dot, rather than the first dot + diff --git a/website/docs/docs/dbt-versions/release-notes/11-feb-2023/no-partial-parse-config.md b/website/docs/docs/dbt-versions/release-notes/11-feb-2023/no-partial-parse-config.md new file mode 100644 index 00000000000..0727c5eb339 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/11-feb-2023/no-partial-parse-config.md @@ -0,0 +1,11 @@ +--- +title: "Disable partial parsing in dbt Cloud job commands" +id: "no-partial-parse-config" +description: "You can now disable partial parsing in dbt Cloud job commands." +sidebar_label: "New: Disable partial parsing in job commands" +tags: [Feb-2023] +--- + +You can now use the `--no-partial-parse` flag to disable partial parsing in your dbt Cloud job commands.  + +Previously, the [`--no-partial-parse` global config](/reference/global-configs/parsing) was only available in dbt Core. For more information, refer to [partial parsing](/reference/parsing#partial-parsing). diff --git a/website/docs/docs/dbt-versions/release-notes/12-jan-2023/ide-updates.md b/website/docs/docs/dbt-versions/release-notes/12-jan-2023/ide-updates.md new file mode 100644 index 00000000000..e8b3dcc4caa --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/12-jan-2023/ide-updates.md @@ -0,0 +1,43 @@ +--- +title: "Jan IDE updates and fixes" +id: "ide-updates" +description: "Jan 2023 release note: We've enhanced the IDE with improved syntax highlighting, faster and snappier IDE, improved error messaging, view repo status, added an easter egg, and more." +sidebar_label: "Update and fixes: IDE" +tags: [Jan-2023, IDE] +--- + +In the spirit of continuing to improve our [Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) experience, the dbt Labs team worked on fixing bugs, increasing reliability, and adding new features ✨. + +Learn more about the [January changes](https://getdbt.slack.com/archives/C03SAHKKG2Z/p1675272600286119) and what's coming soon. + +## New features + +- Improved syntax highlighting within the IDE for better Jinja-SQL combination (double quotes now show proper syntax highlight!) +- Adjusted the routing URL for the IDE page and removed the `next` from the URL +- Added a *new* easter egg within the IDE 🐶🦆 + +## Product refinements + +- Performance improvements and reduced IDE slowness. The IDE should feel faster and snappier. +- Reliability improvements – Improved error handling that previously put IDE in a bad state +- Corrected the list of dropdown options for the Build button +- Adjusted startup page duration +- Added code snippets for `unique` and `not_null` tests for yml files +- Added code snippets for metrics based on environment dbt versions +- Changed “commit and push” to “commit and sync” to better reflect the action +- Improved error message when saving or renaming files to duplicate names + +## Bug fixes + +- You no longer arbitrarily encounter an `RPC server got an unknown async ID` message +- You can now see the build button dropdown, which had been hidden behind the placeholder DAG screen +- You can now close toast notifications for command failure when the history drawer is open +- You no longer encounter a `Something went wrong` message when previewing a model +- You can now see repository status in the IDE, and the IDE finds the SSH folder +- Scroll bars and download CSV no longer flicker within the preview pane + +## Coming soon + +- dbt Labs will roll out a series of features that should improve the quality of life within the IDE over the next several months, such as autosave, the ability to revert individual files, and user experience improvements, like right-clicking. + +- dbt Labs is researching ways to include Linting/SQL Fluff in the IDE. If anyone is interested in sharing how you're using SQL Fluff today, please reach out to [dbt Labs IDE team](mailto:cloud-ide-feedback@dbtlabs.com). diff --git a/website/docs/docs/dbt-versions/release-notes/23-dec-2022/default-thread-value.md b/website/docs/docs/dbt-versions/release-notes/23-dec-2022/default-thread-value.md new file mode 100644 index 00000000000..90f3156599c --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/23-dec-2022/default-thread-value.md @@ -0,0 +1,15 @@ +--- +title: "Threads default value changed to 4" +id: "default-thread-value" +description: "Threads now default to 4 users' profile." +sidebar_label: "Enhancement: Threads default value changed to 4 " +tags: [Dec-2022] +--- + +Threads help parallelize node execution in the dbt directed acyclic graph [(DAG)](https://docs.getdbt.com/terms/dag). + +Previously, the thread value defaulted to 1, which can increase the runtime of your project. To help reduce the runtime of your project, the default value for threads in user profiles is now set to 4 threads. + +You can supply a custom thread count if you'd prefer more or less parallelization. + +For more information, read [Understanding threads](/docs/running-a-dbt-project/using-threads). diff --git a/website/docs/docs/dbt-versions/release-notes/23-dec-2022/new-jobs-default-as-off.md b/website/docs/docs/dbt-versions/release-notes/23-dec-2022/new-jobs-default-as-off.md new file mode 100644 index 00000000000..bdc89b4abde --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/23-dec-2022/new-jobs-default-as-off.md @@ -0,0 +1,14 @@ +--- +title: "Creating a new job no longer triggers a run by default" +id: "new-jobs-default-as-off.md" +description: "You need to click **Run on schedule** before a job will be scheduled to run" +sidebar_label: "Improvement: New jobs no longer run by default" +tags: [Dec-2022] +--- + +To help save compute time, new jobs will no longer be triggered to run by default. When you create a new job in dbt Cloud, you can trigger the job to run by selecting **Run on schedule** and completing the desired schedule and timing information. + +For more information, refer to [Deploy jobs](/docs/deploy/deploy-jobs). + + + diff --git a/website/docs/docs/dbt-versions/release-notes/23-dec-2022/private-packages-clone-git-token.md b/website/docs/docs/dbt-versions/release-notes/23-dec-2022/private-packages-clone-git-token.md new file mode 100644 index 00000000000..75db9d1888b --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/23-dec-2022/private-packages-clone-git-token.md @@ -0,0 +1,25 @@ +--- +title: "Private packages must be cloned using access tokens provided by environment variables" +description: "Private GitHub packages must be cloned using access tokens provided by environment variables." +sidebar_label: "Deprecation: Private packages must be cloned using access tokens" +tags: [Dec-2022] +--- + +The supported method for cloning private GitHub packages is the [git token method](/docs/build/packages#git-token-method), where an appropriate access token is passed into the package repository URL with an environment variable. + +A small number of people have been able to clone private packages using dbt's native GitHub application without explicitly providing an access token. This functionality is being deprecated as it’s limited in flexibility. + +If you have been using a package hosted in a private repository on GitHub, you must start passing an access token into the URL. + +An example of passing an access token: + + + +```yaml + +packages: +- git: "https://{{env_var('DBT_ENV_SECRET_GIT_CREDENTIAL')}}@github.com/dbt-labs/awesome_repo.git" + +``` + + diff --git a/website/docs/docs/dbt-versions/release-notes/24-nov-2022/dbt-databricks-unity-catalog-support.md b/website/docs/docs/dbt-versions/release-notes/24-nov-2022/dbt-databricks-unity-catalog-support.md new file mode 100644 index 00000000000..25d5ca5205f --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/24-nov-2022/dbt-databricks-unity-catalog-support.md @@ -0,0 +1,13 @@ +--- +title: "The dbt Cloud + Databricks experience is getting even better" +id: "cloud-databricks-unitycatalog" +description: "The dbt-databricks adapter is now available for use with dbt Cloud" +sidebar_label: "Enhancement: Support for Databricks Unity Catalog with dbt-databricks" +tags: [Nov-2022, v1.1.66.15] +--- + +dbt Cloud is the easiest and most reliable way to develop and deploy a dbt project. It helps remove complexity while also giving you more features and better performance. A simpler Databricks connection experience with support for Databricks’ Unity Catalog and better modeling defaults is now available for your use. + +For all the Databricks customers already using dbt Cloud with the dbt-spark adapter, you can now [migrate](https://docs.getdbt.com/guides/migration/tools/migrating-from-spark-to-databricks#migration) your connection to the [dbt-databricks adapter](https://docs.getdbt.com/reference/warehouse-setups/databricks-setup) to get the benefits. [Databricks](https://www.databricks.com/blog/2022/11/17/introducing-native-high-performance-integration-dbt-cloud.html) is committed to maintaining and improving the adapter, so this integrated experience will continue to provide the best of dbt and Databricks. + +Check out our [live blog post](https://www.getdbt.com/blog/dbt-cloud-databricks-experience/) to learn more. diff --git a/website/docs/docs/dbt-versions/release-notes/24-nov-2022/ide-features-ide-deprecation.md b/website/docs/docs/dbt-versions/release-notes/24-nov-2022/ide-features-ide-deprecation.md new file mode 100644 index 00000000000..87a7b179b6d --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/24-nov-2022/ide-features-ide-deprecation.md @@ -0,0 +1,37 @@ +--- + +title: "New IDE features and classic IDE deprecation" +id: "ide-features-ide-deprecation" +description: "Nov 2022 release note: We've enhanced the IDE by adding a button to automatically format your SQL. Added dark mode, Git diff view, and 4 new autocomplete options. We have deprecated the classic IDE." +sidebar_label: "Enhancement and deprecation: New IDE features and classic IDE deprecation" +tags: + - Nov-2022 + - v1.1.67.0 + - IDE + +--- + +### Extra features in new and refreshed IDE + +The refreshed version of the dbt Cloud IDE has launched four brand-new additional features, making it easier and faster for you to develop in the IDE. + +The new features are: + +- **Formatting** — Format your dbt SQL files to a single code style with a click of a button. This uses the tool [sqlfmt](https://github.com/tconbeer/sqlfmt). +- **Git diff view** — Highlights the changes in a file before opening a pull request. +- **dbt autocomplete** — There are four new types of autocomplete features to help you develop faster: + - Use `ref` to autocomplete your model names + - Use `source` to autocomplete your source name + table name + - Use `macro` to autocomplete your arguments + - Use `env var` to autocomplete env var +- **Dark mode** — Use dark mode in the dbt Cloud IDE for low-light environments. + +Read more about all the [Cloud IDE features](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#cloud-ide-features). + +### Classic IDE deprecation notice + +In December 2022, dbt Labs will deprecate the classic IDE. The [new and refreshed IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) will be available for _all_ dbt Cloud users. You will no longer be able to access the classic IDE and dbt Labs might introduce changes that break the classic IDE. + +With deprecation, dbt Labs will only support the refreshed version of the dbt Cloud IDE. + +Virtual Private Cloud (VPC) customers with questions about when this change will affect your account can contact your account team or support contact for assistance. diff --git a/website/docs/docs/dbt-versions/release-notes/25-oct-2022/cloud-integration-azure.md b/website/docs/docs/dbt-versions/release-notes/25-oct-2022/cloud-integration-azure.md new file mode 100644 index 00000000000..513c2ef1258 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/25-oct-2022/cloud-integration-azure.md @@ -0,0 +1,15 @@ +--- +title: "Announcing dbt Cloud’s native integration with Azure DevOps" +id: "cloud-integration-azure" +description: "dbt Cloud native integration with Azure DevOps" +sidebar_label: "Improvement: Native integration with Azure DevOps" +tags: [Oct-2022, v1.1.64] +--- + +dbt Cloud now offers a native integration with Azure DevOps for dbt Cloud customers on the enterprise plan. We built this integration to remove friction, increase security, and unlock net new product experiences for our customers. [Setting up the Azure DevOps integration](/docs/cloud/git/connect-azure-devops) in dbt Cloud provides: + - easy dbt project set up, + - an improved security posture, + - repo permissions enforcement in dbt Cloud IDE, and + - dbt Cloud Slim CI. + +Check out our [live blog post](https://www.getdbt.com/blog/dbt-cloud-integration-azure-devops/) to learn more! diff --git a/website/docs/docs/dbt-versions/release-notes/25-oct-2022/new-ide-launch.md b/website/docs/docs/dbt-versions/release-notes/25-oct-2022/new-ide-launch.md new file mode 100644 index 00000000000..aaf5f670e09 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/25-oct-2022/new-ide-launch.md @@ -0,0 +1,21 @@ +--- +title: "Enhancement: New Cloud IDE launch" +id: "new-ide-launch" +description: "Oct 2022 release note: Introducing a new dbt Cloud IDE that's faster, has performance upgrades, ergonomics improvements, and other delightful enhancements." +sidebar_label: "Snappier, faster, and new Cloud IDE" +tags: [Oct-2022, IDE] +--- + +## Introducing a snappier, improved, and powerful Cloud IDE + +The new version of the Cloud IDE makes it easy for you to build data models without thinking much about environment setup and configuration. + +The new Cloud IDE includes performance upgrades, ergonomics improvements, and some delightful enhancements! + +Some of the improvements include: + +- Improved Cloud IDE startup time (starting the IDE), interaction time (saving and committing), and reliability. +- Better organization and navigation with features like drag and drop of files, breadcrumb, build button drop-down, and more. +- You can use new features like auto-format your file, auto-complete model names, and git diff view to see your changes before making a pull request. + +Read more about the new [Cloud IDE features](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#cloud-ide-features) and check out [New and improved Cloud IDE](https://www.getdbt.com/blog/new-improved-cloud-ide/) blog for more info! diff --git a/website/docs/docs/dbt-versions/release-notes/26-sept-2022/liststeps-endpoint-deprecation.md b/website/docs/docs/dbt-versions/release-notes/26-sept-2022/liststeps-endpoint-deprecation.md new file mode 100644 index 00000000000..545847efd90 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/26-sept-2022/liststeps-endpoint-deprecation.md @@ -0,0 +1,15 @@ +--- +title: "List Steps API endpoint deprecation warning" +id: "liststeps-endpoint-deprecation.md" +description: "List Steps API deprecation" +sidebar_label: "Deprecation: List Steps API endpoint" +tags: [Sept-2022] +--- + +On October 14th, 2022 dbt Labs is deprecating the [List Steps](https://docs.getdbt.com/dbt-cloud/api-v2-legacy#tag/Runs/operation/listSteps) API endpoint. From October 14th, any GET requests to this endpoint will fail. Please prepare to stop using the List Steps endpoint as soon as possible. + +dbt Labs will continue to maintain the [Get Run](https://docs.getdbt.com/dbt-cloud/api-v2-legacy#tag/Runs/operation/getRunById) endpoint, which is a viable alternative depending on the use case. + +You can fetch run steps for an individual run with a GET request to the following URL, replacing `YOUR_ACCESS_URL` with the [appropriate Access URL](/docs/cloud/about-cloud/regions-ip-addresses) for your region and plan: + +`https://YOUR_ACCESS_URL/api/v2/accounts/{accountId}/runs/{runId}/?include_related=["run_steps"]` diff --git a/website/docs/docs/dbt-versions/release-notes/26-sept-2022/metadata-api-data-retention-limits.md b/website/docs/docs/dbt-versions/release-notes/26-sept-2022/metadata-api-data-retention-limits.md new file mode 100644 index 00000000000..3ced6fecedd --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/26-sept-2022/metadata-api-data-retention-limits.md @@ -0,0 +1,11 @@ +--- +title: "Query the previous three months of data using the metadata API" +id: "metadata-api-data-retention-limits.md" +description: "Metadata API data retention limits" +sidebar_label: "Fix: Metadata API data retention limits" +tags: [Sept-2022] +--- + +In order to make the metadata API more scalable and improve its latency, we’ve implemented data retention limits. The metadata API can now query data from the previous three months. For example, if today was March 1, you could query data back to January 1st. + +For more information, see [Metadata API](/docs/dbt-cloud-apis/discovery-api) diff --git a/website/docs/docs/dbt-versions/release-notes/27-aug-2022/ide-improvement-beta.md b/website/docs/docs/dbt-versions/release-notes/27-aug-2022/ide-improvement-beta.md new file mode 100644 index 00000000000..aa2c5ee4fa7 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/27-aug-2022/ide-improvement-beta.md @@ -0,0 +1,36 @@ +--- +title: "Enhancement: New Cloud IDE beta" +id: "ide-improvements-beta.md" +description: "Aug 2022 release note: Launch of the IDE beta, which focuses on performance and reliability improvements." +sidebar_label: "Enhancement: New cloud IDE beta" +tags: [Aug-2022, IDE] +--- + +:::info Beta feature +Read more about the [Cloud IDE beta](https://www.getdbt.com/blog/staging-highlights-the-latest-from-dbt-labs/) and [submit your expression of interest](https://docs.google.com/forms/d/e/1FAIpQLSdlU65gqTZPyGAUc16SkxqTc50NO9vdq_KGx1Mjm_4FB_97FA/viewform) to join the new Cloud IDE beta group. + +::: + +Building code on the dbt Cloud integrated development environment (IDE) should be seamless, and the tool that you’re using should not add more distractions to the data work that is often already confusing and difficult. + +We're now excited to start rolling out the IDE Beta version, which focuses on performance and reliability improvements. + +The classic IDE currently has severe performance and reliability issues. It takes a long time to start up the IDE, and the interactions (saving or committing) are slow. + +Our focus is on performance and reliability, particularly around the following four metrics: + +- Spinner time for cold start = Time that you see a spinner in a brand new session. +- Spinner time for hot start = Time that you see a spinner when you resume an existing session (return within 3 hours). +- Time to save = Time between saving a file and the IDE being ready for edit. +- Time to git = Time between making a commit and the IDE being ready for edit. + +**Improvements:** + +To address the issue, we rebuilt the IDE and made some significant architectural changes to the way we work. These changes will help improve the IDE performance and reliability: + +- Your IDE spinner and interaction time will be faster, regardless of the size of your dbt project. + - Instead of fetching and downloading all the contents for the files during any change, the IDE only needs the file tree/name. This means that starting up the IDE should no longer depend on the size of the dbt project. This also helps make the interaction with the IDE (saving files and committing changes) more snappy. + +- Your IDE spinner time will be quicker as you can access it without needing to wait for the rpc server to finish getting ready. + + diff --git a/website/docs/docs/dbt-versions/release-notes/27-aug-2022/support-redshift-ra3.md b/website/docs/docs/dbt-versions/release-notes/27-aug-2022/support-redshift-ra3.md new file mode 100644 index 00000000000..f48a8539923 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/27-aug-2022/support-redshift-ra3.md @@ -0,0 +1,18 @@ +--- +title: "Enhancement: Support for cross-database sources on Redshift RA3 instances" +id: "support-redshift-ra3.md" +description: "Adding support for cross-database queries for RA3" +sidebar_label: "Enhancement: Support for RA3" +tags: [Aug-2022, 1.1.61.5] + +--- + +Cross-database queries for RA3 instances are now supported by dbt Cloud projects using a Redshift connection. + +With cross-database queries, you can seamlessly query data from any database in the cluster, regardless of which database you are connected to with dbt. + +The [connection configuration](https://docs.getdbt.com/reference/warehouse-profiles/redshift-profile) `ra3_node` has been defaulted to `true`. This allows users to: + +- benefit from the full RA3 nodes’ capabilities, +- generate appropriate dbt documentation. + diff --git a/website/docs/docs/dbt-versions/release-notes/28-july-2022/render-lineage-feature.md b/website/docs/docs/dbt-versions/release-notes/28-july-2022/render-lineage-feature.md new file mode 100644 index 00000000000..2d0488d4488 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/28-july-2022/render-lineage-feature.md @@ -0,0 +1,16 @@ +--- +title: "Enhancement: Large DAG feature" +id: "render-lineage-feature" +description: "Jul 2022 release note: Use the Render Lineage button to visualize large DAGs" +sidebar_label: "Enhancement: Large DAG feature" +tags: [July-2022, v1.1.56, IDE] + +--- + +You can now select **Render Lineage** to visualize large DAGs. + +Large DAGs can take a long time (10 or more seconds, if not minutes) to render and can cause browsers to crash. + +The new button prevents large DAGs from rendering automatically. Instead, you can select **Render Lineage** to load the visualization. This should affect about 15% of the DAGs. + + diff --git a/website/docs/docs/dbt-versions/release-notes/29-may-2022/gitlab-auth.md b/website/docs/docs/dbt-versions/release-notes/29-may-2022/gitlab-auth.md new file mode 100644 index 00000000000..4e7c9ec3ea3 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/29-may-2022/gitlab-auth.md @@ -0,0 +1,11 @@ +--- +title: "Refresh expired access tokens in the IDE when using GitLab" +id: "gitlab-auth" +description: "Adding support for expiring OAuth access tokens." +sidebar_label: "Enhancement: Refreshing GitLab OAuth Access" +tags: [May-2022, v1.1.52] +--- + +On May 22, GitLab changed how they treat [OAuth access tokens that don't expire](https://docs.gitlab.com/ee/update/deprecations.html#oauth-tokens-without-expiration). We updated our IDE logic to handle OAuth token expiration more gracefully. Now, the first time your token expires after 2 hours of consecutive IDE usage, you will have to re-authenticate in GitLab to refresh your expired OAuth access token. We will handle subsequent refreshes for you if you provide the authorization when you re-authenticate. + +This additional security layer in the IDE is available only to the dbt Cloud enterprise plan. diff --git a/website/docs/docs/dbt-versions/release-notes/30-april-2022/audit-log.md b/website/docs/docs/dbt-versions/release-notes/30-april-2022/audit-log.md new file mode 100644 index 00000000000..d0cc9208e1e --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/30-april-2022/audit-log.md @@ -0,0 +1,13 @@ +--- +title: "Audit log" +id: "audit-log" +description: "Review actions performed by people in your organization." +sidebar_label: "New: Audit log" + +tags: [April-26-2022] +--- + + +To review actions performed by people in your organization, dbt provides logs of audited user and system events. The dbt Cloud audit log lists events triggered in your organization within the last 90 days. + +The audit log includes details such as who performed the action, what the action was, and when it was performed. For more details, review [the audit log for dbt Cloud Enterprise](/docs/cloud/manage-access/audit-log) documentation. diff --git a/website/docs/docs/dbt-versions/release-notes/30-april-2022/credentials-saved.md b/website/docs/docs/dbt-versions/release-notes/30-april-2022/credentials-saved.md new file mode 100644 index 00000000000..388f133e861 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/30-april-2022/credentials-saved.md @@ -0,0 +1,9 @@ +--- +title: "Credentials no longer accidentally wiped when editing an environment" +id: "credentials-saved" +description: "Credentials are now saved when editing an environment." +sidebar_label: "Fix: Credentials saved" +tags: [April-29-2022, v1.1.51] +--- + +We resolved a bug where when updating unencrypted fields (e.g. threads, schema name) in an environment setting would cause secret fields (e.g. password, keypair, credential details) to be deleted from that environment. Now users can freely update environment settings without fear of unintentionally wiping credentials. diff --git a/website/docs/docs/dbt-versions/release-notes/30-april-2022/email-verification.md b/website/docs/docs/dbt-versions/release-notes/30-april-2022/email-verification.md new file mode 100644 index 00000000000..4beff1b5506 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/30-april-2022/email-verification.md @@ -0,0 +1,10 @@ +--- +title: "Email verification" +description: "dbt Cloud users created via SAML JIT will now confirm identity via email to activate their account." +sidebar_label: "New: Email verification" +sidebar_position: 1 + +tags: [April-26-2022, v1.1.48] +--- + +To enhance the security of user creation, dbt Cloud users created using SAML Just-in-Time (JIT) will now confirm identity via email to activate their account. Using email to confirm identity ensures the user still has access to the same email address they use to login via SAML. diff --git a/website/docs/docs/dbt-versions/release-notes/30-april-2022/scheduler-improvements.md b/website/docs/docs/dbt-versions/release-notes/30-april-2022/scheduler-improvements.md new file mode 100644 index 00000000000..02d3f1bf401 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/30-april-2022/scheduler-improvements.md @@ -0,0 +1,9 @@ +--- +title: "Scheduler performance improvements" +id: "scheduler-improvements" +description: "scheduler, which has much faster prep times, especially at the top of the hour." +sidebar_label: "Improvement: Scheduler performance" +tags: [April-19-2022] +--- + +We rolled out our new distributed scheduler, which has much faster prep times, especially at the top of the hour. We share more about our work and improvements in our [product news blog post](https://www.getdbt.com/blog/a-good-problem-to-have/). diff --git a/website/docs/docs/dbt-versions/release-notes/31-march-2022/ide-timeout-message.md b/website/docs/docs/dbt-versions/release-notes/31-march-2022/ide-timeout-message.md new file mode 100644 index 00000000000..307786c6b85 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/31-march-2022/ide-timeout-message.md @@ -0,0 +1,13 @@ +--- +title: "Spotty internet issues no longer cause a session time out message" +id: "ide-timeout-message" +description: "Mar 2022 release note: We fixed an issue where a spotty internet connection could cause the “IDE session timed out” message to appear unexpectedly. People using a VPN were most likely to see this issue." +sidebar_label: "Fix: Session time out" +tags: [v1.1.47, March-10-2022, IDE] +--- + +We fixed an issue where a spotty internet connection could cause the “IDE session timed out” message to appear unexpectedly. People using a VPN were most likely to see this issue. + +We updated the health check logic so it now excludes client-side connectivity issues from the IDE session check. If you lose your internet connection, we no longer update the health-check state. Now, losing internet connectivity will no longer cause this unexpected message. + + diff --git a/website/docs/docs/dbt-versions/release-notes/31-march-2022/prep-and-waiting-time.md b/website/docs/docs/dbt-versions/release-notes/31-march-2022/prep-and-waiting-time.md new file mode 100644 index 00000000000..9ff5986b4da --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/31-march-2022/prep-and-waiting-time.md @@ -0,0 +1,12 @@ +--- +title: "Dividing queue time into waiting and prep time" +id: "prep-and-waiting-time" +description: "dbt Cloud now shows waiting time and prep time for a run." +sidebar_label: "Enhancement: Waiting time and prep time" + +tags: [v1.1.46, March-02-2022] +--- + +dbt Cloud now shows "waiting time" and "prep time" for a run, which used to be expressed in aggregate as "queue time". Waiting time captures the time dbt Cloud waits to run your job if there isn't an available run slot or if a previous run of the same job is still running. Prep time represents the time it takes dbt Cloud to ready your job to run in your cloud data warehouse. + + diff --git a/website/docs/docs/dbt-versions/release-notes/32-february-2022/dag-updates-more.md b/website/docs/docs/dbt-versions/release-notes/32-february-2022/dag-updates-more.md new file mode 100644 index 00000000000..6557d080394 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/32-february-2022/dag-updates-more.md @@ -0,0 +1,13 @@ +--- +title: "DAG updates and performance improvements" +description: "Feb 2022 release note: Clicking a node in the DAG opens a model or config file in new tab in the IDE." +id: "DAG-updates-more" +sidebar_label: "DAG updates and more" +tags: [v1.1.44, February-02-2022, IDE] +--- + +Love the DAG in the IDE as much as we do? Now when you click on a node in the DAG, the model or config file will open as a new tab in the IDE, so you can directly view or edit the code. We'll continue to ship better developer ergonomic functionality throughout the year. + +#### Performance improvements and enhancements + +* Updated recommended dbt commands in the IDE to include dbt Core v1.0 commands, such as "build" and the "--select" argument. \ No newline at end of file diff --git a/website/docs/docs/dbt-versions/release-notes/32-february-2022/service-tokens-more.md b/website/docs/docs/dbt-versions/release-notes/32-february-2022/service-tokens-more.md new file mode 100644 index 00000000000..5ad241d1365 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/32-february-2022/service-tokens-more.md @@ -0,0 +1,17 @@ +--- +title: "Service tokens and bug fixes" +is: "service-tokens-more" +description: "Service tokens can now be assigned granular permissions to enforce least privilege access." +sidebar_label: "Service tokens and more" +tags: [v1.1.45, February-16-2022] +--- + +Service tokens can now be assigned granular permissions to enforce least privilege access. If you're on Enterprise, you can assign any enterprise permission set to newly issued service tokens. If you're on Teams, you can assign the Job Admin permission set to newly issued service tokens. We highly recommend you re-issue service tokens with these new permissions to increase your security posture! See docs [here](https://docs.getdbt.com/docs/dbt-cloud-apis/service-tokens#permissions-for-service-account-tokens). + +#### New products and features + +- We are joining the [GitHub secret scanning partner program](https://docs.github.com/en/developers/overview/secret-scanning-partner-program) to better secure your token against accidental public exposure and potential fraudulent usage. + +#### Bug fixes + +- Credentials are no longer accidentally deleted when a user updates an environment setting. diff --git a/website/docs/docs/dbt-versions/release-notes/33-january-2022/ide-autocomplete-more.md b/website/docs/docs/dbt-versions/release-notes/33-january-2022/ide-autocomplete-more.md new file mode 100644 index 00000000000..c29ed790e8e --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/33-january-2022/ide-autocomplete-more.md @@ -0,0 +1,14 @@ +--- +title: "Autocomplete snippets for SQL and YAML files in IDE" +id: "ide-timeout-message" +description: "Jan 2022 release note: Autocomplete snippets for SQL and YAML files in IDE" +sidebar_label: "Autocomplete in IDE and more" +tags: [v1.1.43, January-19-2022, IDE] +--- + +Some noteworthy improvements include autocomplete snippets for sql and YAML files in the IDE, which are available for use now! We also added a [new metric layer page](https://docs.getdbt.com/docs/dbt-cloud/using-dbt-cloud/cloud-metrics-layer) to docs.getdbt.com to help you begin thinking about the metrics layer in dbt Cloud. + +#### Performance improvements and enhancements + +* Branch names now default to "main" instead of "master" in new managed and unmanaged Git repositories. +* Update IDE autocomplete snippets. diff --git a/website/docs/docs/dbt-versions/release-notes/33-january-2022/model-timing-more.md b/website/docs/docs/dbt-versions/release-notes/33-january-2022/model-timing-more.md new file mode 100644 index 00000000000..6f49fe5d336 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/33-january-2022/model-timing-more.md @@ -0,0 +1,15 @@ +--- +title: "Model timing for Multi-tenant Team and Enterprise accounts" +id: "model-timing-more" +description: "Model timing is now available for Multi-tenant Team and Enterprise accounts, plus performance improvements." +sidebar_label: "Model timing and more" + +tags: [v1.1.42, January-05-2022] +--- +#### New products and features + +We started the new year with a gift! Multi-tenant Team and Enterprise accounts can now use the new [Model timing](https://docs.getdbt.com/docs/dbt-cloud/using-dbt-cloud/cloud-model-timing-tab) tab in dbt Cloud. You can use this tab to further explore long-running models to see if they need refactoring or rescheduling. + +#### Performance improvements and enhancements + +* We added client-side naming validation for file or folder creation.