diff --git a/docs/release-info/31.0_copy_results_as_sql.png b/docs/release-info/31.0_copy_results_as_sql.png new file mode 100644 index 000000000000..616578eacb6b Binary files /dev/null and b/docs/release-info/31.0_copy_results_as_sql.png differ diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index 82e371e8406f..42b5ad3adcb1 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -24,16 +24,16 @@ title: "Release notes" -Apache Druid {{DRUIDVERSION}} contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors. +Apache Druid 31.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors. -See the [complete set of changes](https://github.com/apache/druid/issues?q=is%3Aclosed+milestone%3A{{MILESTONE}}+sort%3Aupdated-desc+) for additional details, including bug fixes. +See the [complete set of changes](https://github.com/apache/druid/issues?q=is%3Aclosed+milestone%3A31.0.0+sort%3Aupdated-desc+) for additional details, including bug fixes. -Review the [upgrade notes](#upgrade-notes) and [incompatible changes](#incompatible-changes) before you upgrade to Druid {{DRUIDVERSION}}. -If you are upgrading across multiple versions, see the [Upgrade notes](upgrade-notes.md) page, which lists upgrade notes for the most recent Druid versions. +Review the [upgrade notes](#upgrade-notes) and [incompatible changes](#incompatible-changes) before you upgrade to Druid 31.0.0. +If you are upgrading across multiple versions, see the [Upgrade notes](https://druid.apache.org/docs/latest/release-info/upgrade-notes) page, which lists upgrade notes for the most recent Druid versions. @@ -105,15 +647,61 @@ In Druid 32.0.0, the front coded dictionaries feature will be turned on by defau Once this feature is on, you cannot easily downgrade to an earlier version that does not support the feature. -For more information, see [Migration guide: front-coded dictionaries](./migr-front-coded-dict.md). +For more information, see [Migration guide: front-coded dictionaries](https://druid.apache.org/docs/latest/release-info/migr-front-coded-dict). If you're already using this feature, you don't need to take any action. +#### Storage improvement features + +Druid 31.0.0 includes storage improvements that are not backwards compatible. Once you enable [flexible segment sorting](#flexible-segment-sorting) or [compression for complex columns](#compression-for-complex-columns), you cannot downgrade to a release earlier than 31.0.0. Thoroughly test these features before rolling them out to your production cluster. ### Incompatible changes +#### Removed the scan query legacy mode + +The native scan query legacy mode has been removed. It was introduced in Druid 0.11 to maintain compatibility during an upgrade from older versions of Druid where the scan query was part of a `contrib` extension. + +[#16659](https://github.com/apache/druid/pull/16659) + +Hard-coded `"legacy":false` following removal of the legacy mode to prevent error during rolling upgrades or downgrades. + +[#16793](https://github.com/apache/druid/pull/16793) + +#### ZK-based segment loading + +ZK-based segment loading is now disabled. ZK `servedSegmentsPath` was deprecated in Druid 0.7.1. This legacy path has been replaced by `liveSegmentsPath`. + +Segment-serving processes such as Peons, Historicals and Indexers no longer create ZK `loadQueuePath` entries. The `druid.zk.paths.loadQueuePath` and `druid.zk.paths.servedSegmentsPath` properties are no longer used. + +**Move to HTTP-based segment loading first and then perform the version upgrade.** + ### Developer notes +- Fixed the `hibernate-validator` warning displayed during maven build [#16746](https://github.com/apache/druid/pull/16746) +- Replaced the `StorageAdapter` interface with the `CursorHolder` interface [#17024](https://github.com/apache/druid/pull/17024) + +### `StorageAdapter` changes + +If you write custom extensions, specifically query engines or anything else involving the `StorageAdapter` interface, 31.0.0 includes changes to low-level APIs that may impact you. All methods on `StorageAdapter` now throw a Druid exception. + +Prepare for these changes before upgrading to 31.0.0 or later. For more information, see the following pull requests: + +- [#16985](https://github.com/apache/druid/pull/16985) +- [#16849](https://github.com/apache/druid/pull/16849) +- [#16533](https://github.com/apache/druid/pull/16533) + #### Dependency updates -The following dependencies have had their versions bumped: \ No newline at end of file +Bumped the versions of the following dependencies: + +- Apache Avro to 1.11.4 [#17230](https://github.com/apache/druid/pull/17230) +- Apache Calcite to 1.37.0 [#16621](https://github.com/apache/druid/pull/16621) +- Delta Kernel from 3.1.0 to 3.2.1 [#16513](https://github.com/apache/druid/pull/16513) [#17179](https://github.com/apache/druid/pull/17179) +- MySQL Java connector to 8.2.0 [#16024](https://github.com/apache/druid/pull/16024) +- OpenRewrite maven plugin from 5.27.0 to 5.31.0 [#16477](https://github.com/apache/druid/pull/16477) +- Scala library from 2.13.11 to 2.13.14 [#16364](https://github.com/apache/druid/pull/16364) +- Apache Curator reverted to 5.3.0 from 5.5.0 [#16688](https://github.com/apache/druid/pull/16688) +- Updated Blueprint dependencies from v4 to v5 in web console [#16756](https://github.com/apache/druid/pull/16756) +- `io.grpc:grpc-netty-shaded` from 1.57.2 to 1.65.1 [#16731](https://github.com/apache/druid/pull/16731) +- `jclouds.version` from 2.5.0 to 2.6.0 [#16796](https://github.com/apache/druid/pull/16796) +- Axios to 1.7.4 [#16898](https://github.com/apache/druid/pull/16898) \ No newline at end of file diff --git a/docs/release-info/upgrade-notes.md b/docs/release-info/upgrade-notes.md index d3c6ce1cba3c..3a627ee5ce2b 100644 --- a/docs/release-info/upgrade-notes.md +++ b/docs/release-info/upgrade-notes.md @@ -38,6 +38,66 @@ For more information, see [Migration guide: front-coded dictionaries](./migr-fro If you're already using this feature, you don't need to take any action. +## 31.0.0 + +### Upgrade notes + +#### Array ingest mode now defaults to array + +The SQL-based ingestion query context flag `arrayIngestMode` now defaults to `array` instead of `mvd`. This means that SQL `VARCHAR ARRAY` types is no longer implicitly translated and stored in `VARCHAR` columns, but is instead stored as `VARCHAR ARRAY`. This change permits other array types such as `BIGINT ARRAY` and `DOUBLE ARRAY` to be inserted with MSQ task engine into their respective array column types instead of failing as they do in `mvd` mode. + +To continue to store multi-value strings, modify any insert/replace queries to wrap the array types with the `ARRAY_TO_MV` operator. + +Validation is in place to prevent mixing `VARCHAR` and `VARCHAR ARRAY` columns in the same table, so any ingestions affected by this change will fail and provide a descriptive error message instead of exhibiting unexpected behavior. + +The `arrayIngestMode` option of `none` has been removed. It was introduced prior to the table validation logic as a means for cluster operators to force query writers to explicitly set `array` or `mvd` on their query contexts, but provides little utility in Druid 31. + +See the following topics for more information: +* [Ingest multi-value dimensions](https://druid.apache.org/docs/latest/querying/multi-value-dimensions.md#sql-based-ingestion) for how to ingest multi-value strings. +* [Ingest arrays](https://druid.apache.org/docs/latest/querying/arrays.md#sql-based-ingestion) for ingesting arrays. + +[#16789](https://github.com/apache/druid/pull/16789) + +#### Removed task action audit logging + +The deprecated task action audit logging has been removed. This change includes the following updates: + +- The endpoint `/indexer/v1/task/{taskId}/segments` is no longer supported. +- Druid doesn't write to or read from the metadata table `druid_taskLog`. +- Druid ignores the property `druid.indexer.auditlog.enabled`. +- Druid doesn't emit the metric `task/action/log/time`. + +These changes are backward compatible with all existing metadata storage extensions. + +[#16309](https://github.com/apache/druid/pull/16309) + +#### Removed Firehose and FirehoseFactory + +Removed Firehose and FirehoseFactory and remaining implementations. +Apache deprecated support for Druid firehoses in version 0.17. Support for firehose ingestion was removed in version 26.0. + +[#16758](https://github.com/apache/druid/pull/16758) + +### Incompatible changes + +#### Removed the scan query legacy mode + +The native scan query legacy mode has been removed. It was introduced in Druid 0.11 to maintain compatibility during an upgrade from older versions of Druid where the scan query was part of a `contrib` extension. + +[#16659](https://github.com/apache/druid/pull/16659) + +Hard-coded `"legacy":false` following removal of the legacy mode to prevent error during rolling upgrades or downgrades. + +[#16793](https://github.com/apache/druid/pull/16793) + +#### ZK-based segment loading + +ZK-based segment loading is now disabled. ZK `servedSegmentsPath` was deprecated in Druid 0.7.1. This legacy path has been replaced by `liveSegmentsPath`. + +Segment-serving processes such as Peons, Historicals and Indexers no longer create ZK `loadQueuePath` entries. The `druid.zk.paths.loadQueuePath` and `druid.zk.paths.servedSegmentsPath` properties are no longer used. + +Move to HTTP-based segment loading first and then perform the version upgrade. + ## 30.0.0 ### Upgrade notes diff --git a/docs/release-info/web-console-31-rn-copy-results-as-sql.png b/docs/release-info/web-console-31-rn-copy-results-as-sql.png new file mode 100644 index 000000000000..616578eacb6b Binary files /dev/null and b/docs/release-info/web-console-31-rn-copy-results-as-sql.png differ diff --git a/docs/release-info/web-console-31-rn-dart.png b/docs/release-info/web-console-31-rn-dart.png new file mode 100644 index 000000000000..6307e26b4d3c Binary files /dev/null and b/docs/release-info/web-console-31-rn-dart.png differ diff --git a/docs/release-info/web-console-31-rn-new_stages.png b/docs/release-info/web-console-31-rn-new_stages.png new file mode 100644 index 000000000000..f0dc5d849131 Binary files /dev/null and b/docs/release-info/web-console-31-rn-new_stages.png differ diff --git a/website/.spelling b/website/.spelling index 6e204b8be8db..aa6322d08e9b 100644 --- a/website/.spelling +++ b/website/.spelling @@ -233,6 +233,7 @@ Splunk SqlParameter SslContextFactory StatsD +stateful SYSTEM_TABLE TaskRunner TabItem @@ -254,6 +255,7 @@ UTF8 XMLs ZK ZSTD +Axios accessor ad-hoc aggregator @@ -323,6 +325,7 @@ deserialization deserialize deserialized deserializes +deserializing downtimes druid druid–kubernetes-extensions @@ -345,6 +348,7 @@ filterValue firefox firehose firehoses +FirehoseFactory fromPigAvroStorage frontends granularities @@ -536,8 +540,10 @@ stdout storages stringDictionaryEncoding stringified +structs sub-conditions subarray +submenu subnet subqueries subquery @@ -637,6 +643,7 @@ IS_NULLABLE JDBC_TYPE MIDDLE_MANAGER MILLIS_TO_TIMESTAMP +NPE NULLable NUMERIC_PRECISION NUMERIC_PRECISION_RADIX @@ -712,6 +719,7 @@ total_size useApproximateCountDistinct useGroupingSetForExactDistinct useApproximateTopN +UX wikipedia your-table enableTimeBoundaryPlanning @@ -1091,6 +1099,7 @@ reverseLoadingCacheSpec OAuth Okta OpenID +OpenRewrite pac4j Env POD_NAME @@ -1223,6 +1232,8 @@ chunk2 stddev t1 t2 +v4 +v5 variance1 variance2 varianceFold @@ -1632,6 +1643,7 @@ un 5MB 64KiB 8GiB +GBs G1GC GroupBys QoS-type @@ -2204,6 +2216,7 @@ PiB protobuf Golang multiValueHandling +multivalue _n_ KLL KllFloatsSketch