[META] Set OpenSearch 3.0.0 baseline JDK version to JDK-21

[reta]

Please describe the end goal of this project

Currently, the 3.0.0 baseline JDK version is set to JDK-11. The Apache Lucene 10 is going to require JDK-21 and it would make sense to align the OpenSearch JDK baseline requirements with that.

In order to make the migration as painless as possible, the suggestion is to start from dependent components (plugins, clients, extensions, libraries) and finish up with core, for the following reasons:

• if we change core to JDK-21 baseline (bytecode), it will break all and every dependent component
• however, because we always build core with JDK-21 (and bundle it in as well with JDK-21), there won't be any issues dealing the dependent components that are on JDK-21 baseline

Supporting References

Please see #10745

Issues

• [FEATURE] Set OpenSearch Core 3.0.0 baseline JDK version to JDK-21 #14010
• [FEATURE] Set common-utils 3.0.0 baseline JDK version to JDK-21 common-utils#668
• [FEATURE] Set security plugin 3.0.0 baseline JDK version to JDK-21 security#4407
• [FEATURE] Set job-scheduler plugin 3.0.0 baseline JDK version to JDK-21 job-scheduler#636
• [FEATURE] Set reporting plugin 3.0.0 baseline JDK version to JDK-21 reporting#1002
• [FEATURE] Set observability plugin 3.0.0 baseline JDK version to JDK-21 observability#1830
• [FEATURE] Set sql plugin 3.0.0 baseline JDK version to JDK-21 sql#2713
• [FEATURE] Set security-analytics plugin 3.0.0 baseline JDK version to JDK-21 security-analytics#1056
• [FEATURE] Set data-prepper plugin 3.0.0 baseline JDK version to JDK-21 data-prepper#4605
• [FEATURE] Set anomaly-detection plugin 3.0.0 baseline JDK version to JDK-21 anomaly-detection#1223
• [FEATURE] Set k-NN plugin 3.0.0 baseline JDK version to JDK-21 k-NN#1729
• [FEATURE] Set opensearch-plugin-template-java plugin 3.0.0 baseline JDK version to JDK-21 opensearch-plugin-template-java#70
• [FEATURE] Set alerting plugin 3.0.0 baseline JDK version to JDK-21 alerting#1565
• [FEATURE] Set performance-analyzer-commons 2.0.0 baseline JDK version to JDK-21 performance-analyzer-commons#75
• [FEATURE] Set ml-commons plugin 3.0.0 baseline JDK version to JDK-21 ml-commons#2503
• [FEATURE] Set cross-cluster-replication plugin 3.0.0 baseline JDK version to JDK-21 cross-cluster-replication#1393
• [FEATURE] Set neural-search plugin 3.0.0 baseline JDK version to JDK-21 neural-search#775
• [FEATURE] Set flow-framework plugin 3.0.0 baseline JDK version to JDK-21 flow-framework#729
• [FEATURE] Set opensearch-sdk-java baseline JDK version to JDK-21 opensearch-sdk-java#998
• [FEATURE] Set performance-analyzer-rca 2.0.0 baseline JDK version to JDK-21 performance-analyzer-rca#555
• [FEATURE] Set opensearch-hadoop 2.0.0 baseline JDK version to JDK-21 opensearch-hadoop#472
• [FEATURE] Set search-processor plugin 3.0.0 baseline JDK version to JDK-21 search-processor#225
• [FEATURE] Set custom-codecs plugin 3.0.0 baseline JDK version to JDK-21 custom-codecs#153
• [FEATURE] Set asynchronous-search plugin 3.0.0 baseline JDK version to JDK-21 asynchronous-search#581
• [FEATURE] Set opensearch-learning-to-rank-base plugin 3.0.0 baseline JDK version to JDK-21 opensearch-learning-to-rank-base#36
• [FEATURE] Set geospatial plugin 3.0.0 baseline JDK version to JDK-21 geospatial#659
• [FEATURE] Set opensearch-oci-object-storage plugin 3.0.0 baseline JDK version to JDK-21 opensearch-oci-object-storage#60
• [FEATURE] Set notifications plugin 3.0.0 baseline JDK version to JDK-21 notifications#924
• [FEATURE] Set index-management plugin 3.0.0 baseline JDK version to JDK-21 index-management#1181
• [FEATURE] Set performance-analyzer plugin 3.0.0 baseline JDK version to JDK-21 performance-analyzer#659
• [FEATURE] Set query-insights plugin 3.0.0 baseline JDK version to JDK-21 query-insights#134

Related component

Build

[reta]

@msfroh @andrross @dblock @getsaurabh02 fyi folks :) kicking off the campaign ...

[dbwiddis]

@reta Quick question on this.

In response to this campaign, I updated our compatibility from JDK 11 to JDK 21 like this:

java {
    targetCompatibility = JavaVersion.VERSION_21
    sourceCompatibility = JavaVersion.VERSION_21
}

Our normal development flow is to merge to main and backport to 2.x, which still supports earlier JDK's. I can see the potential for someone introducing code from a JDK between 12 and 20 (logically, 17) such as text blocks, switch expressions, records, and my new favorite stream operator, toList(). They'd pass muster on all our main branch CI but fail the backport and require substituting alternate code. Currently we limit this just to a very few differences to keep backporting simple.

I'm curious what the best approach here is. Can we keep targetCompatibility as 21 to address this campaign while still requiring sourceCompatibility of 11 to make sure backporting goes smoothly and doesn't diverge too much?

Or if we're keeping both, it seems this style may be more broadly applicable and recommended here. Thoughts?

tasks.withType(JavaCompile) {
    options.release.set(21)
}

[andrross]

@dbwiddis See some more discussion here: opensearch-project/custom-codecs#154 (comment)

The risk is that some new language features will be exposed by a library using JDK21 as a baseline (i.e. Lucene) and that won't work smoothly if your source compatibility is lower than that. I'm not totally sure what those risks look like in practice though.

[dbwiddis]

The risk is that some new language features will be exposed by a library using JDK21 as a baseline (i.e. Lucene) and that won't work smoothly if your source compatibility is lower than that. I'm not totally sure what those risks look like in practice though.

Agreed that it's not clear what the risks are. I think there are other options (e.g., ForbiddenAPIs) for enforcing lower-JDK source standards, so I'll look into those to address the risk of "dang, my backport failed". And evaluate whether it's more effort to prevent it than to just fix it when it breaks...

[reta]

I'm curious what the best approach here is. Can we keep targetCompatibility as 21 to address this campaign while still requiring sourceCompatibility of 11 to make sure backporting goes smoothly and doesn't diverge too much?

Thank you @dbwiddis , additionally to what @andrross said, I think the benefits of using new language features (as well as standard library) outweigh the inconvenience of manual backports (otherwise the legacy will always hold us back). Surely, there is also an option to use baseline constructs (JDK-11) without lowering the source compatibility levels.

And evaluate whether it's more effort to prevent it than to just fix it when it breaks...

I think this is sane approach, additionally we could also use something like https://github.com/openrewrite/rewrite (or similar) to "downgrade" source code during automatic backports.