Skip to content

Commit

Permalink
Update to Apache Lucene 10 for 3.0.0 (#16366)
Browse files Browse the repository at this point in the history
* Update to Apache Lucene 10 for 3.0.0

Signed-off-by: Andriy Redko <[email protected]>

* More Lucene 10 build fixes

Core compiles now. (Plugins/modules don't yet.)

Major areas covered:

1. MultiTermQuery now takes RewriteMethod via constructor instead of
   setter.
2. Scorable no longer has docId.
3. ScoreCachingWrappingScorer now takes + returns LeafCollector.
4. MMapDirectory allows configuring preload per file (so we don't have
   to).
5. Concurrent search can now operate at sub-segment level (but we don't
   yet).

Signed-off-by: Michael Froh <[email protected]>

* Fix more compilation failures

Signed-off-by: Andriy Redko <[email protected]>

* Fix more compilation failures

Signed-off-by: Andriy Redko <[email protected]>

* Fix more compilation failures

Signed-off-by: Andriy Redko <[email protected]>

* Fix more compilation failures

Signed-off-by: Andrew Ross <[email protected]>

* Replace DocValuesFieldExistsQuery with FieldExistsQuery

Signed-off-by: Andrew Ross <[email protected]>

* Fix signature changes in o.a.l.search.BooleanQuery

Signed-off-by: Andrew Ross <[email protected]>

* Fix more compilation failures, update shadow Gradle plugin and fix loggerUsageCheck Gradle task

Signed-off-by: Andriy Redko <[email protected]>

* Fix more compilation failures

Signed-off-by: Andriy Redko <[email protected]>

* Fix analysis-common compilation issues

Signed-off-by: Andriy Redko <[email protected]>

* Fix more compilation failures

Signed-off-by: Andriy Redko <[email protected]>

* Fix more compilation failures

Signed-off-by: Andriy Redko <[email protected]>

* More compilation fixes

Signed-off-by: Andrew Ross <[email protected]>

* More compilation fixes

Signed-off-by: Andrew Ross <[email protected]>

* Fix assemble tasks

Signed-off-by: Andriy Redko <[email protected]>

* Fix more compilation failures

Signed-off-by: Andriy Redko <[email protected]>

* Fix more compilation failures, add CHANGELOG.md

Signed-off-by: Andriy Redko <[email protected]>

* Fix more compilation failures

Signed-off-by: Andriy Redko <[email protected]>

* Fixed Lucene latest codec name

Signed-off-by: Andriy Redko <[email protected]>

* Refactor preload tests in FsDirectoryFactoryTests

Also more compilation fixes.

Signed-off-by: Andrew Ross <[email protected]>

* Fix more compilation failures

Signed-off-by: Andrew Ross <[email protected]>

* Fix precommit failures

Signed-off-by: Andrew Ross <[email protected]>

* Fix analysis test cases

Signed-off-by: Andriy Redko <[email protected]>

* Fix FieldData & ParentJoinAggregator ordinal traversal

Signed-off-by: Andriy Redko <[email protected]>

* Fix IpFieldMapper & CollapsingDocValuesSource ordinal traversal

Signed-off-by: Andriy Redko <[email protected]>

* Fix MultiValueMode, GlobalOrdinalValuesSource, BinaryRangeAggregator, DiversifiedOrdinalsSamplerAggregator, GlobalOrdinalsStringTermsAggregator & CardinalityAggregator ordinal traversal

Signed-off-by: Andriy Redko <[email protected]>

* Fix MultiValueModeTests, BinaryRangeAggregatorTests, IncludeExcludeTests & MissingValuesTests ordinal traversal

Signed-off-by: Andriy Redko <[email protected]>

* Fix ICUCollationKeywordFieldMapper & VectorFieldTests tests

Signed-off-by: Andriy Redko <[email protected]>

* Fix org.opensearch.common.xcontent.support.XContentMapValuesTests test cases

Signed-off-by: Andriy Redko <[email protected]>

* Fix FunctionScoreQuery::CustomBoostFactorWeight::scorerSupplier that may throw NPE

Signed-off-by: Andriy Redko <[email protected]>

* Fix CodecTests and non-concurrent ContextIndexSearcher::search implementation for aggregations

Signed-off-by: Andriy Redko <[email protected]>

* Fixed GlobalOrdinalMapping ordinals iteration and SortedWiderNumericSortField implementation of missingValueAsComparableLong

Signed-off-by: Andriy Redko <[email protected]>

* Fix FieldData and SortedSetDocValuesWriterWrapperTests

Signed-off-by: Andriy Redko <[email protected]>

* Fix org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests test cases

Signed-off-by: Andriy Redko <[email protected]>

* Fix org.opensearch.search.aggregations.bucket.composite.CompositeAggregatorTests test cases

Signed-off-by: Andriy Redko <[email protected]>

* Fix MultiValueMode ordinal traversal

Signed-off-by: Andriy Redko <[email protected]>

* Fix GlobalOrdinalMapping and GlobalOrdinalsStringTermsAggregator ordinal traversals

Signed-off-by: Andriy Redko <[email protected]>

* Fix QueryStringQueryBuilderTests and QueryStringQueryBuilder fuzzy query default rewrite method detection logic

Signed-off-by: Andriy Redko <[email protected]>

* Fix StarTree*DocValueFormatTests

The Composite912DocValuesReader was not honoring the IOContext
correctly. The metadata file should be opened with READONCE and
immediately be closed (on the same thread). The data file needs
to be opened with ReadAdvice.NORMAL to ensure that it's
closeable on another thread if necessary.

Signed-off-by: Michael Froh <[email protected]>

* Fix various MultiTermQuery-related tests

Now that we use CONSTANT_SCORE_BLENDED_REWRITE by default for most
MTQs, we need to update the tests. Also FuzzyQuery should default to
TopTermsBlendedFreqScoringRewrite (taken from Lucene).

Signed-off-by: Michael Froh <[email protected]>

* Fix Spotless

Signed-off-by: Michael Froh <[email protected]>

* Fix RegEx automaton processing using Operations.determinize(...)

Signed-off-by: Andriy Redko <[email protected]>

* Fix compilation issues (post merge)

Signed-off-by: Andriy Redko <[email protected]>

* Fix UnsignedLongComparator::sortableBytesToLong encoding

Signed-off-by: Andriy Redko <[email protected]>

* Fix HalfFloatComparator::sortableBytesToLong and HalfFloatComparator::missingValueAsComparableLong

Signed-off-by: Andriy Redko <[email protected]>

* Fix infinite recursion in CollapsingTopDocsCollectorTests

Signed-off-by: Michael Froh <[email protected]>

* Fix SingleDimensionValuesSourceTests

Can no longer mock IndexReader as it's sealed.

Signed-off-by: Michael Froh <[email protected]>

* Fix a few more MultiTermQuery-related test failures

Signed-off-by: Michael Froh <[email protected]>

* Fixed QueryPhaseTests#minScoreOptimisation by not delegating setWeight to underlying delegate

Signed-off-by: expani <[email protected]>

* Fix DerivedFieldMapperQueryTests

Needed to add doc values to date and IP fields.

Signed-off-by: Michael Froh <[email protected]>

* Fix SortedNumericDocValuesWriterWrapperTests

Lucene's SortedNumericDocValuesWriter no longer throws an exception if
you try to iterate past a document's last value. (Instead, it happily
leaks the next document's value.)

Signed-off-by: Michael Froh <[email protected]>

* Fix IpRangeAggregatorTests

Rewrote a for-loop in BinaryRangeAggregator to be less confusing. As a
side-effect, it fixed the test.

Signed-off-by: Michael Froh <[email protected]>

* Fix IpFieldMapperTests

I broke these tests as a side-effect for my fix to
DerivedFieldMapperQueryTests.

Signed-off-by: Michael Froh <[email protected]>

* Fix MissingValues ordinal traversal and MultiOrdinalsTests tests

Signed-off-by: Andriy Redko <[email protected]>

* Prevent cascading test failures in SearchServiceTests

I'm not really fixing the failing tests, but ensuring that the two
tests that fail don't cause a bunch of other tests to fail by
leaving transient settings around. (Clean up the transient settings
even if those tests fail.)

Signed-off-by: Michael Froh <[email protected]>

* Fix DerivedFieldTypeTests

I broke these when fixing DerivedFieldMapperQueryTests.

Signed-off-by: Michael Froh <[email protected]>

* Fix TransferManagerRemoteDirectoryReaderTests

Needed to update mocked calls.

Signed-off-by: Michael Froh <[email protected]>

* Add missing javadoc

Signed-off-by: Andriy Redko <[email protected]>

* Uploaded Index generated from Lucene 8.x ( OpenSearch 1.3.0 ) to be used by UnitTests

Signed-off-by: expani <[email protected]>

* Moved path to a constant

Signed-off-by: expani <[email protected]>

* Update Apache Lucene to 10.1.0

Signed-off-by: Andriy Redko <[email protected]>

* Fixed unit tests for lucene 10 upgrade

Signed-off-by: expani <[email protected]>

* Fix posting format for CompletionFieldMapper

Signed-off-by: Andriy Redko <[email protected]>

* Fix org.opensearch.search.SearchServiceTests test cases

Signed-off-by: Andriy Redko <[email protected]>

* Removed extra iteration of DISI and added stored field option to bypass modified check in lucene Field

Signed-off-by: expani <[email protected]>

* Changed to use advance and init to use docId() instead of nextDoc()

Signed-off-by: expani <[email protected]>

* Increased number of indexed docs to match segment count

Signed-off-by: expani <[email protected]>

* Reduced segment count to match assertion instead of increasing docs

Signed-off-by: expani <[email protected]>

* Revert to make 90_search_after integ pass as hits are not shown

Signed-off-by: expani <[email protected]>

* Making Half Float failures work with it's conversions

Signed-off-by: expani <[email protected]>

* Changing max long to signed from unsigned

Signed-off-by: expani <[email protected]>

* Addressed PR Comments

Signed-off-by: expani <[email protected]>

* Trying another approach for gracefully handling unsigned long conversions

Signed-off-by: expani <[email protected]>

* Simplified unsigned to signed long conversion

Signed-off-by: expani <[email protected]>

* Reverted changes as need to think more on this

Signed-off-by: expani <[email protected]>

* Fix azure-fixture & s3-fixture Docker images

Signed-off-by: Andriy Redko <[email protected]>

* Fix 'search/90_search_after/numeric skipping logic with competitive missing value' test cases

Signed-off-by: Andriy Redko <[email protected]>

* Fix gcs-fixture Docker images

Signed-off-by: Andriy Redko <[email protected]>

* Updated to use modern Romanian Unicode

Signed-off-by: expani <[email protected]>

* Fix org.opensearch.wildfly.WildflyIT test cases

Signed-off-by: Andriy Redko <[email protected]>

* Fix org.opensearch.search.aggregations.bucket.MinDocCountIT test cases

Signed-off-by: Andriy Redko <[email protected]>

* Fix CardinalityAggregator.OrdinalsCollector ordinals traversal

Signed-off-by: Andriy Redko <[email protected]>

* Fix org.opensearch.search.query.ScriptScoreQueryIT test cases

Signed-off-by: Andriy Redko <[email protected]>

* Fix org.opensearch.search.scriptfilter.ScriptQuerySearchIT test cases

Signed-off-by: Andriy Redko <[email protected]>

* Fix UnsignedLongComparator top/bottom as comparable long conversions

Signed-off-by: Andriy Redko <[email protected]>

* Fix post-merge (with main) compilation issues

Signed-off-by: Andriy Redko <[email protected]>

* Fix UnsignedLongComparator with respect to sortable bytes to long conversion

Signed-off-by: Andriy Redko <[email protected]>

* Fix FlatObjectFieldTypeTests test cases

Signed-off-by: Andriy Redko <[email protected]>

---------

Signed-off-by: Andriy Redko <[email protected]>
Signed-off-by: Michael Froh <[email protected]>
Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: expani <[email protected]>
Co-authored-by: Michael Froh <[email protected]>
Co-authored-by: Andrew Ross <[email protected]>
Co-authored-by: expani <[email protected]>
  • Loading branch information
4 people authored Jan 22, 2025
1 parent 8a5c3b5 commit 7c46f8f
Show file tree
Hide file tree
Showing 519 changed files with 3,466 additions and 3,050 deletions.
17 changes: 3 additions & 14 deletions .github/workflows/assemble.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ jobs:
runs-on: ${{ matrix.os }}
strategy:
matrix:
java: [ 11, 17, 21 ]
java: [ 21, 23 ]
os: [ubuntu-latest, windows-latest, macos-13, ubuntu-24.04-arm]
steps:
- uses: actions/checkout@v4
Expand All @@ -16,17 +16,6 @@ jobs:
with:
java-version: ${{ matrix.java }}
distribution: temurin
- name: Set up JDK 17
# See please https://docs.gradle.org/8.10/userguide/upgrading_version_8.html#minimum_daemon_jvm_version
if: matrix.java == 11
uses: actions/setup-java@v4
with:
java-version: 17
distribution: temurin
- name: Set JAVA${{ matrix.java }}_HOME
shell: bash
run: |
echo "JAVA${{ matrix.java }}_HOME=$JAVA_HOME_${{ matrix.java }}_${{ runner.arch }}" >> $GITHUB_ENV
- name: Setup docker (missing on MacOS)
id: setup_docker
if: runner.os == 'macos'
Expand All @@ -44,8 +33,8 @@ jobs:
shell: bash
if: runner.os != 'macos'
run: |
./gradlew assemble --parallel --no-build-cache -PDISABLE_BUILD_CACHE -Druntime.java=${{ matrix.java }}
./gradlew assemble --parallel --no-build-cache -PDISABLE_BUILD_CACHE
- name: Run Gradle (assemble)
if: runner.os == 'macos' && steps.setup_docker.outcome == 'success'
run: |
./gradlew assemble --parallel --no-build-cache -PDISABLE_BUILD_CACHE -Druntime.java=${{ matrix.java }}
./gradlew assemble --parallel --no-build-cache -PDISABLE_BUILD_CACHE
15 changes: 2 additions & 13 deletions .github/workflows/precommit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ jobs:
runs-on: ${{ matrix.os }}
strategy:
matrix:
java: [ 11, 17, 21, 23 ]
java: [ 21, 23 ]
os: [ubuntu-latest, windows-latest, macos-latest, macos-13, ubuntu-24.04-arm]
steps:
- uses: actions/checkout@v4
Expand All @@ -17,18 +17,7 @@ jobs:
java-version: ${{ matrix.java }}
distribution: temurin
cache: gradle
- name: Set up JDK 17
# See please https://docs.gradle.org/8.10/userguide/upgrading_version_8.html#minimum_daemon_jvm_version
if: matrix.java == 11
uses: actions/setup-java@v4
with:
java-version: 17
distribution: temurin
- name: Set JAVA${{ matrix.java }}_HOME
shell: bash
run: |
echo "JAVA${{ matrix.java }}_HOME=$JAVA_HOME_${{ matrix.java }}_${{ runner.arch }}" >> $GITHUB_ENV
- name: Run Gradle (precommit)
shell: bash
run: |
./gradlew javadoc precommit --parallel -Druntime.java=${{ matrix.java }}
./gradlew javadoc precommit --parallel
1 change: 1 addition & 0 deletions CHANGELOG-3.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Views, simplify data access and manipulation by providing a virtual layer over one or more indices ([#11957](https://github.com/opensearch-project/OpenSearch/pull/11957))

### Dependencies
- Update Apache Lucene to 10.1.0 ([#16366](https://github.com/opensearch-project/OpenSearch/pull/16366))
- Bump Apache HttpCore5/HttpClient5 dependencies from 5.2.5/5.3.1 to 5.3.1/5.4.1 to support ExtendedSocketOption in HttpAsyncClient ([#16757](https://github.com/opensearch-project/OpenSearch/pull/16757))

### Changed
Expand Down
43 changes: 2 additions & 41 deletions benchmarks/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -85,44 +85,5 @@ spotless {
}
}

if (BuildParams.runtimeJavaVersion >= JavaVersion.VERSION_20) {
// Add support for incubator modules on supported Java versions.
run.jvmArgs += ['--add-modules=jdk.incubator.vector']
run.classpath += files(jar.archiveFile)
run.classpath -= sourceSets.main.output
evaluationDependsOn(':libs:opensearch-common')

sourceSets {
java20 {
java {
srcDirs = ['src/main/java20']
}
}
}

configurations {
java20Implementation.extendsFrom(implementation)
}

dependencies {
java20Implementation sourceSets.main.output
java20Implementation project(':libs:opensearch-common').sourceSets.java20.output
java20AnnotationProcessor "org.openjdk.jmh:jmh-generator-annprocess:$versions.jmh"
}

compileJava20Java {
targetCompatibility = JavaVersion.VERSION_20
options.compilerArgs.addAll(["-processor", "org.openjdk.jmh.generators.BenchmarkProcessor"])
}

jar {
metaInf {
into 'versions/20'
from sourceSets.java20.output
}
manifest.attributes('Multi-Release': 'true')
}

// classes generated by JMH can use all sorts of forbidden APIs but we have no influence at all and cannot exclude these classes
disableTasks('forbiddenApisJava20')
}
// Add support for incubator modules on supported Java versions.
run.jvmArgs += ['--add-modules=jdk.incubator.vector']
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
public class RoundableSupplier implements Supplier<Roundable> {
private final Supplier<Roundable> delegate;

RoundableSupplier(String type, long[] values, int size) throws ClassNotFoundException {
RoundableSupplier(String type, long[] values, int size) {
switch (type) {
case "binary":
delegate = () -> new BinarySearcher(values, size);
Expand All @@ -22,7 +22,8 @@ public class RoundableSupplier implements Supplier<Roundable> {
delegate = () -> new BidirectionalLinearSearcher(values, size);
break;
case "btree":
throw new ClassNotFoundException("BtreeSearcher is not supported below JDK 20");
delegate = () -> new BtreeSearcher(values, size);
break;
default:
throw new IllegalArgumentException("invalid type: " + type);
}
Expand Down
10 changes: 5 additions & 5 deletions buildSrc/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,8 @@ processResources {
* Java version *
*****************************************************************************/

if (JavaVersion.current() < JavaVersion.VERSION_11) {
throw new GradleException('At least Java 11 is required to build opensearch gradle tools')
if (JavaVersion.current() < JavaVersion.VERSION_21) {
throw new GradleException('At least Java 21 is required to build opensearch gradle tools')
}

sourceSets {
Expand Down Expand Up @@ -111,7 +111,7 @@ dependencies {
api 'org.apache.rat:apache-rat:0.15'
api "commons-io:commons-io:${props.getProperty('commonsio')}"
api "net.java.dev.jna:jna:5.14.0"
api 'com.github.johnrengelman:shadow:8.1.1'
api 'com.gradleup.shadow:shadow-gradle-plugin:8.3.5'
api 'org.jdom:jdom2:2.0.6.1'
api "org.jetbrains.kotlin:kotlin-stdlib-jdk8:${props.getProperty('kotlin')}"
api 'de.thetaphi:forbiddenapis:3.8'
Expand Down Expand Up @@ -171,8 +171,8 @@ if (project != rootProject) {

allprojects {
java {
targetCompatibility = JavaVersion.VERSION_11
sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_21
sourceCompatibility = JavaVersion.VERSION_21
}
}

Expand Down
4 changes: 2 additions & 2 deletions buildSrc/reaper/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@
apply plugin: 'java'

java {
targetCompatibility = JavaVersion.VERSION_11
sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_21
sourceCompatibility = JavaVersion.VERSION_21
}

jar {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,6 @@
# specific language governing permissions and limitations
# under the License.
#
OPENSEARCH_BUILD_JAVA=openjdk11
OPENSEARCH_RUNTIME_JAVA=java11
OPENSEARCH_BUILD_JAVA=openjdk21
OPENSEARCH_RUNTIME_JAVA=java21
GRADLE_TASK=build
Original file line number Diff line number Diff line change
Expand Up @@ -61,10 +61,6 @@ java.nio.channels.FileChannel#read(java.nio.ByteBuffer, long)
@defaultMessage Use Lucene.parseLenient instead it strips off minor version
org.apache.lucene.util.Version#parseLeniently(java.lang.String)

@defaultMessage Spawns a new thread which is solely under lucenes control use ThreadPool#relativeTimeInMillis instead
org.apache.lucene.search.TimeLimitingCollector#getGlobalTimerThread()
org.apache.lucene.search.TimeLimitingCollector#getGlobalCounter()

@defaultMessage Don't interrupt threads use FutureUtils#cancel(Future<T>) instead
java.util.concurrent.Future#cancel(boolean)

Expand Down
2 changes: 1 addition & 1 deletion buildSrc/src/main/resources/minimumCompilerVersion
Original file line number Diff line number Diff line change
@@ -1 +1 @@
17
21
2 changes: 1 addition & 1 deletion buildSrc/src/main/resources/minimumRuntimeVersion
Original file line number Diff line number Diff line change
@@ -1 +1 @@
11
21
4 changes: 2 additions & 2 deletions buildSrc/src/testKit/thirdPartyAudit/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -46,11 +46,11 @@ dependencies {
}

tasks.register("empty", ThirdPartyAuditTask) {
targetCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_21
signatureFile = file('third-party-audit-empty.txt')
}

tasks.register("absurd", ThirdPartyAuditTask) {
targetCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_21
signatureFile = file('third-party-audit-absurd.txt')
}
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ public void testCompressesResponseIfRequested() throws IOException {
SearchResponse searchResponse = execute(searchRequest, highLevelClient()::search, highLevelClient()::searchAsync, requestOptions);

assertThat(searchResponse.status().getStatus(), equalTo(200));
assertEquals(1L, searchResponse.getHits().getTotalHits().value);
assertEquals(1L, searchResponse.getHits().getTotalHits().value());
assertEquals(SAMPLE_DOCUMENT, searchResponse.getHits().getHits()[0].getSourceAsString());
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,7 @@ public void testDeleteByQuery() throws Exception {
assertEquals(0, bulkResponse.getSearchFailures().size());
assertEquals(
2,
highLevelClient().search(new SearchRequest(sourceIndex), RequestOptions.DEFAULT).getHits().getTotalHits().value
highLevelClient().search(new SearchRequest(sourceIndex), RequestOptions.DEFAULT).getHits().getTotalHits().value()
);
}
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,7 @@ public void testSearchScroll() throws IOException {
RequestOptions.DEFAULT
);
assertEquals(mockSearchResponse.getScrollId(), searchResponse.getScrollId());
assertEquals(0, searchResponse.getHits().getTotalHits().value);
assertEquals(0, searchResponse.getHits().getTotalHits().value());
assertEquals(5, searchResponse.getTotalShards());
assertEquals(5, searchResponse.getSuccessfulShards());
assertEquals(100, searchResponse.getTook().getMillis());
Expand Down
Loading

0 comments on commit 7c46f8f

Please sign in to comment.