[PROPOSAL] Modularize/remove/make optional the dependency on opensearch-rest-client

[dblock]

What are you proposing?

Remove dependency on opensearch-rest-client, or anything else that may be imported from http://github.com/opensearch-project/OpenSearch.

How did you come up with this proposal?

The opensearch-java depends on code exported by the server, which creates a kind of undesirable tight coupling where the client needs to be released with newer versions of the server. Instead, the client should be independent, mostly generated from spec (opensearch-project/opensearch-clients#19), and supporting at least two major versions of the server (opensearch-project/opensearch-clients#17).

We had a similar discussion for AWS components, #177 (comment) and wanted #157 to build with JDK8, but OpenSearch dependences prevented us from doing so.

What is the user experience going to be?

The opensearch-java will become completely independent and live its own life.

Why should it be built? Any reason not to?

I don't see any.

What will it take to execute?

Start by changing this in https://github.com/opensearch-project/opensearch-java/blob/main/java-client/build.gradle.kts.

implementation("org.opensearch.client", "opensearch-rest-client", opensearchVersion)
testImplementation("org.opensearch.test", "framework", opensearchVersion)

What are remaining open questions?

Probably many.

[dblock]

Curious what @reta and @mtimmerm think about this?

[mtimmerm]

You should probably still provide the capability, which would mean moving the RestClient transport (and probably mine) into separate release packages, because people will want to mix-and-match versions.

I think it's a good idea. The most significant issue I can see is that it requires a major version bump no matter what you do, since the main java client will no longer work by itself in situations where it works currently. That means that your version numbers will no longer correspond to OpenSearch version numbers, which you may or may not consider to be a problem.

[reta]

+1 to move towards client generation from specs, it also would be easier (more straightforward) to catch any backward incompatible changes beforehand.

[dlvenable]

It appears that this project just needs a new implementation of OpenSearchTransport which does not depend on opensearch-rest-client. An initial implementation could possibly just copy-and-paste the code from that project and abstract it appropriately (package protected).

The opensearch-java project could possibly retain the existing RestClientTransport and make opensearch-rest-client an optional dependency. This could help with migration.

[dblock]

@dlvenable Yes. Also note #177.

[VachaShah]

I believe this is done for the next major version where RestClientTransport is optional? @reta Please correct me if I am wrong.

[reta]

I believe this is done for the next major version where RestClientTransport is optional? @reta Please correct me if I am wrong.

Correct, thanks @VachaShah !