[Segment Replication] Create new Segment Replication client to support running Integ Tests with Segment Replication enabled

[Rishikesh1159]

Describe the bug
Currently all the integ tests in opensearch codebase in one way or another inherit from single parent class OpenSearchIntegTestCase. So we need to create a SegRep client in OpenSearchIntegTestCase class so that all integ tests inheriting this class can be ran against segment replication (with waiting until replica catching up behaviour.)

This issue is being tracked as a sub task and part of meta issue #6761

Expected behavior
Creating a new segment replication client similar to Randomized Client, and then we override the search or refresh methods in client to wait until replica shard has caught up.

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[Rishikesh1159]

I have already started working on this issue on my branch: https://github.com/Rishikesh1159/OpenSearch/tree/test-all-integtests-segrep. Currently I am verifying if creating a new segrep client (with wait until replica caught up behaviour) would resolve integ test failures with segment replication. Once we have enough confidence on this branch https://github.com/Rishikesh1159/OpenSearch/tree/test-all-integtests-segrep in resolving all integtest failures with segrep, we can raise a PR to merge this branch.

[Rishikesh1159]

Instead of using a new segrep client that makes additional api call, we are planning to Mock NRTEngine itself, so that we can wait until replica's have caught up with in the engine itself. We don't have to make an additonal API call as we need for segrep client.

The work for adding a new NRT MockEngine for testing is currently tracked in this branch. We are soon planning to merge this into feature branch, before merging it to main.

The work done on segrep client is present in this branch, which can be used in future if needed. Closing this issue for now as we have decided on using Mock NRTEngine

[Rishikesh1159]

Closing this issue because we are planning to move forward with another approach. More details discussed here. In step 2 we will be using approach 2 so closing this.