[Meta] Remote Vector Index Build Component in OpenSearch Vector Engine

[jed326]

Description

This is a meta issue to track the work needed to design and implement the component described by the previous RFC:

• [RFC] Remote Vector Index Build Feature with OpenSearch Vector Engine #2294

Background

Following content is copied from previous RFCs for reference

• [RFC] : Boosting OpenSearch Vector Engine Performance using GPUs #2293
• [RFC] Remote Vector Index Build Feature with OpenSearch Vector Engine #2294

Components Definition

1. Vector Index Build Component: This is new component that is being proposed as part of this design. It will be responsible for building a vector index for a segment and then notifying Vector Engine that index is ready to download. This index build component can have both GPU and CPU machines or any customer hardware to build the vector index.
2. Opensearch Vector Engine(k-NN plugin): This nothing but k-NN plugin which is responsible for providing vector related capabilities in Opensearch.
3. Object Store/Intermediate Vector Store: This an intermediate storage component which will temporarily store the vectors and index for different components. The store will ensure that it has right deletion polices to remove these vectors. This is not the same store where we store the segments for Remote store feature in Opensearch. This will be a separate store.

Assumptions

Below are some of the assumptions taken while designing the integration of Vector Engine with remote Index Build Service.

1. This design assumes that there is an IndexBuildService hosted at a endpoint where Vector Engine can submit the request of create Index.
2. Vector engine has all the details on how to connect to object store to stream vectors and download index.
3. Add what will be provided from customer
4. The building of an index via a remote endpoint will supported with indices that uses NativeKNNVectorsFormat and with old indices which uses DocValuesFormat(old indices format due to the limitations on access of different index/mapping related attributes at the codec level)

Roles and Responsibilities

1. Given the details of the object store, upload vectors data to the store and download the index once it is created.
2. Given any endpoint for Index build service call the remote endpoint to create the vector index per segment.
3. Have intelligent logic to take a decision when to use Index build service endpoint vs Local compute to build the index.
4. Once the index is built download the index from remote store and put it along with other segment related files.

Tasks

High Level Designs:

• Remote Vector Index Build Component — Object Store Upload/Download #2392
• Remote Vector Index Build Component — Remote Vector Service Client #2393

POC Implementations:

• POC - Remote Object Store Upload/Download
• POC - Remote Vector Build Service Client

Low Level Designs:

• LLD - Bringing It All Together

Final Implementations (PR Tracking)