Search pipelines added #172

Tokesh · 2023-12-30T23:27:46Z

Description

Adding Search Pipelines specs

Issues Resolved

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Tokesh <[email protected]>

nhtruong · 2024-01-02T21:44:27Z

This one is only concerned with Creating and Retrieving a search pipeline, correct?
Looks like we also need to update the search endpoint to support the search_pipeline querystring parameter as well.
There's also an endpoint to retrieve pipeline stats.

You can add those in another PR.

Also maybe we should put these endpoints in the search_pipeline namespace? (the operation groups will then be search_pipeline.create, search_pipeline.get, search_pipeline.metrics... and move the files under ./search_pipeline folder)

Signed-off-by: Tokesh <[email protected]>

Tokesh · 2024-01-03T17:40:40Z

Looks like we also need to update the search endpoint to support the search_pipeline querystring parameter as well.
There's also an endpoint to retrieve pipeline stats.

Yeap, I need some more time to do it (I hope this weekends). I'll create another PR

Also maybe we should put these endpoints in the search_pipeline namespace? (the operation groups will then be search_pipeline.create, search_pipeline.get, search_pipeline.metrics... and move the files under ./search_pipeline folder)

Yes, good idea! You can check it now

Signed-off-by: Tokesh <[email protected]>

model/search_pipeline/create_search_pipeline/operations.smithy

model/search_pipeline/get_search_pipeline/operations.smithy

Signed-off-by: Tokesh <[email protected]>

model/search_pipeline/get/operations.smithy

model/search_pipeline/get/structures.smithy

Xtansia · 2024-01-19T00:20:22Z

model/search_pipeline/search_pipeline.smithy

+structure RequestProcessors {
+    filter_query: FilterQuery,
+    neural_query_enricher: NeuralQueryEnricher,
+    script: SearchScript
+}
+
+structure ResponseProcessors {
+    personalize_search_ranking: PersonalizeSearchRanking,
+    rename_field: RenameField,
+}


Really these are more like unions rather than structures, but not sure how the OpenAPI conversion handles unions.

Union will become oneOf in OpenAPI

I never used unions. Do i need to implement it here?

Xtansia · 2024-01-19T00:21:15Z

model/search_pipeline/search_pipeline.smithy

+structure SearchPipelineStructure{
+    version: Integer,
+    request_processors: RequestProcessorsList,
+    response_processors: ResponseProcessorsList


There's also a concept of a phase_results_processors:
https://opensearch.org/docs/latest/search-plugins/search-pipelines/search-processors/#search-phase-results-processors
https://opensearch.org/docs/latest/search-plugins/search-pipelines/normalization-processor/#creating-a-search-pipeline

dblock · 2024-01-19T15:43:44Z

Are we supposed to be adding tests as well?

nhtruong · 2024-01-22T22:44:41Z

@dblock The current test framework for this repo is extremely labor intensive for very little benefits. We need a more streamlined framework to avoid discoursing people from contributing to this project.

I also discussed with others earlier today on this topics. @Xtansia has a proposal for the new test framework that he will describe in a new issue. We also discussed testing the specs through the generated clients. And eventually (hopefully not too far into the future), the Specs should be the point of origin of for every new feature, and the cluster should be tested against the Specs, not the other way around.

dblock · 2024-02-07T14:43:28Z

@Tokesh looks like there are a few questions/issues above, care to address/resolve? Thanks.

…result processor Signed-off-by: Tokesh <[email protected]>

Tokesh · 2024-02-07T16:05:29Z

@Tokesh looks like there are a few questions/issues above, care to address/resolve? Thanks.

I was busy restoring my health, was a little distracted and didn’t notice how I was blocking the rest of the PR. I'm sorry!
Fixed the errors mentioned above
It’s better now, ready to correct any mistakes if needed.

dblock · 2024-02-07T21:24:29Z

@Xtansia @nhtruong all yours!

saimedhi · 2024-02-28T19:52:18Z

@nhtruong, @Xtansia, could you kindly review this PR? There's been interest from users regarding the search pipeline API in OpenSearch-py. Relevant issues: #668, #474.

model/search_pipeline/search_pipeline.smithy

Co-authored-by: Thomas Farr <[email protected]> Signed-off-by: Niyazbek Torekeldi <[email protected]>

Xtansia

@Tokesh Thanks for making those changes, please just run ./gradlew spotlessApply to fix formatting, and also rebase your branch onto the latest main so that the API Coverage workflow can pass

github-actions · 2024-03-22T18:40:05Z

API specs implemented for 241/649 (37%) APIs.

github-actions · 2024-03-22T18:42:48Z

API specs implemented for 241/649 (37%) APIs.

github-actions · 2024-03-22T18:44:31Z

API specs implemented for 241/649 (37%) APIs.

Signed-off-by: Niyazbek Torekeldi <[email protected]>

github-actions · 2024-03-22T18:48:18Z

API specs implemented for 241/649 (37%) APIs.

Tokesh · 2024-03-22T19:11:10Z

@Tokesh Thanks for making those changes, please just run ./gradlew spotlessApply to fix formatting, and also rebase your branch onto the latest main so that the API Coverage workflow can pass

I got this! you can check it now

Search pipelines added

01f29a7

Signed-off-by: Tokesh <[email protected]>

Tokesh requested review from dblock, harshavamsi, pgtgrly, sachetalva, nhtruong, Xtansia and VachaShah as code owners December 30, 2023 23:27

search_pipeline folder added

65a5585

Signed-off-by: Tokesh <[email protected]>

small fix of operation group in create method

68d5a1e

Signed-off-by: Tokesh <[email protected]>

nhtruong reviewed Jan 8, 2024

View reviewed changes

model/search_pipeline/create_search_pipeline/operations.smithy Outdated Show resolved Hide resolved

nhtruong reviewed Jan 8, 2024

View reviewed changes

model/search_pipeline/get_search_pipeline/operations.smithy Outdated Show resolved Hide resolved

Tokesh added 2 commits January 9, 2024 10:18

rename search pipeline folders

6ad4aaa

Signed-off-by: Tokesh <[email protected]>

renaming search pipeline to resolve conflict with main branch

ceaaaad

Signed-off-by: Tokesh <[email protected]>

Tokesh requested a review from nhtruong January 9, 2024 04:25

nhtruong previously approved these changes Jan 10, 2024

View reviewed changes

Xtansia requested changes Jan 19, 2024

View reviewed changes

AbitraryYu mentioned this pull request Feb 7, 2024

Added Tests for Search Pipeline and Notifications Plugin opensearch-project/opensearch-py#668

Merged

adding map for main structure, main search pipeline API spces, phase …

d5c2961

…result processor Signed-off-by: Tokesh <[email protected]>

Tokesh dismissed nhtruong’s stale review via d5c2961 February 7, 2024 15:56

Tokesh requested review from Xtansia and nhtruong February 7, 2024 15:58

Xtansia requested changes Mar 5, 2024

View reviewed changes

model/search_pipeline/search_pipeline.smithy Outdated Show resolved Hide resolved

Update model/search_pipeline/search_pipeline.smithy

6faa509

Co-authored-by: Thomas Farr <[email protected]> Signed-off-by: Niyazbek Torekeldi <[email protected]>

Tokesh requested a review from Xtansia March 16, 2024 14:32

Xtansia requested changes Mar 21, 2024

View reviewed changes

Merge branch 'opensearch-project:main' into search-pipelines

d4425ba

spotless Apply

1a901cd

Signed-off-by: Niyazbek Torekeldi <[email protected]>

Tokesh force-pushed the search-pipelines branch from 5ad74c4 to 1a901cd Compare March 22, 2024 18:47

Tokesh requested a review from Xtansia March 22, 2024 18:49

Xtansia approved these changes Mar 24, 2024

View reviewed changes

Xtansia merged commit 4ef4b7d into opensearch-project:main Mar 24, 2024
5 checks passed

saimedhi mentioned this pull request Mar 25, 2024

[BUG]API generator failing opensearch-project/opensearch-py#710

Closed

nhtruong mentioned this pull request Apr 2, 2024

[FEATURE] Add Search Pipeline #171

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Search pipelines added #172

Search pipelines added #172

Tokesh commented Dec 30, 2023

nhtruong commented Jan 2, 2024

Tokesh commented Jan 3, 2024 •

edited

Loading

Xtansia Jan 19, 2024

nhtruong Feb 7, 2024

Tokesh Feb 8, 2024 •

edited

Loading

Xtansia Jan 19, 2024

dblock commented Jan 19, 2024

nhtruong commented Jan 22, 2024 •

edited

Loading

dblock commented Feb 7, 2024

Tokesh commented Feb 7, 2024

dblock commented Feb 7, 2024

saimedhi commented Feb 28, 2024

Xtansia left a comment

github-actions bot commented Mar 22, 2024

github-actions bot commented Mar 22, 2024

github-actions bot commented Mar 22, 2024

github-actions bot commented Mar 22, 2024

Tokesh commented Mar 22, 2024

Search pipelines added #172

Search pipelines added #172

Conversation

Tokesh commented Dec 30, 2023

Description

Issues Resolved

nhtruong commented Jan 2, 2024

Tokesh commented Jan 3, 2024 • edited Loading

Xtansia Jan 19, 2024

Choose a reason for hiding this comment

nhtruong Feb 7, 2024

Choose a reason for hiding this comment

Tokesh Feb 8, 2024 • edited Loading

Choose a reason for hiding this comment

Xtansia Jan 19, 2024

Choose a reason for hiding this comment

dblock commented Jan 19, 2024

nhtruong commented Jan 22, 2024 • edited Loading

dblock commented Feb 7, 2024

Tokesh commented Feb 7, 2024

dblock commented Feb 7, 2024

saimedhi commented Feb 28, 2024

Xtansia left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 22, 2024

github-actions bot commented Mar 22, 2024

github-actions bot commented Mar 22, 2024

github-actions bot commented Mar 22, 2024

Tokesh commented Mar 22, 2024

Tokesh commented Jan 3, 2024 •

edited

Loading

Tokesh Feb 8, 2024 •

edited

Loading

nhtruong commented Jan 22, 2024 •

edited

Loading