[Feature] Publish plugins SNAPSHOT artifacts (along with SNASPHOT distribution)

[reta]

Is your feature request related to a problem? Please describe

At the moment, we do not publish plugins SNAPSHOT artifacts but min distribution only (see please https://build.ci.opensearch.org/job/publish-opensearch-min-snapshots/). However, it becomes increasingly demanded to have plugins SNAPSHOT artifacts published as well.

Describe the solution you'd like

Publish plugins SNAPSHOT artifacts under S3 bucket https://artifacts.opensearch.org/snapshots/plugins/ so OpenSearch tooling (opensearch-plugin, etc) could be used to install them for SNAPSHOT distributions.

Describe alternatives you've considered

Publish ZIPs to Apache Maven but that would change completely the distribution model for releases and snapshots.

Additional context

See please opensearch-project/OpenSearch#16219

[reta]

CC @prudhvigodithi

[prudhvigodithi]

We have been pushing the created core-plugins already to https://ci.opensearch.org but not in https://artifacts.opensearch.org/snapshots/plugins/ format.

Example:
https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/tar/builds/opensearch/core-plugins/analysis-icu-3.0.0.zip

https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/rpm/builds/opensearch/core-plugins/analysis-icu-3.0.0.zip

https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/deb/builds/opensearch/core-plugins/analysis-icu-3.0.0.zip

and we do the same for arm64

@peterzhuamazon @cwperks @reta can we just the leverage this ? :)

Thank you

[reta]

@peterzhuamazon @cwperks @reta can we just the leverage this ? :)

Thanks @prudhvigodithi . So this is somewhat inconsistent: all these 3 URLs point to the same plugin (at least they should) because the distributions of the plugins do not depend on the OpenSearch packaging. We could leverage that, indeed, but looks more like a workaround, not a solution.

[peterzhuamazon]

At the moment we just let every distribution build the native plugins zip, but when release we only release x64/tar version.

[peterzhuamazon]

So that we do not need to change the workflow too much.

[dblock]

Will this address my problem in opensearch-project/opensearch-api-specification#610?

I am using the 3.0 docker build and need to install one of the optional plugins.

Normal docker reference:

FROM opensearchstaging/opensearch:2.18.0@sha256:f311da085b968663fadf041bafc78b24dc21556f0726e625caf3560f7323a020

# attempt to install the plugin here
RUN ./bin/opensearch-plugin install https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.18.0/10380/linux/x64/tar/builds/opensearch/core-plugins/analysis-phonenumber-2.18.0.zip

The above example is problematic because (it doesn't work), but I also need to manually match the version to the docker image SHA somehow.

[reta]

Will this address my problem in opensearch-project/opensearch-api-specification#610?

It will @dblock

[peterzhuamazon]

After some talk on the triage meeting I fully understand what @reta is talking about.

Similar to:
https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/<>.tar.gz

Do a:
https://artifacts.opensearch.org/snapshots/plugins/opensearch/3.0.0-SNAPSHOT/<>.zip

We can start this with opensearch first, as I just checked we havent done any core snapshot for OSD core on artifacts.opensearch.org for a long time.

Thanks.

[Divyaasm]

Acceptance criteria:
Publish native plugins along side the core snapshot ->
https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/<>.tar.gz

[peterzhuamazon]

We also need to add the checksum files alongside the native plugin zips.
Probably need to use the same folder structure for snapshot as the release ones.

[peterzhuamazon]

Hi @reta if we put the snapshot native plugins zips on s3, I assume you can directly run opensearch-plugin install <>? Or do we need to make changes in core to make it happen.

[reta]

Hi @reta if we put the snapshot native plugins zips on s3, I assume you can directly run opensearch-plugin install <>? Or do we need to make changes in core to make it happen.

Thanks @peterzhuamazon, we only need to know the location, if it is different from the released versions fe

[peterzhuamazon]

Take a look in the next two days, assuming we should be able to add it as part of min snapshot workflows.

[peterzhuamazon]

After offline sync, we will use this folder structure:

https://artifacts.opensearch.org/snapshots/plugins/analysis-icu/2.10.0-SNAPSHOT/analysis-icu-2.10.0-SNAPSHOT.zip

[peterzhuamazon]

PRs:

• Add core plugins snapshots publishing alongside min snapshots opensearch-build-libraries#556
• Upload core plugins snapshots / gradle check updates in lib 7.3.2 #5165

[peterzhuamazon]

The above wont affect linux installation as that is already pushed.

[peterzhuamazon]

Weird I dont see the sig files for uploads.

[reta]

Yeah, me neither

[peterzhuamazon]

Out of no where the core-plugins location being null:

Running /var/jenkins/workspace/publish-opensearch-min-snapshots/src/run_sign.py null --sigtype .sig ...

[peterzhuamazon]

A re-run in test branch passed somehow, will just re-run production as I suspect just hiccups.

[peterzhuamazon]

Even worse, now the new macos agents cannot find pipenv, due to py39 not exist in macports?:

==> amazon-ebs.Jenkins-Agent-MacOS13-ARM64-Mac2M2proMetal-13.6.7-2024-11-08T09-08-06Z: Error: Port py39-python-install not found
    amazon-ebs.Jenkins-Agent-MacOS13-ARM64-Mac2M2proMetal-13.6.7-2024-11-08T09-08-06Z: Selecting 'python39' for 'python' failed: The specified group 'python' does not exist.
    amazon-ebs.Jenkins-Agent-MacOS13-ARM64-Mac2M2proMetal-13.6.7-2024-11-08T09-08-06Z: Selecting 'python39' for 'python3' failed: The specified group 'python3' does not exist.

[peterzhuamazon]

py39-python-install has been deleted as part of the macports:
https://ports.macports.org/port/py39-python-install/

[peterzhuamazon]

Even more PRs:

• Replace py39-python-installer with python39 due to port removal opensearch-ci#504

[peterzhuamazon]

Seems like it is very random that var passed to signing will become null

Running /var/jenkins/workspace/publish-opensearch-min-snapshots/src/run_sign.py null --sigtype .sig ...

[peterzhuamazon]

On the side note windows takes forever to complete the signing.
I think we need to remove signing from normal artifacts now and only reserve for core plugins.

[peterzhuamazon]

Even Even more:

• Only sign core plugins and ignore min artifacts opensearch-build-libraries#564

[peterzhuamazon]

One more PR:

• Publish Min Artifacts with new 7.3.4 libs #5180

[peterzhuamazon]

I thinks this time it is publishing:

Hi @reta could you help confirm from your side on the installation? Thanks!

[reta]

Hi @reta could you help confirm from your side on the installation? Thanks!

Works like a charm, @peterzhuamazon !

$ /home/opensearch-3.0.0-SNAPSHOT$ ./bin/opensearch-plugin install transport-nio
-> Installing transport-nio
-> Downloading transport-nio from opensearch
[=================================================] 100%   
-> Installed transport-nio with folder name transport-nio

$ ll plugins/transport-nio/
total 3.3M
-rw-r--r-- 1 user group  12K Nov 12 14:15 LICENSE.txt
-rw-r--r-- 1 user group 329K Nov 12 14:15 netty-buffer-4.1.114.Final.jar
-rw-r--r-- 1 user group 344K Nov 12 14:15 netty-codec-4.1.114.Final.jar
-rw-r--r-- 1 user group 653K Nov 12 14:15 netty-codec-http-4.1.114.Final.jar
-rw-r--r-- 1 user group 679K Nov 12 14:15 netty-common-4.1.114.Final.jar
-rw-r--r-- 1 user group 564K Nov 12 14:15 netty-handler-4.1.114.Final.jar
-rw-r--r-- 1 user group  37K Nov 12 14:15 netty-resolver-4.1.114.Final.jar
-rw-r--r-- 1 user group 510K Nov 12 14:15 netty-transport-4.1.114.Final.jar
-rw-r--r-- 1 user group  44K Nov 12 14:15 netty-transport-native-unix-common-4.1.114.Final.jar
-rw-r--r-- 1 user group  16K Nov 12 14:15 NOTICE.txt
-rw-r--r-- 1 user group  59K Nov 12 14:15 opensearch-nio-3.0.0-SNAPSHOT.jar
-rw-r--r-- 1 user group 1.9K Nov 12 14:15 plugin-descriptor.properties
-rw-r--r-- 1 user group 1.7K Nov 12 14:15 plugin-security.policy
-rw-r--r-- 1 user group  66K Nov 12 14:15 transport-nio-client-3.0.0-SNAPSHOT.jar