[Remote Store] Add support to restrict creation & deletion if system repository and mutation of immutable settings of system repository #9839

psychbot · 2023-09-06T19:12:09Z

Description

Today with system repository being introduced as part of remote store we allow user to update all the settings of a repository. This can be catastrophic for remote store if a user midway updates the critical settings like location for FS repository all the indices will start failing as the new location will not have any of the historical information.

If a remote store repository is registered and being used against an index it will have to remain entact/unmodified(atleast some of the attributes of a repository) till the index lifecycle.

Related Issues

#8623
#9105

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-09-06T19:50:32Z

Compatibility status:

Checks if related components are compatible with change 402da4c

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-09-06T19:53:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24736/
CommitID: 73dc332
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-06T20:10:27Z

Compatibility status:

Checks if related components are compatible with change f82142d

Incompatible components

Incompatible components: [https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/neural-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-06T20:29:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24746/
CommitID: f82142d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-07T13:42:28Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24914/
CommitID: 2c7b244
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-07T13:46:31Z

Compatibility status:

Checks if related components are compatible with change 2c7b244

Incompatible components

Incompatible components: [https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/sql.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

CHANGELOG.md

server/src/main/java/org/opensearch/repositories/RepositoriesService.java

github-actions · 2023-09-11T08:33:37Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25277/
CommitID: 14f8f3a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-12T12:00:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25378/
CommitID: 68359ce
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-14T20:17:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25641/
CommitID: 92dcccc
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-14T20:27:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25642/
CommitID: d88a3ef
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-18T06:30:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25763/
CommitID: 265b0c4
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-18T07:55:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25768/
CommitID: dbf5fe6
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Bukhtawar

Have we evaluated https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/rest/action/admin/cluster/RestReloadSecureSettingsAction.java#L89
dynamic reload of repository. All repository already implement ReloadablePlugin interface

Signed-off-by: Dharmesh 💤 <[email protected]>

… response Signed-off-by: Dharmesh 💤 <[email protected]>

Signed-off-by: Dharmesh 💤 <[email protected]>

github-actions · 2023-10-04T06:43:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26839/
CommitID: 772ddd2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-04T07:37:06Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26846/
CommitID: 772ddd2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/cluster/metadata/RepositoriesMetadata.java

github-actions · 2023-10-04T10:10:05Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/26860/
CommitID: 772ddd2

github-actions · 2023-10-04T10:18:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.snapshots.CloneSnapshotIT.testCloneShallowSnapshotIndex

URL: https://build.ci.opensearch.org/job/gradle-check/26861/
CommitID: 4d6d1b9
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Dharmesh 💤 <[email protected]>

github-actions · 2023-10-04T12:35:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteLargeBlob

URL: https://build.ci.opensearch.org/job/gradle-check/26874/
CommitID: 402da4c
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-10-04T12:49:53Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-9839-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 699d235a7edad81be19dabbb57e2933ad4f1b6d6
# Push it to GitHub
git push --set-upstream origin backport/backport-9839-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9839-to-2.x.

…ion if system repository and mutation of immutable settings of system repository (opensearch-project#9839) --------- Signed-off-by: Dharmesh 💤 <[email protected]> (cherry picked from commit 699d235) Signed-off-by: Dharmesh 💤 <[email protected]>

…ion if system repository and mutation of immutable settings of system repository (#9839) (#10364) --------- (cherry picked from commit 699d235) Signed-off-by: Dharmesh 💤 <[email protected]>

…repository and mutation of immutable settings of system repository (opensearch-project#9839) --------- Signed-off-by: Dharmesh 💤 <[email protected]>

…repository and mutation of immutable settings of system repository (opensearch-project#9839) --------- Signed-off-by: Dharmesh 💤 <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

gbbafna reviewed Sep 7, 2023

View reviewed changes

server/src/main/java/org/opensearch/repositories/RepositoriesService.java Outdated Show resolved Hide resolved

psychbot force-pushed the system-repo-restricted-settings branch from 2c7b244 to 14f8f3a Compare September 11, 2023 08:12

psychbot changed the title ~~[Remote Store] Introducing concept of RestrictedSystemRepositorySettings for system repository~~ [Remote Store] Adding support to restrict creation & deletion if system repository and mutation of immutable settings of system repository Sep 12, 2023

psychbot force-pushed the system-repo-restricted-settings branch from 92dcccc to d88a3ef Compare September 14, 2023 19:38

psychbot force-pushed the system-repo-restricted-settings branch from 265b0c4 to dbf5fe6 Compare September 18, 2023 07:04

Bukhtawar reviewed Sep 19, 2023

View reviewed changes

psychbot force-pushed the system-repo-restricted-settings branch from dbf5fe6 to 0f16a6b Compare September 19, 2023 21:47

psychbot marked this pull request as ready for review September 19, 2023 21:47

psychbot requested review from reta, anasalkouz, andrross, CEHENKLE, dblock, setiah, kartg, kotwanikunal and mch2 as code owners September 19, 2023 21:47

psychbot added 6 commits October 4, 2023 11:27

Adding Integ Tests

d2fe439

Signed-off-by: Dharmesh 💤 <[email protected]>

Fixing one integ test and removing unnecessary cleanup code

8c94edf

Signed-off-by: Dharmesh 💤 <[email protected]>

Changes to not surface out system_repository in rest get repositories…

42ef348

… response Signed-off-by: Dharmesh 💤 <[email protected]>

Improving codecov

9e4908f

Signed-off-by: Dharmesh 💤 <[email protected]>

Fixing test

4af05ea

Signed-off-by: Dharmesh 💤 <[email protected]>

Updating message

772ddd2

Signed-off-by: Dharmesh 💤 <[email protected]>

psychbot force-pushed the system-repo-restricted-settings branch from 5a15c06 to 772ddd2 Compare October 4, 2023 05:57

gbbafna reviewed Oct 4, 2023

View reviewed changes

server/src/main/java/org/opensearch/cluster/metadata/RepositoriesMetadata.java Show resolved Hide resolved

gbbafna approved these changes Oct 4, 2023

View reviewed changes

Update CHANGELOG.md

402da4c

Signed-off-by: Dharmesh 💤 <[email protected]>

psychbot force-pushed the system-repo-restricted-settings branch from 4d6d1b9 to 402da4c Compare October 4, 2023 11:49

sachinpkale merged commit 699d235 into opensearch-project:main Oct 4, 2023
12 of 13 checks passed

sachinpkale added the backport 2.x Backport to 2.x branch label Oct 4, 2023

opensearch-trigger-bot bot added the backport-failed label Oct 4, 2023

rramachand21 added the v2.11.0 Issues and PRs related to version 2.11.0 label Oct 4, 2023

psychbot deleted the system-repo-restricted-settings branch October 4, 2023 16:28

psychbot mentioned this pull request Oct 4, 2023

[Backport 2.x][Remote Store] Add support to restrict creation & deletion if system repository and mutation of immutable settings of system repository #10364

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Store] Add support to restrict creation & deletion if system repository and mutation of immutable settings of system repository #9839

[Remote Store] Add support to restrict creation & deletion if system repository and mutation of immutable settings of system repository #9839

psychbot commented Sep 6, 2023 •

edited

Loading

github-actions bot commented Sep 6, 2023 •

edited

Loading

github-actions bot commented Sep 6, 2023

github-actions bot commented Sep 6, 2023

github-actions bot commented Sep 6, 2023

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 11, 2023

github-actions bot commented Sep 12, 2023

github-actions bot commented Sep 14, 2023

github-actions bot commented Sep 14, 2023

github-actions bot commented Sep 18, 2023

github-actions bot commented Sep 18, 2023

Bukhtawar left a comment

github-actions bot commented Oct 4, 2023

github-actions bot commented Oct 4, 2023

github-actions bot commented Oct 4, 2023

github-actions bot commented Oct 4, 2023

github-actions bot commented Oct 4, 2023

opensearch-trigger-bot bot commented Oct 4, 2023

[Remote Store] Add support to restrict creation & deletion if system repository and mutation of immutable settings of system repository #9839

[Remote Store] Add support to restrict creation & deletion if system repository and mutation of immutable settings of system repository #9839

Conversation

psychbot commented Sep 6, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Sep 6, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 6, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 7, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 7, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 11, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 12, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 14, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 14, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 18, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 18, 2023

Gradle Check (Jenkins) Run Completed with:

Bukhtawar left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 4, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 4, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 4, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 4, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 4, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Oct 4, 2023

psychbot commented Sep 6, 2023 •

edited

Loading

github-actions bot commented Sep 6, 2023 •

edited

Loading