Register system index descriptors through SystemIndexPlugin.getSystemIndexDescriptors #1584

cwperks · 2024-06-21T19:13:24Z

Issue #, if available:

This PR registers the system indices in this plugin through the SystemIndexPlugin extension point in core. These indices will not be functionally different than they are today, its just a formal registration as a system index.

Related to: opensearch-project/security#4439

CheckList:

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…IndexDescriptors Signed-off-by: Craig Perkins <[email protected]>

cwperks · 2024-06-21T19:14:26Z

alerting/src/main/kotlin/org/opensearch/alerting/alerts/AlertIndices.kt

@@ -117,6 +117,8 @@ class AlertIndices(
        /** The in progress alert history index. */
        const val ALERT_INDEX = ".opendistro-alerting-alerts"

+        const val ALERT_CONFIG_INDEX = ".opendistro-alerting-config"


I added this here because I see it in the security repo, but I'm not able to find references to it in this repo. Where is this index declared and how is it used in the alerting plugin?

https://github.com/opensearch-project/common-utils/blob/889bdb8eb47b334eb1fb8c0e20465f0934403af0/src/main/kotlin/org/opensearch/commons/alerting/model/ScheduledJob.kt#L17

Thanks @bowenlan-amzn! I updated the PR to refer to this constant.

Signed-off-by: Craig Perkins <[email protected]>

cwperks · 2024-07-02T18:22:07Z

@bowenlan-amzn Can this be merged?

bowenlan-amzn · 2024-07-02T23:36:02Z

@cwperks I notice there's one test failure

REPRODUCE WITH: ./gradlew ':alerting:integTest' --tests "org.opensearch.alerting.DocumentMonitorRunnerIT.test execute monitor generates alerts and findings with per alert execution for actions" -Dtests.seed=E48D87D2E1142E72 -Dtests.security.manager=false -Dtests.locale=ta-IN -Dtests.timezone=Pacific/Niue -Druntime.java=17

org.opensearch.alerting.DocumentMonitorRunnerIT > test execute monitor generates alerts and findings with per alert execution for actions FAILED
    org.opensearch.client.WarningFailureException: method [POST], host [http://[::1]:43439], URI [/_refresh?expand_wildcards=open%2Chidden], status line [HTTP/1.1 200 OK]
    Warnings: [this request accesses system indices: [.opendistro-alerting-alert-history-2024.07.02-1], but in a future major version, direct access to system indices will be prevented by default, this request accesses system indices: [.opendistro-alerting-alerts], but in a future major version, direct access to system indices will be prevented by default, this request accesses system indices: [.opensearch-notifications-config], but in a future major version, direct access to system indices will be prevented by default]
    {"_shards":{"total":15,"successful":13,"failed":0}}
        at __randomizedtesting.SeedInfo.seed([E48D87D2E1142E72:1A6999FE631E4BFA]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:381)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:355)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:330)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.refreshAllIndices(OpenSearchRestTestCase.java:718)
        at app//org.opensearch.alerting.DocumentMonitorRunnerIT.test execute monitor generates alerts and findings with per alert execution for actions(DocumentMonitorRunnerIT.kt:687)

can you fix this? probably a common problem, not sure if the refreshAllIndices method needs to be updated
https://github.com/opensearch-project/OpenSearch/blob/a1faafed3ad88dd8f863e40766b7f035df0ed99b/test/framework/src/main/java/org/opensearch/test/rest/OpenSearchRestTestCase.java#L706-L716

cwperks · 2024-07-03T13:28:08Z

@bowenlan-amzn looking into it. I will get back shortly.

cwperks · 2024-07-03T14:09:59Z

~~@bowenlan-amzn I get the same test failure when I revert this change. That message is a warning.~~

I was getting a ktlint error. The test passes with this change reverted.

cwperks · 2024-07-03T14:33:21Z

FYI I see another error as well. Does this need to be addressed?

» ERROR][o.o.a.t.TransportDocLevelMonitorFanOutAction] [integTest-0] Optional finding callback failed
»  java.lang.IllegalStateException: failed to find action [org.opensearch.action.ActionType@de1e9751] to execute
»       at org.opensearch.client.node.NodeClient.transportAction(NodeClient.java:147) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:112) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:99) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:482) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.commons.alerting.AlertingPluginInterface.publishFinding(AlertingPluginInterface.kt:258) ~[common-utils-3.0.0.0-SNAPSHOT.jar:?]
»       at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.publishFinding(TransportDocLevelMonitorFanOutAction.kt:562) ~[opensearch-alerting-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
»       at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.createFindings(TransportDocLevelMonitorFanOutAction.kt:525) [opensearch-alerting-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
»       at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.access$createFindings(TransportDocLevelMonitorFanOutAction.kt:133) [opensearch-alerting-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
»       at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction$createFindings$1.invokeSuspend(TransportDocLevelMonitorFanOutAction.kt) [opensearch-alerting-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
»       at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
»       at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
»       at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
»       at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
»       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
» WARN ][o.o.n.i.NotificationConfigIndex] [integTest-0] notifications:getNotificationConfig - jl_ueJABFEYUG2kq6hJI not found; response:{"_index":".opensearch-notifications-config","_id":"jl_ueJABFEYUG2kq6hJI","found":false}

cwperks · 2024-07-03T15:09:30Z

@bowenlan-amzn I opened up a PR in core to address this: opensearch-project/OpenSearch#14635

bowenlan-amzn · 2024-07-03T15:40:23Z

@cwperks commented on Jul 3, 2024, 7:33 AM PDT:

FYI I see another error as well. Does this need to be addressed?

» ERROR][o.o.a.t.TransportDocLevelMonitorFanOutAction] [integTest-0] Optional finding callback failed
»  java.lang.IllegalStateException: failed to find action [org.opensearch.action.ActionType@de1e9751] to execute
»       at org.opensearch.client.node.NodeClient.transportAction(NodeClient.java:147) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:112) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:99) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:482) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.commons.alerting.AlertingPluginInterface.publishFinding(AlertingPluginInterface.kt:258) ~[common-utils-3.0.0.0-SNAPSHOT.jar:?]
»       at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.publishFinding(TransportDocLevelMonitorFanOutAction.kt:562) ~[opensearch-alerting-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
»       at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.createFindings(TransportDocLevelMonitorFanOutAction.kt:525) [opensearch-alerting-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
»       at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.access$createFindings(TransportDocLevelMonitorFanOutAction.kt:133) [opensearch-alerting-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
»       at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction$createFindings$1.invokeSuspend(TransportDocLevelMonitorFanOutAction.kt) [opensearch-alerting-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
»       at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
»       at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
»       at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
»       at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
»       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
» WARN ][o.o.n.i.NotificationConfigIndex] [integTest-0] notifications:getNotificationConfig - jl_ueJABFEYUG2kq6hJI not found; response:{"_index":".opensearch-notifications-config","_id":"jl_ueJABFEYUG2kq6hJI","found":false}

If this is not causing a test to fail, it should be fine for now.

…IndexDescriptors (#1584) (cherry picked from commit a5bdc23) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…IndexDescriptors (#1584)

Register system index descriptors through SystemIndexPlugin.getSystem…

737f052

…IndexDescriptors Signed-off-by: Craig Perkins <[email protected]>

cwperks requested review from lezzago, AWSHurneyt, sbcd90, eirsep, getsaurabh02, praveensameneni, bowenlan-amzn, rishabhmaurya, engechas, riysaxen-amzn and jowg-amazon as code owners June 21, 2024 19:13

cwperks commented Jun 21, 2024

View reviewed changes

cwperks added 5 commits June 21, 2024 15:24

Remove wildcard import

8aaea50

Signed-off-by: Craig Perkins <[email protected]>

Fix lint issue

68006a2

Signed-off-by: Craig Perkins <[email protected]>

Fix ktlint on JDK11

b095f39

Signed-off-by: Craig Perkins <[email protected]>

Use constants from common-utils

1784eff

Signed-off-by: Craig Perkins <[email protected]>

Remove unused import

5a65033

Signed-off-by: Craig Perkins <[email protected]>

shikharj05 approved these changes Jun 26, 2024

View reviewed changes

bowenlan-amzn approved these changes Jun 26, 2024

View reviewed changes

cwperks added the backport 2.x label Jun 26, 2024

cwperks mentioned this pull request Jul 3, 2024

Allow system index warning in OpenSearchRestTestCase.refreshAllIndices opensearch-project/OpenSearch#14635

Merged

3 tasks

AWSHurneyt approved these changes Jul 3, 2024

View reviewed changes

bowenlan-amzn merged commit a5bdc23 into opensearch-project:main Jul 3, 2024
16 of 19 checks passed

opensearch-trigger-bot bot mentioned this pull request Jul 3, 2024

[Backport 2.x] Register system index descriptors through SystemIndexPlugin.getSystemIndexDescriptors #1592

Closed

bowenlan-amzn mentioned this pull request Jul 3, 2024

[Backport 2.x] Register system index descriptors through SystemIndexPlugin.getSystemIndexDescriptors #1594

Merged

1 task

jowg-amazon pushed a commit that referenced this pull request Jul 17, 2024

Register system index descriptors through SystemIndexPlugin.getSystem…

f5b7142

…IndexDescriptors (#1584)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Register system index descriptors through SystemIndexPlugin.getSystemIndexDescriptors #1584

Register system index descriptors through SystemIndexPlugin.getSystemIndexDescriptors #1584

cwperks commented Jun 21, 2024

cwperks Jun 21, 2024

bowenlan-amzn Jun 22, 2024

cwperks Jun 24, 2024

cwperks commented Jul 2, 2024

bowenlan-amzn commented Jul 2, 2024

cwperks commented Jul 3, 2024

cwperks commented Jul 3, 2024 •

edited

Loading

cwperks commented Jul 3, 2024 •

edited

Loading

cwperks commented Jul 3, 2024

bowenlan-amzn commented Jul 3, 2024

Register system index descriptors through SystemIndexPlugin.getSystemIndexDescriptors #1584

Register system index descriptors through SystemIndexPlugin.getSystemIndexDescriptors #1584

Conversation

cwperks commented Jun 21, 2024

cwperks Jun 21, 2024

Choose a reason for hiding this comment

bowenlan-amzn Jun 22, 2024

Choose a reason for hiding this comment

cwperks Jun 24, 2024

Choose a reason for hiding this comment

cwperks commented Jul 2, 2024

bowenlan-amzn commented Jul 2, 2024

cwperks commented Jul 3, 2024

cwperks commented Jul 3, 2024 • edited Loading

cwperks commented Jul 3, 2024 • edited Loading

cwperks commented Jul 3, 2024

bowenlan-amzn commented Jul 3, 2024

cwperks commented Jul 3, 2024 •

edited

Loading

cwperks commented Jul 3, 2024 •

edited

Loading