[BUG] Error Creating Detector [security_analytics_exception] normalizer [lowercase_keyword] not found for field

[jsun64128]

What is the bug?
When trying to create a security analytics detector I get an error "[security_analytics_exception] normalizer [lowercase_keyword] not found for field 'x'". x being _ Ex. 25-000873_J6yHbIsBON-lwPZfCF7R

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Go to 'Security Analytics'
2. Click on 'Create Detector'
3. Configuration is for all elastic beats data sources "elastic_windows*"
4. Log Type 'windows'
5. Select all Rules
6. All mappings automatically applied except for 'windows-message' which was manually mapped to 'event.original'
7. Alert Trigger used default settings did not add channel during testing
8. Click 'Create Detector'
9. See error
10. "[security_analytics_exception] normalizer [lowercase_keyword] not found for field"

What is the expected behavior?
When creating indexes we already apply normalizers like lowercase_keyword. I pasted the code we utilize in the additional context section.

What is your host/environment?

• OS: Windows 11
• Version: OpenSearch 2.8
• Plugins:

Do you have any screenshots?

Do you have any additional context?
When we create an index, we added the lowercase_keyword normalizer and set it as the normalizer for the message field
{ "mappings": { "properties": { "message": { "type": "keyword", "normalizer": "lowercase_keyword" } } }, "settings": { "analysis": { "normalizer": { "lowercase_keyword": { "type": "custom", "filter": [ "lowercase" ] } } } } }

[mvanderlee]

I think this is related to the fact that we don't have component templates but just regular index templates.
As soon as we tried enabling a Detector, this project injected it's own component template and completely overwrote our own indices.

This causes multiple issues:

1. Detector doesn't work
2. ISM policies no longer work, so our indices have been growing at a rapid pace without rolling over
3. Indices no longer have the correct mapping causing search errors.

In short, attempting to enable Detectors completely broke our OS cluster.

"You'll own nothing and be happy" comes to mind, even with our own data.

[mvanderlee]

Follow up, the broken indices and ISM policy were because use templates instead of index_templates. We now realize that templates were marked as legacy in 7.8.
Still, it'd be great if an error would be thrown preventing the creation of overriding index_templates. The error should tell the user that in order to use Detectors, they will have to update to index_templates, preferably with a link to documentation or even a migration guide.

We have now updated all our templates to index_templates and fixed the ISM policies.

Detectors still don't work
I've tested on a dockerized version of 2.11.0 and still get the same error shown in the bug description.
Steps to reproduce

1. Start a new 2.11.0 cluster - docker compose up see attached docker-compose.yml file
2. Create index template

   curl -X PUT -H 'Content-Type: application/json' --data "@schema.json" http://127.0.0.1:9200/_index_template/elastic_windows

3. Create index <elastic_windows-{now/d}-000001>

   curl -X PUT http://127.0.0.1:9200/%3Celastic_windows-%7Bnow%2Fd%7D-000001%3E

4. Create a Detector on the new index. Enable rule Sticky Key Like Backdoor Usage and for mapping select winlog.message -> event.original
5. Save and observe the error

   [security_analytics_exception] normalizer [lowercase_keyword] not found for field [31-000001_uSNfhYsB4Y4-buH1DJSG]
   

GitHub doesn't allow uploading of .yml files, so I added the .txt.
docker-compose.yml.txt
schema.json

[sbcd90]

hi @mvanderlee @jsun64128 , this is a known issue with using custom analyzers in doc-level monitors which the Security Analytics detectors internally create.
opensearch-project/alerting#961
We are looking to fix this bug in future releases.

As a workaround for this, can you please try to use standard analyzers that OpenSearch support?

[mvanderlee]

@sbcd90 the alerting team just nonchalantly closed it and from the message I'm inferring a "Won't fix attitude"
This would be a major shortcoming of OpenSearch. How will the security team handle this?

[michael-markevich]

I would upvote this. I just ran into the same issue, which seems to be utterly unexpected behaviour. I was lucky to spot Security Analytics module overwriting my index templates, and when I configured a different type of detector, it started complaining.

opensearch-node | Suppressed: java.lang.IllegalArgumentException: index template [.opensearch-sap-alias-mappings-index-template-my-] has index patterns [my-] matching patterns from existing templates [my_default] with patterns (my_default => [my-os-, my-app-*]) that have the same priority [0], multiple index templates may not match during index creation, please use a different priority

What is a suggested workaround or when we can expect a fix?
Even with the suggested setip, over time, this forcedly created alias mappings run out of sync and the whole detector stops working unexpectedly.

[duzvik]

Hit the same issue here.

[avidanov]

We encountered this issue with one of our customers. A possible workaround is to use analyzers instead of normalizers.