[BUG] Index field does not show up in the unmapped_index_fields array of mappings view API response

[amsiglan]

What is the bug?
Log index field not returned as part of the unmapped_index_fields array in the response for get mappings view API.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Create an index for google workspace log type. For that we can simply ingest below document

POST /gworkspace/_doc
{
  "google_workspace.drive.visibility": "shared_externally",
  "google_workspace.drive.target": "conference",
  "google_workspace.event": "conference"
}

2. Create below detection rule for log type gworkspace:

id: 25b9c01c-350d-4b95-bed1-836d04a4f324
logsource:
  product: gworkspace
title: my rule
description: Drive visibility checker
tags: []
falsepositives: []
level: critical
status: experimental
references: []
author: unknown
detection:
  condition: Selection_1
  Selection_1:
    google_workspace.drive.visibility|contains:
      - shared

3. Go to Create detector page and select data source gworkspace and log type gworkspace. The response from the view mappings API is

{
    "ok": true,
    "response": {
        "properties": {},
        "unmapped_index_fields": [
            "google_workspace.event",
            "google_workspace.drive.target"
        ],
        "unmapped_field_aliases": [
            "google_workspace.event.name",
            "google_workspace.admin.service.name",
            "google_workspace.admin.new_value"
        ]
    }
}

which means the field google_workspace.drive.visibility has been automatically mapped and is correct.
4. Abandon the create detector flow
5. Edit the detection rule from step 2 above and change it to

id: 25b9c01c-350d-4b95-bed1-836d04a4f324
logsource:
  product: gworkspace
title: my rule
description: Drive visibility checker
tags: []
falsepositives: []
level: critical
status: experimental
references: []
author: unknown
detection:
  condition: Selection_1
  Selection_1:
    google_workspace.drive-visibility|contains:
      - shared

6. Repeat step 3 and check the response of the view mappings API.

{
    "ok": true,
    "response": {
        "properties": {},
        "unmapped_index_fields": [
            "google_workspace.event",
            "google_workspace.drive.target"
        ],
        "unmapped_field_aliases": [
            "google_workspace.event.name",
            "google_workspace.admin.service.name",
            "google_workspace.admin.new_value",
            "google_workspace.drive-visibility"
        ]
    }
}

We see that now we have google_workspace.drive-visibility under unmapped_field_aliases but the log index field google_workspace.drive.visibility is not present under the unmapped_index_fields array.

The expected response is

{
    "ok": true,
    "response": {
        "properties": {},
        "unmapped_index_fields": [
            "google_workspace.event",
            "google_workspace.drive.target",
            "google_workspace.drive.visibility"
        ],
        "unmapped_field_aliases": [
            "google_workspace.event.name",
            "google_workspace.admin.service.name",
            "google_workspace.admin.new_value",
            "google_workspace.drive-visibility"
        ]
    }
}

What is your host/environment?

• Opensearch 2.10

Do you have any screenshots?
N/A

Do you have any additional context?
N/A

[sbcd90]

this is a known issue https://discuss.elastic.co/t/remove-unmap-a-field-alias/159258/2

[tomuk5]

Hi,

i'm also having the same issue with log index fields missing under unmapped_field_aliases and obsolete/non-used entries also under unmapped_field_aliases after misconfiguring some custom detection rules.

i'm lucky that im able to drop the datastream, but even after this the missing and obselete fields under unmapped_field_aliases from the Mappings view API still persist(!)

how can i flush these entries out?

Many Thanks

Running Opensearch 2.11

[amsiglan]

Hi @tomuk5 can you share the response of the getMappingsView API and the source index mapping that you are using?

[engechas]

Closing as resolved by #652