Unable to mount searchable snapshot for backing indices of .fleet-actions-results system data stream

[romain-chanu]

Elasticsearch Version

8.15.0

Installed Plugins

No response

Java Version

bundled

OS Version

ESS

Problem Description

Issue observed in the field

Users can edit the managed ILM policy .fleet-actions-results-ilm-policy to include a frozen phase.

When a backing index from the .fleet-actions-results enters the frozen phase, Elasticsearch is unable to mount the respective searchable snapshot.

Such error message is observed in the ILM explain result:

        "ilm": {
          "phase": "frozen",
          "failed_step": "mount-snapshot",
          "phase_definition": "{\"policy\":\".fleet-actions-results-ilm-policy\",\"phase_definition\":{\"min_age\":\"0d\",\"actions\":{\"searchable_snapshot\":{\"snapshot_repository\":\"found-snapshots\",\"force_merge_index\":true}}},\"version\":2,\"modified_date_in_millis\":1733296846017}",
          "snapshot_repository": "found-snapshots",
          "snapshot_name": "2024.12.04-.ds-.fleet-actions-results-2024.12.04-000001-.fleet-actions-results-ilm-policy-6i5yigyat6cymxkb7ub37g",
          "phase_time": "1733297097263",
          "step_info": "{\"type\":\"illegal_argument_exception\",\"reason\":\"requested system indices [.ds-.fleet-actions-results-2024.12.04-000001], but system indices can only be restored as part of a feature state\"}",
          "creation_date": "1733297037388",
          "step_time": "1733297097463",
          "action_time": "1733297038142",
          "is_auto_retryable_error": "true",
          "failed_step_retry_count": "3",
          "snapshot_index_name": ".ds-.fleet-actions-results-2024.12.04-000001",
          "action": "searchable_snapshot",
          "step": "ERROR"
        },

This opens the conversation around searchable snapshots and system data streams/indices (part of feature states) and whether we should even allow users to configure cold/frozen phases for ILM policies that are potentially attached to these.

Could the team review this? Do we consider this as an issue and do we want to allow / restrict such use case?

Thank you team 🙏

Steps to Reproduce

1. Create an ESS cluster with 1 hot node, 1 frozen node, one Integrations Server

2. Create a role to access .fleet* indices:

POST _security/role/fleet_superuser
{
  "indices": [
    {
      "names": [
        ".fleet*"
      ],
      "privileges": [
        "all"
      ],
      "allow_restricted_indices": true
    }
  ]
}

3. Create a user with that role:

POST _security/user/temp_user
{
  "password": "temp_password",
  "roles": [
    "superuser",
    "fleet_superuser"
  ]
}

4. Login to Kibana and edit the .fleet-actions-results-ilm-policy ILM policy to include a frozen phase (feel free to modify the rollover conditions and ILM poll interval for testing purposes)

5. Deploy an Elastic Agent with the Osquery Manager integration.

6. Run a few live Osquery queries so that documents are ingested in the .fleet-actions-results data stream

7. Once the initial backing index has rolled over and has entered the frozen phase, check the ILM explain for the same index:

        "ilm": {
          "phase": "frozen",
          "failed_step": "mount-snapshot",
          "phase_definition": "{\"policy\":\".fleet-actions-results-ilm-policy\",\"phase_definition\":{\"min_age\":\"0d\",\"actions\":{\"searchable_snapshot\":{\"snapshot_repository\":\"found-snapshots\",\"force_merge_index\":true}}},\"version\":2,\"modified_date_in_millis\":1733296846017}",
          "snapshot_repository": "found-snapshots",
          "snapshot_name": "2024.12.04-.ds-.fleet-actions-results-2024.12.04-000001-.fleet-actions-results-ilm-policy-6i5yigyat6cymxkb7ub37g",
          "phase_time": "1733297097263",
          "step_info": "{\"type\":\"illegal_argument_exception\",\"reason\":\"requested system indices [.ds-.fleet-actions-results-2024.12.04-000001], but system indices can only be restored as part of a feature state\"}",
          "creation_date": "1733297037388",
          "step_time": "1733297097463",
          "action_time": "1733297038142",
          "is_auto_retryable_error": "true",
          "failed_step_retry_count": "3",
          "snapshot_index_name": ".ds-.fleet-actions-results-2024.12.04-000001",
          "action": "searchable_snapshot",
          "step": "ERROR"
        },

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)