Add support for alerts based on Elasticsearch/Opensearch metrics

[markdboyd]

In order to give customers more visibility into performance issues on their Elasticsearch/Opensearch domains, we would like to add support for creating alerts based on a pre-defined set of metrics

Acceptance Criteria

• Testing demonstrates that an existing Elasticsearch service can be updated with alerts/metrics
• Testing demonstrates that a new Elasticsearch service can be created with alerts/metrics
• Testing demonstrates that an existing Opensearch service can be updated with alerts/metrics
• Testing demonstrates that a new Opensearch service can be created with alerts/metrics

---

Security considerations

There should be no security implications of this feature. We are giving customers the ability to be notified about changes in the performance of their Elasticsearch/Opensearch domains.

Implementation sketch

This is a rough implementation sketch that may be subject to revision

• Add code to support creating Cloudwatch alarms based on two metrics:
  • CPU Utilization
  • Average Search Latency
• Add code to support creating a custom SNS topic to handle notifications from the Cloudwatch alarm
  • Useful documentation: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/US_SetupSNS.html
• Add support for allowing user to specify the desired email recipient of notifications from the SNS topic
• Possible structure for new parameters for the service CLI:
  • `{ "enable_metrics": { "recipient": "[email protected]" } }