Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HPCC-32734 Add ElasticSearch metric sink configuration #19247

Merged
merged 1 commit into from
Nov 21, 2024
Merged

HPCC-32734 Add ElasticSearch metric sink configuration #19247

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 111 additions & 0 deletions helm/examples/metrics/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,3 +177,114 @@ HPCC metrics can be easily routed to Azure Insights for application level monito
11/5/2021, 9:02:00.000 PM prometheus esp_requests_active 0 {"app":"eclservices","namespace":"default","pod_name":"eclservices-778477d679-vgpj2"}
11/5/2021, 9:02:00.000 PM prometheus esp_requests_active 3 {"app":"eclservices","namespace":"default","pod_name":"eclservices-778477d679-vgpj2"}
```
### ElasticSearch Support
HPCC metrics can be routed to ElasticSearch for advanced metrics processing and
alerting. This process involves two requirements: enabling the ElasticSearch metric
sink, and configuring the index in ElasticSearch to receive the metrics.

Since the metrics configuration is common across all HPCC components, the ElasticSearch
sink will report metrics from all components to the same index. Therefore, the
index must be configured to receive metrics from all components.

#### Index Configuration
The index must be created in ElasticSearch before metrics can be reported to it. The name is passed
to the sink as a configuration setting. The index must be created with the following settings:

##### Dynamic Mapping
The index must be created with dynamic mapping enabled. Dynamic mapping allows framework metric
data types to be stored in the index in their native types. Without dynamic mapping, the ElasticSearch
default mapping does not properly map value to unsigned 64-bit integers.

To create an index with dynamic mapping, use the following object when creating the index:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this write up seems very useful.
Is the goal still to handle the creation of the indices programmatically in later versions of the sink?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently no plans to create the index from the sink. If that was something we added, there are several considerations that must be taken into account. These are

  • If the cluster is using the same index for all metric reporting, creating and configuring the index must be coordinated somehow. If each component creates their own, there still needs to be coordination.
  • The sink would also need to add mapping templates in order to map values to the correct type. This could still use wildcard dynamic templates, or could be specific to the metrics registered for each metric framework instance
  • If the sink creates the index, we need to worry about life cycle management.

I felt like just telling the sink what index to use allows the policy of the ElasticSearch instance to remain there instead of pushing it down to the sink.

If we decide to push that responsibility to the sink, we can add it as an improvement addressing the concerns from above

```code json
{
"mappings": {
"dynamic_templates": [
{
"hpcc_metric_count_to_unsigned_long": {
"match": "*_count",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this cause problems if other, non-HPCC, components are publishing values with these suffixes to elastic stack?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly, however only when publishing to the same index with these dynamic mapping settings. The json object is meant to be representative of how you'd add dynamic mapping. I will update the description to mention that _count can be set to what makes sense for the index taking any additional mappings into account.

"mapping": {
"type": "unsigned_long"
}
}
},
{
"hpcc_metric_gauge_to_unsigned_long": {
"match": "*_gauge",
"mapping": {
"type": "unsigned_long"
}
}
},
{
"hpcc_metric_histogram_to_histogram": {
"match": "*_histogram",
"mapping": {
"type": "histogram"
}
}
}
]
}
}
```

Note that there may be other means for adding the required dynamic mapping to the index.

The _match_ values above are representative of typical values. The actual values may
vary depending on the cluster configuration and shared use of the index across multiple clusters.
In all cases, the configuration of the sink must match that of the dynamic mappings in the index.

#### Enabling ElasticSearch Metrics Sink for Kubernetes
To enable reporting of metrics to ElasticSearch, add the metric configuration settings to
the helm chart.

The provided HPCC helm chart provides all global settings from its values.yml file to all components.
To enable metrics reporting, either include the metrics configuration as a permanent part of
your HPCC helm chart values.yml file, or add it as command line settings at chart installation.
To enable the ElasticSearch sink on the command line, use the following to add the ElasticSearch
settings:

```code
helm install mycluster ./hpcc -f <path>/elasticsearch_metrics.yml
```
An example _yml_ file can be found in the repository at helm/examples/metrics/elasticsearch_metrics.yml.
Make a copy and modify as needed for your installation.

##### Configuration Settings
The ElasticSearch sink defines the following settings:

* hostProtocol - The protocol used to connect to the ElasticSearch server. (default: https)
* hostName - The host name or IP address of the ElasticSearch server. (required)
* hostPort - The port number of the ElasticSearch server. (default: 9200)
* indexName - The name of the index to which metrics are reported. (required)
* countMetricSuffix - The suffix used to identify count metrics. (default: count)
* gaugeMetricSuffix - The suffix used to identify gauge metrics. (default: gauge)
* histogramMetricSuffix - The suffix used to identify histogram metrics. (default: histogram)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Does the elastic stack support mtls?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. HPCC-32968 was opened to add security configuration to the sink. It provides for TLS and basic auth. As time progresses I suspect that we will add additional authentication methods to the sink.


Standard periodic metric sink settings are also available.

#### Enabling ElasticSearch Metrics Sink for Bare Metal
To enable reporting of metrics to ElasticSearch, add the metric configuration settings to
the environment configuration file (enviroment.xml). These settings must be added manually
since there is no support in the config manager.

Add the following to the environment configuration file (note only the required
settings are shown):

```code xml

<Environment>
<Software>
<metrics name="mymetricsconfig">
<sinks name="myelasticsink" type="elastic">
<settings period="30" ignoreZeroMetrics="1">
<host name="<hostname>" port="<port>" protocol="http|htps"/>
<index name="<index>"/>
<settings/>
</sinks>
</metrics>
</Software>
</Environment>
```
See section above for additional settings that can be added to the ElasticSearch sink.
36 changes: 36 additions & 0 deletions helm/examples/metrics/elasticsearch_metrics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#
# Defines an elastic sink for reporting metrics to an ElasticSearch instance
# Settings:
# type - sink type (must be elastic for ElasticSearch support)
# name - name for the sink instance
# settings.countMetricSuffix - suffix for count metrics (default: count)
rpastrana marked this conversation as resolved.
Show resolved Hide resolved
# settings.gaugeMetricSuffix - suffix for gauge metrics (default: gauge)
# settings.histogramMetricSuffix - suffix for histogram metrics (default: histogram)
# settings.host - ElasticSearch host settings
# settings.host.protocol - protocol to use, http or https (default)
# settings.host.name - host name
# settings.host.port - port number (default 9200)
# settings.index - ElasticSearch index settings
# settings.index.name - index name
#
# If not overridden, the following suffixes are used by default:
# countMetricSuffix: count
# gaugeMetricSuffix: gauge
# histogramMetricSuffix: histogram

global:
metrics:
sinks:
- type: elastic
name: myelasticsink
settings:
countMetricSuffix: count
gaugeMetricSuffix: gauge
histogramMetricSuffix: histogram
host:
protocol: https
name: hostname
port: 9200
index:
name: hpccmetrics

65 changes: 61 additions & 4 deletions system/metrics/sinks/elastic/elasticSink.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
############################################################################## */

#include "elasticSink.hpp"

#include "nlohmann/json.hpp"

//including cpp-httplib single header file REST client
Expand Down Expand Up @@ -43,8 +42,66 @@ ElasticMetricSink::ElasticMetricSink(const char *name, const IPropertyTree *pSet
PeriodicMetricSink(name, "elastic", pSettingsTree)
{
ignoreZeroMetrics = pSettingsTree->getPropBool("@ignoreZeroMetrics", true);
pSettingsTree->getProp("@elasticHost", elasticHost);
pSettingsTree->getProp("@indexName", indexName);

StringBuffer hostName;
StringBuffer hostProtocol;
StringBuffer hostPort;

Owned<IPropertyTree> pHostConfigTree = pSettingsTree->getPropTree("host");
if (pHostConfigTree)
{
pHostConfigTree->getProp("@name", hostName);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor future point: If the tree is guaranteed to stay active then more common is to use queryProp() and avoid cloning the strings. In this case the code is only executed once, so it doesn't matter.

const char * protocol = pHostConfigTree->queryProp("@protocol");
if (!protocol)
   protocol = "https";


if (!pHostConfigTree->getProp("@protocol", hostProtocol))
{
hostProtocol.append("https");
}

if (!pHostConfigTree->getProp("@port", hostPort))
{
hostPort.append("9200");
}
}

if (!hostName.isEmpty() && !hostPort.isEmpty() && !hostProtocol.isEmpty())
{
elasticHostUrl.append(hostProtocol).append("://").append(hostName).append(":").append(hostPort);
}
else
{
WARNLOG("ElasticMetricSink: Host configuration missing or invalid");
}

Owned<IPropertyTree> pIndexConfigTree = pSettingsTree->getPropTree("index");
if (pIndexConfigTree)
{
pSettingsTree->getProp("@name", indexName);
}

if (indexName.isEmpty())
{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: We do not normally use {} if an if statement only has a single statement - here and in several other places in this file. It makes it slightly harder to read.

WARNLOG("ElasticMetricSink: Index configuration missing or invalid");
}


// Both a host url and an index name are required
configurationValid = !elasticHostUrl.isEmpty() && !indexName.isEmpty();

// Initialize standard suffixes
if (!pSettingsTree->getProp("@countMetricSuffix", countMetricSuffix))
{
countMetricSuffix.append("count");
}

if (!pSettingsTree->getProp("@gaugeMetricSuffix", gaugeMetricSuffix))
{
gaugeMetricSuffix.append("gauge");
}

if (!pSettingsTree->getProp("@histogramMetricSuffix", histogramMetricSuffix))
{
histogramMetricSuffix.append("histogram");
}
}


Expand All @@ -62,6 +119,6 @@ void ElasticMetricSink::doCollection()

void ElasticMetricSink::collectingHasStopped()
{
;

}

6 changes: 5 additions & 1 deletion system/metrics/sinks/elastic/elasticSink.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,5 +38,9 @@ class ELASTICSINK_API ElasticMetricSink : public hpccMetrics::PeriodicMetricSink
protected:
StringBuffer indexName;
bool ignoreZeroMetrics = false;
StringBuffer elasticHost;
StringBuffer elasticHostUrl;
StringBuffer countMetricSuffix;
StringBuffer gaugeMetricSuffix;
StringBuffer histogramMetricSuffix;
bool configurationValid = false;
};
Loading