Skip to content

Commit

Permalink
Chore: update the confluent collector docs
Browse files Browse the repository at this point in the history
  • Loading branch information
@jcountsNR
jcountsNR committed Jan 24, 2024
1 parent 75a4c40 commit e1eb88c
Showing 1 changed file with 103 additions and 108 deletions.
211 changes: 103 additions & 108 deletions ...llector-configuration-examples/opentelemetry-collector-kafka-confluentcloud.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,9 @@ metaDescription: You can collect Kafka metrics from Confluent using the OpenTele

You can collect metrics about your Confluent Cloud-managed Kafka deployment with the OpenTelemetry Collector. The collector is a component of OpenTelemetry that collects, processes, and exports telemetry data to New Relic (or any observability backend).

Complete the steps below to collect Kafka metrics from Confluent using an OpenTelemetry collector running in docker.
This integration works by running a prometheus receiver configuration inside the OpenTelemetry collector, which scrapes [Confluent Cloud's metrics API](https://api.telemetry.confluent.cloud/docs/descriptors/datasets/cloud?_ga=2.183807142.1264186867.1705940186-6520871.1686857317&_gl=1*1te8jue*_ga*NjUyMDg3MS4xNjg2ODU3MzE3*_ga_D2D3EGKSGD*MTcwNjAzNzYwOS41Ni4wLjE3MDYwMzc2MDkuNjAuMC4w) and exports that data to New Relic.

Complete the steps below to collect Kafka metrics from Confluent and export them to New Relic.

<Steps>
<Step>
Expand All @@ -21,7 +23,8 @@ Complete the steps below to collect Kafka metrics from Confluent using an OpenTe

* You have a docker daemon running
* You have [Docker Compose](https://github.com/newrelic/newrelic-opentelemetry-examples/tree/main/other-examples/collector/confluentcloud) installed
* You have a [Confluent Cloud account](https://www.confluent.io/get-started/)
* You have a [Confluent Cloud account](https://www.confluent.io/get-started/)
* You have your [Confluent Cloud API key & secret](https://docs.confluent.io/confluent-cli/current/command-reference/api-key/confluent_api-key_create.html) available

</Step>

Expand All @@ -31,128 +34,120 @@ Complete the steps below to collect Kafka metrics from Confluent using an OpenTe
Download [New Relic's OpenTelemetry Examples repo](https://github.com/newrelic/newrelic-opentelemetry-examples)as this setup uses its example collector configuration. Once installed, open the [Confluent Cloud example](https://github.com/newrelic/newrelic-opentelemetry-examples/tree/main/other-examples/collector/confluentcloud) directory. For more information, you can check the `README` there as well.

</Step>
<Step>
## Add the authentication files

This example setup uses TLS to authenticate the request to Confluent Cloud. There are multiple methods to authenticate, so you should follow your company best practices and authentication methods.

* TLS/SSL requires you to create keys and certificates, create your own Certificate Authority (CA), and sign the certificate.
* Doing this should leave you with three files which need to be added to this directory.
* Those files are referenced in this example as the follwing files: `key.pem`, `cert.pem`, `ca.pem`.

<Callout variant="important">
For more information about TLS authentication with Confluent Cloud, check the [documentation on authenticating with TLS](https://docs.confluent.io/platform/current/kafka/authentication_ssl.html) as well as the [security tutorial](https://docs.confluent.io/platform/current/security/security_tutorial.html).

For dev/test Confluent environments, you can simplify this by using plain text authentication.

</Callout>

</Step>
<Step>
## Set environment variables and run the collector
Export the following variables or add them in a `.env` file, then run the `docker compose up` command.
* Set the API key & Secret variables for both Confluent Cloud and New relic in the `.env` file
* Set the `Cluster_ID` variable with the target kafka cluster ID
* (Optional) To monitor connectors or schema registry's managed by Confluent Cloud, you can un-comment the configuration in the `collector.yaml` file and set the corresponding ID in the `.env` file

```bash
# Open the confluent cloud example directory
cd newrelic-opentelemetry-examples/other-examples/collector/confluentcloud

# Set environment variables.
export NEW_RELIC_API_KEY=<YOUR_API_KEY>
export NEW_RELIC_OTLP_ENDPOINT=https://otlp.nr-data.net
export CLUSTER_ID=<YOUR_CLUSTER_ID>
export CLUSTER_API_KEY=<YOUR_CLUSTER_API_KEY>
export CLUSTER_API_SECRET=<YOUR_CLUSTER_API_SECRET>
export CLUSTER_BOOTSTRAP_SERVER=<YOUR_CLUSTER_BOOTSTRAP_SERVER>

# run the collector in docker
docker compose up
```

### Local Variable information

<table>
<thead>
<tr>
<th style={{ width: "200px"}}>
Variable
</th>
<th>
Description
</th>
<th>
Docs
</th>
</tr>
</thead>
<tbody>
<tr>
<td>
`NEW_RELIC_API_KEY`
</td>
<td>
New Relic Ingest API Key
</td>
<td>
[API Key docs](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/)
</td>
</tr>
<tr>
<td>
`NEW_RELIC_OTLP_ENDPOINT`
</td>
<td>
New Relic OTLP endpoint is https://otlp.nr-data.net:4318
</td>
<td>
[OTLP endpoint config docs](/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/get-started/opentelemetry-set-up-your-app/#review-settings)
</td>
</tr>
<tr>
<td>
`CLUSTER_ID`
</td>
<td>
ID of cluster from Confluent Cloud
</td>
<td>
Available in your Confluent cluster settings
</td>
</tr>
<tr>
<td>
`CONFLUENT_API_ID`
</td>
<td>
Cloud API key
</td>
<td>
[Cloud API key docs](https://docs.confluent.io/cloud/current/monitoring/metrics-api.html#metrics-quick-start)
</td>
</tr>
<tr>
<td>
`CONFLUENT_API_SECRET`
</td>
<td>
Cloud API secret
</td>
<td>
[Cloud API key docs](https://docs.confluent.io/cloud/current/monitoring/metrics-api.html#metrics-quick-start)
</td>
</tr>
<tr>
<td>
`CLUSTER_BOOTSTRAP_SERVER`
</td>
<td>
Bootstrap Server for cluster
</td>
<td>
Available in your cluster settings
</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th style={{ width: "200px"}}>
Variable
</th>
<th>
Description
</th>
<th>
Docs
</th>
</tr>
</thead>
<tbody>
<tr>
<td>
`NEW_RELIC_API_KEY`
</td>
<td>
New Relic Ingest API Key
</td>
<td>
[API Key docs](/docs/apis/intro-apis/new-relic-api-keys/)
</td>
</tr>
<tr>
<td>
`NEW_RELIC_OTLP_ENDPOINT`
</td>
<td>
Default US New Relic OTLP endpoint is https://otlp.nr-data.net:4318
</td>
<td>
[OTLP endpoint config docs](/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/get-started/opentelemetry-set-up-your-app/#review-settings)
</td>
</tr>
<tr>
<td>
`CLUSTER_ID`
</td>
<td>
ID of cluster from Confluent Cloud
</td>
<td>
[Docs for list cluster ID command](https://docs.confluent.io/confluent-cli/current/command-reference/kafka/cluster/confluent_kafka_cluster_list.html#description)
</td>
</tr>
<tr>
<td>
`CONFLUENT_API_KEY`
</td>
<td>
Cloud API key
</td>
<td>
[Cloud API key docs](https://docs.confluent.io/cloud/current/monitoring/metrics-api.html#metrics-quick-start)
</td>
</tr>
<tr>
<td>
`CONFLUENT_API_SECRET`
</td>
<td>
Cloud API secret
</td>
<td>
[Cloud API key docs](https://docs.confluent.io/cloud/current/monitoring/metrics-api.html#metrics-quick-start)
</td>
</tr>
<tr>
<td>
`CONNECTOR_ID`
</td>
<td>
(OPTIONAL) You can monitor your Confluent connectors by specifying the ID here
</td>
<td>
[Docs for list connector ID command](https://docs.confluent.io/confluent-cli/current/command-reference/connect/cluster/confluent_connect_cluster_list.html)
</td>
</tr>
<tr>
<td>
`SCHEMA_REGISTRY_ID`
</td>
<td>
(OPTIONAL) You can monitor your Confluent Schema Registry by specifying the ID here
</td>
<td>
[Docs for list connector ID command](https://docs.confluent.io/confluent-cli/current/command-reference/schema-registry/schema/confluent_schema-registry_schema_list.html)
</td>
</tr>
</tbody>
</table>

</Step>
<Step>

Expand Down

0 comments on commit e1eb88c

Please sign in to comment.