Skip to content

Commit

Permalink
chore(examples): updating elastic examples (#3963)
Browse files Browse the repository at this point in the history 
* chore(examples): updating elastic examples

* chore(examples): updating elastic examples

* updating pr comments
  • Loading branch information
@xoscar
xoscar authored Aug 14, 2024
1 parent 298aef1 commit c154c25
Show file tree
Hide file tree
Showing 24 changed files with 304 additions and 1,034 deletions.
278 changes: 36 additions & 242 deletions ...ocs/examples-tutorials/recipes/running-tracetest-with-elastic-opentelemetry.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,6 @@ keywords:
image: https://res.cloudinary.com/djwdcmwdz/image/upload/v1698686403/docs/Blog_Thumbnail_14_rsvkmo.jpg
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import CodeBlock from '@theme/CodeBlock';

:::note
[Check out the source code on GitHub here.](https://github.com/kubeshop/tracetest/tree/main/examples/tracetest-elasticapm-with-otel)
:::
Expand All @@ -36,19 +32,12 @@ import CodeBlock from '@theme/CodeBlock';

This is a simple quick start on how to configure a Node.js app to use OpenTelemetry with Elastic and Tracetest for enhancing your E2E and integration tests with trace-based testing. The infrastructure will use Elastic APM and Elasticsearch as the trace data store and OpenTelemetry Collector to receive traces from the Node.js app and send them to Elasticsearch.

```mdx-code-block
<Tabs groupId="running-tracetest-without-a-trace-data-store">
<TabItem value="Tracetest" label="Cloud-based Managed Tracetest" default>
```

## Prerequisites

**Tracetest Account**:

- Sign up to [`app.tracetest.io`](https://app.tracetest.io) or follow the [get started](/getting-started/installation) docs.
- Create an [environment](/concepts/environments).
- Create an [environment token](/concepts/environment-tokens).
- Have access to the environment's [agent API key](/configuration/agent).
- Have access to the environment's [agent API key](https://app.tracetest.io/retrieve-token).

**Docker**: Have [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) installed on your machine.

Expand All @@ -66,19 +55,17 @@ cd tracetest/examples/tracetest-elasticapm-with-otel
Follow these instructions to run the quick start:

1. Copy the `.env.template` file to `.env`.
2. Log into the [Tracetest app](https://app.tracetest.io/).
3. Fill out the [token](https://docs.tracetest.io/concepts/environment-tokens) and [API key](https://docs.tracetest.io/concepts/agent) details by editing your `.env` file. You can find these values in the Settings area for your environment.
4. Run `docker compose -f ./docker-compose.agent.yaml up -d`.
5. This example is configured to use the Elastic Tracing Backend. Ensure the environment you're using to run this example is configured to use the Elastic Tracing Backend by clicking on Settings, Tracing Backend, Elastic, Save. Or, use the CLI as explained below.
6. Run tests from the Tracetest Web UI by accessing the app with the URL `http://app:8080/`.
2. Fill out the [TRACETEST_TOKEN and ENVIRONMENT_ID](https://app.tracetest.io/retrieve-token) details by editing your `.env` file.
3. Run `docker compose run tracetest-run`.
4. Follow the links in the output to view the test results.

Follow along with the sections below for an in detail breakdown of what the example you just ran did and how it works.

## Project Structure

The project contains [Tracetest Agent](/getting-started/installation#install-the-tracetest-agent), [Elastic APM server](https://github.com/elastic/apm-server), [Elasticsearch](https://github.com/elastic/elasticsearch), [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector-contrib), and a Node.js app.

The `docker-compose.agent.yaml` file in the root directory of the quick start runs the Node.js app, Elasticsearch, Elastic APM server, OpenTelemetry Collector, and the [Tracetest Agent](/concepts/agent) setup.
The `docker-compose.yaml` file in the root directory of the quick start runs the Node.js app, Elasticsearch, Elastic APM server, OpenTelemetry Collector, and the [Tracetest Agent](/concepts/agent) setup.

## Configuring the Node.js App

Expand All @@ -87,107 +74,39 @@ The Node.js app is a simple Express app, contained in [the `app.js` file](https:
Configure the `.env` like shown below.

```bash
TRACETEST_API_KEY="<YOUR_TRACETEST_API_KEY>"
```
# Get the required information here: https://app.tracetest.io/retrieve-token

The OpenTelemetry tracing is contained in the `tracing.otel.grpc.js` or `tracing.otel.http.js` files. Traces will be sent to OpenTelemetry Collector and forwarded from there to Elastic.
TRACETEST_TOKEN="<YOUR_TRACETEST_TOKEN>"
TRACETEST_ENVIRONMENT_ID="<YOUR_ENV_ID>"

Choosing [the `tracing.otel.grpc.js` file](https://github.com/kubeshop/tracetest/blob/main/examples/tracetest-elasticapm-with-otel/tracing.otel.grpc.js) will send traces to OpenTelemetry Collector's `GRPC` endpoint.
# Password for the 'elastic' user (at least 6 characters)
ELASTIC_PASSWORD=changeme

Enabling the tracer is done by preloading the trace file. As seen in the `package.json`.
# Password for the 'kibana_system' user (at least 6 characters)
KIBANA_PASSWORD=changeme

```json
"scripts": {
"app-with-grpc-tracer": "node -r ./tracing.otel.grpc.js app.js",
},
```
# Version of Elastic products
STACK_VERSION=8.6.0

## Configuring Elastic
# Set the cluster name
CLUSTER_NAME=docker-cluster

Configure Elastic as a Tracing Backend:
# Set to 'basic' or 'trial' to automatically start the 30-day trial
LICENSE=basic

```yaml title=tracetest-tracing-backend.yaml
---
type: DataStore
spec:
id: current
name: elasticapm
type: elasticapm
elasticapm:
addresses:
- https://es01:9200
username: elastic
password: changeme
index: traces-apm-default
insecureSkipVerify: true
```
```bash
tracetest config -t <YOUR_API_TOKEN>
tracetest apply datastore -f ./tracetest-tracing-backend.yaml
```
# Port to expose Elasticsearch HTTP API to the host
ES_PORT=9200

## Run the Node.js App, Elasticsearch, Elastic APM, and OpenTelemetry Collector with Docker Compose
# Port to expose Kibana to the host
KIBANA_PORT=5601

The [`docker-compose.yaml` file](https://github.com/kubeshop/tracetest/blob/main/examples/tracetest-elasticapm-with-otel/docker-compose.agent.yaml) and [`Dockerfile`](https://github.com/kubeshop/tracetest/blob/main/examples/tracetest-elasticapm-with-otel/Dockerfile) are in the root directory contain the Node.js app.
# Increase or decrease based on the available host memory (in bytes)
MEM_LIMIT=1073741824

The [`docker-compose.agent.yaml` file](https://github.com/kubeshop/tracetest/blob/main/examples/tracetest-elasticapm-with-otel/docker-compose.agent.yaml) also contains the Tracetest Agent, OpenTelemetry Collector, Elasticsearch, Kibana, and Elastic APM server.

To start it, run this command:

```bash
docker compose -f ./docker-compose.agent.yaml up -d
# Project namespace (defaults to the current folder name if not set)
# COMPOSE_PROJECT_NAME=test
```

This will start the Node.js app, OpenTelemetry Collector, and Elastic APM server, and send the traces to Elasticsearch.

## Run Tracetest Tests

1. Open [Tracetest](https://app.tracetest.io/).
2. [Configure Elastic as a tracing backend](/configuration/connecting-to-data-stores/elasticapm) if you have not already as explained above.
3. Start creating tests! Make sure to use the `http://app:8080/` URL in your test creation.
4. To trigger tests in the CLI, first [install the CLI](/cli/cli-installation-reference), [configure it](/cli/configuring-your-cli), and [run a test](/cli/running-tests). From the root of the quick start directory, run:

```bash
tracetest configure -t <YOUR_API_TOKEN>
tracetest run test -f ./test-api.yaml
```

```mdx-code-block
</TabItem>
<TabItem value="Tracetest Core" label="Hobby Open-Source Tracetest Core">
```

## Prerequisites

You will need [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) installed on your machine to run this quick start app!

## Project Structure

- `docker-compose.yml` - Docker Compose file that starts the whole environment.
- Elastic stack single node cluster with Elasticsearch, Kibana and, the APM Server.
- OTel collector to support Tracetest.
- Tracetest instance.
- `app.js` - Sample NodeJS application listening on port 8080 and instrumented with Elastic Nodejs APM agent.

The project is built with Docker Compose.

### 1. Node.js App

The `docker-compose.yaml` contains a service called `app` for the Node.js app.

### 2. Tracetest

The `docker-compose.yaml` file, `tracetest-provision.yaml`, and `tracetest-config.yaml` contain configs for setting up Tracetest, Elastic APM, Elasticsearch, and Kibana.

### Docker Compose Network

All `services` in the `docker-compose.yaml` are on the same network and will be reachable by hostname from within other services. For example, `apm-server:8200` in the `elastic-apm-agent.js` will map to the `apm-server` service, where the port `8200` is the port where the Elastic APM accepts traces. And, `https://es01:9200` in the `tracetest-provision.yaml` will map to the `es01` service and port `9200` where Tracetest will fetch trace data from Elasticsearch.

## Node.js App

The Node.js app is a simple Express app, contained in the `app.js` file.

The OpenTelemetry tracing is contained in the `tracing.otel.grpc.js` or `tracing.otel.http.js` files. Traces will be sent to OpenTelemetry Collector and forwarded from there to Elastic.

Choosing [the `tracing.otel.grpc.js` file](https://github.com/kubeshop/tracetest/blob/main/examples/tracetest-elasticapm-with-otel/tracing.otel.grpc.js) will send traces to OpenTelemetry Collector's `GRPC` endpoint.
Expand All @@ -200,148 +119,23 @@ Enabling the tracer is done by preloading the trace file. As seen in the `packag
},
```

To start the server, run this command:

```bash
npm run app-with-grpc-tracer
```

As you can see the `Dockerfile` uses the command above.

```Dockerfile
FROM node:slim
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 8080
CMD [ "npm", "run", "app-with-grpc-tracer" ]
```

And, the `docker-compose.yaml` contains just one service for the Node.js app.

```yaml
app:
image: quick-start-nodejs-elastic-apm
hostname: app
build: .
ports:
- "8080:8080"
depends_on:
apm-server:
condition: service_started
```
## Tracetest
The `docker-compose.yaml` contains 6 services that configure Tracetest and Elastic.

- **Postgres** - Postgres is a prerequisite for Tracetest to work. It stores trace data when running the trace-based tests.
- [**Elasticsearch**](https://www.elastic.co/elasticsearch/) - Data store and search engine. (Also contains a `setup` service to configure Elasticsearch).
- [**Elastic APM**](https://www.elastic.co/observability/application-performance-monitoring) - Elastic application performance monitoring and traces.
- [**Kibana**](https://www.elastic.co/kibana/) - Kibana is a free and open user interface that lets you visualize your Elasticsearch data and navigate the Elastic Stack.
- [**OpenTelemetry Collector**](https://opentelemetry.io/docs/collector/) - A vendor-agnostic implementation of how to receive, process and export telemetry data.
- [**Tracetest**](https://tracetest.io/) - Trace-based testing that generates end-to-end tests automatically from traces.

They will start in this order:

1. Postgres & Setup
2. Elasticsearch
3. Kibana
4. Elastic APM Server
5. OpenTelemetry Collector
6. Tracetest

[View the entire `docker-compose.yaml` file here](https://github.com/kubeshop/tracetest/blob/main/examples/tracetest-elasticapm-with-otel/docker-compose.yaml).

Tracetest depends on Postgres and the OpenTelemetry Collector. The OpenTelemetry Collector depends on the Elastic APM Server that then depends on Elasticsearch and Kibana.

Both Tracetest and the OpenTelemetry Collector require config files to be loaded via a volume.

Elasticsearch, Kibana and Elastic APM use a `.env` file to load their config.

## Steps to Start the Environment

To start the environment run this command:

```bash
docker compose up -d
```

## Connecting Tracetest to Elastic APM

But how does Tracetest fetch traces?

Tracetest connects to Elastic APM to fetch trace data.

In the Web UI, open settings, and select Elastic APM.

![](https://res.cloudinary.com/djwdcmwdz/image/upload/v1674566041/Blogposts/Docs/screely-1674566018046_ci0st9.png)

Or, if you prefer using the CLI, you can use this file config.

```yaml
---
type: DataStore
spec:
name: elasticapm
type: elasticapm
default: true
elasticapm:
addresses:
- https://es01:9200
username: elastic
password: changeme
index: traces-apm-default
insecureSkipVerify: true
```

Proceed to run this command in the terminal and specify the file above.

```bash
tracetest apply datastore -f my/data-store/file/location.yaml
```

### Step-by-step Guide

Open `http://localhost:11633/` to configure the connection to Elasticsearch:

1. In Settings, configure Elastic APM as the Data Store.
2. Set `traces-apm-default` as the Index name.
3. Add the Address and set it to `https://es01:9200`.
4. Set the Username to `elastic` and password to `changeme`.
5. You will need to download the CA certificate from the Docker image and upload it to the config under "Upload CA file".
- The command to download the `ca.crt` file is:
`docker cp tracetest-elasticapm-with-elastic-agent-es01-1:/usr/share/elasticsearch/config/certs/ca/ca.crt .`
- Alternatively, you can skip CA certificate validation by setting the `Enable TLS but don't verify the certificate` option.
6. Test the connection and Save it, if all is successful.

Create a new test:

1. Use the "HTTP Request" option. Hit Next.
2. Name your test and add a description. Hit Next.
3. Configure the GET URL to be `http://app:8080` since the tests will be running in docker compose network. Hit Create.
4. Running the test should succeed.
## Run the Node.js App, Elasticsearch, Elastic APM, and OpenTelemetry Collector with Docker Compose

## Open Kibana
The [`docker-compose.yaml` file](https://github.com/kubeshop/tracetest/blob/main/examples/tracetest-elasticapm-with-otel/docker-compose.yaml) also contains the Tracetest Agent, OpenTelemetry Collector, Elasticsearch, Kibana, and Elastic APM server.

Open `https://localhost:5601` and login using `elastic:changeme` credentials. The credentials can be changed in the `.env` file. Navigate to APM (upper lefthand corner menu) -> Services and you should see the `tracetest` service with the rest of the details.
## Run Tracetest Tests

## Steps to Stop the Environment
To execute the tests, run this command:

```bash
docker compose down -v
# Remove the built app docker image
docker rmi quick-start-nodejs:latest
docker compose run tracetest-run
```

This will also delete the Docker image.

```mdx-code-block
</TabItem>
</Tabs>
```
This will:
1. Start the Node.js app, the OpenTelemetry Collector, and send the traces to Elastic APM.
2. Start the Tracetest Agent.
3. Configure the tracing backend and create tests in your environment.
4. Run the tests.

## Learn More

Expand Down
Loading

0 comments on commit c154c25

Please sign in to comment.