-
Notifications
You must be signed in to change notification settings - Fork 74
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs(examples+recipe): add pw synth mon recipe
- Loading branch information
1 parent
9f9bd95
commit 6e2fda8
Showing
28 changed files
with
705 additions
and
28 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
333 changes: 333 additions & 0 deletions
333
...xamples-tutorials/recipes/synthetic-monitoring-trace-based-playwright-tests.mdx
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,333 @@ | ||
| --- | ||
| id: synthetic-monitoring-trace-based-playwright-tests | ||
| title: Synthetic Monitoring with Trace-based Playwright Tests | ||
| description: Quickstart on how to create True End-To-End Trace-Based Tests with the Tracetest Playwright Engine Trigger and run them as Synthetic Monitors. | ||
| hide_table_of_contents: false | ||
| keywords: | ||
| - tracetest | ||
| - trace-based testing | ||
| - observability | ||
| - distributed tracing | ||
| - end-to-end testing | ||
| - tracetest | ||
| - playwright | ||
| - trace-based-testing | ||
| - synthetic monitoring | ||
| - synthetic testing | ||
| image: https://res.cloudinary.com/djwdcmwdz/image/upload/v1698686403/docs/Blog_Thumbnail_14_rsvkmo.jpg | ||
| --- | ||
|
|
||
| :::info Version Compatibility | ||
| The features described here are compatible with the [Tracetest CLI v1.4.1](https://github.com/kubeshop/tracetest/releases/tag/v1.4.1) and above. | ||
| ::: | ||
|
|
||
| :::note | ||
| [Check out the source code on GitHub here.](https://github.com/kubeshop/tracetest/tree/main/examples/tracetest-synthetic-monitoring) | ||
| ::: | ||
|
|
||
| [Tracetest](https://tracetest.io/) is a synthetic monitoring and testing tool based on [OpenTelemetry](https://opentelemetry.io/) that allows you to test distributed apps. You can use data from distributed traces generated by OpenTelemetry to validate and assert the functionality of your apps. | ||
|
|
||
| [Playwright](https://playwright.dev/) is an open-source automation framework developed by Microsoft that enables cross-browser automation for web applications. It provides a set of APIs and libraries for automating interactions with web browsers such as Chrome, Firefox, and Microsoft Edge. | ||
|
|
||
| ## Why is this important? | ||
|
|
||
| This recipe uses the Tracetest Playwright Engine trigger with Tracetest Monitors for synthetic monitoring. | ||
|
|
||
| With these two working together you can combine the power of end-to-end tests with trace-based testing to easily capture a full distributed trace from your OpenTelemetry instrumented front-end and back-end system, but also run them on a schedule with enabled alerting when tests fail. | ||
|
|
||
| Benefits of using traces as test specs alongside synthetic monitoring: | ||
|
|
||
| - Get faster MTTR for failing performance tests | ||
| - Assert against the Playwright test execution and the system under test | ||
| - Validate functionality of other parts of your system that may be broken, even when end-to-end tests are passing | ||
| - Create synthetic tests that run in defined intervals | ||
| - Get alerted when synthetic tests fail via a Webhook | ||
|
|
||
| ## Requirements | ||
|
|
||
| **Tracetest Account**: | ||
|
|
||
| - Sign up to [`app.tracetest.io`](https://app.tracetest.io) or follow the [get started](/getting-started/installation) docs. | ||
| - Create an [environment](/concepts/environments). | ||
| - Create an [environment token](/concepts/environment-tokens) with **admin role**. | ||
| - Copy the environment id to your clipboard. | ||
|
|
||
| **Docker**: Have [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) installed on your machine. | ||
|
|
||
| ## Run This Example | ||
|
|
||
| :::info View Playwright recipe | ||
| This recipe uses the [official Playwright Engine recipe](/examples-tutorials/recipes/running-tests-with-tracetest-playwright-engine). | ||
| ::: | ||
|
|
||
| The example below is provided as part of the Tracetest GitHub repo. You can download and run the example by following these steps: | ||
|
|
||
| ```bash | ||
| git clone https://github.com/kubeshop/tracetest | ||
| cd tracetest/examples/tracetest-synthetic-monitoring | ||
| ``` | ||
|
|
||
| Follow these instructions to run the example: | ||
|
|
||
| 1. Log into the [Tracetest app](https://app.tracetest.io/). | ||
| 2. Copy the `.env.template` file to `.env`. | ||
| 3. Fill out the [TRACETEST_API_TOKEN](/concepts/environment-tokens) and [TRACETEST_ENVIRONMENT_ID](/concepts/agent) details by editing your `.env` file. You can find these values in the Settings area for your environment. | ||
| 4. Run `docker compose run tracetest-apply`. | ||
| 5. Follow the link in the terminal with the results to view the Monitor run. | ||
|
|
||
| ## Project Structure | ||
|
|
||
| The project structure for running Tracetest Playwright Engine tests is as follows: | ||
|
|
||
| ```bash | ||
| .env.template | ||
| .gitignore | ||
| .Dockerfile | ||
| collector.config.yaml | ||
| docker-compose.yaml | ||
| /resources | ||
| apply.sh | ||
| datastore.yaml | ||
| import-pokemon.yaml | ||
| script.js | ||
| ``` | ||
|
|
||
| The [Pokeshop Demo App](/live-examples/pokeshop/overview) is a complete example of a distributed application using different back-end and front-end services. We will be launching it and running tests against it as part of this example. | ||
| The `docker-compose.yaml` file in the root directory of the quick start runs the Pokeshop Demo app, the OpenTelemetry Collector, Jaeger, and the [Tracetest Agent](/concepts/agent) setup. | ||
|
|
||
| The Tracetest resource definitions and scripts are defined under the `/resources` directory. The resources include tests and the tracing backend definition, while the scripts include the `apply.sh` script to apply the resources. | ||
|
|
||
| ## Provisioned Resources | ||
|
|
||
| The example provisions the following resources: | ||
|
|
||
| ### Import Pokemon Test | ||
|
|
||
| ```yaml title="resources/import-pokemon.yaml" | ||
| type: Test | ||
| spec: | ||
| id: import-pokemon | ||
| name: Import Pokemon | ||
| trigger: | ||
| type: playwrightengine | ||
| playwrightEngine: | ||
| target: http://api:8081 | ||
| script: ./script.js | ||
| method: importPokemon | ||
| specs: | ||
| - selector: span[tracetest.span.type="general" name="documentLoad"] | ||
| name: Document Load Should be fast | ||
| assertions: | ||
| - attr:tracetest.span.duration < 500ms | ||
| - selector: span[tracetest.span.type="http" http.scheme="http"] | ||
| name: All HTTP request should return 200 | ||
| assertions: | ||
| - attr:http.status_code = 200 | ||
| - selector: span[tracetest.span.type="messaging" name="queue.synchronizePokemon | ||
| process" messaging.system="rabbitmq" | ||
| messaging.destination="queue.synchronizePokemon" | ||
| messaging.operation="process"] | ||
| name: The worker should be processed | ||
| assertions: | ||
| - attr:tracetest.selected_spans.count = 1 | ||
| - selector: span[tracetest.span.type="database"] | ||
| name: "All Database Spans: Processing time is less than 100ms" | ||
| assertions: | ||
| - attr:tracetest.span.duration < 250ms | ||
| ``` | ||
| ### Playwright Script | ||
| ```javascript title="resources/script.js" | ||
| const { expect } = require("@playwright/test"); | ||
|
|
||
| async function importPokemon(page) { | ||
| expect(await page.getByText("Pokeshop")).toBeTruthy(); | ||
|
|
||
| await page.click("text=Import"); | ||
| await page.getByLabel("ID").fill("143"); | ||
|
|
||
| await Promise.all([ | ||
| page.waitForResponse((resp) => resp.url().includes("/pokemon/import") && resp.status() === 200), | ||
| page.getByRole("button", { name: "OK", exact: true }).click(), | ||
| ]); | ||
| } | ||
|
|
||
| module.exports = { importPokemon }; | ||
| ``` | ||
|
|
||
| ### Jaeger Tracing Backend | ||
|
|
||
| ```yaml title="resources/datastore.yaml" | ||
| type: DataStore | ||
| spec: | ||
| id: current | ||
| name: jaeger | ||
| type: jaeger | ||
| default: true | ||
| jaeger: | ||
| endpoint: jaeger:16685 | ||
| headers: | ||
| "": "" | ||
| tls: | ||
| insecure: true | ||
| ``` | ||
| ### The Apply Script | ||
| The apply script configures and provisions the resources in the Tracetest environment: | ||
| ```bash title="resources/apply.sh" | ||
| #!/bin/sh | ||
|
|
||
| set -e | ||
|
|
||
| TOKEN=$TRACETEST_API_KEY | ||
| ENVIRONMENT_ID=$TRACETEST_ENVIRONMENT_ID | ||
|
|
||
| apply() { | ||
| echo "Configuring Tracetest" | ||
| tracetest configure --token $TOKEN --environment $ENVIRONMENT_ID | ||
|
|
||
| echo "Applying Resources" | ||
| tracetest apply datastore -f /resources/datastore.yaml | ||
| tracetest apply test -f /resources/import-pokemon.yaml | ||
| tracetest apply monitor -f /resources/monitor.yaml | ||
| tracetest list monitor | ||
| } | ||
|
|
||
| apply | ||
| ``` | ||
|
|
||
| ## Setting the Environment Variables | ||
|
|
||
| Copy the `.env.template` file to `.env` and add the Tracetest API token and environment id to the `TRACETEST_API_TOKEN` and `TRACETEST_ENVIRONMENT_ID` variables. | ||
|
|
||
| ## Running the Full Example | ||
|
|
||
| Creating the resources is automated for you. You only need to run the following command: | ||
|
|
||
| ```bash | ||
| docker compose run tracetest-apply | ||
| ``` | ||
|
|
||
| This command will run the `apply.sh` script to provision the resources and create a synthetic monitor. | ||
|
|
||
| ## Viewing the Created Resources | ||
|
|
||
| The output from the Tracetest resource apply script should be visible in the console log after running the `apply` command. | ||
|
|
||
| ```bash title="Output" | ||
| [+] Running 2/2 | ||
| ✔ api Pulled 2.4s | ||
| ✔ worker Pulled 2.6s | ||
| [+] Creating 8/0 | ||
| ✔ Container tracetest-synthetic-monitoring-playwright-engine-queue-1 Running 0.0s | ||
| ✔ Container tracetest-synthetic-monitoring-playwright-engine-postgres-1 Running 0.0s | ||
| ✔ Container tracetest-synthetic-monitoring-playwright-engine-cache-1 Running 0.0s | ||
| ✔ Container tracetest-synthetic-monitoring-playwright-engine-otel-collector-1 Running 0.0s | ||
| ✔ Container tracetest-synthetic-monitoring-playwright-engine-jaeger-1 Running 0.0s | ||
| ✔ Container tracetest-synthetic-monitoring-playwright-engine-tracetest-agent-1 Running 0.0s | ||
| ✔ Container tracetest-synthetic-monitoring-playwright-engine-worker-1 Running 0.0s | ||
| ✔ Container tracetest-synthetic-monitoring-playwright-engine-api-1 Running 0.0s | ||
| [+] Running 3/3 | ||
| ✔ Container tracetest-synthetic-monitoring-playwright-engine-queue-1 Hea... 1.0s | ||
| ✔ Container tracetest-synthetic-monitoring-playwright-engine-postgres-1 Healthy 1.0s | ||
| ✔ Container tracetest-synthetic-monitoring-playwright-engine-cache-1 Hea... 1.0s | ||
| [+] Running 2/2 | ||
| ✔ api Pulled 1.8s | ||
| ✔ worker Pulled 1.9s | ||
| Configuring Tracetest | ||
| SUCCESS Successfully configured Tracetest CLI | ||
| Applying Resources | ||
| type: DataStore | ||
| spec: | ||
| id: current | ||
| name: jaeger | ||
| type: jaeger | ||
| default: true | ||
| createdAt: 2024-05-13T14:22:27.325353Z | ||
| jaeger: | ||
| endpoint: jaeger:16685 | ||
| headers: | ||
| "": "" | ||
| tls: | ||
| insecure: true | ||
|
|
||
| type: Test | ||
| spec: | ||
| id: import-pokemon | ||
| name: Import Pokemon | ||
| trigger: | ||
| type: playwrightengine | ||
| playwrightEngine: | ||
| target: http://api:8081 | ||
| script: "const { expect } = require(\"@playwright/test\");\n\nasync function addPokemon(page) {\n expect(await page.getByText(\"Pokeshop\")).toBeTruthy();\n\n await page.click(\"text=Add\");\n\n await page.getByLabel(\"Name\").fill(\"Charizard\");\n await page.getByLabel(\"Type\").fill(\"Flying\");\n await page\n .getByLabel(\"Image URL\")\n .fill(\"https://upload.wikimedia.org/wikipedia/en/1/1f/Pok%C3%A9mon_Charizard_art.png\");\n await page.getByRole(\"button\", { name: \"OK\", exact: true }).click();\n}\n\nasync function deletePokemon(page) {\n expect(await page.getByText(\"Pokeshop\")).toBeTruthy();\n\n await page.locator('[data-cy=\"pokemon-list\"]');\n await page.locator('[data-cy=\"pokemon-card\"]').first().click();\n await page.locator('[data-cy=\"pokemon-card\"] [data-cy=\"delete-pokemon-button\"]').first().click();\n}\n\nasync function importPokemon(page) {\n expect(await page.getByText(\"Pokeshop\")).toBeTruthy();\n\n await page.click(\"text=Import\");\n await page.getByLabel(\"ID\").fill(\"143\");\n\n await Promise.all([\n page.waitForResponse((resp) => resp.url().includes(\"/pokemon/import\") && resp.status() === 200),\n page.getByRole(\"button\", { name: \"OK\", exact: true }).click(),\n ]);\n}\n\nmodule.exports = { addPokemon, deletePokemon, importPokemon };\n" | ||
| method: importPokemon | ||
| specs: | ||
| - selector: span[tracetest.span.type="general" name="documentLoad"] | ||
| name: Document Load Should be fast | ||
| assertions: | ||
| - attr:tracetest.span.duration < 500ms | ||
| - selector: span[tracetest.span.type="http" http.scheme="http"] | ||
| name: All HTTP request should return 200 | ||
| assertions: | ||
| - attr:http.status_code = 200 | ||
| - selector: span[tracetest.span.type="messaging" name="queue.synchronizePokemon process" messaging.system="rabbitmq" messaging.destination="queue.synchronizePokemon" messaging.operation="process"] | ||
| name: The worker should be processed | ||
| assertions: | ||
| - attr:tracetest.selected_spans.count = 1 | ||
| - selector: span[tracetest.span.type="database"] | ||
| name: "All Database Spans: Processing time is less than 100ms" | ||
| assertions: | ||
| - attr:tracetest.span.duration < 250ms | ||
|
|
||
| type: Monitor | ||
| spec: | ||
| id: playwright-monitor | ||
| name: Playwright Synthetic Monitor | ||
| enabled: true | ||
| variableSetId: "" | ||
| tokenId: "" | ||
| tests: | ||
| - import-pokemon | ||
| schedule: | ||
| cron: "*/5 * * * *" | ||
| timeZone: Etc/UTC | ||
| alerts: | ||
| - id: slack-webhook-message | ||
| type: webhook | ||
| webhook: | ||
| body: "{\n \"text\": \"Monitor ${.Monitor.Name} has failed, follow the link to find the <${.URL}|results>\"\n}" | ||
| method: POST | ||
| url: <your-webhook-url> | ||
| headers: | ||
| - key: Content-Type | ||
| value: application/json | ||
| events: | ||
| - FAILED | ||
|
|
||
| ID NAME VERSION RUNS LAST RUN TIME LAST RUN STATE URL | ||
| ---------------------- --------------------------------- --------- ------ --------------------- ---------------- ------------------------------------------------------- | ||
| playwright-monitor Playwright Synthetic Monitor 1 0 https://api.tracetest.io/monitor/playwright-monitor-3 | ||
| ``` | ||
|
|
||
| ## Running Synthetic Monitoring | ||
|
|
||
| 1. Select `Monitors` in the [Tracetest](https://app.tracetest.io/) sidebar. | ||
| 2. You'll see the Monitor that was created in the `apply` script. Click "Run Now". | ||
|  | ||
| 3. Select an access token for your Monitor and save the configuration. | ||
|  | ||
|
|
||
| With this setup, your Monitor will trigger the Playwright test every 5 minutes. | ||
|
|
||
| ## What's Next? | ||
|
|
||
| After running the test, you can click the run link, update the assertions, and run the scripts once more. This flow enables complete a trace-based TDD flow. | ||
|
|
||
|  | ||
|
|
||
| ## Learn More | ||
|
|
||
| Please visit our [examples in GitHub](https://github.com/kubeshop/tracetest/tree/main/examples) and join our [Slack Community](https://dub.sh/tracetest-community) for more info! |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
12 changes: 12 additions & 0 deletions
12
examples/tracetest-synthetic-monitoring-github-actions/Dockerfile
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| FROM golang:alpine as builder | ||
| ENV GO111MODULE=on | ||
| RUN apk update && apk add --no-cache git | ||
| WORKDIR /app | ||
| COPY go.mod ./ | ||
| COPY go.sum ./ | ||
| RUN go mod download | ||
| COPY . . | ||
| RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o ./bin/main . | ||
| FROM scratch | ||
| COPY --from=builder /app/bin/main . | ||
| CMD ["./main"] |
Oops, something went wrong.