Skip to content

Commit

Permalink
chore(docs): update pokeshop and kafka trigger docs (#3129)
Browse files Browse the repository at this point in the history
* wip - update pokeshop docs

* Updating Pokeshop + Kafka docs

* Apply suggestions from code review

Co-authored-by: Julianne Fermi <julianne@kubeshop.io>
Co-authored-by: Adnan Rahić <adnan@kubeshop.io>

---------

Co-authored-by: Julianne Fermi <julianne@kubeshop.io>
Co-authored-by: Adnan Rahić <adnan@kubeshop.io>
3 people authored Sep 8, 2023
1 parent 84198fc commit 44a1a34
Showing 12 changed files with 131 additions and 16 deletions.
Binary file added docs/docs/img/choose-trigger-0.13.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/docs/img/create-test-0.13.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
31 changes: 19 additions & 12 deletions docs/docs/live-examples/pokeshop/overview.md
Original file line number Diff line number Diff line change
@@ -2,7 +2,7 @@

As a testing ground, the team at Tracetest has implemented a sample instrumented API around the [PokeAPI](https://pokeapi.co/).

The idea is to have a microservice-divided system that behaves like a typical scenario by having async processes ([RabbitMQ](https://www.rabbitmq.com/)), cache layers ([Redis](https://redis.io/)), database storage ([Postgres](https://www.postgresql.org/)) and simple CRUD interfaces for Pokemons.
The idea is to have a microservice-divided system that behaves like a typical scenario by having async processes ([RabbitMQ](https://www.rabbitmq.com/) and [Kafka](https://kafka.apache.org/)), cache layers ([Redis](https://redis.io/)), database storage ([Postgres](https://www.postgresql.org/)) and simple CRUD interfaces for Pokemons.

With this, users can get familiar with the Tracetest tool by focusing on creating assertions, visualizing the trace and identifying the different data that comes from the Collector ([Jaeger](https://www.jaegertracing.io/)). Users will learn about basic instrumentation practices: what tools to use, what data to send, when, and what suggested standards need to be followed.

@@ -18,15 +18,17 @@ We have three use cases that use each component of this structure and that can b
- [Get Pokemon by ID](./use-cases/get-pokemon-by-id.md): Given a Pokemon ID, this endpoint returns the data of a Pokemon. If the same Pokemon was queried, the API will use its cache to return it.
- [List Pokemon](./use-cases/list-pokemon.md): Lists all Pokemons registered into Pokeshop.
- [Import Pokemon](./use-cases/import-pokemon.md): Given a Pokemon ID, this endpoint does an async process, going to PokeAPI to get Pokemon data and adding it to the database.
- [Import Pokemon from Stream](./use-cases/import-pokemon-from-stream.md): Listening to a Stream, this use case also does an async process, going to PokeAPI to get Pokemon data and adding it to the database.

## System Architecture

The system is divided into two components:

- an **API** that serves client requests,
- a **Worker** who deals with background processes.
- a **Queue Worker** who deals with background processes, receiving data from the API
- a **Stream Worker** who handles import events sent from a stream

The communication between the API and Worker is made using a `RabbitMQ` queue, and both services emit telemetry data to Jaeger and communicate with a Postgres database.
The communication between the API and Queue Worker is made using a `RabbitMQ` queue, and both services emit telemetry data to Jaeger and communicate with a Postgres database. Additionally, a Stream Worker listens to a `Kafka` stream to see if there is any import event sent on it to execute.

A diagram of the system structure can be seen here:

@@ -36,16 +38,21 @@ flowchart TD
B[(Postgres)]
C(Node.js API)
D(RabbitMQ)
E(Worker)
E(Queue Worker)
F(Jaeger)
C --> |IORedis| A
C --> |Sequelize| B
C --> |ampqlib| D
D --> |ampqlib| E
E --> |Sequelize| B
C --> |OpenTelemetry Node.js SDK| F
E --> |OpenTelemetry Node.js SDK| F
G(Kafka)
H(Stream Worker)
C --> A
C --> B
C --> D
D --> E
E --> B
C --> F
E --> F
G --> H
H --> B
H --> F
```

In our live tests, we are deploying into a single Kubernetes namespace, deployed via a [Helm chart](https://github.com/kubeshop/pokeshop/blob/master/docs/installing.md#run-on-a-kubernetes-cluster).
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# Pokeshop API - Import Pokemon from Stream

This use case showcases a more complex scenario involving an async process. Usually, when working with microservices, there are use cases where some of the processing needs to happen asynchronously. For example, when triggering a user notification, generating reports or processing a payment order. With this endpoint, we provide an example of how users can implement trace-based testing for such scenarios.

Here the process listens to a stream, and whenever an event is read from it, the following process is triggered:
```mermaid
sequenceDiagram
participant Stream as Kafka
participant Worker as Stream Worker
participant ExternalAPI as PokeAPI
participant Database as Postgres
Stream->>Worker: read "import" message
Worker->>ExternalAPI: get pokemon info
ExternalAPI-->>Worker: pokemon info
Worker->>Database: save pokemon
Database-->>Worker: pokemon saved
```

You can trigger this use case by sending a message to Kafka on the `pokemon` topic with the following message value:
```json
{
"id": 143
}
```

## Building a Test for This Scenario

Using Tracetest, we can [create a test](../../../web-ui/creating-tests.md) that will produce a message to Kafka on `pokemon` topic and validate the following properties:
- The worker should read the import task.
- PokeAPI should return a valid response.
- The database should respond with low latency (< 200ms).

### Traces

Running these tests for the first time will create a distributed trace like the image below, where you can see spans for the stream messaging, the PokeAPI (external API) call and database calls.

![](../images/import-pokemon-from-stream-trace.png)

### Assertions

With this trace, we can build [assertions](../../../concepts/assertions.md) with Tracetest and validate the API and Worker behavior:

- **A message was received from Kafka stream:**
![](../images/import-pokemon-from-stream-message-received.png)

- **Import Pokemon use case was triggered**:
- ![](../images/import-pokemon-from-stream-use-case-executed.png)

- **PokeAPI should return a valid response:**
![](../images/import-pokemon-from-stream-get-pokeapi.png)

- **The database should respond with low latency (< 200ms):**
![](../images/import-pokemon-from-stream-database-latency.png)

Now you can validate this entire use case.

### Test Definition

If you want to replicate this entire test with Tracetest, you can replicate these steps in the Web UI or using the CLI, saving the following test definition as the file `test-definition.yml` and running:

```sh
tracetest run test -f test-definition.yml
```

```yaml
type: Test
spec:
id: a97syfdkjad
name: Import a Pokemon reading a Stream
description: Import a Pokemon via Stream
trigger:
type: kafka
kafka:
brokerUrls:
- stream:9092
topic: pokemon
headers: []
messageKey: snorlax-key
messageValue: "{\"id\":143}"
specs:
- selector: span[tracetest.span.type="messaging" name="pokemon process" messaging.system="kafka" messaging.destination="pokemon" messaging.destination_kind="topic" messaging.operation="process"]
name: A message was received from Kafka stream
assertions:
- attr:messaging.system = "kafka"
- selector: span[tracetest.span.type="general" name="import pokemon"]
name: Import Pokemon use case was triggered
assertions:
- attr:name = "import pokemon"
- selector: span[tracetest.span.type="http" name="GET" http.method="GET"]
name: PokeAPI should return a valid response
assertions:
- attr:http.response.body = '{"name":"snorlax"}'
- attr:http.status_code = 200
- selector: span[tracetest.span.type="database"]
name: The database should respond with low latency (< 200ms)
assertions:
- attr:tracetest.span.duration <= 200ms

```
Original file line number Diff line number Diff line change
@@ -26,7 +26,7 @@ sequenceDiagram
```mermaid
sequenceDiagram
participant Queue as RabbitMQ
participant Worker as Worker
participant Worker as Queue Worker
participant ExternalAPI as PokeAPI
participant Database as Postgres
7 changes: 4 additions & 3 deletions docs/docs/web-ui/creating-tests.md
Original file line number Diff line number Diff line change
@@ -8,19 +8,20 @@ Click the **Create** button and select **Create New Test** in the drop down:

The "Create New Test" dialog appears:

![Create a Test](../img/create-test-0.11.png)
![Create a Test](../img/create-test-0.13.png)

The option to choose the kind of trigger to initiate the trace is presented:

- HTTP Request - Create a basic HTTP request.
- GRPC Request - Test and debug your GRPC request.
- cURL Command - Define your HTTP test via a cURL command.
- Postman Collection - Define your HTTP request via a Postman collection.
- TraceID - Define you test via a TraceID.
- TraceID - Define your test via a TraceID.
- Kafka - Test consumers with Kafka messages

Choose the trigger and click **Next**:

![Choose Trigger](../img/choose-trigger-0.11.png)
![Choose Trigger](../img/choose-trigger-0.13.png)

In this example, HTTP Request has been chosen.

5 changes: 5 additions & 0 deletions docs/sidebars.js
Original file line number Diff line number Diff line change
@@ -56,6 +56,11 @@ const sidebars = {
id: "live-examples/pokeshop/use-cases/import-pokemon",
label: "Import Pokemon",
},
{
type: "doc",
id: "live-examples/pokeshop/use-cases/import-pokemon-from-stream",
label: "Import Pokemon from Stream",
},
],
},
],

0 comments on commit 44a1a34

Please sign in to comment.