From ffee9251fdbfa151d5205f8b57163f17a8ff0e6a Mon Sep 17 00:00:00 2001 From: David Pequegnot Date: Fri, 5 Jul 2024 08:35:17 +0000 Subject: [PATCH] feat(profiling): add section about profiling with pyroscope --- docs/workshop.md | 148 ++++++++++++++++++++++++++++ scripts/download-pyroscope-agent.sh | 23 +++++ 2 files changed, 171 insertions(+) create mode 100755 scripts/download-pyroscope-agent.sh diff --git a/docs/workshop.md b/docs/workshop.md index 22f8f1f..59864ab 100644 --- a/docs/workshop.md +++ b/docs/workshop.md @@ -2095,3 +2095,151 @@ In this section, we will configure the Tempo data source to link our traces to t This was the last correlation dimension we wanted to show you! +## Profiling (Bonus) + +The OpenTelemetry project standardizes telemetry signals, particularly the logs, metrics, and traces we have seen so far. +However, just a few months ago, [they announced their work on a fourth signal: profiling](https://opentelemetry.io/blog/2024/profiling/). + +Profiling involves measuring the performance characteristics of your application, such as execution time, CPU utilization, or memory usage. It helps identify bottlenecks and optimize resource usage in your application. + +If you're familiar with Java, you may already know [async_profiler](https://github.com/async-profiler/async-profiler) for HotSpot JVMs. It defines itself as a "low overhead sampling profiler for Java." + +You may have also heard about eBPF, a technology embedded in the Linux kernel that allows running code in a sandbox within the kernel space. This technology is gaining traction in service meshes and in continuous profiling. + +Continuous profiling is an ongoing area of interest in the observability field, aimed at finding additional performance improvements. + +### (Grafana) Pyroscope + +Pyroscope was an open-source project for continuous profiling. It consists of a server that receives profiling samples, which can then be analyzed and displayed as a [flamegraph](https://www.brendangregg.com/flamegraphs.html). + +In the Java landscape, it offers a Java Agent based on *async_profiler*, compatible with other agents such as the OpenTelemetry agent. Phew! + +In 2023, Grafana acquired Pyroscope and merged it with its own solution, Phlare. Welcome to [Grafana Pyroscope](https://grafana.com/docs/pyroscope/latest/)! + +If you want to know more about Continuous Profiling and what it can bring to you, you may want to check out the [Grafana Pyroscope documentation](https://grafana.com/docs/pyroscope/latest/introduction/profiling/). + +### Objectives + +In this section, we aim to show you: +* What profiling is, +* What a flamegraph is, +* How it integrates in Grafana. + +### Enable Continuous Profiling + +#### Start the Pyroscope server + +๐Ÿ› ๏ธ We will use the `grafana/pyroscope` container image: we already defined a pyroscope service in our `compose.yml` file, but it is not yet enabled. You can start it by enabling the `continuous-profiling` profile: + +```bash +$ docker compose --profile=continuous-profiling up -d +``` + +โœ… It should start a new service on port `4040`. + +๐Ÿ› ๏ธ Go to Grafana Pyroscope dashboard UI on port `4040`: + +* You should see Pyroscope self-profiling and a new graph type: a flamegraph. +* On the top of the dashboard you can select the type of information you want to display: + * CPU profiling, + * Memory, + * Goroutines (itโ€™s a Go process), + * Etc. +* You can also filter your data by tagsโ€ฆ +* And there is a query language: you should be used to this by now! ๐Ÿ˜‰ + +#### Setup easypay-service for Continuous Profiling + +Letโ€™s use an agent again to profile our application. + +๐Ÿ› ๏ธ First, download the [agent](https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/java/). You can use the provided script to download it as `instrumentation/pyroscope.jar`: + +```bash +$ bash ./scripts/download-pyroscope-agent.sh + + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 +100 9782k 100 9782k 0 0 10.2M 0 --:--:-- --:--:-- --:--:-- 13.9M +Grafana Pyroscope agent downloaded successfully in ./scripts/../instrumentation +``` + +๐Ÿ“ Just like for traces, we should modify the `easypay-service/src/main/docker/Dockerfile` file: + +```Dockerfile +# ... +# Copy Java Agent into the container +COPY instrumentation/grafana-opentelemetry-java.jar /app/grafana-opentelemetry-java.jar + +# Copy pyroscope Java Agent +COPY instrumentation/pyroscope.jar /app/pyroscope.jar + +# Add the -javagent flag to setup the JVM to start with our Java Agent +ENTRYPOINT ["java", "-javaagent:/app/grafana-opentelemetry-java.jar", "-javaagent:/app/pyroscope.jar", "-cp","app:app/lib/*","com.worldline.easypay.EasypayServiceApplication"] +``` + +๐Ÿ“ Pyroscope can be configured with [environment variables](https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/java/#configuration-options). Letโ€™s do it directly in the `compose.yml` file: + +```yaml + easypay-service: + # ... + environment: + # ... + PYROSCOPE_APPLICATION_NAME: easypay-service # (1) + PYROSCOPE_FORMAT: jfr # (2) + PYROSCOPE_PROFILER_EVENT: wall # (3) + PYROSCOPE_PROFILER_LOCK: 10ms # (4) + PYROSCOPE_PROFILER_ALLOC: 512k # (5) + PYROSCOPE_LOG_LEVEL: debug + PYROSCOPE_SERVER_ADDRESS: http://pyroscope:4040 # (6) + # ... +``` +1. Define an application name (this will create the `service_name` label), +2. Set format: JFR allows to have multiple events to be recorded, +3. Type of event to profile: `wall` allows to record the time spent in methods. Other valid values are `itimer` and `cpu`. +4. Threshold to record lock events, +5. Threshord to record memory events, +6. Server address. + +๐Ÿ› ๏ธ Rebuild and redeploy `easypay-service`: + +```bash +$ docker compose build easypay-service +$ docker compose up -d easypay-service +``` + +โœ… Check logs for correct startup: + +```bash +docker compose logs -f easypay-service +``` + +You should see additional logs related to Pyroscope. + +๐Ÿ‘€ Go back to the Pyroscope dashboard (port `4040`): + +* In the top menu, you should be able to select the `easypay-service` application, +* Try to display wall profiling. + +### Grafana setup + +> aside positive +> +> We already configured the Pyroscope data source in Grafana. +> You can take a look at its configuration in the `Connections` > `Data sources` section. + +๐Ÿ‘€ Letโ€™s go to the Grafana `Explore` dashboard: + +* Select the `Pyroscope` data source, +* For the profiling type, select `wall` > `wall`, +* In the field next to the profiling type, enter a filter to get profiling of the `easypay-service`: `{service_name="easypay-service"}`, +* Select another profiling type (such as process CPU sampling). + +๐Ÿ› ๏ธ Generate some load with `k6`: + +```bash +$ k6 run -u 1 -d 5m k6/02-payment-smartbank.js +``` + +๐Ÿ‘€ Try to find what is taking the most time in the `ProcessPayment.processPayment` method: +* Use the sandwich view to focus! diff --git a/scripts/download-pyroscope-agent.sh b/scripts/download-pyroscope-agent.sh new file mode 100755 index 0000000..73e03be --- /dev/null +++ b/scripts/download-pyroscope-agent.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash + +PYROSCOPE_VERSION=v0.14.0 +PYROSCOPE_OTEL_VERSION=0.10.1.11 + +# Define the URL of the Grafana OpenTelemetry agent +PYROSCOPE_URL="https://github.com/grafana/pyroscope-java/releases/download/${PYROSCOPE_VERSION}/pyroscope.jar" +PYROSCOPE_OTEL_URL="https://repo1.maven.org/maven2/io/pyroscope/otel/${PYROSCOPE_OTEL_VERSION}/otel-${PYROSCOPE_OTEL_VERSION}.jar" + +# Define the path to the instrumentation directory +#INSTRUMENTATION_DIR="$(dirname "$(dirname "${BASH_SOURCE[0]}")")/instrumentation" + +INSTRUMENTATION_DIR="$(dirname "${BASH_SOURCE[0]}")/../instrumentation" + +# Create the instrumentation directory if it doesn't exist +mkdir -p "$INSTRUMENTATION_DIR" + +# Download the Grafana OpenTelemetry agent and save it in the instrumentation directory +curl -L "$PYROSCOPE_URL" -o "$INSTRUMENTATION_DIR/pyroscope.jar" +curl -L "$PYROSCOPE_OTEL_URL" -o "$INSTRUMENTATION_DIR/pyroscope-otel.jar" + +# Print a success message +echo "Grafana Pyroscope agent downloaded successfully in $INSTRUMENTATION_DIR" \ No newline at end of file