From 9ce81034d2d2e62bb60f3b7c4475f1c2ff667459 Mon Sep 17 00:00:00 2001 From: YaphetKG <45075777+YaphetKG@users.noreply.github.com> Date: Wed, 29 Nov 2023 09:02:52 -0500 Subject: [PATCH 1/5] Update monitoring.md --- docs/deployment-guide/monitoring.md | 37 +++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/docs/deployment-guide/monitoring.md b/docs/deployment-guide/monitoring.md index 4294e4d6..b0b8a000 100644 --- a/docs/deployment-guide/monitoring.md +++ b/docs/deployment-guide/monitoring.md @@ -12,3 +12,40 @@ T.B.A. (Tim Putnam to elaborate) # Telemetry Given the distributed nature of Biomedical Translator knowledge processing components, tracing the flow of queries through the system represents a challenge. Observability is the practice of measuring the state of a system by its various component outputs. [OpenTelemetry](https://opentelemetry.io/) is an open source observability framework. Applying elements of OpenTelemetry to Translator will help in the challenge of query auditing, for quality assurance or performance. An overview of OpenTelemetry concepts is provided [here](https://docs.google.com/presentation/d/1OjcE1gVhx8u9EvvHGn6h50otBKmpd-9HidlTNppXXy0/edit#slide=id.g27ee40efb83_0_3) and a small Translator demo of the concept using the Jaeger telemetry collector, is [here](https://github.com/TranslatorSRI/Jaeger-demo). + +### Telemetery Frequently Asked Questions +#### When I instrument my application should I trace incoming requests or outgoing ? + In a microservices environment such as Translator system, tracing both incoming and outgoing requests is key. Incoming tracing reveals user journeys, while outgoing tracing uncovers dependencies between services—both are crucial for comprehensive visibility and issue diagnosis. + +#### When tracing outgoing requests should it be logged if the request is bound to external services ? +Logging outgoing requests bound to external services can be essential for capturing communication details, aiding in troubleshooting, performance monitoring, and understanding dependencies outside your system. + +#### In Development enviroment where there is no provisioned Jaeger instance , what is the best way to test my OTEL implementation? + +To use a local Jaeger instance for testing your OpenTelemetry implementation, you can follow these general steps: +
    +
  1. Install Jaeger: Download and install Jaeger locally. You can use Docker to quickly set up a Jaeger instance: +```bash +docker run -d --name jaeger -p 16686:16686 -p 6831:6831/udp jaegertracing/all-in-one:latest +``` +This command pulls the latest Jaeger image and runs it, exposing the Jaeger UI on port 16686. +
    2. +
  2. Configure OpenTelemetry SDK: Use the OpenTelemetry SDK in your application to send traces to your local Jaeger instance. Configure your OpenTelemetry instrumentation to send data to localhost on the relevant Jaeger ports (usually 6831 for UDP and 16686 for HTTP). +
    3. +
  3. Instrument Your Code: Instrument your application using OpenTelemetry APIs to create traces. Ensure you've set up the instrumentation to export traces to your local Jaeger instance. +
    4. +
  4. Verify Traces: Execute your application's workflows or requests that should generate traces. Then, access the Jaeger UI at http://localhost:16686 in your browser to view the traces generated by your application.
    5. +
+ +Remember to adapt the OpenTelemetry SDK configuration in your code to use the address and ports where your local Jaeger instance is running. + +This approach provides a local environment for testing OpenTelemetry traces with Jaeger without needing a remote Jaeger instance. + +Once ready for deployment, in the ITRB managed enviroments Jaeger can be found at `jaeger.sri:6831`. This stays the same in all ITRB managed enviroments, pointing to an instance in the enviroment your application is deployed to. + +#### Where can I see my traces in ITRB enviroments once deployed? +Jaeger UI can be accessed in the following links: +* [CI Jaeger](https://translator-otel.ci.transltr.io/search) +* [Test Jaeger](https://translator-otel.test.transltr.io/search) +* [Prod Jaeger](https://translator-otel.transltr.io/search) + From 47d0a82d33d90e2e5fd02e14324cbc5ab87d7f93 Mon Sep 17 00:00:00 2001 From: YaphetKG <45075777+YaphetKG@users.noreply.github.com> Date: Wed, 29 Nov 2023 10:05:52 -0500 Subject: [PATCH 2/5] Update monitoring.md adding aragorn as an example. --- docs/deployment-guide/monitoring.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/deployment-guide/monitoring.md b/docs/deployment-guide/monitoring.md index b0b8a000..b3b06aec 100644 --- a/docs/deployment-guide/monitoring.md +++ b/docs/deployment-guide/monitoring.md @@ -17,6 +17,8 @@ Given the distributed nature of Biomedical Translator knowledge processing compo #### When I instrument my application should I trace incoming requests or outgoing ? In a microservices environment such as Translator system, tracing both incoming and outgoing requests is key. Incoming tracing reveals user journeys, while outgoing tracing uncovers dependencies between services—both are crucial for comprehensive visibility and issue diagnosis. + As an example ARAGORN, an ARA that recieves requests from the ARS and preforms subsequent requests to downstream components makes use of FastAPI instrumentation to trace incoming requests and httpx instrumentation for tracing outgoing requests. This [code snippet](https://github.com/ranking-agent/aragorn/blob/main/src/otel_config.py) shows how ARAGORN traces both incoming and outgoing requests. + #### When tracing outgoing requests should it be logged if the request is bound to external services ? Logging outgoing requests bound to external services can be essential for capturing communication details, aiding in troubleshooting, performance monitoring, and understanding dependencies outside your system. From 214dcfef8465c95c1f68b0f62549b43442c23a30 Mon Sep 17 00:00:00 2001 From: YaphetKG <45075777+YaphetKG@users.noreply.github.com> Date: Wed, 29 Nov 2023 11:33:21 -0500 Subject: [PATCH 3/5] Update monitoring.md --- docs/deployment-guide/monitoring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deployment-guide/monitoring.md b/docs/deployment-guide/monitoring.md index b3b06aec..05636f20 100644 --- a/docs/deployment-guide/monitoring.md +++ b/docs/deployment-guide/monitoring.md @@ -17,7 +17,7 @@ Given the distributed nature of Biomedical Translator knowledge processing compo #### When I instrument my application should I trace incoming requests or outgoing ? In a microservices environment such as Translator system, tracing both incoming and outgoing requests is key. Incoming tracing reveals user journeys, while outgoing tracing uncovers dependencies between services—both are crucial for comprehensive visibility and issue diagnosis. - As an example ARAGORN, an ARA that recieves requests from the ARS and preforms subsequent requests to downstream components makes use of FastAPI instrumentation to trace incoming requests and httpx instrumentation for tracing outgoing requests. This [code snippet](https://github.com/ranking-agent/aragorn/blob/main/src/otel_config.py) shows how ARAGORN traces both incoming and outgoing requests. + As an example ARAGORN, an ARA that recieves requests from the ARS and performs subsequent requests to downstream components makes use of FastAPI instrumentation to trace incoming requests and httpx instrumentation for tracing outgoing requests. This [code snippet](https://github.com/ranking-agent/aragorn/blob/main/src/otel_config.py) shows how ARAGORN traces both incoming and outgoing requests. #### When tracing outgoing requests should it be logged if the request is bound to external services ? Logging outgoing requests bound to external services can be essential for capturing communication details, aiding in troubleshooting, performance monitoring, and understanding dependencies outside your system. From bda52a7c299ed736827ae6487f2791efba757fa4 Mon Sep 17 00:00:00 2001 From: YaphetKG <45075777+YaphetKG@users.noreply.github.com> Date: Wed, 29 Nov 2023 12:47:25 -0500 Subject: [PATCH 4/5] Update monitoring.md grammar and typo edits --- docs/deployment-guide/monitoring.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/deployment-guide/monitoring.md b/docs/deployment-guide/monitoring.md index 05636f20..ddc15329 100644 --- a/docs/deployment-guide/monitoring.md +++ b/docs/deployment-guide/monitoring.md @@ -11,18 +11,18 @@ T.B.A. (Tim Putnam to elaborate) # Telemetry -Given the distributed nature of Biomedical Translator knowledge processing components, tracing the flow of queries through the system represents a challenge. Observability is the practice of measuring the state of a system by its various component outputs. [OpenTelemetry](https://opentelemetry.io/) is an open source observability framework. Applying elements of OpenTelemetry to Translator will help in the challenge of query auditing, for quality assurance or performance. An overview of OpenTelemetry concepts is provided [here](https://docs.google.com/presentation/d/1OjcE1gVhx8u9EvvHGn6h50otBKmpd-9HidlTNppXXy0/edit#slide=id.g27ee40efb83_0_3) and a small Translator demo of the concept using the Jaeger telemetry collector, is [here](https://github.com/TranslatorSRI/Jaeger-demo). +Given the distributed nature of Biomedical Translator knowledge processing components, tracing the flow of queries through the system represents a challenge. Observability is the practice of measuring the state of a system by its various component outputs. [OpenTelemetry](https://opentelemetry.io/) is an open-source observability framework. Applying elements of OpenTelemetry to Translator will help in the challenge of query auditing, for quality assurance or performance. An overview of OpenTelemetry concepts is provided [here](https://docs.google.com/presentation/d/1OjcE1gVhx8u9EvvHGn6h50otBKmpd-9HidlTNppXXy0/edit#slide=id.g27ee40efb83_0_3) and a small Translator demo of the concept using the Jaeger telemetry collector, is [here](https://github.com/TranslatorSRI/Jaeger-demo). -### Telemetery Frequently Asked Questions -#### When I instrument my application should I trace incoming requests or outgoing ? - In a microservices environment such as Translator system, tracing both incoming and outgoing requests is key. Incoming tracing reveals user journeys, while outgoing tracing uncovers dependencies between services—both are crucial for comprehensive visibility and issue diagnosis. +### Telemetry Frequently Asked Questions +#### When I instrument my application should I trace incoming requests or outgoing? + In a microservices environment such as the Translator system, tracing both incoming and outgoing requests is key. Incoming tracing reveals user journeys, while outgoing tracing uncovers dependencies between services—both are crucial for comprehensive visibility and issue diagnosis. - As an example ARAGORN, an ARA that recieves requests from the ARS and performs subsequent requests to downstream components makes use of FastAPI instrumentation to trace incoming requests and httpx instrumentation for tracing outgoing requests. This [code snippet](https://github.com/ranking-agent/aragorn/blob/main/src/otel_config.py) shows how ARAGORN traces both incoming and outgoing requests. + As an example ARAGORN, an ARA that receives requests from the ARS and performs subsequent requests to downstream components makes use of FastAPI instrumentation to trace incoming requests and httpx instrumentation for tracing outgoing requests. This [code snippet](https://github.com/ranking-agent/aragorn/blob/main/src/otel_config.py) shows how ARAGORN traces both incoming and outgoing requests. -#### When tracing outgoing requests should it be logged if the request is bound to external services ? +#### When tracing outgoing requests should it be logged if the request is bound to external services? Logging outgoing requests bound to external services can be essential for capturing communication details, aiding in troubleshooting, performance monitoring, and understanding dependencies outside your system. -#### In Development enviroment where there is no provisioned Jaeger instance , what is the best way to test my OTEL implementation? +#### In a Development environment with no provisioned Jaeger instance, what is the best way to test my OTEL implementation? To use a local Jaeger instance for testing your OpenTelemetry implementation, you can follow these general steps:
    @@ -34,7 +34,7 @@ This command pulls the latest Jaeger image and runs it, exposing the Jaeger UI o
  1. Configure OpenTelemetry SDK: Use the OpenTelemetry SDK in your application to send traces to your local Jaeger instance. Configure your OpenTelemetry instrumentation to send data to localhost on the relevant Jaeger ports (usually 6831 for UDP and 16686 for HTTP).
    2. -
  2. Instrument Your Code: Instrument your application using OpenTelemetry APIs to create traces. Ensure you've set up the instrumentation to export traces to your local Jaeger instance. +
  3. Instrument Your Code: Instrument your application using OpenTelemetry APIs to create traces. Please make sure you've set up the instrumentation to export traces to your local Jaeger instance.
  4. Verify Traces: Execute your application's workflows or requests that should generate traces. Then, access the Jaeger UI at http://localhost:16686 in your browser to view the traces generated by your application.
@@ -43,9 +43,9 @@ Remember to adapt the OpenTelemetry SDK configuration in your code to use the ad This approach provides a local environment for testing OpenTelemetry traces with Jaeger without needing a remote Jaeger instance. -Once ready for deployment, in the ITRB managed enviroments Jaeger can be found at `jaeger.sri:6831`. This stays the same in all ITRB managed enviroments, pointing to an instance in the enviroment your application is deployed to. +Once ready for deployment, in the ITRB-managed environments Jaeger can be found at `jaeger.sri:6831`. This stays the same in all ITRB-managed environments, pointing to an instance in the environment where your application is deployed. -#### Where can I see my traces in ITRB enviroments once deployed? +#### Where can I see my traces in ITRB environments once deployed? Jaeger UI can be accessed in the following links: * [CI Jaeger](https://translator-otel.ci.transltr.io/search) * [Test Jaeger](https://translator-otel.test.transltr.io/search) From df55e640e2c9ad2a563ea01f9f596ded48880e51 Mon Sep 17 00:00:00 2001 From: YaphetKG <45075777+YaphetKG@users.noreply.github.com> Date: Wed, 29 Nov 2023 13:44:09 -0500 Subject: [PATCH 5/5] Update monitoring.md Full mark-down for list and code block in local setup steps. --- docs/deployment-guide/monitoring.md | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/docs/deployment-guide/monitoring.md b/docs/deployment-guide/monitoring.md index ddc15329..4d7e4309 100644 --- a/docs/deployment-guide/monitoring.md +++ b/docs/deployment-guide/monitoring.md @@ -25,19 +25,18 @@ Logging outgoing requests bound to external services can be essential for captur #### In a Development environment with no provisioned Jaeger instance, what is the best way to test my OTEL implementation? To use a local Jaeger instance for testing your OpenTelemetry implementation, you can follow these general steps: -
    -
  1. Install Jaeger: Download and install Jaeger locally. You can use Docker to quickly set up a Jaeger instance: -```bash -docker run -d --name jaeger -p 16686:16686 -p 6831:6831/udp jaegertracing/all-in-one:latest -``` -This command pulls the latest Jaeger image and runs it, exposing the Jaeger UI on port 16686. -
    2. -
  2. Configure OpenTelemetry SDK: Use the OpenTelemetry SDK in your application to send traces to your local Jaeger instance. Configure your OpenTelemetry instrumentation to send data to localhost on the relevant Jaeger ports (usually 6831 for UDP and 16686 for HTTP). -
    3. -
  3. Instrument Your Code: Instrument your application using OpenTelemetry APIs to create traces. Please make sure you've set up the instrumentation to export traces to your local Jaeger instance. -
    4. -
  4. Verify Traces: Execute your application's workflows or requests that should generate traces. Then, access the Jaeger UI at http://localhost:16686 in your browser to view the traces generated by your application.
    5. -
+ +1. **Install Jaeger:** Download and install Jaeger locally. You can use Docker to quickly set up a Jaeger instance: + ```bash + docker run -d --name jaeger -p 16686:16686 -p 6831:6831/udp jaegertracing/all-in-one:latest + ``` + This command pulls the latest Jaeger image and runs it, exposing the Jaeger UI on port 16686. +2. **Configure OpenTelemetry SDK:** Use the OpenTelemetry SDK in your application to send traces to your local Jaeger instance. Configure your OpenTelemetry instrumentation to send data to localhost on the relevant Jaeger ports (usually 6831 for UDP and 16686 for HTTP). + +3. **Instrument Your Code:** Instrument your application using OpenTelemetry APIs to create traces. Please make sure you've set up the instrumentation to export traces to your local Jaeger instance. + +4. **Verify Traces:** Execute your application's workflows or requests that should generate traces. Then, access the Jaeger UI at http://localhost:16686 in your browser to view the traces generated by your application. + Remember to adapt the OpenTelemetry SDK configuration in your code to use the address and ports where your local Jaeger instance is running.