Jaeger Query integration with Prometheus on OCP4

[fabry00]

Hi

we have configured Opentelemetry collector to send the metrics to Jaeger and Prometheus (deployed at cluster level in OCP - Configuring the monitoring stack) to scrape the metrics of the Opentelemry collector:

otel-collector.yaml

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679  

exporters:
  logging:
    loglevel: debug
  prometheus:
    endpoint: "0.0.0.0:8889"    
  jaeger:
    endpoint: http://jaeger.mynamespace.svc.cluster.local:14250
    tls:
      insecure: true

service:
  telemetry:
    logs:
      level: "debug"
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, prometheus]

ServiceMonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  ......
spec:
  endpoints:
  - interval: 10s
    port: metrics
    scheme: http
  - interval: 10s
    port: otel-metrics
    scheme: http
 ......
  selector:
    matchLabels:
      component: otel-collector

In the otel-collector I can see that Prometheus is scraping the logs:

2023-04-05T06:03:57.182Z debug [email protected]/accumulator.go:279 Accumulator collect called {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}

and I can query in OCP Observe the metrics with PromQL. e.g: otelcol_process_uptime

Now I would like to configure jaeger query to get the metrics from Prometheus:

Jaeger env variables:

env:
          - name: COLLECTOR_OTLP_ENABLED
            value: "true"
          - name: MEMORY_MAX_TRACES
            value: {{ .Values.traces.memory.max | quote }}
          - name: METRICS_STORAGE_TYPE
            value: "prometheus"
          - name: PROMETHEUS_SERVER_URL
            value: "https://thanos-querier.openshift-monitoring.svc:9091"
          - name: PROMETHEUS_TLS_ENABLED
            value: "true"
          - name: PROMETHEUS_TLS_SKIP_HOST_VERIFY
            value: "true"

What I am missing is how to configure the barer token for thanos authentication in Jaeger Query (https://thanos-querier.openshift-monitoring.svc:9091) because now I can see in the Jaeger logs:

"level":"error","ts":1680674889.211048,"caller":"app/http_handler.go:494","msg":"HTTP handler, Internal Server Error","error":"failed executing metrics query: client_error: client error: 403"

thank you in advance

[yurishkuro]

I don't think we have this as a configuration. There is a way to propagate bearer token from the UI requests all the way down to storage. I assume that means you need to run some reverse reverse proxy with auth that sets the token. I suggest searching for the original issue / PR where the mechanism was added, perhaps the authors can elaborate about the setup.

[albertteoh]

Perhaps I'm mistaken, but I thought this capability was added in #4295:

--prometheus.token-file=<path to file where your bearer token is stored>

It appears we're missing documentation for this. Issue created to address this: jaegertracing/documentation#624.

[fabry00]

@albertteoh Thank you for the answer
I tried to add this env variable to Jaeger deployment:

- name: PROMETHEUS_TOKEN_FILE
   value: "/etc/prometheus/token"

where /etc/prometheus/token contains the token for prometheus authentication (that is retrieved by the same secret that grafana uses to query prometheus so it has to work) but unfortunatelly still same issue: 403

Forgot to mention that I am using: jaegertracing/all-in-one:1.43

Thank you in advance

[albertteoh]

Ah then it's my mistake, @fabry00. That PR was merged 3 weeks ago, after jaegertracing/all-in-one:1.43 was tagged. I believe version 1.44 will be tagged next week so please try again with jaegertracing/all-in-one:1.44 after it's tagged, or you could also try using jaegertracing/all-in-one:latest in the meantime.

[albertteoh]

@fabry00 version 1.44 was just released. Please try again with jaegertracing/all-in-one:1.44.