[Bug]:

[rbaumgar]

What happened?

I am running a all-in-one jaeger instance in a kubernetes conatiner.
Try to connect to Prometheus/Thaos with https.

x509: certificate signed by unknown authority
tried to use PROMETHEUS_TLS_SKIP-HOST-VERIFY=TRUE and PROMETHEUS_TLS_INSECURE=TRUE
without any success.

How can I solve this problem?

{"level":"error","ts":1675529217.7555947,"caller":"app/http_handler.go:495","msg":"HTTP handler, Internal Server Error","error":"failed executing metrics query: Post \"https://thanos-querier.openshift-monitoring.svc.cluster.local:9091/api/v1/query_range\": x509: certificate signed by unknown authority","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\t/remote-source/jaeger/app/cmd/query/app/http_handler.go:495\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).metrics\n\t/remote-source/jaeger/app/cmd/query/app/http_handler.go:351\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).calls\n\t/remote-source/jaeger/app/cmd/query/app/http_handler.go:318\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2109\ngithub.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\t/remote-source/jaeger/deps/gomod/pkg/mod/github.com/opentracing-contrib/[email protected]/nethttp/server.go:154\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/li...

Steps to reproduce

deploy jaeger with operator
set environment
- name: METRICS_STORAGE_TYPE
value: prometheus
- name: PROMETHEUS_SERVER_URL
value: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091'

Expected behavior

true to get data into the new Service Performance Monitoring (SPM)

Relevant log output

curl works.
curl -k -v https://thanos-querier.openshift-monitoring.svc.cluster.local:9091/api/v1/query_range 
 Trying 172.30.30.101...
* TCP_NODELAY set
* Connected to thanos-querier.openshift-monitoring.svc.cluster.local (172.30.30.101) port 9091 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, [no content] (0):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [21 bytes data]
* TLSv1.3 (IN), TLS handshake, [no content] (0):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2762 bytes data]
* TLSv1.3 (IN), TLS handshake, [no content] (0):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, [no content] (0):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=thanos-querier.openshift-monitoring.svc
*  start date: Oct 14 11:40:54 2022 GMT
*  expire date: Oct 13 11:40:55 2024 GMT
*  issuer: CN=openshift-service-serving-signer@1631619628
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.

Screenshot

No response

Additional context

No response

Jaeger backend version

1.39

SDK

No response

Pipeline

No response

Stogage backend

No response

Operating system

No response

Deployment model

OpenShift 4.12, Kubernetes 1.25

Deployment configs

No response

[yurishkuro]

curl -k tells curl to skip validation of the server certificate. Since you're using self-signed cert, you can pass a corresponding CAcert to Jaeger. I don't know if Go tls lib supports skipping cert validation - even if it does we do not expose that option in Jaeger CLI flags.

[yurishkuro]

Alternatively, you can find a way to install your signing cert as a trusted CA in the OS (e.g. this is where Go looks for those: https://go.dev/src/crypto/x509/root_linux.go)

[rbaumgar]

@yurishkuro Now I have added to my environment
- name: PROMETHEUS_TLS_CA
value: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
- name: PROMETHEUS_TLS_ENABLED
value: 'TRUE'

-> client error: 401"

How do add the bearer token?

    - name: PROMETHEUS_TLS_KEY
      value: /run/secrets/kubernetes.io/serviceaccount/token

Does not help! (PROMETHEUS_TLS_KEY=Path to a TLS Private Key file, used to identify this process to the remote server(s))

{"level":"error","ts":1675572383.3949995,"caller":"app/http_handler.go:495","msg":"HTTP handler, Internal Server Error","error":"failed executing metrics query: client_error: client error: 401","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\t/remote-source/jaeger/app/cmd/query/app/http_handler.go:495\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).metrics\n\t/remote-source/jaeger/app/cmd/query/app/http_handler.go:351\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).errors\n\t/remote-source/jaeger/app/cmd/query/app/http_handler.go:326\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2109\ngithub.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\t/remote-source/jaeger/deps/gomod/pkg/mod/github.com/opentracing-contrib/[email protected]/nethttp/server.go:154\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2109\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.g...

[rbaumgar]

@yurishkuro BTW kubeconfig has a simple switch "--insecure-skip-tls-verify". So I believe it is not a problem to implement in go.