SEVERE: Failed to export spans. The request could not be executed. Full error message: Failed to connect to http://simplest-collector:4317/:4317

[jcarranzan]

Hi, I've configured 2 simplest yamls to deploy on Openshift with my application.

• My jaeger yml :

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: my-jaeger

• My collector yml:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: simplest
  namespace: opentelemetry-project
spec:
  mode: deployment
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
    exporters:
      jaeger:
        endpoint: my-jaeger-collector-headless.opentelemetry-project.svc:14250
        tls:
          ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt"
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [jaeger]

My application code to export the spans :

  String jaegerEndpoint= "http://my-jaeger-collector-opentelemetry-project.apps.ocp12.my.cluster";  **//What would be the correct endpoint to my jaeger instance on openshift?**
    Resource serviceNameResource = Resource.getDefault()
      .merge(Resource.create(Attributes.of(ResourceAttributes.SERVICE_NAME, "test-traced-service")));
    // Export traces to Jaeger over OTLP
    OtlpGrpcSpanExporter jaegerOtlpExporter =
      OtlpGrpcSpanExporter.builder()
        .setEndpoint(jaegerEndpoint)
        .setTimeout(30, TimeUnit.SECONDS)
        .build();


    // Set to process the spans by the Jaeger Exporter
    SdkTracerProvider tracerProvider =
      SdkTracerProvider.builder()
        .addSpanProcessor(BatchSpanProcessor.builder(jaegerOtlpExporter).build())
        .setResource(Resource.getDefault().merge(serviceNameResource))
        .build();

    OpenTelemetrySdk openTelemetry =
      OpenTelemetrySdk.builder().setTracerProvider(tracerProvider).build();

    // it's always a good idea to shut down the SDK cleanly at JVM exit.
    Runtime.getRuntime().addShutdownHook(new Thread(tracerProvider::close));

    vertx = Vertx.vertx( new VertxOptions()
                           .setTracingOptions(new OpenTelemetryOptions(openTelemetry)));

    Tracer tracer = openTelemetry.getTracer("io.vertx");

    Span span = tracer.spanBuilder("another span use case").startSpan();
    span.setAttribute("attribute3", "I am the attribute3");
    span.addEvent("Event 3");
    Thread.sleep(800);
    span.addEvent("Event 4");
    span.end();

My logs :

2023/04/10 10:23:36 provider.go:129: Defaulting client-id to system:serviceaccount:opentelemetry-project:my-jaeger-ui-proxy
2023/04/10 10:23:36 provider.go:134: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token
2023/04/10 10:23:36 oauthproxy.go:203: mapping path "/" => upstream "http://localhost:16686/"
2023/04/10 10:23:36 oauthproxy.go:230: OAuthProxy configured for  Client ID: system:serviceaccount:opentelemetry-project:my-jaeger-ui-proxy
2023/04/10 10:23:36 oauthproxy.go:240: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> samesite: refresh:disabled
2023/04/10 10:23:36 http.go:61: HTTP: listening on 127.0.0.1:4180
2023/04/10 10:23:36 http.go:107: HTTPS: listening on [::]:8443
I0410 10:23:36.015642       1 dynamic_serving_content.go:130] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key

2023-04-10T10:23:41.499Z	info	service/telemetry.go:110	Setting up own telemetry...
2023-04-10T10:23:41.499Z	info	service/telemetry.go:140	Serving Prometheus metrics	{"address": ":8888", "level": "basic"}
2023-04-10T10:23:41.500Z	info	service/service.go:89	Starting otelcol...	{"Version": "0.63.1", "NumCPU": 4}
2023-04-10T10:23:41.500Z	info	extensions/extensions.go:42	Starting extensions...
2023-04-10T10:23:41.500Z	info	pipelines/pipelines.go:74	Starting exporters...
2023-04-10T10:23:41.500Z	info	pipelines/pipelines.go:78	Exporter is starting...	{"kind": "exporter", "data_type": "traces", "name": "jaeger"}
2023-04-10T10:23:41.500Z	info	[email protected]/exporter.go:185	State of the connection with the Jaeger Collector backend	{"kind": "exporter", "data_type": "traces", "name": "jaeger", "state": "IDLE"}
2023-04-10T10:23:41.500Z	info	pipelines/pipelines.go:82	Exporter started.	{"kind": "exporter", "data_type": "traces", "name": "jaeger"}
2023-04-10T10:23:41.500Z	info	pipelines/pipelines.go:86	Starting processors...
2023-04-10T10:23:41.500Z	info	pipelines/pipelines.go:98	Starting receivers...
2023-04-10T10:23:41.500Z	info	pipelines/pipelines.go:102	Receiver is starting...	{"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2023-04-10T10:23:41.500Z	info	otlpreceiver/otlp.go:71	Starting GRPC server	{"kind": "receiver", "name": "otlp", "pipeline": "traces", "endpoint": "0.0.0.0:4317"}
2023-04-10T10:23:41.500Z	info	otlpreceiver/otlp.go:89	Starting HTTP server	{"kind": "receiver", "name": "otlp", "pipeline": "traces", "endpoint": "0.0.0.0:4318"}
2023-04-10T10:23:41.500Z	info	pipelines/pipelines.go:106	Receiver started.	{"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2023-04-10T10:23:41.500Z	info	service/service.go:106	Everything is ready. Begin running and processing data.
2023-04-10T10:23:42.500Z	info	[email protected]/exporter.go:185	State of the connection with the Jaeger Collector backend	{"kind": "exporter", "data_type": "traces", "name": "jaeger", "state": "READY"}

And the error that I got:
SEVERE: Failed to export spans. The request could not be executed. Full error message: Failed to connect to localhost/0:0:0:0:0:0:0:1:4317

Thanks!

[jcarranzan]

Also, I commented here and you can see more info from this thread:
open-telemetry/opentelemetry-operator#635 (comment)

[albertteoh]

Looks like a routing problem.

Perhaps you can try the following to confirm the service URL: https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/. If you found a valid service URL to the OTEL collector, try telneting to that URL on port 4317.

[jcarranzan]

oc get svc
NAME                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                    AGE
my-jaeger-agent                 ClusterIP   None             <none>        5775/UDP,5778/TCP,6831/UDP,6832/UDP                        112s
my-jaeger-collector             ClusterIP   ipclusternumber   <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP,4317/TCP,4318/TCP   112s
my-jaeger-collector-headless    ClusterIP   None             <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP,4317/TCP,4318/TCP   112s
my-jaeger-query                 ClusterIP   ipclusternumber    <none>        443/TCP,16685/TCP                                          112s
simplest-collector              ClusterIP   ipclusternumber   <none>        4317/TCP,4318/TCP,55681/TCP                                107s
simplest-collector-headless     ClusterIP   None             <none>        4317/TCP,4318/TCP,55681/TCP                                107s
simplest-collector-monitoring   ClusterIP   ipclusternumber    <none>        8888/TCP                                                   107s
vertx-opentelemetry             ClusterIP   ipclusternumber     <none>        80/TCP                                                     101s

I tested with a bunch of combinations but any successful results :

• http://simplest-collector:4317/

SEVERE: Failed to export spans. The request could not be executed. Full error message: simplest-collector: Name or service not known

• http://my-jaeger-collector:4317/

SEVERE: Failed to export spans. The request could not be executed. Full error message: my-jaeger-collector: Name or service not known

• http://simplest-collector" + openshift.getRoute("vertx-opentelemetry").getSpec().getHost() + ":4317

SEVERE: Failed to export spans. The request could not be executed. Full error message: connect timed out
...
If I do a curl Inside the simples-collector pod works :

oc rsh simplest-collector-857cd4bbd5-mb8gv
sh-4.4$ curl -v http://simplest-collector:4317

• Rebuilt URL to: http://simplest-collector:4317/
• Trying 172.30.233.8...
• TCP_NODELAY set
• Connected to simplest-collector (172.30.233.8) port 4317 (#0)

GET / HTTP/1.1
Host: simplest-collector:4317
User-Agent: curl/7.61.1
Accept: /

But I can't do it externally (logged from my project/namespace on openshift)...

[jcarranzan]

Do you have any idea why can't do curl -v http://simplest-collector:4317/ @albertteoh ? Thanks