Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenTelemetry exporter fails with message Cannot read property 'Resolver' of undefined #281

Open
jussisjostrom opened this issue Sep 26, 2024 · 5 comments
Labels
bug Something isn't working untriaged Issue needs to be reviewed for validity

Comments

@jussisjostrom
Copy link

Environment

  • Telemetry Streaming Version: 1.36.0-1
  • BIG-IP Version: 17.1.1.4

Summary

Opentelemetry_Exporter type of consumer fails to send the collected metrics.
Verifying by tcpdump on the target shows that no attempt to open the tcp connection to send the report is made.
In the /var/log/restnoded/restnoded.log file this line is printed:

Thu, 26 Sep 2024 08:25:09 GMT - severe: [telemetry.service.ConsumersService.OpenTelemetry_Exporter.f5telemetry_default::SRV01_OTEL_GRPC_NOSSL] error: Cannot read property 'Resolver' of undefined

Impact is that I cannot use gRPC to send Opentelemetry reports from BIG-IP.

Steps To Reproduce

Steps to reproduce the behavior:

  1. Submit the following declaration:
{
    "$schema": "https://raw.githubusercontent.com/F5Networks/f5-telemetry-streaming/master/src/schema/latest/base_schema.json",
    "class": "Telemetry",
    "schemaVersion": "1.36.0",
    "controls": {
        "class": "Controls",
        "logLevel": "debug",
        "debug": false,
        "memoryMonitor": {
            "logFrequency": 60
        },
        "memoryThresholdPercent": 90
    },
    "Listener": {
        "class": "Telemetry_Listener",
        "port": 6514,
        "enable": true,
        "trace": false,
        "match": "",
        "actions": [
            {
                "setTag": {
                    "tenant": "`T`",
                    "application": "`A`"
                },
                "enable": true
            }
        ]
    },
    "defaultPoller": {
        "class": "Telemetry_System_Poller",
        "interval": 60,
        "host": "localhost",
        "port": 8100,
        "protocol": "http",
        "allowSelfSignedCert": true,
        "enable": true,
        "workers": 5,
        "chunkSize": 30
    },
    "telemetrySystem": {
        "class": "Telemetry_System",
        "enable": true,
        "systemPoller": [
            "defaultPoller"
        ],
        "host": "localhost",
        "port": 8100,
        "protocol": "http",
        "enableHostConnectivityCheck": false,
        "allowSelfSignedCert": true
    },
    "SRV01_OTEL_GRPC_NOSSL": {
        "class": "Telemetry_Consumer",
        "type": "OpenTelemetry_Exporter",
        "enable": true,
        "trace": true,
        "host": "IP-address-redacted",
        "port": 4317,
        "exporter": "grpc",
        "convertBooleansToMetrics": false,
        "useSSL": false
    }
}
  1. Declaration is accepted, response is 200 OK:
{
    "message": "success",
    "declaration": {
...
  1. Wait for the poller to collect data and observe the restnoded.log file for result. Also, verify with tcpdump if any connection is attempted to the consumer.

Expected Behavior

The consumer of type OpenTelemetry_Exporter would be succesful in sending telemetry reports using gRPC.

Actual Behavior

No report is sent out. Based on logs, data is collected, but sending to the consumer fails even before a TCP connection is attempted:

Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Starting polling cycle
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully completed "WAITING" step
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Transitioning from step "WAITING" to "COLLECT"
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller.Loader[localhost]] Task queue "TaskQueue_9dd6d" configured with concurrency = 5
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller.context.Collector] Starting stats collection
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller.context.Collector] Task queue "TaskQueue_4667f" configured with concurrency = 5
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller.stats.Collector] Starting stats collection
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller.stats.Collector] Task queue "TaskQueue_918df" configured with concurrency = 5
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully collected stats
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully completed "COLLECT" step
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Transitioning from step "COLLECT" to "SEND_REPORT"
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully processed stats report
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully completed "SEND_REPORT" step
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Transitioning from step "SEND_REPORT" to "DONE"
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully collected and processed stats. Polling Cycle duration - 0 sec.
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully completed "DONE" step
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Transitioning from step "DONE" to "SCHEDULE"
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Next polling cycle starts on 2024-09-26T08:26:09.029Z (in 59 s.)
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully scheduled next execution date
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully completed "SCHEDULE" step
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Transitioning from step "SCHEDULE" to "WAITING"
Thu, 26 Sep 2024 08:25:09 GMT - severe: [telemetry.service.ConsumersService.OpenTelemetry_Exporter.f5telemetry_default::SRV01_OTEL_GRPC_NOSSL] error: Cannot read property 'Resolver' of undefined
@jussisjostrom jussisjostrom added bug Something isn't working untriaged Issue needs to be reviewed for validity labels Sep 26, 2024
@pgouband
Copy link

Hi @jussisjostrom,

New versions of OTEL library can't be used because node.js is too old and OTEL libs has been updated and it fails now. So it even fails on the version that worked in the past: the server side was updated and as result the client side (TS) tries to trigger the code that even does not exist in the library or does not work on node.js 8.11.1

@petrov-serg
Copy link
Contributor

@jussisjostrom

you can try http/protobuf instead of grpc if you server configured to accept it

@PatricDahl
Copy link

Same problem here after upgrading to Telemetry 1.37

"f5-collector": {
"class": "Telemetry_Consumer",
"type": "OpenTelemetry_Exporter",
"host": "10.41.85.46",
"port": 4317,
"trace": true,
"enable": true,
"headers": [
{
"name": "x-access-token",
"value": "YOUR_TOKEN"
}
],
"convertBooleansToMetrics": true,
"exporter": "grpc",
"useSSL": false,
"allowSelfSignedCert": true
},

What's the plan? should we revert back to previous version or will someone fix issue?

@PatricDahl
Copy link

Telemetry 1.36 has the same issue.

@pgouband
Copy link

Hi @PatricDahl

New versions of OTEL library can't be used because node.js is too old and OTEL libs has been updated and it fails now. So it even fails on the version that worked in the past: the server side was updated and as result the client side (TS) tries to trigger the code that even does not exist in the library or does not work on node.js 8.11.1.

Can you try http/protobuf instead of grpc if you server configured to accept it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working untriaged Issue needs to be reviewed for validity
Projects
None yet
Development

No branches or pull requests

4 participants