Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support pushing traces to otel collector #261

Merged
merged 4 commits into from
Mar 19, 2024

Conversation

adam-cattermole
Copy link
Member

@adam-cattermole adam-cattermole commented Feb 20, 2024

This adds support to push traces to an otel collector, configured via new command line options, --tracing-host and --tracing-port.

Currently this is configured to use grpc without TLS however this could be added as a follow up.

Verification:

For this example I'm running Redis as the datastore to verify.

Run Redis service

docker run --rm -it -p 6379:6379 redis

Run Jaeger with OTLP

docker run --rm -it --name jaeger -e COLLECTOR_OTLP_ENABLED=true -p 16686:16686 -p 4317:4317 -p 4318:4318 jaegertracing/all-in-one:latest
firefox -private-window "localhost:16686"

Build and run from this branch

cargo build --bin limitador-server
./target/debug/limitador-server --tracing-endpoint rpc://localhost:4317 -vvvv --grpc-reflection-service limitador-server/sandbox/limits.yaml redis redis://127.0.0.1

Send Requests

grpcurl -plaintext -d @ 127.0.0.1:8081 envoy.service.ratelimit.v3.RateLimitService.ShouldRateLimit <<EOM
{
    "domain": "test_namespace",
    "hits_addend": 1,
    "descriptors": [
        {
            "entries": [
                {
                    "key": "req.method",
                    "value": "POST"
                }
            ]
        }
    ]
}
EOM

See the spans for the requests under the limitador-server service in the jaeger ui:

image

@adam-cattermole adam-cattermole self-assigned this Feb 20, 2024
@adam-cattermole adam-cattermole linked an issue Feb 20, 2024 that may be closed by this pull request
@adam-cattermole adam-cattermole force-pushed the tracing-otel branch 2 times, most recently from c5aaf7a to 391cfbb Compare February 20, 2024 14:56
.with(telemetry_layer)
.init();
} else {
tracing_subscriber::registry()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this doing? (when exporter is not being defined)

Copy link
Member Author

@adam-cattermole adam-cattermole Feb 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tracing package is being used for the standard logging since #250 - if tracing_host is unset we ensure that the fmt_layer is still applied so we get log output on stdout. Only if it's set we apply the telemetry_layer as well which is sending to the collector

Comment on lines 310 to 311
.with_endpoint(format!("rpc://{}", config.tracing_address()))
.with_protocol(Protocol::Grpc),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'd want to align that with authorino's way of configuring...

Copy link
Member Author

@adam-cattermole adam-cattermole Mar 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've had a look and this .with_protocol doesn't actually effect the behaviour. Either we use .tonic() and it's grpc, or we use .http() and provide our choice of http client. For now I've moved to a --tracing-endpoint option which contains scheme the same as authorino, but it won't use http if provided unless we check the scheme and configure an http client

Copy link
Member

@alexsnaps alexsnaps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, modulo the unnecessary diffs, but I'll let you be the judge of their faith

@adam-cattermole adam-cattermole merged commit de261a8 into Kuadrant:main Mar 19, 2024
9 checks passed
@adam-cattermole adam-cattermole deleted the tracing-otel branch March 19, 2024 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Add support to push traces to collector
3 participants