gRPC TracingServerInterceptor does NOT finish span when RPC stream is cancelled

[osadchuk-roman]

gRPC TracingServerInterceptor does NOT finish span when RPC stream is cancelled

Consider my case:

• there are 2 services: one of them is grpc client, another - grpc server.
• Distributed tracing is not implemented in client service.
• On server side distributed tracing is implemented using spring-cloud-starter-sleuth, spring-cloud-sleuth-zipkin and brave-instrumentation-grpc. TracingServerInterceptor is used in grpc server config.

In server service there is an implementation of grpc server side streaming. So, when client calls grpc request, server starts grps stream and a new root span is created by TracingServerInterceptor.

If, during processing of stream messages, client decides to cancel stream before server has sent all required messages, stream will be closed as cancelled on server and created span WILL NOT BE FINISHED. Since span is not finished, it will never be reported to distributed tracing collector (Zipkin in my case). Same behaviour is reproduced when client has closed/lost grpc connection (stream is closed as cancelled, span is not finished). Also this bug is always reproducible, when server returns an infinite stream to the client, client listens to it during some period of time and cancels it when it is needed .

When server has sent all messages, it closes stream as completed and previously created span is finished by TracingServerInterceptor.

Possible solution might be to finish span in TracingServerInterceptor.TracingServerCallListener#onCancel(). At least, I have created a custom tracing interceptor with this fix and it works in my case.

Libs versions:

1. io.grpc - 1.43.2
2. org.springframework.cloud:spring-cloud-starter-sleuth - 3.1.3
3. org.springframework.cloud:spring-cloud-sleuth-zipkin - 3.1.3
4. io.zipkin.brave:brave-instrumentation-grpc - 5.13.10

Preconditions:

1. Consider having gRPC service implementation that returns stream to the client
2. Distributed tracing is supported only on server side

Steps to Reproduce

Steps to reproduce the behavior:

1. Send rpc request from client
2. Verify rpc stream is started on server
3. Verify new span is created on server
4. Cancel stream from client before server has sent all messages to the stream OR disconnect client from grpc channel
5. Verify stream is closed as cancelled, but span was not finished, as a result it was not reported to distributed tracing system (e. g. Zipkin)

Expected Behaviour

TracingServerInterceptor should finish span, which was opened when gRPC stream was started, when stream is cancelled.

[jcchavezs]

Are you up for a PR on this?

[osadchuk-roman]

Are you up for a PR on this?

@jcchavezs
Created PR #1341