How to handle long-running spans?

[Ayush-Kaushik]

We're looking for some advice on how to approach the integration of OpenTelemetry into a service that has long-running spans by nature, given issues that we've realized with respect to long-running spans (that they get auto-terminated if they remain outstanding for too long).

We have a service which accepts incoming network connections from remote agent systems, where these connections remain active for potentially hours. Occasionally a connection will be utilized to facilitate a request/response interaction to an agent, initiated by a third party that interacts with this service. Our initial thinking was that the maintenance of a network connection by this service constituted a span, as there is ongoing activity (keep-alive messages are exchanged, even if nothing else occurs). A request/response interaction would also be a span as well, linked to the connection span.

Unfortunately, as conceived above, we see that our connection spans are terminated because they persist for too long. Given that a request/response interaction may at some point take up to 30s, it seems it is also in danger of being terminated based on what we've read here: OpenTelemetry Challenges: Handling Long-Running Spans about issues related to long-running spans.

Is there any general advice for handling services like this? Should we not be thinking about spans at all as applying to these activities, or approaching the problem in another way?

[svrnm]

@Ayush-Kaushik the discussions on this repository are meant to talk about the information architecture of the OpenTelemetry website.

On the topic there are discussions in the specification repository, you can take a look here for example: open-telemetry/opentelemetry-specification#373