Trace searching and metrics

[AntPeixe]

Problem Statement

Investigating traces with multiple roots (from distributed services) is quite hard.

Similarly in Discovery it isn't possible to investigate traces, for example to try to figure out duration distribution or how much "wasted" time there is between the root transactions of the trace.
One other idea could be to add a groupby function to the queries. Maybe it's interesting to group transactions with the same value on a field so that you can then apply a function on other tags/field for the whole group.

Solution Brainstorm

Maybe Traces can become an event so that they can be queried.

groupby function so that you can apply functions to whole group (duration of group of transactions for example)

[getsentry-release]

Routing to @getsentry/visibility for triage. ⏲️

[shruthilayaj]

Hi @AntPeixe, Discover should automatically do a groupby on columns that you've selected when you apply an aggregate function like max, count, avg etc. You should also be able to to add trace as a column in discover and query on it. Would you be able to clarify what you want kind of groupby you'd like to apply to functions that isn't already supported?

[AntPeixe]

Hi @shruthilayaj,

We are using Sentry tracing in a distributed environment with custom instrumentation. For this reason our traces have multiple roots.

Our main objective is to get some metrics on the traces. For example we'd like to get a distribution of trace duration, a distribution of number of transactions in traces or how much time is "wasted" in between services. However Discover and Dashboards work mostly on "Events" (errors, transactions).

Thinking about the second example: if I try to count transactions in traces, I have to query transactions but count(trace) is always 1 because each transaction has a single trace. So, I can't say "group all transactions with same trace id and count number of transactions in that group".

Ideally I think it would be better to have the option to query traces and its fields directly. Something like event.type: trace or trace.duration: > 5m, count(trace.events). If that isn't possible then a "groupBy" would probably help. This is also quite common for database query and data analytics, you group data and then operate on the groups instead of the elements alone.

I hope this made it more clear.

[shruthilayaj]

Thanks for the clarification @AntPeixe. Let me start with parts that can be handled in discover today: grouping transactions in a trace. This can currently be done in discover with the following columns: trace and count_unique(id). This groups by trace and applies the count_unique(id) aggregate on the group (you can also apply other aggregates to this). You can then apply any other filters you wish in the query bar.

Some of the other stuff like distribution of trace measurements and calculating time "wasted" is more complex isn't currently supported because of how our data model is set up. Thanks for raising this, it's a great use case, I'll ticket this up and move this issue to the backlog.

[alvercau]

Is there any progress on this?

[Stormix]

+1

[getsantry]

Routing to @getsentry/product-owners-performance for triage ⏲️

[alexjillard]

@alvercau @Stormix @AntPeixe, Apologies for the lack of updates recently, but I do have some good news! We are nearing an LA release of a new tool (Explore) that will allow searching for traces and spans. If you're interested in a preview, connect with @Dhrumil-Sentry and we'll see what we can do.

If you'd like to be included in the EA cohort, make sure the setting is enabled for your org. We should be going to EA early in the new year, with a GA date not long after.

Since this functionality is covered in the new product, I'll close this issue for now. Once you have access to Explore, we'd love to get your feedback on it.

[Dhrumil-Sentry]

Please see this discussion for more details: #81239