Call debug information & Telemetry (in a machine readable way)

[fkwp]

Highlevel overview

We have failures on both full mesh and focus based Element Call that are hard to debug. Any debugging we can do involves manually lining up and trawling through logs from each client and (if applicable) the SFU. This is not really a sustainable solution and it is critical to the success of MatrixRTC and Element Call that we can diagnose and fix failure modes.

Therefore, alongside the above change, we add feedback and debugging, investigating OpenTelemetry to collect call events and logs from all parties (on our own deployments and where permission is granted) in one place.

We need this information on both sides: frontend and backend. Information to report are:

• WebRTC signaling (ICE candidates, tracks advertising/requesting, feeds we have, transceiver directions, basically a dump of the SDP)
• business logic signaling (mute states, requesting media layers)
• also address KPIs
  • video frozen
  • audio drop outs
  • total bandwidth used

Level of detail:

• global debug view
• zoom into participants

Linked Issues

Preview Give feedback

1. Investigate OpenTelemetry with a goal to report (not necessarily all initially)
2. Error events from calls
3. Error events from group calls
4. Undecryptable to-device events
5. Incoming to-device events
6. OpenTelemetry: Approximate audio levels over time on sent & received tracks vector-im/element-call#959
   +0
   
7. Convert / generate spans for current signaling events vector-im/element-call#950
   +0
   
8. Add OpenTelemetry instrumentation matrix-org/waterfall#141
   T-Enhancement +1
   
9. Add Posthog exporter to OpenTelemetry (in order to define our baseline KPI for Element Call) vector-im/element-call#948
   +0
   
10. Add Json Exporter to OpenTelemetry vector-im/element-call#949
    +0
    
11. Add OpenTelemetry spans for media statistics vector-im/element-call#947
    +0
    
12. https://github.com/vector-im/voip-internal/issues/131

Goals

Preview Give feedback

1. We should already know when somebody has been able to see/hear someone on a call (assuming they have granted permission for debugging) without them rageshaking / coming to tell us
2. Add OpenTelemetry spans to rage-shakes and provide a compelling analysis framework

Relation to other Epics

• Telemetry - give users the option to share usage data using Posthog #533