Document: Current status & plans on low latency streaming, webrtc etc.

[shaqq]

Is your feature request related to a problem? Please describe.
We have various real-time streams that we need low-latency visibility on. We need a GUI to view the streams, transform the data, replay the streams, and run model inference on them in real-time. We also need to mix/match the ML models to compare the results. These streams are high throughput, like video, audio, IMU, etc.

Describe the solution you'd like
A UDP protocol in rerun that accepts high throughput streams like video/audio/imu to log samples. WebRTC is a nice balance of a network protocol that also supports codecs.

Describe alternatives you've considered
Building our own solution in WebRTC

I saw #5815 for video codec support, but I wasn't sure to what extent there's broad interest in supporting WebRTC (which is admittedly very challenging).

I know there's TCP streaming, but this isn't quite ideal for us. Low-latency is really important for our use case, and TCP adds enough latency to make it not worth the effort.

[Wumpf]

Thanks for your interest in using Rerun. Unfortunately, we don't have anything planned regarding low latency & lossy transmission in the mid term (next few months) and it's very unlikely that we'll integrate WebRTC anytime soon. Bar any major shifts in priority, we'll stick with TCP (and protocols on top) for connections for data transmission for the immediate future.

What we are going to tackle very soon is basic video streaming with comparable (or better) latency of what we offer right now using raw image logging. Which ofc would still give us a lot worse latencies than what you're asking for.

Looking out a bit further into the future I can imagine this being a valuable puzzle piece where we want to ingest lossy and lossless streams at the same time, allowing to ingest them both into our dataplatform and/or directly to the viewer's in-memory store. Given how aggressively multi-modal our data formats are, it's not unlikely that we'd be rolling our own UDP based protocol in order to support everything we have at that point, adjusting the video streaming format accordingly in order to smoothly bridge over what we're building now. But this is pure speculation at this point, we'll have to do a lot more research into these things.

Leaving this ticket open as a todo for our docs: we had similar questions in the past and should document this somewhere more clearly!