You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've been migrating my app from Twilio and recently had some users reporting performance issues, which appear to be related to using virtual backgrounds in our livekit calls.
I was able to reproduce this on the track processor sample app when applying a cpu slowdown in the chrome dev tools. When using the same cpu slowdown settings on a Twilio virtual background example app I did not see the same long running tasks in performance profiles. I've attached screenshots from the profiles for the Livekit and Twilio example apps below.
The performance difference looks like it comes down to Twilio's library using a webgl pipeline to apply the virtual background when webgl is supported.
Twilio profile:
Livekit profile:
The text was updated successfully, but these errors were encountered:
bcherry
changed the title
Virtual background performance
Adopt WebGL for virtual background when available
Aug 30, 2024
Additionally, by utilizing WebGL2 like Twilio, we can achieve higher-quality effects that align more closely with the methods described by the Google Meet team in their 2020 papers. You can read more about their approach here: Google Research Blog.
The mask is then used to render the video output via WebGL2, with the background blurred or replaced.
For further insights, check out the Rendering Effects section, where they detail their workaround. They mention a Bilateral filter which seems to have been implemented in WebGL by Twilio here
I've been migrating my app from Twilio and recently had some users reporting performance issues, which appear to be related to using virtual backgrounds in our livekit calls.
I was able to reproduce this on the track processor sample app when applying a cpu slowdown in the chrome dev tools. When using the same cpu slowdown settings on a Twilio virtual background example app I did not see the same long running tasks in performance profiles. I've attached screenshots from the profiles for the Livekit and Twilio example apps below.
The performance difference looks like it comes down to Twilio's library using a webgl pipeline to apply the virtual background when webgl is supported.
Twilio profile:
Livekit profile:
The text was updated successfully, but these errors were encountered: