Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adopt WebGL for virtual background when available #44

Open
kyleparrott opened this issue Jul 8, 2024 · 1 comment
Open

Adopt WebGL for virtual background when available #44

kyleparrott opened this issue Jul 8, 2024 · 1 comment

Comments

@kyleparrott
Copy link
Contributor

I've been migrating my app from Twilio and recently had some users reporting performance issues, which appear to be related to using virtual backgrounds in our livekit calls.

I was able to reproduce this on the track processor sample app when applying a cpu slowdown in the chrome dev tools. When using the same cpu slowdown settings on a Twilio virtual background example app I did not see the same long running tasks in performance profiles. I've attached screenshots from the profiles for the Livekit and Twilio example apps below.

The performance difference looks like it comes down to Twilio's library using a webgl pipeline to apply the virtual background when webgl is supported.

Twilio profile:
image

Livekit profile:
image

@bcherry bcherry changed the title Virtual background performance Adopt WebGL for virtual background when available Aug 30, 2024
@lebaudantoine
Copy link
Contributor

lebaudantoine commented Sep 20, 2024

Additionally, by utilizing WebGL2 like Twilio, we can achieve higher-quality effects that align more closely with the methods described by the Google Meet team in their 2020 papers. You can read more about their approach here: Google Research Blog.

The mask is then used to render the video output via WebGL2, with the background blurred or replaced.

For further insights, check out the Rendering Effects section, where they detail their workaround. They mention a Bilateral filter which seems to have been implemented in WebGL by Twilio here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants