[Working] Green Screen Chroma Key Passthrough - Enable OpenXR

[beiller]

I attempted to add green screen pass-through, and got stuck trying to get the pass-through to be "over top" the "video stream". If I move my head real fast I can see the pass-through in the lags that the video is not covering so I think it's progress.

[beiller]

If anyone is able to help me make the passthrough appear over the video stream, even just setting alpha to zero doesn't seem to work. I tried changing all the GLSL shaders to set .a = 0.0 to no avail. Tried various GL clear flags etc.

Once that works I can write code to toggle it as well as chroma key GLSL code. I just can't seem to make it pass through however.

[zmerp]

I believe you will have to set the alpha channel == 0 for every shader in the compositor. The reason I haven't implemented passthrough yet is because I wanted to rewrite the client compositor since the current OpenGL one is pretty messy.

[Vixea]

It's best to use an epsilon when comparing floats btw or alpha < 0.01. If I understand what you're trying to do here zarik

[beiller]

@zarik5 I tried that all the glsl I could find and set .a = 0.0 for all of them and it didn't pass through. I just reverted that change since it didn't seem to work. Only on the outside of the frame I could see slivers of pass-through. Or maybe can you point me to a particular shader of interest?

[zmerp]

@beiller There are a few shaders: gamma correction, FFR, and one last one to split the image between the two eyes. To be honest the code is very old and quite bad, I can't help you much further unless I dig into that more.

[beiller]

Thanks for your comments @zarik5 I will try it out. I will think on it some more because I feel there is something higher level than GLSL that is setting the alpha blending.

[zmerp]

It's useful to disable/bypass some code temporarily, for debug purposes. for example in the shader to render to the final swapchains, discard the texture values and always set (0, 0, 0, 0).

[zmerp]

@beiller, instead of writing raw ffi code in ALVR itslef, are you able to add the passthrough bindings to our openxrs fork? Or you can submit the bindings upstream, Ralith could help you making sure the styling of the code is consistent.

[beiller]

Hey @zarik5 the good news is I don't need all that I guess I was just using it to debug. I have a new working implementation obviously missing a lot and is hard-coded to chromakey green currently. Maybe we can turn it into a feature somehow.

[beiller]

Need way to toggle and maybe some small cleanup before this could ever be considered.

• Need to toggle feature on/off with key combo
• Need to remove required: true for passthrough component in open XR
• Need to test on more devices (I have quest2 only)

[Meister1593]

Does not seem to work on pico 4 (pro, 5.9.1 pui)?... Just black screen

[Vixea]

Does not seem to work on pico 4 (pro, 5.9.1 pui)?... Just black screen

check for shader compile error in logcat

[Meister1593]

Does not seem to work on pico 4 (pro, 5.9.1 pui)?... Just black screen

check for shader compile error in logcat

Any specifics for compiler error message? Don't seem to find much of anything with "shader" in logs

[Vixea]

nah not really just search for err or error

[Meister1593]

From what i can tell, it compiled shaders just fine and runtime recognized passthrough

02-21 13:04:11.107 30004 30032 I [ALVR Native]: Compiling shaders.
02-21 13:04:11.108 30004 30032 I [ALVR Native]: Successfully compiled shader.
02-21 13:04:11.108 30004 30032 I [ALVR Native]: Compiling shaders.
02-21 13:04:11.108 30004 30032 I [ALVR Native]: Successfully compiled shader.
02-21 13:04:11.109 30004 30032 E APxrRuntime:  oxr_session_poll:XRT_COMPOSITOR_EVENT_SEETHROUGH_CHANGE-------:0

[Meister1593]

fb_passthrough is available and being used

[beiller]

Ah so it is just black where the passthrough should be seen? That is at least "working". If the shader has issue the app just crashes :)

I did hack around with it until it works so I think there may be something missing in the code. I can try experimenting with some things if you can help test pico 4.

[Meister1593]

Ah so it is just black where the passthrough should be seen? That is at least "working". If the shader has issue the app just crashes :)

I did hack around with it until it works so I think there may be something missing in the code. I can try experimenting with some things if you can help test pico 4.

Sure thing, i can test it of course

[beiller]

OK maybe it is caused by "com.oculus.feature.PASSTHROUGH" Is there some other feature that needs to be enabled for Pico do we think?

[beiller]

@Meister1593 I found maybe an issue, the features aren't properly returned. Can you try compiling with this commit and let me know? I think I may have had a different build on my device. With the "if" statements uncommented I got similar black screen behavior that you see.

[Meister1593]

@Meister1593 I found maybe an issue, the features aren't properly returned. Can you try compiling with this commit and let me know? I think I may have had a different build on my device. With the "if" statements uncommented I got similar black screen behavior that you see.

just tried, nothing changed, exactly the same thing
Checked one of the passthrough apps (it's unity though) and android props, tried to copy them - nothing changed

[[package.metadata.android.application.meta_data]]
name = "MRCsupport"
value = "1"
[[package.metadata.android.application.meta_data]]
name = "color_Space"
value = "1"
[[package.metadata.android.application.meta_data]]
name = "rendering_mode"
value = "0"

[Meister1593]

From brief discussion in the server i've changed layer mode from OPAQUE to ALPHA_BLEND and it started to work.
I will try to work on providing settings from server to client to make chromakey colour, range and toggle be adjustable.

[JackD83]

I just tried this PR with the latest master. Works great, but I had to manually merge graphics.cpp. I just have a major flaw: If the application running in SteamVR is dropping frames (e.g late start frames), the frame gets cleared to paththrough and the rendered content starts to flicker.
I don't know enough about the inner workings and shader in general to fix this. Is there even a chance to solve this?

[beiller]

I never came across that problem. I'm not sure what it means by late start frames? I've been working on a game engine it runs quite slow at times with no issues like 20-30 fps? Maybe I don't understand that issue. Did you have any luck?

[JackD83]

Sorry for not reporting back. The issue was not caused by the passtrough but a bug in ALVR itself. The bug is already fixed: d42701b

Late start frames are not caused by slow rendering but by a slow cpu. In our case the generation of textures in the unity main thread that takes too long.