-
-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement omnidirectional cameras for real-time reflection probes. #13840
base: main
Are you sure you want to change the base?
Conversation
This commit introduces a new type of camera, the *omnidirectional* camera. These cameras render to a cubemap texture, and as such extract into six different cameras in the render world, one for each side. The cubemap texture can be attached to a reflection probe as usual, which allows reflections to contain moving objects. To use an omnidirectional camera, create an [`OmnidirectionalCameraBundle`]. Because omnidirectional cameras extract to six different sub-cameras in the render world, render world extraction code that targets components present on cameras now needs to be aware of this fact and extract components to the individual sub-cameras, not the root camera component. They also need to run after omnidirectional camera extraction, as only then will the sub-cameras be present in the render world. New plugins, `ExtractCameraComponentPlugin` and `ExtractCameraInstancesPlugin`, are available to assist with this. Each side of an omnidirectional camera can be individually marked as active via the `ActiveCubemapSides` bitfield. This allows for the common technique of rendering only one (or two, or three) sides of the cubemap per frame, to reduce rendering overhead. It also allows for on-demand rendering, so that an application that wishes to optimize further can choose sides to refresh. For example, an application might wish to only rerender sides whose frusta contain moving entities. In addition to real-time reflection probes, this patch introduces much of the infrastructure necessary to support baking reflection probes from within Bevy as opposed to in an external program such as Blender, which has been the status quo up to this point. Even with this patch, there are still missing pieces needed to make this truly convenient, however: 1. Baking a reflection probe requires more than just saving a cubemap: it requires pre-filtering the cubemap into diffuse and specular parts in the same way that the [glTF IBL Sampler] does. This is not yet implemented in Bevy; see bevyengine#9414 for a previous attempt. 2. The cubemap needs to be saved in `.ktx2` format, as that's the only format that Bevy presently knows how to load. There's no comprehensive Rust crate for this, though note that my [glTF IBL Sampler UI] has code to do it for the specific case of cubemaps. 3. An editor UI is necessary for convenience, as otherwise every application will have to create some sort of bespoke tool that arranges scenes and saves the reflection cubemaps. The `reflection_probes` example has been updated in order to add an option to enable dynamic reflection probes, as well as an option to spin the cubes so that the impact of the dynamic reflection probes is visible. Additionally, the static reflection probe, which was previously rendered in Blender, has been changed to one rendered in Bevy. This results in a change in appearance, as Blender and Bevy render somewhat differently. Partially addresses bevyengine#12233. [glTF IBL Sampler]: https://github.com/KhronosGroup/glTF-IBL-Sampler [glTF IBL Sampler UI]: https://github.com/pcwalton/gltf-ibl-sampler-egui
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not super familiar with the lightprobe code itself, but the camera/view stuff looks good. Just a few non-blocking comments on some random things.
/// *Main textures* are used as intermediate targets for rendering, before | ||
/// postprocessing and tonemapping resolves to the final output. | ||
#[derive(Clone, PartialEq, Eq, Hash)] | ||
struct ViewTargetTextureKey { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, these are a really helpful cleanup.
commands | ||
.get_or_spawn(view_entity) | ||
.insert(render_view_light_probes); | ||
if let (Some(view_render_layers), Some(light_probe_render_layers)) = ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In other places, we tend to fall back to default layer 0 if view layers are missing, is that intentionally not the case here?
@@ -823,6 +844,7 @@ pub struct ExtractedCamera { | |||
pub clear_color: ClearColorConfig, | |||
pub sorted_camera_index_for_target: usize, | |||
pub exposure: f32, | |||
pub render_target_layer: Option<NonMaxU32>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nbd, but this field wasn't immediately intuitive to me and part of me feels like it should be a direct property of NormalizedRenderTarget::Image
or a new NormalizedRenderTarget::Cubemap
instead.
This commit introduces a new type of camera, the omnidirectional camera. These cameras render to a cubemap texture, and as such extract into six different cameras in the render world, one for each side. The cubemap texture can be attached to a reflection probe as usual, which allows reflections to contain moving objects. To use an omnidirectional camera, create an
OmnidirectionalCameraBundle
.Because omnidirectional cameras extract to six different sub-cameras in the render world, render world extraction code that targets components present on cameras now needs to be aware of this fact and extract components to the individual sub-cameras, not the root camera component. They also need to run after omnidirectional camera extraction, as only then will the sub-cameras be present in the render world. New plugins,
ExtractCameraComponentPlugin
andExtractCameraInstancesPlugin
, are available to assist with this.Each side of an omnidirectional camera can be individually marked as active via the
ActiveCubemapSides
bitfield. This allows for the common technique of rendering only one (or two, or three) sides of the cubemap per frame, to reduce rendering overhead. It also allows for on-demand rendering, so that an application that wishes to optimize further can choose sides to refresh. For example, an application might wish to only rerender sides whose frusta contain moving entities.In addition to real-time reflection probes, this patch introduces much of the infrastructure necessary to support baking reflection probes from within Bevy as opposed to in an external program such as Blender, which has been the status quo up to this point. Even with this patch, there are still missing pieces needed to make this truly convenient, however:
Baking a reflection probe requires more than just saving a cubemap: it requires pre-filtering the cubemap into diffuse and specular parts in the same way that the glTF IBL Sampler does. This is not yet implemented in Bevy; see GenerateEnvironmentMapLight #9414 for a previous attempt.
The cubemap needs to be saved in
.ktx2
format, as that's the only format that Bevy presently knows how to load. There's no comprehensive Rust crate for this, though note that my glTF IBL Sampler UI has code to do it for the specific case of cubemaps.An editor UI is necessary for convenience, as otherwise every application will have to create some sort of bespoke tool that arranges scenes and saves the reflection cubemaps.
The
reflection_probes
example has been updated in order to add an option to enable dynamic reflection probes, as well as an option to spin the cubes so that the impact of the dynamic reflection probes is visible. Additionally, the static reflection probe, which was previously rendered in Blender, has been changed to one rendered in Bevy. This results in a change in appearance, as Blender and Bevy render somewhat differently.Partially addresses #12233.
Changelog
Added
OmnidirectionalCameraBundle
has been added in order to render to a cubemap. This allows reflection probes to reflect the dynamic scene.