Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement omnidirectional cameras for real-time reflection probes. #13840

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

pcwalton
Copy link
Contributor

@pcwalton pcwalton commented Jun 14, 2024

This commit introduces a new type of camera, the omnidirectional camera. These cameras render to a cubemap texture, and as such extract into six different cameras in the render world, one for each side. The cubemap texture can be attached to a reflection probe as usual, which allows reflections to contain moving objects. To use an omnidirectional camera, create an OmnidirectionalCameraBundle.

Because omnidirectional cameras extract to six different sub-cameras in the render world, render world extraction code that targets components present on cameras now needs to be aware of this fact and extract components to the individual sub-cameras, not the root camera component. They also need to run after omnidirectional camera extraction, as only then will the sub-cameras be present in the render world. New plugins, ExtractCameraComponentPlugin and ExtractCameraInstancesPlugin, are available to assist with this.

Each side of an omnidirectional camera can be individually marked as active via the ActiveCubemapSides bitfield. This allows for the common technique of rendering only one (or two, or three) sides of the cubemap per frame, to reduce rendering overhead. It also allows for on-demand rendering, so that an application that wishes to optimize further can choose sides to refresh. For example, an application might wish to only rerender sides whose frusta contain moving entities.

In addition to real-time reflection probes, this patch introduces much of the infrastructure necessary to support baking reflection probes from within Bevy as opposed to in an external program such as Blender, which has been the status quo up to this point. Even with this patch, there are still missing pieces needed to make this truly convenient, however:

  1. Baking a reflection probe requires more than just saving a cubemap: it requires pre-filtering the cubemap into diffuse and specular parts in the same way that the glTF IBL Sampler does. This is not yet implemented in Bevy; see GenerateEnvironmentMapLight #9414 for a previous attempt.

  2. The cubemap needs to be saved in .ktx2 format, as that's the only format that Bevy presently knows how to load. There's no comprehensive Rust crate for this, though note that my glTF IBL Sampler UI has code to do it for the specific case of cubemaps.

  3. An editor UI is necessary for convenience, as otherwise every application will have to create some sort of bespoke tool that arranges scenes and saves the reflection cubemaps.

The reflection_probes example has been updated in order to add an option to enable dynamic reflection probes, as well as an option to spin the cubes so that the impact of the dynamic reflection probes is visible. Additionally, the static reflection probe, which was previously rendered in Blender, has been changed to one rendered in Bevy. This results in a change in appearance, as Blender and Bevy render somewhat differently.

Partially addresses #12233.

Changelog

Added

  • An OmnidirectionalCameraBundle has been added in order to render to a cubemap. This allows reflection probes to reflect the dynamic scene.

This commit introduces a new type of camera, the *omnidirectional*
camera. These cameras render to a cubemap texture, and as such extract
into six different cameras in the render world, one for each side. The
cubemap texture can be attached to a reflection probe as usual, which
allows reflections to contain moving objects. To use an omnidirectional
camera, create an [`OmnidirectionalCameraBundle`].

Because omnidirectional cameras extract to six different sub-cameras in
the render world, render world extraction code that targets components
present on cameras now needs to be aware of this fact and extract
components to the individual sub-cameras, not the root camera component.
They also need to run after omnidirectional camera extraction, as only
then will the sub-cameras be present in the render world. New plugins,
`ExtractCameraComponentPlugin` and `ExtractCameraInstancesPlugin`, are
available to assist with this.

Each side of an omnidirectional camera can be individually marked as
active via the `ActiveCubemapSides` bitfield. This allows for the common
technique of rendering only one (or two, or three) sides of the cubemap
per frame, to reduce rendering overhead. It also allows for on-demand
rendering, so that an application that wishes to optimize further can
choose sides to refresh. For example, an application might wish to only
rerender sides whose frusta contain moving entities.

In addition to real-time reflection probes, this patch introduces much
of the infrastructure necessary to support baking reflection probes from
within Bevy as opposed to in an external program such as Blender, which
has been the status quo up to this point. Even with this patch, there
are still missing pieces needed to make this truly convenient, however:

1. Baking a reflection probe requires more than just saving a cubemap:
   it requires pre-filtering the cubemap into diffuse and specular parts
   in the same way that the [glTF IBL Sampler] does. This is not yet
   implemented in Bevy; see bevyengine#9414 for a previous attempt.

2. The cubemap needs to be saved in `.ktx2` format, as that's the only
   format that Bevy presently knows how to load. There's no
   comprehensive Rust crate for this, though note that my [glTF IBL
   Sampler UI] has code to do it for the specific case of cubemaps.

3. An editor UI is necessary for convenience, as otherwise every
   application will have to create some sort of bespoke tool that
   arranges scenes and saves the reflection cubemaps.

The `reflection_probes` example has been updated in order to add an
option to enable dynamic reflection probes, as well as an option to spin
the cubes so that the impact of the dynamic reflection probes is
visible. Additionally, the static reflection probe, which was previously
rendered in Blender, has been changed to one rendered in Bevy. This
results in a change in appearance, as Blender and Bevy render somewhat
differently.

Partially addresses bevyengine#12233.

[glTF IBL Sampler]: https://github.com/KhronosGroup/glTF-IBL-Sampler

[glTF IBL Sampler UI]: https://github.com/pcwalton/gltf-ibl-sampler-egui
@pcwalton pcwalton added A-Rendering Drawing game state to the screen C-Feature A new feature, making something new possible labels Jun 14, 2024
@pcwalton pcwalton added this to the 0.15 milestone Jun 14, 2024
@pcwalton pcwalton requested review from IceSentry and robtfm June 14, 2024 03:44
@alice-i-cecile alice-i-cecile added the M-Needs-Release-Note Work that should be called out in the blog due to impact label Jun 14, 2024
@NthTensor NthTensor added the S-Needs-Review Needs reviewer attention (from anyone!) to move forward label Jul 16, 2024
Copy link
Contributor

@tychedelia tychedelia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not super familiar with the lightprobe code itself, but the camera/view stuff looks good. Just a few non-blocking comments on some random things.

/// *Main textures* are used as intermediate targets for rendering, before
/// postprocessing and tonemapping resolves to the final output.
#[derive(Clone, PartialEq, Eq, Hash)]
struct ViewTargetTextureKey {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, these are a really helpful cleanup.

commands
.get_or_spawn(view_entity)
.insert(render_view_light_probes);
if let (Some(view_render_layers), Some(light_probe_render_layers)) = (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In other places, we tend to fall back to default layer 0 if view layers are missing, is that intentionally not the case here?

@@ -823,6 +844,7 @@ pub struct ExtractedCamera {
pub clear_color: ClearColorConfig,
pub sorted_camera_index_for_target: usize,
pub exposure: f32,
pub render_target_layer: Option<NonMaxU32>,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nbd, but this field wasn't immediately intuitive to me and part of me feels like it should be a direct property of NormalizedRenderTarget::Image or a new NormalizedRenderTarget::Cubemap instead.

@alice-i-cecile alice-i-cecile modified the milestones: 0.15, 0.16 Oct 8, 2024
@BenjaminBrienen BenjaminBrienen added D-Complex Quite challenging from either a design or technical perspective. Ask for help! S-Waiting-on-Author The author needs to make changes or address concerns before this can be merged and removed S-Needs-Review Needs reviewer attention (from anyone!) to move forward labels Oct 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-Rendering Drawing game state to the screen C-Feature A new feature, making something new possible D-Complex Quite challenging from either a design or technical perspective. Ask for help! M-Needs-Release-Note Work that should be called out in the blog due to impact S-Waiting-on-Author The author needs to make changes or address concerns before this can be merged
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

5 participants