Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce image data conversion pipeline, taking over existing YUV conversions #7640

Merged
merged 19 commits into from
Oct 9, 2024

Conversation

Wumpf
Copy link
Member

@Wumpf Wumpf commented Oct 8, 2024

What

via pixi run -e py python ./tests/python/chroma_downsample_image/main.py:
https://rerun.io/viewer/pr/7640?url=https%3A%2F%2Fstatic.rerun.io%2Frrd%2F0.19%2Fchroma_formats_ad9697d05933f4104fbbe66af23073ad4e9e4a58.rrd

image

The new functionality is tightly integrated into TextureManager2D which now ingests all incoming data via a new internal method transfer_image_data_to_texture.
This in turn takes care of data upload via GpuReadCpuWriteBelt (new! previously, we would use queue.write_texture in this case) and may if needed run gpu based conversion.

Gpu based conversion is for convenience implemented like a Renderer - while it is not used like a typical Renderer, the lifecycle (create lazily once, store on context, feed with data bundles [...]) was so similar that it made sense to me to use this existing pattern even though it's not a perfect match.

A curious sideeffect of this is that you can now put chroma downsampled textures on custom meshes!

Next steps:

  • support formats required for AV1
    • while on it, check if things could get simpler by us
  • move BGR handling into the new pipeline
    • revisit the re_renderer sided input data description. Not sure if I'm happy about it drifting away from the user facing one!
    • once we're there we're also very close to get rid of SourceImageDataFormat::WgpuCompatible. But it's not strictly necessary, this might yet prove a convenient loophole, although its presence makes me a little bit nervous (details see code comments) 🤔
  • consider exposing color primaries to public API
    • the only thing keeping us from it is adding a new enum to the color format!
    • should it be called "primaries" or "color space" 🤔

Checklist

  • test on web Mac Chrome/Firefox/Safari
  • test on web Window Chrome/Firefox
  • test on web Firefox Chrome/Firefox (@jleibs plz 🥺 )
  • I have read and agree to Contributor Guide and the Code of Conduct
  • I've included a screenshot or gif (if applicable)
  • I have tested the web demo (if applicable):
  • The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG
  • If applicable, add a new check to the release checklist!
  • If have noted any breaking changes to the log API in CHANGELOG.md and the migration guide

To run all checks from main, comment on the PR with @rerun-bot full-check.

@Wumpf Wumpf added 🔺 re_renderer affects re_renderer itself 🚜 refactor Change the code, not the functionality include in changelog labels Oct 8, 2024
Comment on lines -6 to -8
// WARNING! Adding anything else to this shader is very likely to push us over a size threshold that
// causes the failure reported in: https://github.com/rerun-io/rerun/issues/3931
// Make sure any changes are tested in Chrome on Linux using the Intel Mesa driver.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the expectation is that this is no longer the case now.
Is it true though? @jleibs :)

@Wumpf Wumpf force-pushed the andreas/flexible-and-fast-image-conversion branch from 61d287f to 1c31cff Compare October 8, 2024 16:25
@emilk emilk self-requested a review October 8, 2024 16:31
Copy link
Member

@emilk emilk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beautiful! I love all the comments

LGTM if tested with NV12 and YUV on natie, WebGL and WebGPU (I'm paranoid!)


var rgb: vec3f;

switch (primaries) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wat, wgsl supports switch!? NICE! That could be useful in crates/viewer/re_renderer/shader/rectangle_fs.wgsl too 😄

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

learned that today as well 😄. Followed a hunch there - probably I've seen it in the meanwhile somewhere.

I think it didn't back when we started. But Naga & Tint (Dawn's compiler) are happy with it so full steam ahead!

Comment on lines 21 to 52
pub enum ChromaSubsamplingPixelFormat {
/// 4:2:0 subsampling with a separate Y plane, followed by a UV plane.
///
/// Expects single channel texture format.
///
/// First comes entire image in Y in one plane,
/// followed by a plane with interleaved lines ordered as U0, V0, U1, V1, etc.
///
/// width
/// __________
/// | |
/// height | Y |
/// | |
/// |_________|
/// height/2 | U,V,U,… |
/// |_________|
Y_UV12 = 0,

/// YUV 4:2:2 subsampling, single plane.
///
/// Expects single channel texture format.
///
/// The order of the channels is Y0, U0, Y1, V0, all in the same plane.
///
/// width * 2
/// __________________
/// | |
/// height | Y0, U0, Y1, V0… |
/// |_________________|
///
YUYV16 = 1,
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love this 🤩

crates/viewer/re_renderer/src/workspace_shaders.rs Outdated Show resolved Hide resolved
zeroed_texture_sint,
zeroed_texture_uint,
device,
queue,
inner: Default::default(),
}
}

/// Creates a new 2D texture resource and schedules data upload to the GPU.
/// TODO(jleibs): All usages of this should be replaced with `get_or_create`, which is strictly preferable
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like we're only using create in crates/viewer/re_renderer/src/importer/gltf.rs

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and this one is surprisingly hard to get rid of because for that we'd to be sure that we get a unique enough description of the mesh without having to resort to a hash of the texture content within it.

Copy link
Member

@emilk emilk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beautiful! I love all the comments

LGTM if tested with NV12 and YUV on natie, WebGL and WebGPU (I'm paranoid!)

@Wumpf
Copy link
Member Author

Wumpf commented Oct 9, 2024

skipping linux web check for now. still have to do that though!

@Wumpf Wumpf merged commit c3af105 into main Oct 9, 2024
32 of 33 checks passed
@Wumpf Wumpf deleted the andreas/flexible-and-fast-image-conversion branch October 9, 2024 08:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
include in changelog 🔺 re_renderer affects re_renderer itself 🚜 refactor Change the code, not the functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants