xx-splat - 3D Gaussian Splatting for Real-Time Radiance Field Rendering

[bkaradzic]

Port WebGL 3D Gaussian Splatting technique to bgfx as an example:

https://github.com/antimatter15/splat

[Seyedof]

I'm on it

[bkaradzic]

@Seyedof check these articles too:
https://aras-p.info/blog/2023/09/13/Making-Gaussian-Splats-smaller/
https://aras-p.info/blog/2023/09/27/Making-Gaussian-Splats-more-smaller/

Also need to find one or few smaller in data-size data sets for example. We don't want to bloat repo with 40+MiB data sets.

For example these data sets are around 13MiB each:
https://gsplat.tech/materials/
https://gsplat.tech/lego/

[Seyedof]

@bkaradzic
Thanks for the links, yeah I've seen Aras's work on shrinking splat data, he uses textures.
I tend to keep this as simple as I can, since it's just an example.
Regarding data file sizes, the train scene is almost 32mb, the small files you sent are around 13mb.
The output quality is very sensitive to splat density, so for good looking scenes like this Train and the truck we should accept the data size.

There's also a workaround, I can fetch the splat file from http instead of local file system, no need to put it inside repo.

[bkaradzic]

I tend to keep this as simple as I can, since it's just an example.

Yeah, simple is better for examples.

There's also a workaround, I can fetch the splat file from http instead of local file system, no need to put it inside repo.

I would avoid this because it's not guaranteed to be available in the future.

[Seyedof]

@bkaradzic
It's almost done, the only problem is I couldn't make it work in Direct3D, only works in OGL, it that ok to create a pull request?

[bkaradzic]

What's the issue with D3D? Where is your branch? I can take a look.

[ichordev]

What about Vulkan? I thought OpenGL support was EoL

[Seyedof]

What's the issue with D3D? Where is your branch? I can take a look.

@bkaradzic Sorry I was too busy in past two weeks, just pushed my changes to a fork, here you can see it:

https://github.com/Seyedof/bgfx-gaussian

It works in both OpenGL and Vulkan and I assume it works for all the OGL brid (like WebGL,...) as well, but not on Direct3D family. This code needs a final minor cleanup pass to create a PR/MR on bgfx upstream, please take a look into it and let me know if you figure out how to fix d3d problem and I will fix it quickly and raise a PR.

To my understanding the problem might come from difference in D3D and GL matrix layout or the dynamic vertex buffer is not uploaded correctly or per-instance vertex layout is not right (based on some experiments I ran).

[bkaradzic]

D3D doesn't work because semantics used for instanced data here:
https://github.com/Seyedof/bgfx-gaussian/blob/dc033bc0fe2a6358b4a0ccc174945964d78af9b2/examples/50-gaussian-splatting/varying.def.sc#L5-L8

i_data0 should be TEXCOORD7, i_data1 should be TEXCOORD6, etc.
https://bkaradzic.github.io/bgfx/tools.html#vertex-shader-attributes

[Seyedof]

Thanks it worked after fixing this and with some more changes in shader, will clean up the code and raise a PR shortly.

[bkaradzic]

Few other things I noticed:

• use bx::tan
• fix resource leaks at exit