Mesh Distance Fields and maybe Nanite?

[KieranCoppins]

Hi there!

I'm looking into Lumen in Unreal Engine and have discovered that Cesium's tiles do not have any mesh distance fields generated. This causes Lumen to behave primarily in screen space and global illumination doesn't behave as well as it should.

I want to look into building these distance fields at runtime when a tile is loaded. I have been looking at the UCesiumGltfComponent and I can see some references to a static mesh here, but I am unable to tell if this is where the mesh is actually generated? Any insight on where mesh generation actually occurs would be fantastic.

As an addition with this, I would also like to experiment with building Nanite meshes from the Cesium tiles when they are loaded. I understand that tiles are streamed in and the building is meant to happen in editor as it is quite expensive to do at runtime, however my use case is that I have some 3d models that do not have LODs and are loaded as 3d tiles from Cesium Ion. This means that Nanite (and distance fields) only need to be created once it has loaded and it will remain loaded.

What I have already tried is inside the UCesiumGltfComponent::loadPrimitiveGameThreadPart I have taken the reference to the UStaticMesh and set generate distance fields to true as well as enable Nanite. However calling the Build function throws a warning saying that the StaticMesh has no source models, so I feel I'm looking in the wrong place here.

Any help and or insight would be appreciated!

Many thanks,

Kieran

[KieranCoppins]

I have just noticed that issue #1172 is linked to this as well and I would love to take a crack at it

[kring]

I have been looking at the UCesiumGltfComponent and I can see some references to a static mesh here, but I am unable to tell if this is where the mesh is actually generated?

Yes, that's where the UStaticMesh is created.

my use case is that I have some 3d models that do not have LODs and are loaded as 3d tiles from Cesium Ion.

In this case, I suspect it would be much easier to simply download the model from Cesium ion - using the Clips feature if necessary - and then add it to Unreal with the normal workflow. Trying to go via 3D Tiles -> Nanite statically seems like unnecessary complexity to me, but maybe it's justified for a reason that isn't obvious to me. If we could do it at runtime, that would be another story.

However calling the Build function throws a warning saying that the StaticMesh has no source models, so I feel I'm looking in the wrong place here.

It's pretty typical that you need to spend a lot of time spelunking through the Unreal source code in order to figure out how to do things at runtime that the Engine usually does at Editor time. loadPrimitiveGameThreadPart is the correct place to look. But whether setting it to generate distance fields and calling Build is sufficient or whether it's the right approach at all, I'm afraid I don't know.

[KieranCoppins]

In this case, I suspect it would be much easier to simply download the model from Cesium ion - using the Clips feature if necessary - and then add it to Unreal with the normal workflow. Trying to go via 3D Tiles -> Nanite statically seems like unnecessary complexity to me, but maybe it's justified for a reason that isn't obvious to me. If we could do it at runtime, that would be another story.

Unfortunately we can't load the meshes into the Unreal Editor as we want to be able to load different 3d tiles from Cesium Ion and use a config file to tell Unreal which asset ids from Cesium Ion to load. This allows for us to add new 3d tiles to Cesium Ion and update a config file with the new ids without having to rebuild the Unreal application.

It's pretty typical that you need to spend a lot of time spelunking through the Unreal source code in order to figure out how to do things at runtime that the Engine usually does at Editor time. loadPrimitiveGameThreadPart is the correct place to look. But whether setting it to generate distance fields and calling Build is sufficient or whether it's the right approach at all, I'm afraid I don't know.

I stepped through that section with the debugger and it shows that the source mesh is not populated when loading the static mesh. The part that I am struggling to find is where the mesh is actually generated. By my understanding it must download the vertex, uv, normal data from Cesium Ion and use that to generate the mesh, possibly using a FMeshDescriptionBuilder? Or is there some kind of GLTF loader that downloads the 3d tiles as a .glb file and loads that into a UStaticMesh? Either way, it feels that the source mesh property on the UStaticMesh should be populated? With this populated we can then tell Unreal to build mesh distance fields and possibly even tell it to build nanite meshes at runtime - albeit at a performance loss.

[KieranCoppins]

Upon further investigation, I have discovered that the UStaticMesh::Build function is an editor only function. I suspect that generating distance fields in general will be encapsulated in editor only preprocessor defines. I think that for distance fields to be generated, a bespoke algorithm will need to be devised to create and set the data at runtime. After some research, it appears that the voxel plugin managed to generate distance fields at runtime but, I assume, it is a simplified algorithm they have made which is runtime friendly.

[kring]

Unfortunately we can't load the meshes into the Unreal Editor as we want to be able to load different 3d tiles from Cesium Ion and use a config file to tell Unreal which asset ids from Cesium Ion to load. This allows for us to add new 3d tiles to Cesium Ion and update a config file with the new ids without having to rebuild the Unreal application.

That makes sense, but it sounds like it's at odds with baking Nanite meshes at Editor time, no?

I guess you're saying that because you don't have LODs, you only need to create the Nanite mesh once when it's first loaded, not incrementally as new tiles are loaded. If I have that correct, I don't think it solves the fundamental problem: Nanite meshes can only be created in the Editor (as far as I know). Doing the baking per tile isn't particularly hard (we do lots of things per tile). The question is just whether UE allows us to do the baking in a game (versus Editor) at all.

By my understanding it must download the vertex, uv, normal data from Cesium Ion and use that to generate the mesh, possibly using a FMeshDescriptionBuilder?

We do as much of that work as we can off the game thread in order to improve performance. It's in that same file, though. The entry point is loadModelAnyThreadPart. The part that actually populates the position data is in loadPrimitive (other attributes are nearby).

Upon further investigation, I have discovered that the UStaticMesh::Build function is an editor only function.

Yeah that is unfortunately common and not too surprising. It's possible there's a "lower level" way to do it that is available at runtime (perhaps by including an extra module that is usually only used by the Editor?), but there are no guarantees.