Debug problem with new Asset Bundles Converter changes and some scenes

[pravusjif]

We have reports that these scenes are having invisible assets after re-converting with the latest Asset Bundles converter update:

• 25,-120 (the scene above)
• -8, 100 (university scene)
• -10,100 (university scene)
• -13,103 (university scene)
• Carl fravel in -90,15
• deandotland in 138,-3

the AB Converter updates has already been reverted.

We should:

• Re-convert the affected scenes
• Invalidate the AB fetching url cache in the CDN
• Debug why the latest changes in the AB Converter break those conversions
• Ideally add some visual test or test to avoid this case in the future (Since the bug only happens when converting a whole scene, it's very difficult to have an automated test with local assets)
• Re-update asset bundles converter

NOTE
the first 4 scenes have already been re-converted during the weekend and the cache already invalidated so they are looking fine in production.

---

Edit:

• Unfinished PR: fix: asset bundles converter gltfs dependencies unity-renderer#2219
• The commit from this PR breaks the conversion of scene -8, 100: https://github.com/decentraland/unity-renderer/pull/2240/files
• This merged PR could affect the Conversor visual tests: fix: discard pixels below Y plane unity-renderer#2389

Updated Tasks:

• Fix Unity Editor crashing when converting this scene, due to multi-threadding
• Check if the conversor visual tests are affected or not by this merged PR, we may need to instantiate meshes in a higher Y position
• Re-test converting -8, 100, -10,100, -13,103
• Re-update asset bundles converter

[pravusjif]

Debugging findings

• It was reported that several scenes were having invisible asset bundle meshes after updating the Asset Bunbdles Converter.
• We reverted the latest AB Converter update, re-converted the affected scenes and invalidates the CDN ABs cache.
• After debugging the issue for a week using one of the affected scenes, with which the problem can be reproduced converting locally, the following is a report on all the findings until now:

• Some commit hunting shows that the scene is converted fine before the GLTFs Multi-Threadding update was merged in master (569a6a796b24bca9e58650cd94fb49fe21772ec3)
• Debugging was done checking the diff changes introduced in that commit and also testing local conversion with the previous commit (7d22053c23bf35fb3607632db57a918ee1181bc2), that converts the scene fine.
• Debugged converting the following scene:
  -8, 100 (university scene)
  - Isolated asset conversion tested:

            {
                "file": "models/ParisNFT/BoothPremium/PNFT_BoothPremium.gltf",
                "hash": "QmR4NymhuRnJM9X1afKJoDgifkUm6Sg7hHch2GkTDKSnS3"
            }

• When converting an affected asset in an isolated way I could reproduce the problem as well, and after solving that I discovered it was only a problem for isolated assets conversion and didn't fix the whole scene conversion. That means that when that asset is converted in an isolated way it converts fine, but when it is converted along with the whole scene, its conversion is broken and it ends up being invisible.
• Further debugging showed that the affected assets are being created without a mesh and that's why they are invisible.
• Another thing found when examining the visual tests comparison between the GLTF version and the Asset Bundle version of the converted assets is that when the editor imports the GLTF version, it's already invisible, so the actual problem is no in the conversion but in the loading of the original GLTF asset during the conversion process.
• The problem is specifically happening on some GLTFs with external dependencies (buffer and texture files).
• The NullReference exception we see when converting the affected assets is due to the GLTFSceneImporter not finding the previously-injected BUFFER FILES or TEXTURE FILES. Actually they are downloaded but the STREAM object is lost at some point (or not correctly loaded from the PersistentAssetCache)
• If we remove the buffer assets cache list clearing that happens at Core.DumpGltf(), the problem appears to be solved
  • In the pre-multithreadding commit we also cleared the buffer and image caches in Core.DumpGLTF(), but with the multi-threadding commit we moved that to Core.CleanupCache()
  • GLTFSceneImporter - Error - Buffer file not found (/Users/pravus/git/unity-renderer/unity-renderer/Assets/_Downloaded/QmRvnNSpUpyueXX7QUMxLGGNRh4dF3rjYh4eCJ7GWth1Lt/MainEntranceRightDoor.bin) -- MainEntranceRightDoor.bin
    • Triggers the usual NullReference exception when the buffer file is not found
    • When converting this asset in isolation, the buffer file appears to be downloaded/injected after the GLTF, when it should happen before as with the dependency textures
    • In the pre-multithreadding commit, this asset conversion is also broken, so we can skip debugging it.
  • 3 assets not being converted
    • Visual Test Detection: FAILED converting asset -> QmQvNTs1uNYiWty6v6iTEGz72tzLRjvgnFUytkYXZtpZXf
    • Visual Test Detection: FAILED converting asset -> QmW4LiUDdut1ygU9wZErasLpyXg2gBbcYSV9KtcqnkiFVq
    • Visual Test Detection: FAILED converting asset -> QmYtLrvWoAcUhXDTpmETMmLmtBSFGJz53urEzPqcYqtDB9
    • In the pre-multithreadding commit, only 'QmW4LiUDdut1ygU9wZErasLpyXg2gBbcYSV9KtcqnkiFVq' and additionally 'Qmf6YeiSSNrbhvr3WiFwmB26aC13ccebm9SgaGPgYXnDhM' assets fail converting.

Next

• Will remove clearing the buffer cache on every GLTF dump (since with multi-threadding commit we moved that to Core.CleanupCache()`) since that fixes the problem and will test the fix with the other affected scenes.
• Will try to add some test coverage, maybe a visual test with a local gltf in test resources
• Will make a list of all the cases of textures being shared as linear and non-linear, to communicate that to the creator

[pravusjif]

• Tested with the first 4 affected scenes and the result assets are the same as the ones resulting from converting using the pre-multithreadding commit
• Couldn't create an automated test for the dependencies problem as the problem happens when converting a whole scene (not with isolated asset conversions) and that makes it super difficult to automate with local test-resource assets
• Took note of the texture assets that are being shared in the wrong way (linear and srgb at the same time):

-8, 100 (university scene)

GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUStudentVenueDesk_Metal-DCLUStudentVenueDesk_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'ModularCubesSeatBooth_Metal-ModularCubesSeatBooth_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'LongiHighBackSofa_Metal-LongiHighBackSofa_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'PNFT_BoothPremuim_Metal-PNFT_BoothPremuim_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'PNFT_Booth01_Metal-PNFT_Booth01_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUD_SVPillarsStructure_Metal-DCLUD_SVPillarsStructure_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUD_SVRoofDeskSupports_Metal-DCLUD_SVRoofDeskSupports_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUD_SVInteriorWall_Metal-DCLUD_SVInteriorWall_Roughness2.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUD_SVBalcony_Metal.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUD_SVStairs_Metal.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'PNFT_ModularCubesSeatBooth_Metal-PNFT_ModularCubesSeatBooth_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUD_SVWindowDoorFrames_MAT_Metal-DCLUD_SVWindowDoorFrames_MAT_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.

-10,100 (university scene)

GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUD_OutsideProps_metal-DCLUD_OutsideProps_roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'PathBinSignPost_Metal-PathBinSignPost_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUDCareerCentreLift_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'Master_ConcreteWallTiles_Normal.jpg' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUDCareerCentreLiftButtons_Metal-DCLUDCareerCentreLiftButtons_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'CareerCentreRoofTopFrames_MAT_Metal_jpg-CareerCentreRoofTopFrames_MAT_Roughness_jpg.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'CareerCentreBuildingGlass_Metal.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'CareerCentreBuildingGlass_Normal.jpg' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUD_CareerCentreSupports_Metal-DCLUD_CareerCentreSupports_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUD_FrontEntranceFrames_Metal-DCLUD_FrontEntranceFrames_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUD_BackEntranceFrame_Metal-DCLUD_BackEntranceFrame_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'SeaterOfficeSofa_Metal-SeaterOfficeSofa_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.

-13,103 (university scene)

GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUD_UniversityShopFrontWindowsFrames_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUD_WoodFloor01_normal.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUD_UniversityShopFirstFloorWalls_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUD_UniversityShopSupportsTrims_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUD_UniversityShopRoof_MAT_Metal-DCLUD_UniversityShopRoof_MAT_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUD_UnivsersityShopStairs_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.
GLTF IMPORTER WARNING: using same texture as linear and srgb will lead to visual artifacts. If 'DCLUD_UniversityShopFirstFloorPatternWall_Roughness.png' is being used as a normal map or metallic map, make sure it's only used in those material properties on every model.

[pravusjif]

Updated issue description with new tasks list