Vulkan crash: Device Lost during _fulltestsuite_.bv.luac

[galibzon]

A crash occurs when running ASV with -rhi=vulkan. Here is the log with -rhi-device-validation=enable:
asv_vulkan_crash.txt

It occurs randomly, but mostly during EyeMaterialTest.bv.luac.

O3DE Version:
commit 0dc26c6f0628e4b09d7e9be9a4f6408b4dd7e7ce (upstream/development, development)
Merge: b497b30c66 2541a9a312
Author: Danilo Aimini [email protected]
Date: Thu Apr 6 20:44:33 2023 +0100

ASV Version:
commit 5697989 (HEAD -> development, upstream/development)
Author: Tommy Walton [email protected]
Date: Thu Apr 6 08:52:31 2023 -0700

• Device: PC
• OS: Windows
• Version 10
• CPU AMD Ryzen Threadripper 3970X 32-Core Processor 3.70 GHz
• GPU NVidia RTX 3070
• Memory 128GB
• 2TB SSD NVMe PCIe4.0 Seagate FireCuda 520 SSD ZP2000GM30002

[martinwinter-huawei]

I have the "same" issue on Linux at the moment, for me it always hangs at the RHI/IndirectRendering example when running the full testsuite.
When starting the example by hand, it works without any issues.
But as part of the full testsuite, it always hangs there with Device lost.

Log:

<12:38:28> (Automation) - Script: ========= Begin Fast-check RHI/IndirectRendering =========

<12:38:31> (IndirectRenderingExampleComponent) - Got all variants

<12:38:31> [Error] (vkDebugMessage) - [ERROR][Validation] Validation Error: [ VUID-VkFramebufferCreateInfo-flags-04533 ] Object 0: handle = 0x7f0b52a10070, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0xfe6b2428 | vkCreateFramebuffer(): VkFramebufferCreateInfo attachment #1 mip level 0 has width (800) smaller than the corresponding framebuffer width (2560). The Vulkan spec states: If renderpass is not VK_NULL_HANDLE and flags does not include VK_FRAMEBUFFER_CREATE_IMAGELESS_BIT, each element of pAttachments that is used as an input, color, resolve, or depth/stencil attachment by renderPass must have been created with a VkImageCreateInfo::width greater than or equal to width (https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/vkspec.html#VUID-VkFramebufferCreateInfo-flags-04533)
<12:38:31> [Error] (vkDebugMessage) - [ERROR][Validation] Validation Error: [ VUID-VkFramebufferCreateInfo-flags-04534 ] Object 0: handle = 0x7f0b52a10070, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x77e9b3aa | vkCreateFramebuffer(): VkFramebufferCreateInfo attachment #1 mip level 0 has height (600) smaller than the corresponding framebuffer height (1355). The Vulkan spec states: If renderpass is not VK_NULL_HANDLE and flags does not include VK_FRAMEBUFFER_CREATE_IMAGELESS_BIT, each element of pAttachments that is used as an input, color, resolve, or depth/stencil attachment by renderPass must have been created with a VkImageCreateInfo::height greater than or equal to height (https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/vkspec.html#VUID-VkFramebufferCreateInfo-flags-04534)
<12:38:31> [Error] (vkDebugMessage) - [ERROR][Validation] Validation Error: [ VUID-VkFramebufferCreateInfo-flags-04533 ] Object 0: handle = 0x7f0b52a10070, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0xfe6b2428 | vkCreateFramebuffer(): VkFramebufferCreateInfo attachment #1 mip level 0 has width (800) smaller than the corresponding framebuffer width (2560). The Vulkan spec states: If renderpass is not VK_NULL_HANDLE and flags does not include VK_FRAMEBUFFER_CREATE_IMAGELESS_BIT, each element of pAttachments that is used as an input, color, resolve, or depth/stencil attachment by renderPass must have been created with a VkImageCreateInfo::width greater than or equal to width (https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/vkspec.html#VUID-VkFramebufferCreateInfo-flags-04533)
<12:38:31> [Error] (vkDebugMessage) - [ERROR][Validation] Validation Error: [ VUID-VkFramebufferCreateInfo-flags-04534 ] Object 0: handle = 0x7f0b52a10070, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x77e9b3aa | vkCreateFramebuffer(): VkFramebufferCreateInfo attachment #1 mip level 0 has height (600) smaller than the corresponding framebuffer height (1355). The Vulkan spec states: If renderpass is not VK_NULL_HANDLE and flags does not include VK_FRAMEBUFFER_CREATE_IMAGELESS_BIT, each element of pAttachments that is used as an input, color, resolve, or depth/stencil attachment by renderPass must have been created with a VkImageCreateInfo::height greater than or equal to height (https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/vkspec.html#VUID-VkFramebufferCreateInfo-flags-04534)
<12:38:31> (System) - 
==================================================================

<12:38:31> (System) - Trace::Assert
 /data/projects/o3de/engine/Gems/Atom/RHI/Vulkan/Code/Source/RHI/Vulkan.h(96): (139688675949696) 'void AZ::Vulkan::AssertSuccess(VkResult)'

<12:38:31> (System) - ASSERT: Vulkan API method failed: Device lost

<12:38:31> (System) - ------------------------------------------------

<12:38:32> (System) - AZ::Vulkan::AssertSuccess(VkResult) (+0x70) [0x7f0b9f31e430]

<12:38:32> (System) - AZ::Vulkan::Fence::WaitOnCpuInternal() const (+0x7a) [0x7f0b9f49ae0a]

<12:38:32> (System) - AZ::RHI::Fence::WaitOnCpu() const (+0x5c) [0x7f0b9f6bcd0c]

<12:38:32> (System) - AZ::Vulkan::CommandQueueContext::End() (+0x19c) [0x7f0b9f4baa3c]

<12:38:32> (System) - AZ::Vulkan::Device::EndFrameInternal() (+0xb8) [0x7f0b9f4c78a8]

<12:38:32> (System) - AZ::RHI::Device::EndFrame() (+0x6e) [0x7f0b9426043e]

<12:38:32> (System) - AZ::RHI::FrameScheduler::EndFrame() (+0xf4) [0x7f0b9420a924]

<12:38:32> (System) - AZ::RHI::RHISystem::FrameUpdate(AZStd::function<void (AZ::RHI::FrameGraphBuilder&)>) (+0x1a5) [0x7f0b94158dc5]

<12:38:32> (System) - AZ::RPI::RPISystem::RenderTick() (+0x1d6) [0x7f0b93c052c6]

<12:38:32> (System) - AZ::RPI::RPISystemComponent::OnSystemTick() (+0x263) [0x7f0b915ab743]

<12:38:32> (System) - non-virtual thunk to AZ::RPI::RPISystemComponent::OnSystemTick() (+0x26) [0x7f0b915ab7a6]

<12:38:32> (System) - decltype (((*(InvokeTraits::forward<AZ::Internal::HandlerNode<AZ::SystemTickEvents, AZ::SystemTickEvents, AZ::Internal::EBusContainer<AZ::SystemTickEvents, AZ::SystemTickEvents, (AZ::EBusAddressPolicy)0, (AZ::EBusHandlerPolicy)1>::HandlerHolder, false>&>(

<12:38:32> (System) - void AZ::EBusEventProcessingPolicy::Call<void (AZ::SystemTickEvents::*&)(), AZ::Internal::HandlerNode<AZ::SystemTickEvents, AZ::SystemTickEvents, AZ::Internal::EBusContainer<AZ::SystemTickEvents, AZ::SystemTickEvents, (AZ::EBusAddressPolicy)0, (AZ::EBusHa

<12:38:32> (System) - AZ::ComponentApplication::TickSystem() (+0x53) [0x559876a24d23]

<12:38:32> (System) - (anonymous namespace)::RunMainLoop(AzGameFramework::GameApplication&) (+0xa9) [0x55987467f289]

<12:38:32> (System) - O3DELauncher::Run(O3DELauncher::PlatformMainInfo const&) (+0xbf0) [0x55987467efd0]

<12:38:32> (System) - main (+0x162) [0x55987467f452]

<12:38:32> (System) - __libc_init_first (+0x90) [0x7f0bcd43c790]

<12:38:32> (System) - __libc_start_main (+0x8a) [0x7f0bcd43c84a]

<12:38:32> (System) - _start (+0x25) [0x55987467d455]