SWDEV-359379 - catch2: Standalone single exe per file [EXPERIMENTAL- DO NOT MERGE] #86

agunashe · 2022-12-19T11:30:29Z

SWDEV-359379 - catch2: Standalone single exe per file

Use -DSTANDALONE_TESTS=1 to build standalone tests one exe per file. Default option will continue to build one exe for all the files under a submodule.

TODO - This PR fails to execute on Windows. Will update this PR when I have the fix for windows too.
Currently only ABM folder is used for experiment. Please uncomment the folders in [catch/CMakeLists.txt] file to build other folders.

Change-Id: I292765dded4ff745f02d3d284168663875853d76

pvelesko · 2023-01-04T13:10:38Z

@agunashe @gargrahul

It was not clear on what to do this.
Only at the bottom of this page did I see a note about hip-tests https://docs.amd.com/en-US/bundle/HIP-Installation-Guide-v5.4/page/How_to_Build_HIP_from_Source.html

and it seems that hip-tests is not being used anywhere. I modified CMakeLists.txt on HIPAMD to use these tests. Not sure if this is what I was supposed to do but it worked!

What I think I'll do is drop all the changes to tests from ROCm/HIP#2975 and move these changes to hip-tests? Most of these changes are trivial like doing/not doing things based on HT_AMD for example.

pvelesko · 2023-01-08T13:04:49Z

@agunashe @gargrahul

I was able to integrate this into CHIP-SPV but the main issues remain:

The test discovery doesn't happen during compilation. It would be very helpful to be able to toggle this. It would be great to have a -DCATCH_DISCOVER_DURING_COMPILE option.
The test discovery doesn't seem to persist.

consider a test file hipMemcpy.cc here we have multiple sections each corresponding to a ctest case which can either pass or fail. Let's say I am working on Unit_hipMemcpy_PinnedRegMemWithKernelLaunch

I would like to build the code and run
ctest -R Unit_hipMemcpy_PinnedRegMemWithKernelLaunch

As it stands now, ctest will not know where this test is located so it will have to run all the executables (JITing things in the process) until it finds it and then run. If I want to re-run the test case this process will be repeated because the test discovery doesn't seem to persist.

Please note that this is a regression from HIP 4.x where things were working well.

pvelesko · 2023-01-09T09:05:57Z

To illustrate: ~6min to run a single test through cmake and 6min again running the 2nd time (meaning test registration is not saved)

pvelesko@arcticus08:~/space/CHIP-SPV/build> time ctest -R Unit_hipGraphAddMemcpyNode_BasicFunctional
Test project /home/pvelesko/space/CHIP-SPV/build
CMake Warning at /gpfs/jlse-fs0/users/pvelesko/CHIP-SPV/build/catch/catch_tests/script/CatchAddTests.cmake:48 (message):
  Test executable
  '/gpfs/jlse-fs0/users/pvelesko/CHIP-SPV/build/catch/catch_tests/unit/device/hipExtGetLinkTypeAndHopCount'
  contains no tests!



    Start 107: Unit_hipGraphAddMemcpyNode_BasicFunctional
1/1 Test #107: Unit_hipGraphAddMemcpyNode_BasicFunctional ...***Exception: SegFault  0.31 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) = 360.27 sec

The following tests FAILED:
	107 - Unit_hipGraphAddMemcpyNode_BasicFunctional (SEGFAULT)
Errors while running CTest
Output from these tests are in: /home/pvelesko/space/CHIP-SPV/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

real	6m0.285s
user	1m47.095s
sys	0m18.705s
pvelesko@arcticus08:~/space/CHIP-SPV/build>
pvelesko@arcticus08:~/space/CHIP-SPV/build> time ctest -R Unit_hipGraphAddMemcpyNode_BasicFunctional
Test project /home/pvelesko/space/CHIP-SPV/build
CMake Warning at /gpfs/jlse-fs0/users/pvelesko/CHIP-SPV/build/catch/catch_tests/script/CatchAddTests.cmake:48 (message):
  Test executable
  '/gpfs/jlse-fs0/users/pvelesko/CHIP-SPV/build/catch/catch_tests/unit/device/hipExtGetLinkTypeAndHopCount'
  contains no tests!



    Start 107: Unit_hipGraphAddMemcpyNode_BasicFunctional
1/2 Test #107: Unit_hipGraphAddMemcpyNode_BasicFunctional ...***Exception: SegFault  0.30 sec
    Start 308: Unit_hipGraphAddMemcpyNode_BasicFunctional
2/2 Test #308: Unit_hipGraphAddMemcpyNode_BasicFunctional ...Subprocess aborted***Exception:   0.30 sec

0% tests passed, 2 tests failed out of 2

Total Test time (real) = 357.62 sec

The following tests FAILED:
	107 - Unit_hipGraphAddMemcpyNode_BasicFunctional (SEGFAULT)
	308 - Unit_hipGraphAddMemcpyNode_BasicFunctional (Subprocess aborted)
Errors while running CTest
Output from these tests are in: /home/pvelesko/space/CHIP-SPV/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

real	5m57.677s
user	1m46.266s
sys	0m18.949s

Compare this to running the test directly from the executable:

pvelesko@arcticus08:~/space/CHIP-SPV/build> time ./catch/catch_tests/unit/graph/hipGraphAddMemcpyNode "Unit_hipGraphAddMemcpyNode_BasicFunctional"
Filters: Unit_hipGraphAddMemcpyNode_BasicFunctional




Abort was called at 84 line in file:
/home/ubit/rpmbuild/BUILD/compute-runtime-1.3.24347.3/level_zero/core/source/cmdlist/cmdlist.cpp

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
hipGraphAddMemcpyNode is a Catch v2.13.4 host application.
Run with -? for options

-------------------------------------------------------------------------------
Unit_hipGraphAddMemcpyNode_BasicFunctional
  Memcpy with 2D array on default device
-------------------------------------------------------------------------------
/home/pvelesko/space/CHIP-SPV/hip-tests/catch/unit/graph/hipGraphAddMemcpyNode.cc:481
...............................................................................

/home/pvelesko/space/CHIP-SPV/hip-tests/catch/unit/graph/hipGraphAddMemcpyNode.cc:481: FAILED:
due to a fatal error condition:
  SIGABRT - Abort (abnormal termination) signal

===============================================================================
test cases: 1 | 1 failed
assertions: 3 | 2 passed | 1 failed

Aborted

real	0m0.485s
user	0m0.217s
sys	0m0.066s

pvelesko · 2023-01-09T13:19:35Z

I noticed that this issue is also present HIP. It was introduced at some point after 4.5, I believe. Here is a diff that fixes the issue for me:

diff --git a/tests/catch/external/Catch2/cmake/Catch2/Catch.cmake b/tests/catch/external/Catch2/cmake/Catch2/Catch.cmake
index 869a6785..67343c63 100644
--- a/tests/catch/external/Catch2/cmake/Catch2/Catch.cmake
+++ b/tests/catch/external/Catch2/cmake/Catch2/Catch.cmake
@@ -142,24 +142,43 @@ function(catch_discover_tests TARGET)
   # Define rule to generate test list for aforementioned test executable
   set(ctest_include_file "${CMAKE_CURRENT_BINARY_DIR}/${TARGET}_include-${args_hash}.cmake")
   set(ctest_tests_file "${CMAKE_CURRENT_BINARY_DIR}/${TARGET}_tests-${args_hash}.cmake")
-  file(RELATIVE_PATH ctestincludepath ${CMAKE_CURRENT_BINARY_DIR} ${ctest_include_file})
-  file(RELATIVE_PATH ctestfilepath ${CMAKE_CURRENT_BINARY_DIR} ${ctest_tests_file})
-  file(RELATIVE_PATH _workdir ${CMAKE_CURRENT_BINARY_DIR} ${_WORKING_DIRECTORY})
-  file(RELATIVE_PATH _CATCH_ADD_TEST_SCRIPT ${CMAKE_CURRENT_BINARY_DIR} ${ADD_SCRIPT_PATH})
-
   get_property(crosscompiling_emulator
     TARGET ${TARGET}
     PROPERTY CROSSCOMPILING_EMULATOR
   )
+  add_custom_command(
+    TARGET ${TARGET} POST_BUILD
+    BYPRODUCTS "${ctest_tests_file}"
+    COMMAND "${CMAKE_COMMAND}"
+            -D "TEST_TARGET=${TARGET}"
+            -D "TEST_EXECUTABLE=$<TARGET_FILE:${TARGET}>"
+            -D "TEST_EXECUTOR=${crosscompiling_emulator}"
+            -D "TEST_WORKING_DIR=${_WORKING_DIRECTORY}"
+            -D "TEST_SPEC=${_TEST_SPEC}"
+            -D "TEST_EXTRA_ARGS=${_EXTRA_ARGS}"
+            -D "TEST_PROPERTIES=${_PROPERTIES}"
+            -D "TEST_PREFIX=${_TEST_PREFIX}"
+            -D "TEST_SUFFIX=${_TEST_SUFFIX}"
+            -D "TEST_LIST=${_TEST_LIST}"
+            -D "TEST_REPORTER=${_REPORTER}"
+            -D "TEST_OUTPUT_DIR=${_OUTPUT_DIR}"
+            -D "TEST_OUTPUT_PREFIX=${_OUTPUT_PREFIX}"
+            -D "TEST_OUTPUT_SUFFIX=${_OUTPUT_SUFFIX}"
+            -D "CTEST_FILE=${ctest_tests_file}"
+            -P "${_CATCH_DISCOVER_TESTS_SCRIPT}"
+    VERBATIM
+  )

-  set(EXEC_NAME ${TARGET})
-  if(WIN32)
-    set(EXEC_NAME ${EXEC_NAME}.exe)
-  endif()
+  file(RELATIVE_PATH ctestincludepath ${CMAKE_CURRENT_BINARY_DIR} ${ctest_include_file})
+  file(RELATIVE_PATH ctestfilepath ${CMAKE_CURRENT_BINARY_DIR} ${ctest_tests_file})

-  # uses catch_include.cmake.in file to generate the *_include.cmake file
-  # *_include.cmake is used to generate the *_test.cmake during execution of ctest cmd
-  configure_file(${CATCH2_INCLUDE} ${TARGET}_include-${args_hash}.cmake @ONLY)
+  file(WRITE "${ctest_include_file}"
+    "if(EXISTS \"${ctestfilepath}\")\n"
+    "  include(\"${ctestfilepath}\")\n"
+    "else()\n"
+    "  message(WARNING \"Test ${TARGET} not built yet.\")\n"
+    "endif()\n"
+  )

   if(NOT ${CMAKE_VERSION} VERSION_LESS "3.10.0")
     # Add discovered tests to directory TEST_INCLUDE_FILES
@@ -216,9 +235,14 @@ function(hip_add_exe_to_target)
   else ()
     add_executable(${_NAME} EXCLUDE_FROM_ALL ${_TEST_SRC} $<TARGET_OBJECTS:Main_Object>)
     if(HIP_PLATFORM STREQUAL "amd")
-      target_link_libraries(${_NAME} hiprtc)
+      target_link_libraries(${_EXE_NAME} hiprtc)
+    elseif(HIP_PLATFORM STREQUAL "nvidia")
+      target_link_libraries(${_EXE_NAME} nvrtc)
+    elseif(HIP_PLATFORM STREQUAL "spirv")
+      message(FATAL_ERROR "RTC path for SPIRV not yet checked")
+      target_link_libraries(${_EXE_NAME} spirv)
     else()
-      target_link_libraries(${_NAME} nvrtc)
+      message(FATAL_ERROR "Unsupported HIP_PLATFORM: ${HIP_PLATFORM}")
     endif()
   endif()
   catch_discover_tests(${_NAME} PROPERTIES  SKIP_REGULAR_EXPRESSION "HIP_SKIP_THIS_TEST")

gargrahul · 2023-01-11T00:15:32Z

@agunashe @gargrahul

It was not clear on what to do this. Only at the bottom of this page did I see a note about hip-tests https://docs.amd.com/en-US/bundle/HIP-Installation-Guide-v5.4/page/How_to_Build_HIP_from_Source.html

and it seems that hip-tests is not being used anywhere. I modified CMakeLists.txt on HIPAMD to use these tests. Not sure if this is what I was supposed to do but it worked!

What I think I'll do is drop all the changes to tests from ROCm-Developer-Tools/HIP#2975 and move these changes to hip-tests? Most of these changes are trivial like doing/not doing things based on HT_AMD for example.

Hi @pvelesko , instructions for building HIP tests from hip-tests project are at https://github.com/ROCm-Developer-Tools/hip-tests#build-hip-catch-tests.

Yes please move HIP tests (catch ) related changes to this project as a PR.

pvelesko · 2023-01-11T03:31:17Z

Hi @pvelesko , instructions for building HIP tests from hip-tests project are at https://github.com/ROCm-Developer-Tools/hip-tests#build-hip-catch-tests.

I was under the impression that all documentation is moving to docs.amd.com so it would be nice to include these in the "installation guide" or something like that.

agunashe · 2023-01-11T10:41:18Z

@agunashe @gargrahul

I was able to integrate this into CHIP-SPV but the main issues remain:

The test discovery doesn't happen during compilation. It would be very helpful to be able to toggle this. It would be great to have a -DCATCH_DISCOVER_DURING_COMPILE option.

The test discovery doesn't seem to persist.

consider a test file hipMemcpy.cc here we have multiple sections each corresponding to a ctest case which can either pass or fail. Let's say I am working on Unit_hipMemcpy_PinnedRegMemWithKernelLaunch

I would like to build the code and run ctest -R Unit_hipMemcpy_PinnedRegMemWithKernelLaunch

As it stands now, ctest will not know where this test is located so it will have to run all the executables (JITing things in the process) until it finds it and then run. If I want to re-run the test case this process will be repeated because the test discovery doesn't seem to persist.

Please note that this is a regression from HIP 4.x where things were working well.

@pvelesko - your diff in the other comment is the original code from catch framework. This had worked fine for linux but on windows we had issues detecting the executable. It looked like a race condition between generation of exe and using the exe to find the tests names.
One solution was moving the test detection to the execution phase so that executables are sure to be present.
I will continue to investigate into the compilation time test detection.

pvelesko · 2023-01-11T11:01:59Z

@agunashe

your diff in the other comment is the original code from catch framework. This had worked fine for linux but on windows we had issues detecting the executable. It looked like a race condition between generation of exe and using the exe to find the tests names.

Yes - I'm still using the integrated HIP tests rather than hip-tests due to this. Could you integrate this into the current branch for the UNIX path? It would be very helpful and I could switch to using hip-tests.

maybe an option like -DCMAKE_DISCOVER_TESTS_POST_BUILD that can be off by default.

pvelesko · 2024-03-13T23:57:09Z

Standalone has been merged and test discovery has been addressed in #459 - this can be closed.

SWDEV-359379 - catch2: Standalone single exe per file

ce784a7

Change-Id: I292765dded4ff745f02d3d284168663875853d76

agunashe mentioned this pull request Dec 19, 2022

SWDEV-359379 - WIP catch2 exe per src file inplace of one exe per module [TEST ONLY. DO NOT MERGE] ROCm/HIP#3018

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SWDEV-359379 - catch2: Standalone single exe per file [EXPERIMENTAL- DO NOT MERGE] #86

SWDEV-359379 - catch2: Standalone single exe per file [EXPERIMENTAL- DO NOT MERGE] #86

agunashe commented Dec 19, 2022

pvelesko commented Jan 4, 2023

pvelesko commented Jan 8, 2023

pvelesko commented Jan 9, 2023

pvelesko commented Jan 9, 2023

gargrahul commented Jan 11, 2023

pvelesko commented Jan 11, 2023

agunashe commented Jan 11, 2023 •

edited

Loading

pvelesko commented Jan 11, 2023

pvelesko commented Mar 13, 2024

SWDEV-359379 - catch2: Standalone single exe per file [EXPERIMENTAL- DO NOT MERGE] #86

Are you sure you want to change the base?

SWDEV-359379 - catch2: Standalone single exe per file [EXPERIMENTAL- DO NOT MERGE] #86

Conversation

agunashe commented Dec 19, 2022

pvelesko commented Jan 4, 2023

pvelesko commented Jan 8, 2023

pvelesko commented Jan 9, 2023

pvelesko commented Jan 9, 2023

gargrahul commented Jan 11, 2023

pvelesko commented Jan 11, 2023

agunashe commented Jan 11, 2023 • edited Loading

pvelesko commented Jan 11, 2023

pvelesko commented Mar 13, 2024

agunashe commented Jan 11, 2023 •

edited

Loading