-
Notifications
You must be signed in to change notification settings - Fork 663
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ci: debug lidar_centerpoint ci failure #7712
Conversation
Signed-off-by: M. Fatih Cırıt <[email protected]>
Thank you for contributing to the Autoware project! 🚧 If your pull request is in progress, switch it to draft mode. Please ensure:
|
It is "Building tests for lidar_centerpoint" finds cuda too. But for some reason, it is not testing it correctly 🤷♂️ On my machine: 🖱️Click here to expand🔛mfc@mfc-leo:~/projects/autoware$ colcon test --event-handlers console_cohesion+ --mixin coverage-pytest --ctest-args --packages-select lidar_centerpoint
Starting >>> lidar_centerpoint
--- output: lidar_centerpoint
UpdateCTestConfiguration from :/home/mfc/projects/autoware/build/lidar_centerpoint/CTestConfiguration.ini
Parse Config file:/home/mfc/projects/autoware/build/lidar_centerpoint/CTestConfiguration.ini
Site: mfc-leo
Build name: (empty)
Add coverage exclude regular expressions.
Create new tag: 20240626-1415 - Experimental
UpdateCTestConfiguration from :/home/mfc/projects/autoware/build/lidar_centerpoint/CTestConfiguration.ini
Parse Config file:/home/mfc/projects/autoware/build/lidar_centerpoint/CTestConfiguration.ini
Test project /home/mfc/projects/autoware/build/lidar_centerpoint
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
Start 1: copyright
1: Test command: /usr/bin/python3.10 "-u" "/home/mfc/projects/autoware/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/copyright.xunit.xml" "--package-name" "lidar_centerpoint" "--output-file" "/home/mfc/projects/autoware/build/lidar_centerpoint/ament_copyright/copyright.txt" "--command" "/opt/ros/humble/bin/ament_copyright" "--xunit-file" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/copyright.xunit.xml"
1: Test timeout computed to be: 200
1: -- run_test.py: invoking following command in '/home/mfc/projects/autoware/src/universe/autoware.universe/perception/lidar_centerpoint':
1: - /opt/ros/humble/bin/ament_copyright --xunit-file /home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/copyright.xunit.xml
1: No problems found, checked 35 files
1: -- run_test.py: return code 0
1: -- run_test.py: verify result file '/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/copyright.xunit.xml'
1/10 Test #1: copyright ........................ Passed 1.34 sec
test 2
Start 2: cppcheck
2: Test command: /usr/bin/python3.10 "-u" "/home/mfc/projects/autoware/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/cppcheck.xunit.xml" "--package-name" "lidar_centerpoint" "--output-file" "/home/mfc/projects/autoware/build/lidar_centerpoint/ament_cppcheck/cppcheck.txt" "--command" "/opt/ros/humble/bin/ament_cppcheck" "--xunit-file" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/cppcheck.xunit.xml" "--include_dirs" "/home/mfc/projects/autoware/src/universe/autoware.universe/perception/lidar_centerpoint/include" "/home/mfc/projects/autoware/src/universe/autoware.universe/perception/lidar_centerpoint/include"
2: Test timeout computed to be: 300
2: -- run_test.py: invoking following command in '/home/mfc/projects/autoware/src/universe/autoware.universe/perception/lidar_centerpoint':
2: - /opt/ros/humble/bin/ament_cppcheck --xunit-file /home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/cppcheck.xunit.xml --include_dirs /home/mfc/projects/autoware/src/universe/autoware.universe/perception/lidar_centerpoint/include /home/mfc/projects/autoware/src/universe/autoware.universe/perception/lidar_centerpoint/include
2: cppcheck 2.7 has known performance issues and therefore will not be used, set the AMENT_CPPCHECK_ALLOW_SLOW_VERSIONS environment variable to override this.
2: -- run_test.py: return code 0
2: -- run_test.py: verify result file '/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/cppcheck.xunit.xml'
2/10 Test #2: cppcheck ......................... Passed 0.21 sec
test 3
Start 3: lint_cmake
3: Test command: /usr/bin/python3.10 "-u" "/home/mfc/projects/autoware/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/lint_cmake.xunit.xml" "--package-name" "lidar_centerpoint" "--output-file" "/home/mfc/projects/autoware/build/lidar_centerpoint/ament_lint_cmake/lint_cmake.txt" "--command" "/opt/ros/humble/bin/ament_lint_cmake" "--xunit-file" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/lint_cmake.xunit.xml"
3: Test timeout computed to be: 60
3: -- run_test.py: invoking following command in '/home/mfc/projects/autoware/src/universe/autoware.universe/perception/lidar_centerpoint':
3: - /opt/ros/humble/bin/ament_lint_cmake --xunit-file /home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/lint_cmake.xunit.xml
3:
3: No problems found
3: -- run_test.py: return code 0
3: -- run_test.py: verify result file '/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/lint_cmake.xunit.xml'
3/10 Test #3: lint_cmake ....................... Passed 0.18 sec
test 4
Start 4: xmllint
4: Test command: /usr/bin/python3.10 "-u" "/home/mfc/projects/autoware/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/xmllint.xunit.xml" "--package-name" "lidar_centerpoint" "--output-file" "/home/mfc/projects/autoware/build/lidar_centerpoint/ament_xmllint/xmllint.txt" "--command" "/opt/ros/humble/bin/ament_xmllint" "--xunit-file" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/xmllint.xunit.xml"
4: Test timeout computed to be: 60
4: -- run_test.py: invoking following command in '/home/mfc/projects/autoware/src/universe/autoware.universe/perception/lidar_centerpoint':
4: - /opt/ros/humble/bin/ament_xmllint --xunit-file /home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/xmllint.xunit.xml
4: File 'package.xml' is valid
4:
4: File 'launch/lidar_centerpoint.launch.xml' is valid
4:
4: No problems found
4: -- run_test.py: return code 0
4: -- run_test.py: verify result file '/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/xmllint.xunit.xml'
4/10 Test #4: xmllint .......................... Passed 0.49 sec
test 5
Start 5: test_detection_class_remapper
5: Test command: /usr/bin/python3.10 "-u" "/home/mfc/projects/autoware/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_detection_class_remapper.gtest.xml" "--package-name" "lidar_centerpoint" "--output-file" "/home/mfc/projects/autoware/build/lidar_centerpoint/ament_cmake_gtest/test_detection_class_remapper.txt" "--command" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_detection_class_remapper" "--gtest_output=xml:/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_detection_class_remapper.gtest.xml"
5: Test timeout computed to be: 60
5: -- run_test.py: invoking following command in '/home/mfc/projects/autoware/build/lidar_centerpoint':
5: - /home/mfc/projects/autoware/build/lidar_centerpoint/test_detection_class_remapper --gtest_output=xml:/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_detection_class_remapper.gtest.xml
5: [==========] Running 1 test from 1 test suite.
5: [----------] Global test environment set-up.
5: [----------] 1 test from DetectionClassRemapperTest
5: [ RUN ] DetectionClassRemapperTest.MapClasses
5: [ OK ] DetectionClassRemapperTest.MapClasses (0 ms)
5: [----------] 1 test from DetectionClassRemapperTest (0 ms total)
5:
5: [----------] Global test environment tear-down
5: [==========] 1 test from 1 test suite ran. (0 ms total)
5: [ PASSED ] 1 test.
5: -- run_test.py: return code 0
5: -- run_test.py: inject classname prefix into gtest result file '/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_detection_class_remapper.gtest.xml'
5: -- run_test.py: verify result file '/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_detection_class_remapper.gtest.xml'
5/10 Test #5: test_detection_class_remapper .... Passed 0.42 sec
test 6
Start 6: test_ros_utils
6: Test command: /usr/bin/python3.10 "-u" "/home/mfc/projects/autoware/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_ros_utils.gtest.xml" "--package-name" "lidar_centerpoint" "--output-file" "/home/mfc/projects/autoware/build/lidar_centerpoint/ament_cmake_gtest/test_ros_utils.txt" "--command" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_ros_utils" "--gtest_output=xml:/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_ros_utils.gtest.xml"
6: Test timeout computed to be: 60
6: -- run_test.py: invoking following command in '/home/mfc/projects/autoware/build/lidar_centerpoint':
6: - /home/mfc/projects/autoware/build/lidar_centerpoint/test_ros_utils --gtest_output=xml:/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_ros_utils.gtest.xml
6: [==========] Running 4 tests from 1 test suite.
6: [----------] Global test environment set-up.
6: [----------] 4 tests from TestSuite
6: [ RUN ] TestSuite.box3DToDetectedObject
6: [WARN 1719411316.311329629] [lidar_centerpoint]: Unexpected label: UNKNOWN is set. (box3DToDetectedObject() at /home/mfc/projects/autoware/src/universe/autoware.universe/perception/lidar_centerpoint/lib/ros_utils.cpp:40)
6: [ OK ] TestSuite.box3DToDetectedObject (1 ms)
6: [ RUN ] TestSuite.getSemanticType
6: [ OK ] TestSuite.getSemanticType (0 ms)
6: [ RUN ] TestSuite.convertPoseCovarianceMatrix
6: [ OK ] TestSuite.convertPoseCovarianceMatrix (0 ms)
6: [ RUN ] TestSuite.convertTwistCovarianceMatrix
6: [ OK ] TestSuite.convertTwistCovarianceMatrix (0 ms)
6: [----------] 4 tests from TestSuite (1 ms total)
6:
6: [----------] Global test environment tear-down
6: [==========] 4 tests from 1 test suite ran. (1 ms total)
6: [ PASSED ] 4 tests.
6: -- run_test.py: return code 0
6: -- run_test.py: inject classname prefix into gtest result file '/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_ros_utils.gtest.xml'
6: -- run_test.py: verify result file '/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_ros_utils.gtest.xml'
6/10 Test #6: test_ros_utils ................... Passed 0.19 sec
test 7
Start 7: test_nms
7: Test command: /usr/bin/python3.10 "-u" "/home/mfc/projects/autoware/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_nms.gtest.xml" "--package-name" "lidar_centerpoint" "--output-file" "/home/mfc/projects/autoware/build/lidar_centerpoint/ament_cmake_gtest/test_nms.txt" "--command" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_nms" "--gtest_output=xml:/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_nms.gtest.xml"
7: Test timeout computed to be: 60
7: -- run_test.py: invoking following command in '/home/mfc/projects/autoware/build/lidar_centerpoint':
7: - /home/mfc/projects/autoware/build/lidar_centerpoint/test_nms --gtest_output=xml:/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_nms.gtest.xml
7: [==========] Running 1 test from 1 test suite.
7: [----------] Global test environment set-up.
7: [----------] 1 test from NonMaximumSuppressionTest
7: [ RUN ] NonMaximumSuppressionTest.Apply
7: [ OK ] NonMaximumSuppressionTest.Apply (0 ms)
7: [----------] 1 test from NonMaximumSuppressionTest (0 ms total)
7:
7: [----------] Global test environment tear-down
7: [==========] 1 test from 1 test suite ran. (0 ms total)
7: [ PASSED ] 1 test.
7: -- run_test.py: return code 0
7: -- run_test.py: inject classname prefix into gtest result file '/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_nms.gtest.xml'
7: -- run_test.py: verify result file '/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_nms.gtest.xml'
7/10 Test #7: test_nms ......................... Passed 0.18 sec
test 8
Start 8: test_voxel_generator
8: Test command: /usr/bin/python3.10 "-u" "/home/mfc/projects/autoware/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_voxel_generator.gtest.xml" "--package-name" "lidar_centerpoint" "--output-file" "/home/mfc/projects/autoware/build/lidar_centerpoint/ament_cmake_gtest/test_voxel_generator.txt" "--command" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_voxel_generator" "--gtest_output=xml:/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_voxel_generator.gtest.xml"
8: Test timeout computed to be: 60
8: -- run_test.py: invoking following command in '/home/mfc/projects/autoware/build/lidar_centerpoint':
8: - /home/mfc/projects/autoware/build/lidar_centerpoint/test_voxel_generator --gtest_output=xml:/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_voxel_generator.gtest.xml
8: [==========] Running 3 tests from 1 test suite.
8: [----------] Global test environment set-up.
8: [----------] 3 tests from VoxelGeneratorTest
8: [ RUN ] VoxelGeneratorTest.SingleFrame
8: 1719411316.697583 [0] test_voxel: selected interface "lo" is not multicast-capable: disabling multicast
8: [ OK ] VoxelGeneratorTest.SingleFrame (379 ms)
8: [ RUN ] VoxelGeneratorTest.TwoFramesNoTf
8: 1719411317.074784 [0] test_voxel: selected interface "lo" is not multicast-capable: disabling multicast
8: [WARN 1719411317.580253546] [lidar_centerpoint]: "lidar" passed to lookupTransform argument target_frame does not exist. (getTransform() at /home/mfc/projects/autoware/src/universe/autoware.universe/perception/lidar_centerpoint/lib/preprocess/pointcloud_densification.cpp:42)
8: [WARN 1719411318.084682247] [lidar_centerpoint]: "lidar" passed to lookupTransform argument target_frame does not exist. (getTransform() at /home/mfc/projects/autoware/src/universe/autoware.universe/perception/lidar_centerpoint/lib/preprocess/pointcloud_densification.cpp:42)
8: [ OK ] VoxelGeneratorTest.TwoFramesNoTf (1017 ms)
8: [ RUN ] VoxelGeneratorTest.TwoFrames
8: 1719411318.091804 [0] test_voxel: selected interface "lo" is not multicast-capable: disabling multicast
8: [ OK ] VoxelGeneratorTest.TwoFrames (7 ms)
8: [----------] 3 tests from VoxelGeneratorTest (1403 ms total)
8:
8: [----------] Global test environment tear-down
8: [==========] 3 tests from 1 test suite ran. (1403 ms total)
8: [ PASSED ] 3 tests.
8: -- run_test.py: return code 0
8: -- run_test.py: inject classname prefix into gtest result file '/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_voxel_generator.gtest.xml'
8: -- run_test.py: verify result file '/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_voxel_generator.gtest.xml'
8/10 Test #8: test_voxel_generator ............. Passed 1.78 sec
test 9
Start 9: test_preprocess_kernel
9: Test command: /usr/bin/python3.10 "-u" "/home/mfc/projects/autoware/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_preprocess_kernel.xml" "--package-name" "lidar_centerpoint" "--generate-result-on-success" "--command" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_preprocess_kernel"
9: Test timeout computed to be: 60
9: -- run_test.py: invoking following command in '/home/mfc/projects/autoware/build/lidar_centerpoint':
9: - /home/mfc/projects/autoware/build/lidar_centerpoint/test_preprocess_kernel
9: [==========] Running 4 tests from 1 test suite.
9: [----------] Global test environment set-up.
9: [----------] 4 tests from PreprocessKernelTest
9: [ RUN ] PreprocessKernelTest.EmptyVoxelTest
9: [ OK ] PreprocessKernelTest.EmptyVoxelTest (149 ms)
9: [ RUN ] PreprocessKernelTest.BasicTest
9: [ OK ] PreprocessKernelTest.BasicTest (6 ms)
9: [ RUN ] PreprocessKernelTest.OutOfRangeTest
9: [ OK ] PreprocessKernelTest.OutOfRangeTest (7 ms)
9: [ RUN ] PreprocessKernelTest.VoxelOverflowTest
9: [ OK ] PreprocessKernelTest.VoxelOverflowTest (6 ms)
9: [----------] 4 tests from PreprocessKernelTest (168 ms total)
9:
9: [----------] Global test environment tear-down
9: [==========] 4 tests from 1 test suite ran. (168 ms total)
9: [ PASSED ] 4 tests.
9: -- run_test.py: return code 0
9: -- run_test.py: generate result file '/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_preprocess_kernel.xml' with successful test
9/10 Test #9: test_preprocess_kernel ........... Passed 0.33 sec
test 10
Start 10: test_postprocess_kernel
10: Test command: /usr/bin/python3.10 "-u" "/home/mfc/projects/autoware/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_postprocess_kernel.xml" "--package-name" "lidar_centerpoint" "--generate-result-on-success" "--command" "/home/mfc/projects/autoware/build/lidar_centerpoint/test_postprocess_kernel"
10: Test timeout computed to be: 60
10: -- run_test.py: invoking following command in '/home/mfc/projects/autoware/build/lidar_centerpoint':
10: - /home/mfc/projects/autoware/build/lidar_centerpoint/test_postprocess_kernel
10: [==========] Running 4 tests from 1 test suite.
10: [----------] Global test environment set-up.
10: [----------] 4 tests from PostprocessKernelTest
10: [ RUN ] PostprocessKernelTest.EmptyTensorTest
10: [ OK ] PostprocessKernelTest.EmptyTensorTest (478 ms)
10: [ RUN ] PostprocessKernelTest.SingleDetectionTest
10: [ OK ] PostprocessKernelTest.SingleDetectionTest (2 ms)
10: [ RUN ] PostprocessKernelTest.InvalidYawTest
10: [ OK ] PostprocessKernelTest.InvalidYawTest (3 ms)
10: [ RUN ] PostprocessKernelTest.CircleNMSTest
10: [ OK ] PostprocessKernelTest.CircleNMSTest (3 ms)
10: [----------] 4 tests from PostprocessKernelTest (486 ms total)
10:
10: [----------] Global test environment tear-down
10: [==========] 4 tests from 1 test suite ran. (486 ms total)
10: [ PASSED ] 4 tests.
10: -- run_test.py: return code 0
10: -- run_test.py: generate result file '/home/mfc/projects/autoware/build/lidar_centerpoint/test_results/lidar_centerpoint/test_postprocess_kernel.xml' with successful test
10/10 Test #10: test_postprocess_kernel .......... Passed 0.65 sec
100% tests passed, 0 tests failed out of 10
Label Time Summary:
copyright = 1.34 sec*proc (1 test)
cppcheck = 0.21 sec*proc (1 test)
gtest = 2.57 sec*proc (4 tests)
lint_cmake = 0.18 sec*proc (1 test)
linter = 2.21 sec*proc (4 tests)
xmllint = 0.49 sec*proc (1 test)
Total Test time (real) = 5.76 sec
---
Finished <<< lidar_centerpoint [6.25s]
Summary: 1 package finished [7.35s] On the CI: It is missing so many things.
My machine:
|
Weirdly enough, non-cuda tests are also the same:
|
Found the bug 💡 https://github.com/autowarefoundation/autoware.universe/actions/caches
Cuda version is using the cached build and install folders from the noncuda version which caches it. They should have been using separate keys but they are not. This goes all the way back to: |
Signed-off-by: M. Fatih Cırıt <[email protected]>
Signed-off-by: M. Fatih Cırıt <[email protected]>
Signed-off-by: M. Fatih Cırıt <[email protected]>
Signed-off-by: M. Fatih Cırıt <[email protected]>
Signed-off-by: M. Fatih Cırıt <[email protected]>
da824c4
to
fc78221
Compare
Signed-off-by: M. Fatih Cırıt <[email protected]>
Description
This PR is for testing and debugging the CI for this package
Related links
Parent Issue:
How was this PR tested?
Notes for reviewers
None.
Interface changes
None.
Effects on system behavior
None.