feat(lidar_transfusion): add lidar_transfusion 3D detection package

[amadeuszsz]

Description

The PR adds a new lidar_transfusion package which brings a new 3D detection component based on TransFusion[1].

lidar_transfusion_demo-2024-04-25_15.56.14.mp4

Related links

[1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu and Chiew-Lan Tai. "TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers." arXiv preprint arXiv:2203.11496 (2022)
[2] rosbag (TIER IV internal link)
[3] helper files (TIER IV internal link)
[4] TransFusion onnx model (TIER IV internal link)

Tests performed

• Model metrics:

------------- T4Metric results -------------
04/25 09:28:12 - mmengine - INFO - | class_name | mAP   | [email protected] | [email protected] | [email protected] | [email protected] | error@trans_err | error@scale_err | error@orient_err | error@vel_err | error@attr_err |
04/25 09:28:12 - mmengine - INFO - |----------------------------------------------------------------------------------------------------------------------------------------------------|
04/25 09:28:12 - mmengine - INFO - | car        | 0.788 | 0.609   | 0.798   | 0.859   | 0.884   | 0.26            | 0.159           | 0.103            | 0.779         | 1              |
04/25 09:28:12 - mmengine - INFO - | truck      | 0.59  | 0.315   | 0.577   | 0.7     | 0.766   | 0.379           | 0.149           | 0.0516           | 0.77          | 1              |
04/25 09:28:12 - mmengine - INFO - | bus        | 0.554 | 0.347   | 0.532   | 0.628   | 0.709   | 0.363           | 0.137           | 0.0762           | 1.69          | 1              |
04/25 09:28:12 - mmengine - INFO - | bicycle    | 0.236 | 0.192   | 0.241   | 0.251   | 0.261   | 0.283           | 0.248           | 0.27             | 0.839         | 1              |
04/25 09:28:12 - mmengine - INFO - | pedestrian | 0.628 | 0.57    | 0.613   | 0.642   | 0.688   | 0.253           | 0.281           | 0.424            | 0.626         | 1              |
04/25 09:28:12 - mmengine - INFO - | Total mAP: 0.559

• ROS 2 node performance with RTX 4090:

                   | pre-process [ms] | inference [ms] | post-process [ms] | total [ms]  |
------------------------------------------------------------------------------------------
lidar_transfusion  |   1.52 ± 0.14    |  2.29 ± 0.37   |    0.20 ± 0.15    | 4.07 ± 0.44 |
lidar_centerpoint  |        -         |       -        |         -         | 5.53 ± 0.55 |

Notes for reviewers

The package can best tested with rosbag file. If data needed, you can use this rosbag[2] and copy helper files[3] to the launch directory of lidar_transfusion package before building. The default path for onnx model is ~/autoware_data/lidar_transfusion/transfusion.onnx. The model awaits for deployment, temporary please use attached link[4].

To start, run commands:

# terminal 1
ros2 launch lidar_transfusion data_pipeline.launch.py 

# terminal 2
ros2 bag play trailer_yaw_ticket_data_jpntaxi7/01d060d9-8d25-45e0-b45e-9fd7201ac27b_2023-05-26-11-30-03_p0900_3.db3 --loop --clock

# terminal 3
rviz2 --ros-args -p use_sim_time:=True 

# terminal 4 (for convenience you can add <param name="use_sim_time" value="true"/> to the node in xml file)
ros2 launch lidar_transfusion lidar_transfusion.launch.xml

Interface changes

Effects on system behavior

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

• I've confirmed the contribution guidelines.
• The PR follows the pull request guidelines.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

• The PR follows the pull request guidelines.
• The PR has been properly tested.
• The PR has been reviewed by the code owners.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

• There are no open discussions or they are tracked via tickets.
• The PR is ready for merge.

After all checkboxes are checked, anyone who has write access can merge the PR.

[knzo25]

Comments:

• When the number of voxels falls outside the expected parameters, the output is 200 NaN objects (200 being the max number of objects). This can happen when the top lidar is missing in the taxi, for example. We should consider lowering the value, and/or aborting the inference + adding an error message.
• We need to add transfusion to the perception related launchers. I did it in my local environment, so I will either pass them to the PR's author, commit them myself, or explain him the procedure.
• In my environment, I have cuda memory errors sporadically. It happens inside the first kernel of the preprocess step.
• Not confirmed, but the stream used in the preprocessing may not be the one we are expecting (since adding device synchronizations instead of streams changed the behavior)

[knzo25]

@scepter914
The PR can still be reviewed to some degree, but I think it is better to wait until the errors disappear 🙏

[taikitanaka3]

@knzo25
I think you wanted to mention @scepter914

[knzo25]

Apologies 🙇

[amadeuszsz]

Thank you @knzo25 for your review. Let me investigate the kernels first to find mentioned leak. Regarding the NaN output, as you suggested we will:

• change the optimization profile,
• handle the cases with not sufficient number of voxels after preprocessing for optimization profile.

[amadeuszsz]

@knzo25
Recent fix solves the issue with cuda memory and NaNs. Please check if the issue disappeared on your machine as well.

[scepter914]

Thank you for PR.
Overall very readable code and rich documentation is so great. 👍

I took a quick look at whole code and I ask you to add some members for maintainers because development will be robuster and smoother if multiple people can maintain it.
As the rest of my work, I'll test with some rosbag and I'll approve after I confirm the operation with some Rosbag.

[scepter914]

@amadeuszsz

I apologize for the inconvenience, would you fix DCO?
We need to pass CI include DCO for merge (this specification is so invonvenient...).

[scepter914]

I confirm the perform with other rosbags.
After kenzo-san's review and uploading model, we can merge this PR. 👍