-
Notifications
You must be signed in to change notification settings - Fork 659
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(lidar_transfusion): add lidar_transfusion 3D detection package (#…
…6890) * feat(lidar_transfusion): add lidar_transfusion 3D detection package Signed-off-by: amadeuszsz <[email protected]> * style(pre-commit): autofix * style(lidar_transfusion): cpplint Signed-off-by: amadeuszsz <[email protected]> * style(lidar_transfusion): cspell Signed-off-by: Amadeusz Szymko <[email protected]> * fix(lidar_transfusion): CUDA mem allocation & inference input Signed-off-by: amadeuszsz <[email protected]> * style(pre-commit): autofix * fix(lidar_transfusion): arrays size Signed-off-by: amadeuszsz <[email protected]> * style(pre-commit): autofix * chore(lidar_transfusion): update maintainers Co-authored-by: Satoshi Tanaka <[email protected]> Signed-off-by: amadeuszsz <[email protected]> * fix(lidar_transfusion): array size & grid idx Signed-off-by: amadeuszsz <[email protected]> * chore(lidar_transfusion): update maintainer email Signed-off-by: amadeuszsz <[email protected]> * chore: added transfusion to the respective launchers Signed-off-by: Kenzo Lobos-Tsunekawa <[email protected]> * refactor(lidar_transfusion): rename config Signed-off-by: amadeuszsz <[email protected]> * refactor(lidar_transfusion): callback access specifier Signed-off-by: amadeuszsz <[email protected]> * refactor(lidar_transfusion): pointers initialziation Signed-off-by: amadeuszsz <[email protected]> * refactor(lidar_transfusion): change macros for constexpr Signed-off-by: amadeuszsz <[email protected]> * refactor(lidar_transfusion): consts & uniform initialization Signed-off-by: amadeuszsz <[email protected]> * refactor(lidar_transfusion): change to unique ptr & uniform initialization Signed-off-by: amadeuszsz <[email protected]> * style(pre-commit): autofix * refactor(lidar_transfusion): use of config params Signed-off-by: amadeuszsz <[email protected]> * refactor(lidar_transfusion): remove unnecessary condition Signed-off-by: amadeuszsz <[email protected]> * style(lidar_transfusion): switch naming (CPU to HOST) Signed-off-by: amadeuszsz <[email protected]> * refactor(lidar_transfusion): remove redundant device sync Signed-off-by: amadeuszsz <[email protected]> * style(lidar_transfusion): intensity naming Signed-off-by: amadeuszsz <[email protected]> * feat(lidar_transfusion): full network shape validation Signed-off-by: amadeuszsz <[email protected]> * feat(lidar_transfusion): validate objects' orientation in host processing Signed-off-by: amadeuszsz <[email protected]> * feat(lidar_transfusion): add json schema Signed-off-by: amadeuszsz <[email protected]> * style(pre-commit): autofix * style(lidar_transfusion): affine matrix naming Signed-off-by: amadeuszsz <[email protected]> * style(lidar_transfusion): transformed point naming Signed-off-by: amadeuszsz <[email protected]> * refactor(lidar_transfusion): add param descriptor & arrays size check Signed-off-by: amadeuszsz <[email protected]> * style(lidar_transfusion): affine matrix naming Signed-off-by: amadeuszsz <[email protected]> * feat(lidar_transfusion): caching cloud input as device ptr Signed-off-by: amadeuszsz <[email protected]> * fix(lidar_transfusion): logging Signed-off-by: amadeuszsz <[email protected]> * chore(tier4_perception_launch): revert to centerpoint Signed-off-by: amadeuszsz <[email protected]> * fix(lidar_transfusion): typo Signed-off-by: amadeuszsz <[email protected]> * docs(lidar_transfusion): use hook for param description Signed-off-by: amadeuszsz <[email protected]> * fix(lidar_transfusion): interpret eigen matrix as col major Signed-off-by: amadeuszsz <[email protected]> * feat(lidar_transfusion): update to autware_msgs Signed-off-by: amadeuszsz <[email protected]> --------- Signed-off-by: Amadeusz Szymko <[email protected]> Signed-off-by: Kenzo Lobos-Tsunekawa <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Kenzo Lobos Tsunekawa <[email protected]>
- Loading branch information
1 parent
f93e5b3
commit d891ef7
Showing
37 changed files
with
3,587 additions
and
4 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,156 @@ | ||
cmake_minimum_required(VERSION 3.14) | ||
project(lidar_transfusion) | ||
|
||
find_package(autoware_cmake REQUIRED) | ||
autoware_package() | ||
|
||
add_compile_options(-Wno-deprecated-declarations) | ||
|
||
option(CUDA_VERBOSE "Verbose output of CUDA modules" OFF) | ||
|
||
# set flags for CUDA availability | ||
option(CUDA_AVAIL "CUDA available" OFF) | ||
find_package(CUDA) | ||
if(CUDA_FOUND) | ||
find_library(CUBLAS_LIBRARIES cublas HINTS | ||
${CUDA_TOOLKIT_ROOT_DIR}/lib64 | ||
${CUDA_TOOLKIT_ROOT_DIR}/lib | ||
) | ||
if(CUDA_VERBOSE) | ||
message("CUDA is available!") | ||
message("CUDA Libs: ${CUDA_LIBRARIES}") | ||
message("CUDA Headers: ${CUDA_INCLUDE_DIRS}") | ||
endif() | ||
# Note: cublas_device was depreciated in CUDA version 9.2 | ||
# https://forums.developer.nvidia.com/t/where-can-i-find-libcublas-device-so-or-libcublas-device-a/67251/4 | ||
# In LibTorch, CUDA_cublas_device_LIBRARY is used. | ||
unset(CUDA_cublas_device_LIBRARY CACHE) | ||
set(CUDA_AVAIL ON) | ||
else() | ||
message("CUDA NOT FOUND") | ||
set(CUDA_AVAIL OFF) | ||
endif() | ||
|
||
# set flags for TensorRT availability | ||
option(TRT_AVAIL "TensorRT available" OFF) | ||
# try to find the tensorRT modules | ||
find_library(NVINFER nvinfer) | ||
find_library(NVONNXPARSER nvonnxparser) | ||
if(NVINFER AND NVONNXPARSER) | ||
if(CUDA_VERBOSE) | ||
message("TensorRT is available!") | ||
message("NVINFER: ${NVINFER}") | ||
message("NVONNXPARSER: ${NVONNXPARSER}") | ||
endif() | ||
set(TRT_AVAIL ON) | ||
else() | ||
message("TensorRT is NOT Available") | ||
set(TRT_AVAIL OFF) | ||
endif() | ||
|
||
# set flags for CUDNN availability | ||
option(CUDNN_AVAIL "CUDNN available" OFF) | ||
# try to find the CUDNN module | ||
find_library(CUDNN_LIBRARY | ||
NAMES libcudnn.so${__cudnn_ver_suffix} libcudnn${__cudnn_ver_suffix}.dylib ${__cudnn_lib_win_name} | ||
PATHS $ENV{LD_LIBRARY_PATH} ${__libpath_cudart} ${CUDNN_ROOT_DIR} ${PC_CUDNN_LIBRARY_DIRS} ${CMAKE_INSTALL_PREFIX} | ||
PATH_SUFFIXES lib lib64 bin | ||
DOC "CUDNN library." | ||
) | ||
if(CUDNN_LIBRARY) | ||
if(CUDA_VERBOSE) | ||
message(STATUS "CUDNN is available!") | ||
message(STATUS "CUDNN_LIBRARY: ${CUDNN_LIBRARY}") | ||
endif() | ||
set(CUDNN_AVAIL ON) | ||
else() | ||
message("CUDNN is NOT Available") | ||
set(CUDNN_AVAIL OFF) | ||
endif() | ||
|
||
if(TRT_AVAIL AND CUDA_AVAIL AND CUDNN_AVAIL) | ||
find_package(ament_cmake_auto REQUIRED) | ||
ament_auto_find_build_dependencies() | ||
|
||
include_directories( | ||
include | ||
${CUDA_INCLUDE_DIRS} | ||
) | ||
|
||
if(CMAKE_BUILD_TYPE STREQUAL "Debug") | ||
set(CMAKE_CUDA_FLAGS ${CMAKE_CUDA_FLAGS} "-g -G") | ||
set(CUDA_NVCC_FLAGS "-g -G") | ||
endif() | ||
|
||
ament_auto_add_library(transfusion_lib SHARED | ||
lib/detection_class_remapper.cpp | ||
lib/network/network_trt.cpp | ||
lib/postprocess/non_maximum_suppression.cpp | ||
lib/preprocess/voxel_generator.cpp | ||
lib/preprocess/pointcloud_densification.cpp | ||
lib/ros_utils.cpp | ||
lib/transfusion_trt.cpp | ||
) | ||
|
||
cuda_add_library(transfusion_cuda_lib SHARED | ||
lib/postprocess/circle_nms_kernel.cu | ||
lib/postprocess/postprocess_kernel.cu | ||
lib/preprocess/preprocess_kernel.cu | ||
) | ||
|
||
target_link_libraries(transfusion_lib | ||
${NVINFER} | ||
${NVONNXPARSER} | ||
${CUDA_LIBRARIES} | ||
${CUBLAS_LIBRARIES} | ||
${CUDA_curand_LIBRARY} | ||
${CUDNN_LIBRARY} | ||
transfusion_cuda_lib | ||
) | ||
|
||
target_include_directories(transfusion_lib | ||
PUBLIC | ||
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include> | ||
$<INSTALL_INTERFACE:include> | ||
) | ||
|
||
# To suppress unknown-pragmas error. The root-cause is CUB library in CUDA 11.6. | ||
# This issue was fixed by https://github.com/NVIDIA/cub/commit/7d608bf1dc14553e2fb219eabeed80b76621b6fe | ||
target_include_directories(transfusion_lib | ||
SYSTEM PUBLIC | ||
${CUDA_INCLUDE_DIRS} | ||
) | ||
|
||
ament_auto_add_library(lidar_transfusion_component SHARED | ||
src/lidar_transfusion_node.cpp | ||
) | ||
|
||
target_link_libraries(lidar_transfusion_component | ||
transfusion_lib | ||
) | ||
|
||
rclcpp_components_register_node(lidar_transfusion_component | ||
PLUGIN "lidar_transfusion::LidarTransfusionNode" | ||
EXECUTABLE lidar_transfusion_node | ||
) | ||
|
||
if(BUILD_TESTING) | ||
find_package(ament_lint_auto REQUIRED) | ||
ament_lint_auto_find_test_dependencies() | ||
endif() | ||
|
||
ament_auto_package( | ||
INSTALL_TO_SHARE | ||
launch | ||
config | ||
) | ||
|
||
else() | ||
find_package(ament_cmake_auto REQUIRED) | ||
ament_auto_find_build_dependencies() | ||
|
||
ament_auto_package( | ||
INSTALL_TO_SHARE | ||
launch | ||
) | ||
endif() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,113 @@ | ||
# lidar_transfusion | ||
|
||
## Purpose | ||
|
||
The `lidar_transfusion` package is used for 3D object detection based on lidar data (x, y, z, intensity). | ||
|
||
## Inner-workings / Algorithms | ||
|
||
The implementation bases on TransFusion [1] work. It uses TensorRT library for data process and network inference. | ||
|
||
We trained the models using <https://github.com/open-mmlab/mmdetection3d>. | ||
|
||
## Inputs / Outputs | ||
|
||
### Input | ||
|
||
| Name | Type | Description | | ||
| -------------------- | ------------------------------- | ----------------- | | ||
| `~/input/pointcloud` | `sensor_msgs::msg::PointCloud2` | Input pointcloud. | | ||
|
||
### Output | ||
|
||
| Name | Type | Description | | ||
| -------------------------------------- | ------------------------------------------------ | --------------------------- | | ||
| `~/output/objects` | `autoware_perception_msgs::msg::DetectedObjects` | Detected objects. | | ||
| `debug/cyclic_time_ms` | `tier4_debug_msgs::msg::Float64Stamped` | Cyclic time (ms). | | ||
| `debug/pipeline_latency_ms` | `tier4_debug_msgs::msg::Float64Stamped` | Pipeline latency time (ms). | | ||
| `debug/processing_time/preprocess_ms` | `tier4_debug_msgs::msg::Float64Stamped` | Preprocess (ms). | | ||
| `debug/processing_time/inference_ms` | `tier4_debug_msgs::msg::Float64Stamped` | Inference time (ms). | | ||
| `debug/processing_time/postprocess_ms` | `tier4_debug_msgs::msg::Float64Stamped` | Postprocess time (ms). | | ||
| `debug/processing_time/total_ms` | `tier4_debug_msgs::msg::Float64Stamped` | Total processing time (ms). | | ||
|
||
## Parameters | ||
|
||
### TransFusion | ||
|
||
{{ json_to_markdown("perception/lidar_transfusion/schema/transfusion.schema.json") }} | ||
|
||
### Detection class remapper | ||
|
||
{{ json_to_markdown("perception/lidar_transfusion/schema/detection_class_remapper.schema.json") }} | ||
|
||
### The `build_only` option | ||
|
||
The `lidar_transfusion` node has `build_only` option to build the TensorRT engine file from the ONNX file. | ||
Although it is preferred to move all the ROS parameters in `.param.yaml` file in Autoware Universe, the `build_only` option is not moved to the `.param.yaml` file for now, because it may be used as a flag to execute the build as a pre-task. You can execute with the following command: | ||
|
||
```bash | ||
ros2 launch lidar_transfusion lidar_transfusion.launch.xml build_only:=true | ||
``` | ||
|
||
### The `log_level` option | ||
|
||
The default logging severity level for `lidar_transfusion` is `info`. For debugging purposes, the developer may decrease severity level using `log_level` parameter: | ||
|
||
```bash | ||
ros2 launch lidar_transfusion lidar_transfusion.launch.xml log_level:=debug | ||
``` | ||
|
||
## Assumptions / Known limits | ||
|
||
## Trained Models | ||
|
||
You can download the onnx format of trained models by clicking on the links below. | ||
|
||
- TransFusion: [transfusion.onnx](https://awf.ml.dev.web.auto/perception/models/transfusion/v1/transfusion.onnx) | ||
|
||
The model was trained in TIER IV's internal database (~11k lidar frames) for 20 epochs. | ||
|
||
### Changelog | ||
|
||
## (Optional) Error detection and handling | ||
|
||
<!-- Write how to detect errors and how to recover from them. | ||
Example: | ||
This package can handle up to 20 obstacles. If more obstacles found, this node will give up and raise diagnostic errors. | ||
--> | ||
|
||
## (Optional) Performance characterization | ||
|
||
<!-- Write performance information like complexity. If it wouldn't be the bottleneck, not necessary. | ||
Example: | ||
### Complexity | ||
This algorithm is O(N). | ||
### Processing time | ||
... | ||
--> | ||
|
||
## References/External links | ||
|
||
[1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu and Chiew-Lan Tai. "TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers." arXiv preprint arXiv:2203.11496 (2022). <!-- cspell:disable-line --> | ||
|
||
[2] <https://github.com/wep21/CUDA-TransFusion> | ||
|
||
[3] <https://github.com/open-mmlab/mmdetection3d> | ||
|
||
[4] <https://github.com/open-mmlab/OpenPCDet> | ||
|
||
[5] <https://www.nuscenes.org/nuscenes> | ||
|
||
## (Optional) Future extensions / Unimplemented parts | ||
|
||
<!-- Write future extensions of this package. | ||
Example: | ||
Currently, this package can't handle the chattering obstacles well. We plan to add some probabilistic filters in the perception layer to improve it. | ||
Also, there are some parameters that should be global(e.g. vehicle size, max steering, etc.). These will be refactored and defined as global parameters so that we can share the same parameters between different nodes. | ||
--> |
38 changes: 38 additions & 0 deletions
38
perception/lidar_transfusion/config/detection_class_remapper.param.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
/**: | ||
ros__parameters: | ||
allow_remapping_by_area_matrix: | ||
# NOTE(kl): We turn all vehicles into trailers if they go over 3x12 [m^2]. | ||
# NOTE(kl): We turn cars into trucks if they have an area between 2.2 x 5.5 and 3.0 * 12.0 [m^2] | ||
# row: original class. column: class to remap to | ||
#UNKNOWN, CAR, TRUCK, BUS, TRAILER, MOTORBIKE, BICYCLE,PEDESTRIAN | ||
[0, 0, 0, 0, 0, 0, 0, 0, #UNKNOWN | ||
0, 0, 1, 0, 1, 0, 0, 0, #CAR | ||
0, 0, 0, 0, 1, 0, 0, 0, #TRUCK | ||
0, 0, 0, 0, 1, 0, 0, 0, #BUS | ||
0, 0, 0, 0, 0, 0, 0, 0, #TRAILER | ||
0, 0, 0, 0, 0, 0, 0, 0, #MOTORBIKE | ||
0, 0, 0, 0, 0, 0, 0, 0, #BICYCLE | ||
0, 0, 0, 0, 0, 0, 0, 0] #PEDESTRIAN | ||
|
||
min_area_matrix: | ||
#UNKNOWN, CAR, TRUCK, BUS, TRAILER, MOTORBIKE, BICYCLE, PEDESTRIAN | ||
[ 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, #UNKNOWN | ||
0.000, 0.000, 12.100, 0.000, 36.000, 0.000, 0.000, 0.000, #CAR | ||
0.000, 0.000, 0.000, 0.000, 36.000, 0.000, 0.000, 0.000, #TRUCK | ||
0.000, 0.000, 0.000, 0.000, 36.000, 0.000, 0.000, 0.000, #BUS | ||
0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, #TRAILER | ||
0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, #MOTORBIKE | ||
0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, #BICYCLE | ||
0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000] #PEDESTRIAN | ||
|
||
|
||
max_area_matrix: | ||
#UNKNOWN, CAR, TRUCK, BUS, TRAILER, MOTORBIKE, BICYCLE, PEDESTRIAN | ||
[ 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, #UNKNOWN | ||
0.000, 0.000, 36.000, 0.000, 999.999, 0.000, 0.000, 0.000, #CAR | ||
0.000, 0.000, 0.000, 0.000, 999.999, 0.000, 0.000, 0.000, #TRUCK | ||
0.000, 0.000, 0.000, 0.000, 999.999, 0.000, 0.000, 0.000, #BUS | ||
0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, #TRAILER | ||
0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, #MOTORBIKE | ||
0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, #BICYCLE | ||
0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000] #PEDESTRIAN |
20 changes: 20 additions & 0 deletions
20
perception/lidar_transfusion/config/transfusion.param.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
/**: | ||
ros__parameters: | ||
# network | ||
class_names: ["CAR", "TRUCK", "BUS", "BICYCLE", "PEDESTRIAN"] | ||
trt_precision: fp16 | ||
voxels_num: [5000, 30000, 60000] # [min, opt, max] | ||
point_cloud_range: [-76.8, -76.8, -3.0, 76.8, 76.8, 5.0] # [x_min, y_min, z_min, x_max, y_max, z_max] | ||
voxel_size: [0.3, 0.3, 8.0] # [x, y, z] | ||
onnx_path: "$(var model_path)/transfusion.onnx" | ||
engine_path: "$(var model_path)/transfusion.engine" | ||
# pre-process params | ||
densification_num_past_frames: 1 | ||
densification_world_frame_id: map | ||
# post-process params | ||
circle_nms_dist_threshold: 0.5 | ||
iou_nms_target_class_names: ["CAR"] | ||
iou_nms_search_distance_2d: 10.0 | ||
iou_nms_threshold: 0.1 | ||
yaw_norm_thresholds: [0.3, 0.3, 0.3, 0.3, 0.0] # refers to the class_names | ||
score_threshold: 0.2 |
Oops, something went wrong.