Skip to content

Commit

Permalink
feat(lidar_transfusion): add lidar_transfusion 3D detection package (#…
Browse files Browse the repository at this point in the history
…6890)

* feat(lidar_transfusion): add lidar_transfusion 3D detection package

Signed-off-by: amadeuszsz <[email protected]>

* style(pre-commit): autofix

* style(lidar_transfusion): cpplint

Signed-off-by: amadeuszsz <[email protected]>

* style(lidar_transfusion): cspell

Signed-off-by: Amadeusz Szymko <[email protected]>

* fix(lidar_transfusion): CUDA mem allocation & inference input

Signed-off-by: amadeuszsz <[email protected]>

* style(pre-commit): autofix

* fix(lidar_transfusion): arrays size

Signed-off-by: amadeuszsz <[email protected]>

* style(pre-commit): autofix

* chore(lidar_transfusion): update maintainers

Co-authored-by: Satoshi Tanaka <[email protected]>

Signed-off-by: amadeuszsz <[email protected]>

* fix(lidar_transfusion): array size & grid idx

Signed-off-by: amadeuszsz <[email protected]>

* chore(lidar_transfusion): update maintainer email

Signed-off-by: amadeuszsz <[email protected]>

* chore: added transfusion to the respective launchers

Signed-off-by: Kenzo Lobos-Tsunekawa <[email protected]>

* refactor(lidar_transfusion): rename config

Signed-off-by: amadeuszsz <[email protected]>

* refactor(lidar_transfusion): callback access specifier

Signed-off-by: amadeuszsz <[email protected]>

* refactor(lidar_transfusion): pointers initialziation

Signed-off-by: amadeuszsz <[email protected]>

* refactor(lidar_transfusion): change macros for constexpr

Signed-off-by: amadeuszsz <[email protected]>

* refactor(lidar_transfusion): consts & uniform initialization

Signed-off-by: amadeuszsz <[email protected]>

* refactor(lidar_transfusion): change to unique ptr & uniform initialization

Signed-off-by: amadeuszsz <[email protected]>

* style(pre-commit): autofix

* refactor(lidar_transfusion): use of config params

Signed-off-by: amadeuszsz <[email protected]>

* refactor(lidar_transfusion): remove unnecessary condition

Signed-off-by: amadeuszsz <[email protected]>

* style(lidar_transfusion): switch naming (CPU to HOST)

Signed-off-by: amadeuszsz <[email protected]>

* refactor(lidar_transfusion): remove redundant device sync

Signed-off-by: amadeuszsz <[email protected]>

* style(lidar_transfusion): intensity naming

Signed-off-by: amadeuszsz <[email protected]>

* feat(lidar_transfusion): full network shape validation

Signed-off-by: amadeuszsz <[email protected]>

* feat(lidar_transfusion): validate objects' orientation in host processing

Signed-off-by: amadeuszsz <[email protected]>

* feat(lidar_transfusion): add json schema

Signed-off-by: amadeuszsz <[email protected]>

* style(pre-commit): autofix

* style(lidar_transfusion): affine matrix naming

Signed-off-by: amadeuszsz <[email protected]>

* style(lidar_transfusion): transformed point naming

Signed-off-by: amadeuszsz <[email protected]>

* refactor(lidar_transfusion): add param descriptor & arrays size check

Signed-off-by: amadeuszsz <[email protected]>

* style(lidar_transfusion): affine matrix naming

Signed-off-by: amadeuszsz <[email protected]>

* feat(lidar_transfusion): caching cloud input as device ptr

Signed-off-by: amadeuszsz <[email protected]>

* fix(lidar_transfusion): logging

Signed-off-by: amadeuszsz <[email protected]>

* chore(tier4_perception_launch): revert to centerpoint

Signed-off-by: amadeuszsz <[email protected]>

* fix(lidar_transfusion): typo

Signed-off-by: amadeuszsz <[email protected]>

* docs(lidar_transfusion): use hook for param description

Signed-off-by: amadeuszsz <[email protected]>

* fix(lidar_transfusion): interpret eigen matrix as col major

Signed-off-by: amadeuszsz <[email protected]>

* feat(lidar_transfusion): update to autware_msgs

Signed-off-by: amadeuszsz <[email protected]>

---------

Signed-off-by: Amadeusz Szymko <[email protected]>
Signed-off-by: Kenzo Lobos-Tsunekawa <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Kenzo Lobos Tsunekawa <[email protected]>
  • Loading branch information
3 people authored Jun 7, 2024
1 parent f93e5b3 commit d891ef7
Show file tree
Hide file tree
Showing 37 changed files with 3,587 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

<!-- Lidar parameters -->
<arg name="input/pointcloud"/>
<arg name="lidar_detection_model" default="centerpoint" description="options: `centerpoint`, `apollo`, `pointpainting`, `clustering`"/>
<arg name="lidar_detection_model" default="centerpoint" description="options: `transfusion`, `centerpoint`, `apollo`, `pointpainting`, `clustering`"/>
<arg name="use_object_filter" default="true" description="use object filter"/>
<arg name="pointcloud_container_name" default="pointcloud_container"/>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,41 @@
<launch>
<!-- Lidar parameters -->
<arg name="input/pointcloud"/>
<arg name="lidar_detection_model" default="centerpoint" description="options: `centerpoint`, `apollo`, `clustering`"/>
<arg name="lidar_detection_model" default="centerpoint" description="options: `transfusion`, `centerpoint`, `apollo`, `clustering`"/>

<!-- Lidar detector centerpoint parameters -->
<arg name="centerpoint_model_name" default="centerpoint_tiny" description="options: `centerpoint`, `centerpoint_tiny` or `centerpoint_sigma`"/>
<arg name="centerpoint_model_path" default="$(var data_path)/lidar_centerpoint"/>
<arg name="lidar_model_param_path" default="$(find-pkg-share lidar_centerpoint)/config"/>

<!-- Lidar detector transfusion parameters -->
<arg name="transfusion_model_name" default="transfusion" description="options: `transfusion`"/>
<arg name="transfusion_model_path" default="$(var data_path)/lidar_transfusion"/>

<!-- TransFusion -->
<group if="$(eval &quot;'$(var lidar_detection_model)'=='transfusion'&quot;)">
<push-ros-namespace namespace="transfusion"/>
<arg name="lidar_model_param_path" default="$(find-pkg-share lidar_transfusion)/config"/>

<group>
<include file="$(find-pkg-share lidar_transfusion)/launch/lidar_transfusion.launch.xml">
<arg name="input/pointcloud" value="$(var input/pointcloud)"/>
<arg name="output/objects" value="objects"/>
<arg name="model_name" value="$(var transfusion_model_name)"/>
<arg name="model_path" value="$(var transfusion_model_path)"/>
<arg name="model_param_path" value="$(var lidar_model_param_path)/$(var transfusion_model_name).param.yaml"/>
<arg name="class_remapper_param_path" value="$(var lidar_model_param_path)/detection_class_remapper.param.yaml"/>

<arg name="use_pointcloud_container" value="true"/>
<arg name="pointcloud_container_name" value="$(var pointcloud_container_name)"/>
</include>
</group>
</group>

<!-- CenterPoint -->
<group if="$(eval &quot;'$(var lidar_detection_model)'=='centerpoint'&quot;)">
<push-ros-namespace namespace="centerpoint"/>
<arg name="lidar_model_param_path" default="$(find-pkg-share lidar_centerpoint)/config"/>

<group>
<include file="$(find-pkg-share lidar_centerpoint)/launch/lidar_centerpoint.launch.xml">
<arg name="input/pointcloud" value="$(var input/pointcloud)"/>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@
<arg name="input/pointcloud" default="/sensing/lidar/concatenated/pointcloud" description="The topic will be used in the detection module"/>
<arg name="mode" default="camera_lidar_fusion" description="options: `camera_lidar_radar_fusion`, `camera_lidar_fusion`, `lidar_radar_fusion`, `lidar` or `radar`"/>
<arg name="data_path" default="$(env HOME)/autoware_data" description="packages data and artifacts directory path"/>
<arg name="lidar_detection_model" default="centerpoint" description="options: `centerpoint`, `apollo`, `pointpainting`, `clustering`"/>
<arg name="lidar_detection_model" default="centerpoint" description="options: `transfusion`, `centerpoint`, `apollo`, `pointpainting`, `clustering`"/>
<arg name="image_raw0" default="/sensing/camera/camera0/image_rect_color" description="image raw topic name"/>
<arg name="camera_info0" default="/sensing/camera/camera0/camera_info" description="camera info topic name"/>
<arg name="detection_rois0" default="/perception/object_recognition/detection/rois0" description="detection rois output topic name"/>
Expand Down
156 changes: 156 additions & 0 deletions perception/lidar_transfusion/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
cmake_minimum_required(VERSION 3.14)
project(lidar_transfusion)

find_package(autoware_cmake REQUIRED)
autoware_package()

add_compile_options(-Wno-deprecated-declarations)

option(CUDA_VERBOSE "Verbose output of CUDA modules" OFF)

# set flags for CUDA availability
option(CUDA_AVAIL "CUDA available" OFF)
find_package(CUDA)
if(CUDA_FOUND)
find_library(CUBLAS_LIBRARIES cublas HINTS
${CUDA_TOOLKIT_ROOT_DIR}/lib64
${CUDA_TOOLKIT_ROOT_DIR}/lib
)
if(CUDA_VERBOSE)
message("CUDA is available!")
message("CUDA Libs: ${CUDA_LIBRARIES}")
message("CUDA Headers: ${CUDA_INCLUDE_DIRS}")
endif()
# Note: cublas_device was depreciated in CUDA version 9.2
# https://forums.developer.nvidia.com/t/where-can-i-find-libcublas-device-so-or-libcublas-device-a/67251/4
# In LibTorch, CUDA_cublas_device_LIBRARY is used.
unset(CUDA_cublas_device_LIBRARY CACHE)
set(CUDA_AVAIL ON)
else()
message("CUDA NOT FOUND")
set(CUDA_AVAIL OFF)
endif()

# set flags for TensorRT availability
option(TRT_AVAIL "TensorRT available" OFF)
# try to find the tensorRT modules
find_library(NVINFER nvinfer)
find_library(NVONNXPARSER nvonnxparser)
if(NVINFER AND NVONNXPARSER)
if(CUDA_VERBOSE)
message("TensorRT is available!")
message("NVINFER: ${NVINFER}")
message("NVONNXPARSER: ${NVONNXPARSER}")
endif()
set(TRT_AVAIL ON)
else()
message("TensorRT is NOT Available")
set(TRT_AVAIL OFF)
endif()

# set flags for CUDNN availability
option(CUDNN_AVAIL "CUDNN available" OFF)
# try to find the CUDNN module
find_library(CUDNN_LIBRARY
NAMES libcudnn.so${__cudnn_ver_suffix} libcudnn${__cudnn_ver_suffix}.dylib ${__cudnn_lib_win_name}
PATHS $ENV{LD_LIBRARY_PATH} ${__libpath_cudart} ${CUDNN_ROOT_DIR} ${PC_CUDNN_LIBRARY_DIRS} ${CMAKE_INSTALL_PREFIX}
PATH_SUFFIXES lib lib64 bin
DOC "CUDNN library."
)
if(CUDNN_LIBRARY)
if(CUDA_VERBOSE)
message(STATUS "CUDNN is available!")
message(STATUS "CUDNN_LIBRARY: ${CUDNN_LIBRARY}")
endif()
set(CUDNN_AVAIL ON)
else()
message("CUDNN is NOT Available")
set(CUDNN_AVAIL OFF)
endif()

if(TRT_AVAIL AND CUDA_AVAIL AND CUDNN_AVAIL)
find_package(ament_cmake_auto REQUIRED)
ament_auto_find_build_dependencies()

include_directories(
include
${CUDA_INCLUDE_DIRS}
)

if(CMAKE_BUILD_TYPE STREQUAL "Debug")
set(CMAKE_CUDA_FLAGS ${CMAKE_CUDA_FLAGS} "-g -G")
set(CUDA_NVCC_FLAGS "-g -G")
endif()

ament_auto_add_library(transfusion_lib SHARED
lib/detection_class_remapper.cpp
lib/network/network_trt.cpp
lib/postprocess/non_maximum_suppression.cpp
lib/preprocess/voxel_generator.cpp
lib/preprocess/pointcloud_densification.cpp
lib/ros_utils.cpp
lib/transfusion_trt.cpp
)

cuda_add_library(transfusion_cuda_lib SHARED
lib/postprocess/circle_nms_kernel.cu
lib/postprocess/postprocess_kernel.cu
lib/preprocess/preprocess_kernel.cu
)

target_link_libraries(transfusion_lib
${NVINFER}
${NVONNXPARSER}
${CUDA_LIBRARIES}
${CUBLAS_LIBRARIES}
${CUDA_curand_LIBRARY}
${CUDNN_LIBRARY}
transfusion_cuda_lib
)

target_include_directories(transfusion_lib
PUBLIC
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
$<INSTALL_INTERFACE:include>
)

# To suppress unknown-pragmas error. The root-cause is CUB library in CUDA 11.6.
# This issue was fixed by https://github.com/NVIDIA/cub/commit/7d608bf1dc14553e2fb219eabeed80b76621b6fe
target_include_directories(transfusion_lib
SYSTEM PUBLIC
${CUDA_INCLUDE_DIRS}
)

ament_auto_add_library(lidar_transfusion_component SHARED
src/lidar_transfusion_node.cpp
)

target_link_libraries(lidar_transfusion_component
transfusion_lib
)

rclcpp_components_register_node(lidar_transfusion_component
PLUGIN "lidar_transfusion::LidarTransfusionNode"
EXECUTABLE lidar_transfusion_node
)

if(BUILD_TESTING)
find_package(ament_lint_auto REQUIRED)
ament_lint_auto_find_test_dependencies()
endif()

ament_auto_package(
INSTALL_TO_SHARE
launch
config
)

else()
find_package(ament_cmake_auto REQUIRED)
ament_auto_find_build_dependencies()

ament_auto_package(
INSTALL_TO_SHARE
launch
)
endif()
113 changes: 113 additions & 0 deletions perception/lidar_transfusion/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
# lidar_transfusion

## Purpose

The `lidar_transfusion` package is used for 3D object detection based on lidar data (x, y, z, intensity).

## Inner-workings / Algorithms

The implementation bases on TransFusion [1] work. It uses TensorRT library for data process and network inference.

We trained the models using <https://github.com/open-mmlab/mmdetection3d>.

## Inputs / Outputs

### Input

| Name | Type | Description |
| -------------------- | ------------------------------- | ----------------- |
| `~/input/pointcloud` | `sensor_msgs::msg::PointCloud2` | Input pointcloud. |

### Output

| Name | Type | Description |
| -------------------------------------- | ------------------------------------------------ | --------------------------- |
| `~/output/objects` | `autoware_perception_msgs::msg::DetectedObjects` | Detected objects. |
| `debug/cyclic_time_ms` | `tier4_debug_msgs::msg::Float64Stamped` | Cyclic time (ms). |
| `debug/pipeline_latency_ms` | `tier4_debug_msgs::msg::Float64Stamped` | Pipeline latency time (ms). |
| `debug/processing_time/preprocess_ms` | `tier4_debug_msgs::msg::Float64Stamped` | Preprocess (ms). |
| `debug/processing_time/inference_ms` | `tier4_debug_msgs::msg::Float64Stamped` | Inference time (ms). |
| `debug/processing_time/postprocess_ms` | `tier4_debug_msgs::msg::Float64Stamped` | Postprocess time (ms). |
| `debug/processing_time/total_ms` | `tier4_debug_msgs::msg::Float64Stamped` | Total processing time (ms). |

## Parameters

### TransFusion

{{ json_to_markdown("perception/lidar_transfusion/schema/transfusion.schema.json") }}

### Detection class remapper

{{ json_to_markdown("perception/lidar_transfusion/schema/detection_class_remapper.schema.json") }}

### The `build_only` option

The `lidar_transfusion` node has `build_only` option to build the TensorRT engine file from the ONNX file.
Although it is preferred to move all the ROS parameters in `.param.yaml` file in Autoware Universe, the `build_only` option is not moved to the `.param.yaml` file for now, because it may be used as a flag to execute the build as a pre-task. You can execute with the following command:

```bash
ros2 launch lidar_transfusion lidar_transfusion.launch.xml build_only:=true
```

### The `log_level` option

The default logging severity level for `lidar_transfusion` is `info`. For debugging purposes, the developer may decrease severity level using `log_level` parameter:

```bash
ros2 launch lidar_transfusion lidar_transfusion.launch.xml log_level:=debug
```

## Assumptions / Known limits

## Trained Models

You can download the onnx format of trained models by clicking on the links below.

- TransFusion: [transfusion.onnx](https://awf.ml.dev.web.auto/perception/models/transfusion/v1/transfusion.onnx)

The model was trained in TIER IV's internal database (~11k lidar frames) for 20 epochs.

### Changelog

## (Optional) Error detection and handling

<!-- Write how to detect errors and how to recover from them.
Example:
This package can handle up to 20 obstacles. If more obstacles found, this node will give up and raise diagnostic errors.
-->

## (Optional) Performance characterization

<!-- Write performance information like complexity. If it wouldn't be the bottleneck, not necessary.
Example:
### Complexity
This algorithm is O(N).
### Processing time
...
-->

## References/External links

[1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu and Chiew-Lan Tai. "TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers." arXiv preprint arXiv:2203.11496 (2022). <!-- cspell:disable-line -->

[2] <https://github.com/wep21/CUDA-TransFusion>

[3] <https://github.com/open-mmlab/mmdetection3d>

[4] <https://github.com/open-mmlab/OpenPCDet>

[5] <https://www.nuscenes.org/nuscenes>

## (Optional) Future extensions / Unimplemented parts

<!-- Write future extensions of this package.
Example:
Currently, this package can't handle the chattering obstacles well. We plan to add some probabilistic filters in the perception layer to improve it.
Also, there are some parameters that should be global(e.g. vehicle size, max steering, etc.). These will be refactored and defined as global parameters so that we can share the same parameters between different nodes.
-->
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
/**:
ros__parameters:
allow_remapping_by_area_matrix:
# NOTE(kl): We turn all vehicles into trailers if they go over 3x12 [m^2].
# NOTE(kl): We turn cars into trucks if they have an area between 2.2 x 5.5 and 3.0 * 12.0 [m^2]
# row: original class. column: class to remap to
#UNKNOWN, CAR, TRUCK, BUS, TRAILER, MOTORBIKE, BICYCLE,PEDESTRIAN
[0, 0, 0, 0, 0, 0, 0, 0, #UNKNOWN
0, 0, 1, 0, 1, 0, 0, 0, #CAR
0, 0, 0, 0, 1, 0, 0, 0, #TRUCK
0, 0, 0, 0, 1, 0, 0, 0, #BUS
0, 0, 0, 0, 0, 0, 0, 0, #TRAILER
0, 0, 0, 0, 0, 0, 0, 0, #MOTORBIKE
0, 0, 0, 0, 0, 0, 0, 0, #BICYCLE
0, 0, 0, 0, 0, 0, 0, 0] #PEDESTRIAN

min_area_matrix:
#UNKNOWN, CAR, TRUCK, BUS, TRAILER, MOTORBIKE, BICYCLE, PEDESTRIAN
[ 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, #UNKNOWN
0.000, 0.000, 12.100, 0.000, 36.000, 0.000, 0.000, 0.000, #CAR
0.000, 0.000, 0.000, 0.000, 36.000, 0.000, 0.000, 0.000, #TRUCK
0.000, 0.000, 0.000, 0.000, 36.000, 0.000, 0.000, 0.000, #BUS
0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, #TRAILER
0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, #MOTORBIKE
0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, #BICYCLE
0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000] #PEDESTRIAN


max_area_matrix:
#UNKNOWN, CAR, TRUCK, BUS, TRAILER, MOTORBIKE, BICYCLE, PEDESTRIAN
[ 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, #UNKNOWN
0.000, 0.000, 36.000, 0.000, 999.999, 0.000, 0.000, 0.000, #CAR
0.000, 0.000, 0.000, 0.000, 999.999, 0.000, 0.000, 0.000, #TRUCK
0.000, 0.000, 0.000, 0.000, 999.999, 0.000, 0.000, 0.000, #BUS
0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, #TRAILER
0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, #MOTORBIKE
0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, #BICYCLE
0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000] #PEDESTRIAN
20 changes: 20 additions & 0 deletions perception/lidar_transfusion/config/transfusion.param.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
/**:
ros__parameters:
# network
class_names: ["CAR", "TRUCK", "BUS", "BICYCLE", "PEDESTRIAN"]
trt_precision: fp16
voxels_num: [5000, 30000, 60000] # [min, opt, max]
point_cloud_range: [-76.8, -76.8, -3.0, 76.8, 76.8, 5.0] # [x_min, y_min, z_min, x_max, y_max, z_max]
voxel_size: [0.3, 0.3, 8.0] # [x, y, z]
onnx_path: "$(var model_path)/transfusion.onnx"
engine_path: "$(var model_path)/transfusion.engine"
# pre-process params
densification_num_past_frames: 1
densification_world_frame_id: map
# post-process params
circle_nms_dist_threshold: 0.5
iou_nms_target_class_names: ["CAR"]
iou_nms_search_distance_2d: 10.0
iou_nms_threshold: 0.1
yaw_norm_thresholds: [0.3, 0.3, 0.3, 0.3, 0.0] # refers to the class_names
score_threshold: 0.2
Loading

0 comments on commit d891ef7

Please sign in to comment.