feat(autoware_tensorrt_rtmdet): add tensorrt rtmdet model

[StepTurtle]

Description

This PR adds a instance segmentation method RTMDet to perception pipeline.

Test Video: https://youtu.be/dBJdZtc4BC8

Process Times

	Average Time (ms)	Standard Deviation (ms)
Preprocess	1.61101 ms	0.470295 ms
Inference	1.71506 ms	0.757723 ms
Postprocess	13.4203 ms	0.628055 ms
Visualization	23.4338 ms	6.98011 ms

Following table shows the total time for preprocess, inference and postprocess processes. It don't contain visualization

	Average Time (ms)	Standard Deviation (ms)
Total	15.791 ms	1.58901 ms

Process Times for 8 camera on single GPU

	Average Time (ms)	Standard Deviation (ms)	Min (ms)	Max (ms)
node-0	43.5821 ms	17.0037 ms	16.2328 ms	102.521 ms
node-1	42.7286 ms	15.8802 ms	22.3733 ms	98.7796 ms
node-2	41.4004 ms	15.0862 ms	15.0169 ms	87.0995 ms
node-3	42.3738 ms	15.7069 ms	21.1178 ms	105.505 ms
node-4	36.4766 ms	13.2308 ms	20.3997 ms	81.181 ms
node-5	42.2531 ms	16.1687 ms	16.265 ms	91.1761 ms
node-6	35.9258 ms	13.5001 ms	19.7352 ms	76.6217 ms
node-7	36.4776 ms	15.4815 ms	21.2165 ms	80.9634 ms

Computer Specifications

Device	Model
GPU	GeForce RTX 3090 (24 gB VRAM)
GPU	AMD® Ryzen 7 2700x eight-core processor × 16
Memory	32 gB

Related links

Parent Issue:

• Implement RTMDet to Perception Pipeline #7235

PR for message type which is used in this PR:

• feat(autoware_perception_msgs): add instance segmentation msg autoware_msgs#106

Repository for Plugin:

• https://github.com/autowarefoundation/trt_batched_nms

How was this PR tested?

Testers can follow these steps to test PR.

1) Download the pre-trained model:

gdown https://drive.google.com/drive/folders/1P1JyMrstJIN9Y4F947nUkbKxQ2JDntgF?usp=drive_link -O ~/autoware_data/ --folder

2) For autoware_internal_msgs use following PR:

• feat(SegmentationMask): add instance segmentation message autoware_internal_msgs#23

3) For trt_batched_nms use https://github.com/autowarefoundation/trt_batched_nms:

• Clone the repository into universe/extarnal

4) Build the package and other dependencies of package

cd autoware
colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release --packages-up-to tensorrt_rtmdet

5) Update the topic parameters from launch/rtmdet.launch.xml and run the launch file

ros2 launch autoware_tensorrt_rtmdet rtmdet.launch.xml

Notes for reviewers

🟨 Plugin Loading

Autoware has the same logic that I used to load the TensorRT plugin.

for (const auto & plugin_path : plugin_paths) {
    int32_t flags{RTLD_LAZY};
    void * handle = dlopen(plugin_path.c_str(), flags); // Plugin path is '/path/to/plugin.so'
    if (!handle) {
      logger_.log(nvinfer1::ILogger::Severity::kERROR, "Could not load plugin library");
    }
}

After compilation, a file with the extension '.so' is created. This file stored in build and it should be parameter of dlopen() function.

Is there any information about can we handle this in Cmake. If we cannot, how can I provide the path to the file located inside the 'build' folder?

I was able to load the plugin using the file paths below:

• ./build/tensorrt_rtmdet/libtensorrt_rtmdet_plugin.so (relative path from workspace)
• /home/user/projects/workspace/build/tensorrt_rtmdet/libtensorrt_rtmdet_plugin.so (absolute path)

🟨 TRTBatchedNMS Codebase

The RTMDet model uses the TRTBatchedNMS plugin (modified version of original TRTBatchedNMS by TensorRT) and I put all code base of plugin into src/trt_batched_nms and include/trt_batched_nms but I am not is it suitable or not?

🟨 int8 Precision Option

There are three precision option (fp16, fp32 and int8) and one of them was not work. In the current situation, it is working, but the result is not entirely correct. Watch video to see problem:

Video Link	https://youtu.be/3YlY3a9Xnpk

🟨 Message Type

• feat(SegmentationMask): add instance segmentation message autoware_internal_msgs#23

Interface changes

None.

Effects on system behavior

None.

[github-actions]

Thank you for contributing to the Autoware project!

🚧 If your pull request is in progress, switch it to draft mode.

Please ensure:

• You've checked our contribution guidelines.
• Your PR follows our pull request guidelines.
• All required CI checks pass before marking the PR ready for review.

[github-actions]

Documentation URL: https://autowarefoundation.github.io/autoware.universe/pr-8165/
Modified URLs:

• https://autowarefoundation.github.io/autoware.universe/pr-8165/perception/autoware_tensorrt_rtmdet/

[StepTurtle]

Hi, @mitsudome-r

This PR looks ready to review.

[Owen-Liuyuxuan]

Thank you for your PR. I am considering whether tensorrt_batched_nms should be a standalone package or add to the tensorrt_common. Any ideas?

[StepTurtle]

Thank you for your PR. I am considering whether tensorrt_batched_nms should be a standalone package or add to the tensorrt_common. Any ideas?

Hi @Owen-Liuyuxuan, thanks for your thoughts,

I am not sure if it would be beneficial to use it as a separate package. When we organize it as a separate package, I would expect it to be used by multiple packages or models. Normally, the batchedNMSPlugin is a plugin published within the TensorRT library, and the version we are using has been modified by mmdeploy. Therefore, I don't think it will be used by another model or package in the future. If this plugin is used, it will most likely be in its original form and I guess we can directly reach the original form from TensorRT headers.

Apart from that, if we want to gather all currently used and future plugins within tensorrt_common, this seems reasonable to me, but it is equally reasonable to keep it within the package. So, both of them OK for me (keep current form or put it to tensorrt_common).

[Owen-Liuyuxuan]

@StepTurtle
Your reasoning is solid. I agree to keep it as its current form.

[Shin-kyoto]

@StepTurtle
As an honest opinion from a reviewer, reviewing a PR with more than 5000 lines is quite challenging.

May I suggest one of the following actions? 🙏

• Add unit tests and ensure that they pass in CI.
  • The reviewer will then verify whether the tests are appropriate.
• Split the PR into smaller parts.

cc: @mitsudome-r @kminoda @mojomex
Please give your comments.

[mojomex]

@StepTurtle
Thank you for your PR! Thank you especially for the easy-to-follow PR description and documentation.

@Shin-kyoto

• Add unit tests and ensure that they pass in CI.
  The reviewer will then verify whether the tests are appropriate.
• Split the PR into smaller parts.

cc: @mitsudome-r @kminoda @mojomex
Please give your comments.

• I agree that this PR is proably better split up into parts. I would suggest making trt_batched_nms and related code its own PR as was also proposed by @Owen-Liuyuxuan.
• in calibrator.hpp, remove things like Int8LegacyCalibrator if there is no strong technical reason as to why multiple calibrators are needed.
• I agree that unit tests, or a module test with small sample rosbags as ground truths should be included.

After compilation, a file with the extension '.so' is created. This file stored in build and it should be parameter of dlopen() function.

Is there any information about can we handle this in Cmake. If we cannot, how can I provide the path to the file located inside the 'build' folder?

This is best handled in the launch stage. ROS 2 provides substitutions like $(find-pkg-prefix <pkg_name>) which can be used to get the package's location. There is also $(exec-in-package <exec-name> <package-name>) which looks like it just might work for shared libraries as well (I haven't verified though).

Loading plugins should also definitely be covered by unit tests to ensure they work reliably during runtime.

Thank you for your consideration.

[StepTurtle]

Hey, @mojomex @Shin-kyoto thanks for reviews.

At first, I thought we wouldn’t focus much on the plugin since it was developed by others, but it's nice that you reviewed the plugin code.

If we want to add the plugin as a separate package, should we put it under external folder or somewhere in perception folder? I can create a repository for the plugin under the leo-drive organization and we can clone the repository using the autoware.repos file.

I am working on the other topics you mentioned like calibrator.hpp code, unit tests, and so on. If I encounter any issues, I will let you know.

NOTE: Since there are a lot of TO DO, I switched the PR status as draft

[amadeuszsz]

@StepTurtle
Regarding libtensorrt word, please make a PR for local dictionary here https://github.com/autowarefoundation/autoware.universe/blob/main/.cspell.json#L7

[StepTurtle]

@amadeuszsz

spell-check-differential passed somehow but idk how. Do you think should we still add libtensorrt word to .cspell.json file?

[amadeuszsz]

@amadeuszsz

spell-check-differential passed somehow but idk how. Do you think should we still add libtensorrt word to .cspell.json file?

Test passed because something wrong is with this workflow for now. It will fail after branch update or merge to main. Please, add libtensorrt to local dictionary.

[StepTurtle]

Test passed because something wrong is with this workflow for now. It will fail after branch update or merge to main. Please, add libtensorrt to local dictionary.

Related PR: #9550

[amadeuszsz]

@StepTurtle
We were a bit unlucky with repository transition, could you add rtmdet to our new repository via casual PR? After that you can remove all cSpell ignores for rtmdet in your code. This word is needed as json files can't store comments in same line as desired word, thus spell check fails now.

[StepTurtle]

@amadeuszsz

The PR which adds rtmdet is merged and spell-check looks okay now.

[amadeuszsz]

@StepTurtle
I think we are close to finish review here. The only thing is to address message change

[StepTurtle]

@StepTurtle I think we are close to finish review here. The only thing is to address message change

@amadeuszsz msg type is updated.

[StepTurtle]

Fatih merged the msg type: autowarefoundation/autoware_internal_msgs#29

@amadeuszsz Code is updated with last msg type.

[amadeuszsz]

LGTM!

After some time I noticed there is a lot of boilerplate code which overlaps with other autoware perception modules. This makes code maintenance more challenging. Secondly, reviewer's comments doesn't make sense if after code owner change there is still a piece of code with same issues in different package. All of this suggest us to store common features (CUDA kernels) in common package. I will try to address this idea later.

@StepTurtle I would like to ask you for wait with merge until we complete all of subtasks. Also just before merge, check any universe package.xml <version> and assign appropriate value in your package.xml.

Great work! Thank you for your contribution!