feat(image_projection_based_fusion): add image segmentation_pointcloud_fusion for pointcloud filter #5562

badai-nguyen · 2023-11-13T08:28:27Z

Description

This PR to add a fusion node of pointcloud with image segmentation mask which is output from some kind of 2D detection model such as model to be added by #5553.
The fusion result could be used to filter the unnecessary pointcloud for autonomous driving which is discussed at #5594

One of the idea for node use

Tests performed

Testing filter using front camera semantic segmentation:

Green pointcloud (input): /perception/object_recognition/detection/pointcloud_map_filtered/pointcloud
Pink pointcloud (output): /perception/object_recognition/detection/image_segmentation_filtered/pointcloud (note that it also covers the input green pointcloud)

semantic_segmentation_based_filter_01d060d9-8d25-45e0-b45e-9fd7201ac27b_2023-05-26-11-18-34_p0900_0_.webm

Notes for reviewers

Interface changes

Effects on system behavior

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

I've confirmed the contribution guidelines.
The PR follows the pull request guidelines.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

The PR follows the pull request guidelines.
The PR has been properly tested.
The PR has been reviewed by the code owners.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

There are no open discussions or they are tracked via tickets.
The PR is ready for merge.

After all checkboxes are checked, anyone who has write access can merge the PR.

Signed-off-by: badai-nguyen <[email protected]>

github-actions · 2023-11-21T00:43:49Z

Documentation URL: https://autowarefoundation.github.io/autoware.universe/pr-5562/
Modified URLs:

https://autowarefoundation.github.io/autoware.universe/pr-5562/perception/image_projection_based_fusion/segmentation-pointcloud-fusion/

Signed-off-by: badai-nguyen <[email protected]>

yukkysaito · 2023-11-21T05:08:30Z

Can you share the video and debugging screen with us?

Signed-off-by: badai-nguyen <[email protected]>

badai-nguyen · 2023-11-21T15:53:21Z

Can you share the video and debugging screen with us?

@yukkysaito I have added the debug video in test performed. could you please check it.

Signed-off-by: badai-nguyen <[email protected]>

yukkysaito · 2023-11-22T01:06:00Z

Good 👍
Does this launch by default?

badai-nguyen · 2023-11-22T02:24:19Z

Does this launch by default?

@yukkysaito Not yet.

Since the yolox-s semseg header was trained only on public dataset and need more time for the further improvement so I would to keep this new feature out of default. I hope can merge this feature so we can test autoware for some specific enviroments or weather condition and see how good it works first.

This reverts commit 5dfa160.

Signed-off-by: badai-nguyen <[email protected]>

badai-nguyen · 2023-12-26T08:20:13Z

@yukkysaito As I dicussed with @miursh, since the semseg yolox model is still being improved by @dan-dnn, so I would like to merge the image_pointcloud fusion node first as preparation. The pointcloud filter and perception pipeline change will be considered and updated later when the semseg yolox is ready.
I removed the change of tier4_perception_launch from this PR so could you review this again?

codecov · 2023-12-26T11:08:06Z

Codecov Report

Attention: 67 lines in your changes are missing coverage. Please review.

Comparison is base (1d6e7e8) 15.28% compared to head (ef36ead) 15.28%.
Report is 66 commits behind head on main.

Files	Patch %	Lines
...fusion/src/segmentation_pointcloud_fusion/node.cpp	0.00%	52 Missing ⚠️
.../image_projection_based_fusion/src/fusion_node.cpp	0.00%	12 Missing ⚠️
...ction_based_fusion/src/roi_cluster_fusion/node.cpp	0.00%	1 Missing ⚠️
...sed_fusion/src/roi_detected_object_fusion/node.cpp	0.00%	1 Missing ⚠️
...on_based_fusion/src/roi_pointcloud_fusion/node.cpp	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5562      +/-   ##
==========================================
- Coverage   15.28%   15.28%   -0.01%     
==========================================
  Files        1750     1751       +1     
  Lines      120568   120620      +52     
  Branches    36781    36781              
==========================================
  Hits        18434    18434              
- Misses      81469    81521      +52     
  Partials    20665    20665

Flag	Coverage Δ		*Carryforward flag
differential	`0.00% <0.00%> (?)`
total	`15.29% <ø> (+<0.01%)`	⬆️	Carriedforward from 4c0f353

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

YoshiRi

LGTM in code grammar, but I did not check if it works in real data.

perception/image_projection_based_fusion/src/segmentation_pointcloud_fusion/node.cpp

…tcloud_fusion/node.cpp Co-authored-by: Yoshi Ri <[email protected]>

miursh · 2024-01-09T16:34:50Z

perception/image_projection_based_fusion/src/segmentation_pointcloud_fusion/node.cpp

+    }
+  }
+
+  pcl::toROSMsg(output_cloud, output_pointcloud_msg);


[imo]
Just some comments; not required to change now.
Since we don't use the PCL method, is it possible not to use PCL?
Converting ROS messages to PCL (and vice versa) incurs a significant computational cost, so we should try to avoid it.

We can access each point in directly ros message like this.

miursh · 2024-01-09T16:45:31Z

perception/image_projection_based_fusion/docs/segmentation-pointcloud-fusion.md

+- The pointclouds are projected onto the semantic/instance segmentation mask image which is the output from 2D segmentation neural network.
+- The pointclouds are belong to segment such as road, sidewalk, building, vegetation, sky ... are considered as less important points for autonomous driving and could be removed.
+
+![segmentation_pointcloud_fusion_image](./images/segmentation_pointcloud_fusion.png)


[Q]
The image file has not been committed?

miursh

LGTM with a few comments

…d_fusion for pointcloud filter (autowarefoundation#5562) * fix: roiCallback for Image Signed-off-by: badai-nguyen <[email protected]> * fix: pointcloud filter Signed-off-by: badai-nguyen <[email protected]> * fix: pointcloud filter bugfix Signed-off-by: badai-nguyen <[email protected]> * fix: segmentation poincloud filter Signed-off-by: badai-nguyen <[email protected]> * fix: add option in launch file Signed-off-by: badai-nguyen <[email protected]> * chore: refactor Signed-off-by: badai-nguyen <[email protected]> * docs: update docs Signed-off-by: badai-nguyen <[email protected]> * fix: roi_pointcloud_fusion with new template Signed-off-by: badai-nguyen <[email protected]> * fix: add param of selectable semantic id Signed-off-by: badai-nguyen <[email protected]> * chore: typo Signed-off-by: badai-nguyen <[email protected]> * fix: launch file Signed-off-by: badai-nguyen <[email protected]> * Revert "fix: launch file" This reverts commit 5dfa160. * fix: reverse perception launcher Signed-off-by: badai-nguyen <[email protected]> * style(pre-commit): autofix * Update perception/image_projection_based_fusion/src/segmentation_pointcloud_fusion/node.cpp Co-authored-by: Yoshi Ri <[email protected]> * style(pre-commit): autofix --------- Signed-off-by: badai-nguyen <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Yoshi Ri <[email protected]>

badai-nguyen added 8 commits November 10, 2023 13:30

fix: roiCallback for Image

ef93592

Signed-off-by: badai-nguyen <[email protected]>

fix: pointcloud filter

fcb8169

Signed-off-by: badai-nguyen <[email protected]>

fix: pointcloud filter bugfix

94ba048

Signed-off-by: badai-nguyen <[email protected]>

fix: segmentation poincloud filter

cfa1d63

Signed-off-by: badai-nguyen <[email protected]>

fix: add option in launch file

73e4fd8

Signed-off-by: badai-nguyen <[email protected]>

chore: refactor

949ac3c

Signed-off-by: badai-nguyen <[email protected]>

docs: update docs

0702bf6

Signed-off-by: badai-nguyen <[email protected]>

fix: roi_pointcloud_fusion with new template

55da066

Signed-off-by: badai-nguyen <[email protected]>

github-actions bot added type:documentation Creating or refining documentation. (auto-assigned) component:perception Advanced sensor data processing and environment understanding. (auto-assigned) component:launch Launch files, scripts and initialization tools. (auto-assigned) labels Nov 13, 2023

badai-nguyen mentioned this pull request Nov 13, 2023

feat(tensorrt yolox): inference and publish mask image from yolox model with semantic segmentation header #5553

Merged

7 tasks

badai-nguyen added the run:deploy-docs Mark for deploy-docs action generation. (used-by-ci) label Nov 21, 2023

fix: add param of selectable semantic id

d4ebbb8

Signed-off-by: badai-nguyen <[email protected]>

badai-nguyen marked this pull request as ready for review November 21, 2023 02:36

badai-nguyen requested review from yukkysaito, yukke42, miursh and YoshiRi as code owners November 21, 2023 02:36

chore: typo

2ff0ff3

Signed-off-by: badai-nguyen <[email protected]>

fix: launch file

5dfa160

Signed-off-by: badai-nguyen <[email protected]>

badai-nguyen added 2 commits December 26, 2023 15:55

Revert "fix: launch file"

b999c78

This reverts commit 5dfa160.

fix: reverse perception launcher

6a9331a

Signed-off-by: badai-nguyen <[email protected]>

badai-nguyen requested review from ktro2828 and tzhong518 as code owners December 26, 2023 07:01

Merge branch 'main' into feat/add_image_segmentation_fusion

bf32121

github-actions bot removed the component:launch Launch files, scripts and initialization tools. (auto-assigned) label Dec 26, 2023

style(pre-commit): autofix

4c0f353

badai-nguyen added the run:build-and-test-differential Mark to enable build-and-test-differential workflow. (used-by-ci) label Dec 26, 2023

YoshiRi approved these changes Jan 9, 2024

View reviewed changes

perception/image_projection_based_fusion/src/segmentation_pointcloud_fusion/node.cpp Outdated Show resolved Hide resolved

badai-nguyen and others added 2 commits January 9, 2024 23:49

Update perception/image_projection_based_fusion/src/segmentation_poin…

1499f30

…tcloud_fusion/node.cpp Co-authored-by: Yoshi Ri <[email protected]>

style(pre-commit): autofix

ef36ead

miursh reviewed Jan 9, 2024

View reviewed changes

miursh approved these changes Jan 9, 2024

View reviewed changes

miursh enabled auto-merge (squash) January 9, 2024 16:46

miursh merged commit 39e9acc into autowarefoundation:main Jan 9, 2024
26 of 31 checks passed

tier4-autoware-public-bot bot mentioned this pull request Jan 10, 2024

chore: sync upstream tier4/autoware.universe#1084

Merged

badai-nguyen mentioned this pull request Jan 10, 2025

Add a new pointcloud filter based on image semantic segmentation #5594

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(image_projection_based_fusion): add image segmentation_pointcloud_fusion for pointcloud filter #5562

feat(image_projection_based_fusion): add image segmentation_pointcloud_fusion for pointcloud filter #5562

badai-nguyen commented Nov 13, 2023 •

edited

Loading

github-actions bot commented Nov 21, 2023

yukkysaito commented Nov 21, 2023

badai-nguyen commented Nov 21, 2023

yukkysaito commented Nov 22, 2023 •

edited

Loading

badai-nguyen commented Nov 22, 2023 •

edited

Loading

badai-nguyen commented Dec 26, 2023

codecov bot commented Dec 26, 2023 •

edited

Loading

YoshiRi left a comment

miursh Jan 9, 2024

miursh Jan 9, 2024

miursh Jan 9, 2024

miursh left a comment

feat(image_projection_based_fusion): add image segmentation_pointcloud_fusion for pointcloud filter #5562

feat(image_projection_based_fusion): add image segmentation_pointcloud_fusion for pointcloud filter #5562

Conversation

badai-nguyen commented Nov 13, 2023 • edited Loading

Description

Related links

Tests performed

Notes for reviewers

Interface changes

Effects on system behavior

Pre-review checklist for the PR author

In-review checklist for the PR reviewers

Post-review checklist for the PR author

github-actions bot commented Nov 21, 2023

yukkysaito commented Nov 21, 2023

badai-nguyen commented Nov 21, 2023

yukkysaito commented Nov 22, 2023 • edited Loading

badai-nguyen commented Nov 22, 2023 • edited Loading

badai-nguyen commented Dec 26, 2023

codecov bot commented Dec 26, 2023 • edited Loading

Codecov Report

YoshiRi left a comment

Choose a reason for hiding this comment

miursh Jan 9, 2024

Choose a reason for hiding this comment

miursh Jan 9, 2024

Choose a reason for hiding this comment

miursh Jan 9, 2024

Choose a reason for hiding this comment

miursh left a comment

Choose a reason for hiding this comment

badai-nguyen commented Nov 13, 2023 •

edited

Loading

yukkysaito commented Nov 22, 2023 •

edited

Loading

badai-nguyen commented Nov 22, 2023 •

edited

Loading

codecov bot commented Dec 26, 2023 •

edited

Loading