feat(image_projection_based_fusion): add cache for camera projection

[a-maumau]

Description

This PR introduces a caching feature for calculating raw image projections (unrectifyPoint).
This PR must merged with autowarefoundation/autoware_launch#1275

It introduces the CameraProjection class to handle the related processes. The cache is computed based on specific grid centers, and the projected points are represented using these grid centers. These grid centers are determined by approximation_grid_cell_width and approximation_grid_cell_height.
Pre-cached result will be simply stored in an array and all grid cells on the image will be pre-computed.

For example, when approximation_grid_cell_* is set to 1.0 pixel, the grid centers will be created as shown in the following image:

Note that edge pixels ((w, *), (*, h)) will be rounded to fall outside the image bounds.

memory consumption

When using caching, memory consumption is calculated as:

ceil(image_width / grid_cell_width) * ceil(image_height / grid_cell_height) * (4 * 2) * roi_numbers [bytes]

Here, (4 * 2) represents the bytes for two float values.
For example, for an image with dimensions 1440 × 1080, a grid size of 1 pixel, and caching enabled for 6 cameras, the memory consumption is:

(1440 / 1.0) * (1080 / 1.0) * 4 * 2 * 6 / (1024 * 1024) ≈ 71.2 MiB (theoretical).

Also when using the cache feature, it requires an initialization step (pre-caching). This will make some delay to a node before starting the processing.

Other changes

• This PR fixes a bug where the camera parameters were used before being properly initialized in pointpainting_fusion.
• take a list input instead of single bool for point_project_to_unrectified_image param

Performance: current impl. vs. this PR

• The following result is done with the same rosbag file, but the starting is determined by the (almost) first topic from the map_based_prediction node, so there is no guarantee it is aligned.
• The processing time is timed with TimeKeeper (subCallback is timed).
• CPU usage and memory usage is collected with autoware_tools
• The grid cell size is set to a square (width == height), and the camera (roi) number is set to 6.
• baseline indicates the current implementation, unrectified is the point_project_to_unrectified_image and cache means approximate_camera_projection option in the config.

Processing Time

roi_cluster_fusion

• all processing times
  

• current point_project_to_unrectified_image=false vs. this PR point_project_to_unrectified_image=true, cache=true
  
  This shows the processing time is close to the case of point_project_to_unrectified_image=false case when we use the cache feature.

pointpainting_fusion

• all processing times
  

• current point_project_to_unrectified_image=false vs. this PR point_project_to_unrectified_image=true, cache=true
  

• current point_project_to_unrectified_image=true vs. this PR point_project_to_unrectified_image=true, no cache
  
  This performance difference should be coming from the fix of the bug in FOV filtering (camera_info was not properly initialized).

CPU Usage

roi_cluster_fusion

Plot length differ because of the difference of play rate (0.1x is 3 times longer than 0.3x).
It does not contains the CPU usage of initialization step (which might use 100% of CPU).

Memory Usage

roi_cluster_fusion

Plot length differ because of the difference of play rate (0.1x is 3 times longer than 0.3x).

Visualization

Following images ware not captured at the same frame.

All images are in the case of point_project_to_unrectified_image=true.

Related links

Parent Issue:

• Link

How was this PR tested?

Checked the rviz output and each fusion node's visualization.

For checking by yourself, you need some processing like

# decompress image
ros2 launch autoware_image_transport_decompressor image_transport_decompressor.launch.xml input_topic_name:=/sensing/camera/camera0/image_rect_color/compressed output_topic_name:=/sensing/camera/camera0/image_rect_color

# relay camera info
ros2 run topic_tools relay /sensing/camera/camera0/camera_info /perception/object_recognition/detection/pointpainting/pointpainting/debug/camera_info

and set the debug_mode: true in param. For pointpainting_fusion, you need to set the #if 0 to #if 1 to contain the points in an image.

Notes for reviewers

None.

Interface changes

None.

Effects on system behavior

None.

[github-actions]

Thank you for contributing to the Autoware project!

🚧 If your pull request is in progress, switch it to draft mode.

Please ensure:

• You've checked our contribution guidelines.
• Your PR follows our pull request guidelines.
• All required CI checks pass before marking the PR ready for review.

[technolojin]

Can you also update the schema file?
https://github.com/autowarefoundation/autoware.universe/blob/main/perception/autoware_image_projection_based_fusion/schema/fusion_common.schema.json

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 166 lines in your changes missing coverage. Please review.

Project coverage is 29.78%. Comparing base (932e87f) to head (3b0bbd0).
Report is 53 commits behind head on main.

Files with missing lines	Patch %	Lines
..._projection_based_fusion/src/camera_projection.cpp	0.00%	99 Missing ⚠️
..._image_projection_based_fusion/src/fusion_node.cpp	0.00%	21 Missing ⚠️
...on_based_fusion/src/roi_pointcloud_fusion/node.cpp	0.00%	19 Missing ⚠️
...sed_fusion/src/roi_detected_object_fusion/node.cpp	0.00%	12 Missing ⚠️
...ction_based_fusion/src/roi_cluster_fusion/node.cpp	0.00%	9 Missing ⚠️
...fusion/src/segmentation_pointcloud_fusion/node.cpp	0.00%	3 Missing ⚠️
...ion_based_fusion/src/pointpainting_fusion/node.cpp	0.00%	2 Missing ⚠️
...mage_projection_based_fusion/camera_projection.hpp	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9635      +/-   ##
==========================================
- Coverage   29.80%   29.78%   -0.03%     
==========================================
  Files        1443     1446       +3     
  Lines      108590   108696     +106     
  Branches    42653    42691      +38     
==========================================
+ Hits        32369    32370       +1     
- Misses      73042    73145     +103     
- Partials     3179     3181       +2     

Flag	Coverage Δ		*Carryforward flag
differential	2.54% <0.00%> (?)
total	29.82% <ø> (+0.02%)	⬆️	Carriedforward from c3317ac

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[a-maumau]

@technolojin I changed the terms in bc1e8ed

[technolojin]

Just rename some variables.

• avoid to use too general name such as index: more specify like grid_index
• name as its role: grid_x -> grid_idx_x
• align naming style: since cell_width_ is used, image_w_ -> image_width_

[a-maumau]

@technolojin I applied the suggestions. thank you for pointing out!

[technolojin]

Test the PR in TIER IV INTERNAL CLOUD TESTER

[technolojin]

LGTM

[technolojin]

@badai-nguyen @YoshiRi @yukkysaito
Dear code-owners,

Can you review this PR? This PR has been tested.

[badai-nguyen]

@a-maumau Thank you for your great idea and PR. I have minor comments.

[a-maumau]

@badai-nguyen Thank you for pointing out. I applied your suggestions.

[badai-nguyen]

point_project_to_unrectified_image parameter was missing from previous PR, could also add it?

Suggested change

	},
	},
	"point_project_to_unrectified_image": {
	"type": "array",
	"description": "An array of options indicating whether to project point to unrectified image or not.",
	"default": [false, false, false, false, false, false]
	},

[technolojin]

resolved 3b0bbd0

[github-actions]

Documentation URL: https://autowarefoundation.github.io/autoware.universe/pr-9635/
Modified URLs:

• https://autowarefoundation.github.io/autoware.universe/pr-9635/perception/autoware_image_projection_based_fusion/

[badai-nguyen]

LGTM