feat(prediction): add support of prediction evaluation

[ktro2828]

Signed-off-by: ktro2828 [email protected]

Category

• Perception
  • Prediction

What

Add support of prediction evaluation.
Related to #6

Type of change

• New feature
• Update test
• Update document

Test performed

• sample score results

Frame:
 Total Num: 200
 Skipped Frames: []
 Skipped Frames Count: 0
Ground Truth Num: 13081
ADE: 1.663, FDE: 2.864, Miss Rate: 0.312, Soft mAP: 0.002(Center Distance 3d [m])



Label
car(1.0)
bicycle(1.0)
pedestrian(1.0)
motorbike(1.0)




Predict_num
3771
1189
7526
335


ADE
2.703
nan
0.624
nan


FDE
5.009
nan
0.719
nan


Miss Rate
0.611
nan
0.012
nan


Soft AP
0.000
0.000
0.008
0.000



ADE: 2.247, FDE: 3.198, Miss Rate: 0.260, Soft mAP: 0.056(Center Distance 3d [m])



Label
car(2.0)
bicycle(2.0)
pedestrian(2.0)
motorbike(2.0)




Predict_num
3771
1189
7526
335


ADE
4.049
1.489
1.201
nan


FDE
6.824
1.492
1.278
nan


Miss Rate
0.716
0.000
0.063
nan


Soft AP
0.000
0.181
0.043
0.000



ADE: 1.410, FDE: 2.410, Miss Rate: 0.302, Soft mAP: 0.000(IoU BEV)



Label
car(0.5)
bicycle(0.5)
pedestrian(0.5)
motorbike(0.5)




Predict_num
3771
1189
7526
335


ADE
2.524
nan
0.296
nan


FDE
4.387
nan
0.434
nan


Miss Rate
0.600
nan
0.003
nan


Soft AP
0.000
0.000
0.000
0.000



ADE: 0.255, FDE: 0.427, Miss Rate: 0.000, Soft mAP: 0.000(IoU 3D)



Label
car(0.5)
bicycle(0.5)
pedestrian(0.5)
motorbike(0.5)




Predict_num
3771
1189
7526
335


ADE
nan
nan
0.255
nan


FDE
nan
nan
0.427
nan


Miss Rate
nan
nan
0.000
nan


Soft AP
0.000
0.000
0.000
0.000



ADE: 1.795, FDE: 2.020, Miss Rate: 0.342, Soft mAP: 0.053(Plane Distance [m])



Label
car(2.0)
bicycle(2.0)
pedestrian(2.0)
motorbike(2.0)




Predict_num
3771
1189
7526
335


ADE
2.710
1.477
1.198
nan


FDE
3.300
1.477
1.282
nan


Miss Rate
0.958
0.000
0.069
nan


Soft AP
0.000
0.172
0.042
0.000



ADE: 2.146, FDE: 2.177, Miss Rate: 0.803, Soft mAP: 0.056(Plane Distance [m])



Label
car(3.0)
bicycle(3.0)
pedestrian(3.0)
motorbike(3.0)




Predict_num
3771
1189
7526
335


ADE
2.380
1.886
2.016
2.300


FDE
2.481
1.885
2.041
2.300


Miss Rate
0.994
0.490
0.729
1.000


Soft AP
0.000
0.181
0.044
0.000

Reference

Notes for reviewer

[hayato-m126]

I performed a coupled test with the following PR, but got the following error

tier4/driving_log_replayer_v2#40

[perception_evaluator_node.py-91] [2024-11-13 14:07:46,718] [INFO] [dataset.py:346 get_interpolated_now_frame] Only before frame is available for interpolation
[perception_evaluator_node.py-91] Traceback (most recent call last):
[perception_evaluator_node.py-91]   File "/home/hyt/ros_ws/unlabeled/install/driving_log_replayer_v2/lib/driving_log_replayer_v2/perception_evaluator_node.py", line 372, in <module>
[perception_evaluator_node.py-91]     main()
[perception_evaluator_node.py-91]   File "/home/hyt/ros_ws/unlabeled/install/driving_log_replayer_v2/local/lib/python3.10/dist-packages/driving_log_replayer_v2/evaluator.py", line 203, in wrapper
[perception_evaluator_node.py-91]     executor.spin()
[perception_evaluator_node.py-91]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/executors.py", line 294, in spin
[perception_evaluator_node.py-91]     self.spin_once()
[perception_evaluator_node.py-91]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/executors.py", line 794, in spin_once
[perception_evaluator_node.py-91]     self._spin_once_impl(timeout_sec)
[perception_evaluator_node.py-91]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/executors.py", line 791, in _spin_once_impl
[perception_evaluator_node.py-91]     future.result()
[perception_evaluator_node.py-91]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/task.py", line 94, in result
[perception_evaluator_node.py-91]     raise self.exception()
[perception_evaluator_node.py-91]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/task.py", line 239, in __call__
[perception_evaluator_node.py-91]     self._handler.send(None)
[perception_evaluator_node.py-91]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/executors.py", line 437, in handler
[perception_evaluator_node.py-91]     await call_coroutine(entity, arg)
[perception_evaluator_node.py-91]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/executors.py", line 362, in _execute_subscription
[perception_evaluator_node.py-91]     await await_or_execute(sub.callback, msg)
[perception_evaluator_node.py-91]   File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/executors.py", line 107, in await_or_execute
[perception_evaluator_node.py-91]     return callback(*args)
[perception_evaluator_node.py-91]   File "/home/hyt/ros_ws/unlabeled/install/driving_log_replayer_v2/lib/driving_log_replayer_v2/perception_evaluator_node.py", line 280, in perception_cb
[perception_evaluator_node.py-91]     frame_result: PerceptionFrameResult = self.__evaluator.add_frame_result(
[perception_evaluator_node.py-91]   File "/home/hyt/ros_ws/unlabeled/install/perception_eval/local/lib/python3.10/dist-packages/perception_eval/manager/perception_evaluation_manager.py", line 122, in add_frame_result
[perception_evaluator_node.py-91]     result.evaluate_frame(previous_result=self.frame_results[-1])
[perception_evaluator_node.py-91]   File "/home/hyt/ros_ws/unlabeled/install/perception_eval/local/lib/python3.10/dist-packages/perception_eval/evaluation/result/perception_frame_result.py", line 142, in evaluate_frame
[perception_evaluator_node.py-91]     self.metrics_score.evaluate_prediction(object_results_dict, num_ground_truth_dict)
[perception_evaluator_node.py-91]   File "/home/hyt/ros_ws/unlabeled/install/perception_eval/local/lib/python3.10/dist-packages/perception_eval/evaluation/metrics/metrics.py", line 250, in evaluate_prediction
[perception_evaluator_node.py-91]     prediction_score_ = PredictionMetricsScore(
[perception_evaluator_node.py-91]   File "/home/hyt/ros_ws/unlabeled/install/perception_eval/local/lib/python3.10/dist-packages/perception_eval/evaluation/metrics/prediction/prediction_metrics_score.py", line 54, in __init__
[perception_evaluator_node.py-91]     displacement_err = PathDisplacementError(
[perception_evaluator_node.py-91]   File "/home/hyt/ros_ws/unlabeled/install/perception_eval/local/lib/python3.10/dist-packages/perception_eval/evaluation/metrics/prediction/path_displacement_error.py", line 73, in __init__
[perception_evaluator_node.py-91]     self.ade, self.fde, self.miss_rate = self._calculate_displacement_error(
[perception_evaluator_node.py-91]   File "/home/hyt/ros_ws/unlabeled/install/perception_eval/local/lib/python3.10/dist-packages/perception_eval/evaluation/metrics/prediction/path_displacement_error.py", line 103, in _calculate_displacement_error
[perception_evaluator_node.py-91]     estimation, ground_truth = prepare_path(result, self.top_k)
[perception_evaluator_node.py-91]   File "/home/hyt/ros_ws/unlabeled/install/perception_eval/local/lib/python3.10/dist-packages/perception_eval/evaluation/metrics/prediction/utils.py", line 41, in prepare_path
[perception_evaluator_node.py-91]     estimated_object_.predicted_paths.sort(key=lambda x: x.confidence, reverse=True)
[perception_evaluator_node.py-91] AttributeError: 'NoneType' object has no attribute 'sort'