Reprojection of 3D keypoints

[mshooter]

Hi,

I am reprojecting the keypoint_3d data, however, they do not corresponds to the keypoint_2d. They do correspond to the reproj_kp_2d. But it is slightly off of the keypoint_2d ground truth. Which library did you use to reproject the keypoints?

[luoxue-star]

Have you solved the problem? Looking forward to your reply. Thanks!

[mshooter]

Hi @luoxue-star,
unfortunately no, if you find the solution please update this thread :) !

[ShirleyMaxx]

Hi! @mshooter May I ask how you performed the projection? The JSON annotation files provided by the authors do not include camera intrinsic parameters. When I set empirical values for projection, they do not align with the images. I'm looking forward to your response! Thank you!

[mshooter]

Hi @ShirleyMaxx I have not been able to do so.
I am having a similar issue as you. Are you using Pytorch3D to reproject the keypoints?

[luoxue-star]

`def projection(points, k):
proj = np.dot(k, points.T).T
proj = proj / proj[:, 2][:, None]
print(f"proj points 2d: {proj.astype(np.int32).tolist()}")

data = read_json_file(path) # read data from json file
print(f"reproj points 2d: {data[2]['reproj_kp_2d']}")
trans = np.array(data[2]['trans'])
points_3d = np.array(data[2]['keypoint_3d'])
k = np.eye(3, dtype=np.float32)
k[0, 2], k[1, 2] = data[2]['width'] // 2, data[2]['height'] // 2

k[0, 0], k[1, 1] = 1000., 1000.
projection(points_3d, k)`

@mshooter You can do this. But the projected 3D keypoints can only correspond to reproj_kp_2d, not to 2D keypoints.

[mshooter]

@luoxue-star Indeed that as the approach however, the reprojected 2D keypoints are not as accurate as the true 2D ground truths. Which might affect the networks performance. Thank you though!

[ShirleyMaxx]

@mshooter @luoxue-star Thank you for your reply! I didn't use PyTorch3D, but instead followed @luoxue-star approach and wrote the projection in Python. However, I chose the focal length as (img_w * img_w + img_h * img_h) ** 0.5. When I changed it to 1000 as @luoxue-star suggested, the projected points aligned with the reprojected 2D keypoints, although this is not as accurate as the true 2D ground truths.

[mshooter]

Thank you all for replying and confirming for having similar issues :)